Code

Proxmox - Lessons Learned

Leon SCOTT

• 5 min read
Proxmox - Lessons Learned

Proxmox VE is great, and I mean great. This hypervisor can go from a tinkering arrangement to a full-blown Enterprise deployment with redundancy, high availability, clustering you name it. I've been tinkering with it for months now (since the whole VMware buyout scenario).

But it does have its problems.

My network is an Ubiquiti affair. I have a UDM, and 2 APs that run the entire lot and a few Flex switches. It gives me total control.

I can configure the whole damn network from a single interface, from WAN, redundancy WAN, VLANS, Firewall, NetFlow, SEIM, Wi-Fi, remote access and the list goes on.

IPv6 however is a minor problem. Not for Ubiquiti although the Unifi app for my mobile still doesn't show IPv6 addresses and can be problematic on some features of the Ubiquiti Controller but it's pretty good.

IPv6 in Proxmox is a bastard, to be blunt. I don't want a static address set at the server box, I want this set from the router. If the ISP changes things, I want the whole lot to be dynamic - it's a management thing - all from one spot - easy.

The whole network just reconfigures to what the ISP says is so. Great

Proxmox, nope, it's a server - it should be static. Since I am doing that via the router, we change the configuration as below via the CLI, effectively making it static by using DHCP and DHCP6 or SLAAC.

So, you end up with something like this in your /etc/network/interfaces file

This one is VLAN enabled but only for VLAN 1.

# Modify the following file /etc/network/interfaces.d/vmbr0.cfg

auto lo
iface lo inet loopback
        ndp on
iface enp4s0 inet manual

iface enp4s0 inet6 manual
        accept_ra 2
        ndp on

auto vmbr0
iface vmbr0 inet dhcp
        gateway 192.168.1.1
        dns-nameservers [ipv4 ipv6 addresses]
        dns-search [internal domain suffix]
        bridge-ports enp4s0
        bridge-stp on
        bridge-fd 0
        post-up dhclient -4 vmbr0

iface vmbr0 inet6 auto
        gateway fe80::f4e2:c6ff:feee:63da
        bridge-vlan-aware yes
        bridge-vids 1
        dns-nameservers [ip4 ipv6 addresses]
        dns-search [internal domain suffix]
        accept_ra 2
        ndp on
        post-up dhclient -6 vmbr0
        pre-down dhclient -6 -r vmbr0

source /etc/network/interfaces.d/*

But that is not the only thing you need to look at. Other issues exist.

You need to change the host file to at least contain references for the gateway and the Proxmox server

Then you have to look at the /etc/resolv.conf file. I was having issues with the file losing its configuration data, due in part on how the UC advertises the ipv6 information. So, I ended up with this

search [internal domain]
nameserver [ipv4 gateway ipv4 address]
nameserver [ipv6 gateway ipv6 link-local address]
nameserver [ipv6 gateway ipv6 global-link address]

#lock the file so stop the damn thing from changing
chattr +i /etc/resolv.conf
reboot

Beauty, the server is now holding all the correct information and is doing everything network related correctly. However, you will get an error because I locked the file.

Checking the resolv.conf file on the containers and the information is there, as well which is what is meant to happen.

Hold on, what's going on with LXC containers - what the hell.

So, a bit of digging and I found out I had to do the following as well

So once again - changes are called for

#!/bin/bash

# Path to the configuration file
CONFIG_FILE="/etc/lxc/default.conf"

# New configuration content
NEW_CONTENT="
# Custom configuration
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx
lxc.network.dns.search = [internal domain suffix]
lxc.network.dns.nameservers = [ipv4 and ipv6 addresses]
lxc.network.ipv6.ndp = true
# Enable DHCPv6
lxc.net.0.ipv6.dhcp = true
# Enable SLAAC for automatic address configuration as backup
lxc.net.0.ipv6.auto = 1

# Backup the old configuration file
cp $CONFIG_FILE $CONFIG_FILE.bak

# Replace the contents of the configuration file
echo "$NEW_CONTENT" >> $CONFIG_FILE

# Restart the LXC networking service
systemctl restart lxc-net

echo "The configuration file has been updated and the LXC networking service has been restarted."

# End of script

Can you see where I am going with this. This is quickly becoming very static again

Finally, we are going to ensure that the /etc/sysctl.conf file is as is should be by doing the following

# Modified configuration

kernel.domainname = [internal domain suffix]

net.ipv4.tcp_syncookies=1
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.accept_source_route = 1
net.ipv4.ip_forward=1
net.ipv4.conf.default.forwarding=1
net.ipv4.conf.all.log_martians=1
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1

net.ipv6.conf.all.accept_source_route = 1

# Enable IPv6 forwarding
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.forwarding = 1

# Accept Router Advertisements even if forwarding is enabled
net.ipv6.conf.all.accept_ra = 2
net.ipv6.conf.default.accept_ra = 2

# Enable Router Solicitation
net.ipv6.conf.all.router_solicitations = 1
net.ipv6.conf.default.router_solicitations = 1

# Enable Router Advertisement
net.ipv6.conf.all.accept_ra_pinfo = 1
net.ipv6.conf.default.accept_ra_pinfo = 1

# Enable DHCPv6
net.ipv6.conf.all.accept_ra_defrtr = 1
net.ipv6.conf.default.accept_ra_defrtr = 1

# Enable SLAAC
net.ipv6.conf.all.autoconf = 1
net.ipv6.conf.default.autoconf = 1

But it works and all the containers are networking just fine and are giving the results that they should be by running the following commands, depending on what you are looking for.

# checking networking controls
sysctl net.ipv6.conf.eth0.accept_ra
sysctl net.ipv6.conf.eth0.accept_ra_defrtr
sysctl net.ipv6.conf.all.forwarding
sysctl net.ipv6.conf.default.forwarding
sysctl net.ipv4.conf.all.log_martians
sysctl net.ipv4.conf.default.rp_filter
sysctl net.ipv4.conf.all.rp_filter

# checking the systemd journals
journalctl -u docker.service
journalctl -u networking -xe
journalctl -u dhclient -xe

# looking at nat tables in docker
iptables -t nat -L -v
ip6tables -t nat -L -v

# diagnosis ipv4 and ipv6
rdisc6 [appropraite network name eg... eth0, vmbr0, docker0]
dhclient -6 -v
dhclient -4 -v
mtr -6 ipv6.google.com
ip -6 neigh show
ping6 -c 3 [mac, ipv4, fqdn as required]

# checking firewalls - eg netflow
telnet ipv4.address 2055                 #tcp - closed
nc -zv ipv4.address 2055                 #tcp - closed
nc -zv -u ipv4.address 2055              #udp - open

Then there is the firewall to configure, the docker daemon, the postfix mail relay configuration so all your containers and the server send emails to a designated address every time something goes on that you want to know about.

A quick note on the docker daemon. Do not use UFW. Its incompatible with docker. So, you will need to use the iptables command and if "prefiltering" the rules generated by docker using DOCKER-USER as the chain.

This assumes you are doing this manually and are not using the GUI Firewall interface and have not moved to the new tables. I try to keep it simple.

Reference:

Packet filtering and firewalls
How Docker works with packet filtering, iptables, and firewalls
Docker Documentation

I disable ipv6 in docker. I see no point.

IPv6 networking is a lot harder than I thought, and cybersecurity is even harder

WTH who in Lithuania is sending /bin/bash commands in the URI. You little shit.

Off to Cloudflare and change the WAF Rule or get off your backside and get Bunkerweb going (actually I got it going but I need to figure out a few things and I need and edge certificate, which isn't free apparently).

But it all works for now, and the more I tinker the better I get.

By the way, don't take my word for it. I might be wrong.

#enoughsaid