Learn how to setup a proxmox in a routed network configuration. This is typically required for small labs or non productive servers. It is highly discouraged to run a virtual server like proxmox on only one network interface (NIC) and thus only one network switch port. Good hosting companies make it possible to provide you multiple switch ports along with multiple NICs for your server.
Okay I see you are still not convinced to order one of our proxmox servers 😉 *sigh*
We will help you anyway, read on 😉
What is the routed configuration?
First of all it simulates to the outside, the uplink switch that your server is using only one MAC address. But there does it come from? Each proxmox guest (your virtual machines on proxmox) will get a new generated MAC address. Usually this MAC address is passed to the switch port. So if you have 10 guests on your proxmox server, you will have 11 MAC addresses to register on the uplink switch:
- 1 MAC addresses for your physical NIC which is cabled to the switch
- 10 MAC addresses for your 10 virtual machines/guests on proxmox
Since for security reasons the provider wants to know what MAC addresses are connected, it would require the provider to manage all these 10 additional MAC addresses. This would cost the provider to much work or money, so they simply do not allow this.
Why is routed configuration required and how does it work?
The basic principle behind the routed configuration on activate Proxy ARP on proxmox. Proxy ARP is a very old technique specified in RFC 925 in 1984!
Here proxmox physical NIC gets instructed to answer all packets from internal traffic with the MAC address of the physical NIC. So the outside will only see pakets from one MAC even if you have tons of additional guests on your proxmox.
Why is routed configuration bad for your datacenter?
Usually you wire one server with at least two physical ports on your server. This two ports go with two different wires to two different uplink switches (usually named Top Of Rack (TOR) or only Rack-Switches. And these switches go again to two different uplink switches.
So if we have a fault on one of the paths (software of one NIC crashed, NIC broken, cable broken, TOR-Switch broken, and so on) we still have one working path, which is about the half of bandwidth, but we are still online. Typically for this links is the Spanning Tree Protocol (STP) or alternatives are used.
If you now want to use smooth guest failover to other proxmox servers, you need to be able to handover the guest MAC to the other proxmox server. And latest here you will see that the routed configuration will make the network administrator some lengthy nights to handle all this requirements, with Proxy ARP and routed proxmox network configuration.
You still want to use Proxy ARP? Couldn’t we still not convince you to give us a call 😉 No issue, let us explain further.
How to configure proxmox with Proxy ARP?
The Linux kernel provides this feature out of the box. You can temporarily activate it per NIC.
Login as root to your proxmox server via physical keyboard+VGA, KVM or via SSH
First list your NICs:
ip link
the NIC names usually start on proxmox with the prefix en<something>, like:
- eno1
- ens18
- enp0s31f0
- and so on
lets assume you have 2 NICs and they are named en01 and eno2. To activate on eno1 proxy arp do following:
echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp cat /proc/sys/net/ipv4/conf/eno1/proxy_arp
and the last command should return you a 1 as result. This is indicating that proxy_arp is active now.
So far so good, but we now only have the MAC proxying, and no routing.
Do you still want to …? Okay, okay you want it the hard way, read on 😉
What is still required for proxmox to work properly with the single NIC?
Quick and dirty answer: you will need a IP configured at the physical NIC and a bridge on top of this NIC.
But what the hack is a bridge? Would be a longer story, in short and simple words it is behaving like a simple network ethernet switch but on virtual/local level on your proxmox machine.
You really still want to go on?
Ohh man, maybe you could be a right fit to our team? We are hiring, take a look at our open job positions …
How to configure proxmox for routed IPv4 network configuration?
MAC issue is solved, moving on to IP networking by assigning an accessible IP to the physical NIC eno1.
First of all we need to enable ip forwarding on linux with
echo 1 > /proc/sys/net/ipv4/ip_forward
Now let us edit the network definition file
vi /etc/network/interfaces
in this file ensure this is contained:
auto eno1 iface eno1 inet static address 44.16.24.48 # your IP address netmask 255.255.255.255 # yes, 4 times 255! pointopoint 44.16.24.25 # your gateway IP gateway 44.16.24.25 # your gateway IP up sysctl -p post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp
save the changes and close the file.
What does this?
First we assign the IP 44.16.24.48 to the physical NIC eno1. And we ensure that only this IP is assigned to the NIC with the netmask 255.255.255.255
Also we need to define our routing now. We need to define a pointopoint (PtP) IP and gateway IP. Our upstream gateway in our example is 44.16.24.25.
And finally enforce on starting the interface to
- reload the sysctl
- to activate ip forwarding
- to activate proxy ARP
You can now reboot proxmox or you restart your networking with
service networking restart
Proxmox shall now be accessible on https://44.16.24.48:8006 or alternatively via SSH on root@44.16.24.48
How to enable the networking for the proxmox guests?
We need to define a bridge or a openvswitch (OVS) for proxmox to be able to assign proxmox guests to the network. Even if OVS has some benefits, we will use the more simple setup with a bridge.
vi /etc/network/interfaces
and ensure you have added this also underneath the previous config
auto vmbr0 iface vmbr0 inet static address 44.16.24.48 netmask 32 bridge_ports none bridge_stp off bridge_fd 0
You remember the explanation from previously about STP and so on? Here you have it 😉 I will also not explain them … now …
Simply take the configuration as shown.
Now it depends, you can do a even more complex configuration my NATing all internal IPs to the one 44.16.24.48. This would be the proxmox Masquerading (NAT) with iptables configuration. Or the simpler one to buy additional IPs to your server.
The NATed configuration is highly discouraged, since you bring in senseless complexity. Would recommend you to take additional IPs or subnets.
How to add additional IPs to the routed config?
Assuming your additional IP is 22.77.2.204
Again open the network definition
vi /etc/network/interfaces
change your config to this
auto vmbr0 iface vmbr0 inet static address 44.16.24.48 netmask 32 bridge_ports none bridge_stp off bridge_fd 0 up ip route add 22.77.2.204/32 dev vmbr0
save and restart your networking and your guest can now use the IP 22.77.2.204 on the vmbr0 interface.
How to configure VLAN tagging with the routed network configuration?
For a clean separation of network traffic from e.g. departments (C-level data, accounting, production) it is suggested to separate the networks with VLANs. This increases security for the departments.
But consider also this security flaw, that you want to ensure, that only a very limited amount of admins shall be even able to access management services of your proxmox, even if they have physical access to the server. They can workaround IP based firewall very simply with a different wiring. But if your network is properly configured for VLANs, the task will get harder for them to attack your servers.
Also VLANs help to keep the broadcasts limited to only the devices which really need them.
So for a secure and professional network setup with proxmox it is highly recommended to use VLANs.
How to setup VLANs in routed config will maybe be explained in a later article. Since this makes the setup even more complicated in the routed configuration.
How to configure proxmox for routed IPv6 network configuration?
This is also a matter for an additional article.
You are unhappy with the missing VLAN and IPv6 configuration?
you already know, we are hiring 😉 open job positions
and you can hire us for helping you out of the dark, if you still need support here 😉
And of course you can get servers in our datacenter with the setup for your needs.