Wednesday, November 10, 2010

libvirt & KVM & unnumbered bridge setup

This is an ubuntu/debian recipe to use an 'unnumbered bridge' to save on the amount of IP addresses needed to connect your virtual machines on a libvirt host to other networks.

I'm assuming you want the host to be a router between the VM's and the external network. The advantage of this is that you can firewall traffic between the virtual machines and other networks on the host.

A setup like this can be used if your ISP provides you with a /29 and you want to be able to use every IP address out of that /29, and not waste IP's on the network, broadcast and gateway address.

This image shows the various elements involved:

The /etc/network/interfaces file on the host:
auto virbr0
iface virbr0 inet manual
  bridge-ports none
  bridge_stp off
  bridge_maxwait 1
  post-up ip route add dev virbr0
The above configuration will configure a bridge interface without an IPv4 address and route the /29 that was assigned to you by your ISP to that interface. This will force Linux to ARP for every IP from that /29 on this particular virbr0 interface.

The following virsh commands will remove the default network settings, you can enter them on a root prompt:
virsh net-destroy default
virsh net-undefine default
Virtlib's /etc/qemu-ifup script needs to be replaced with the following:
/sbin/ifconfig $1 up
/usr/sbin/brctl addif ${switch} $1
The above will make sure that any tap interface associated with a virtual machine will be added to the correct bridge. The default /etc/qemu-ifup file will try and guess the correct bridge, which is not desirable.

You can edit the XML describing the virtual machines interface with the following command: 'virsh edit NAME_OF_VM'. You could also just type this command:
virsh attach-interface NAME_OF_VM bridge virbr0
This is what the XML could like like:
<interface type="'bridge'">
  <mac address="'52:54:00:51:8b:ad'/">
  <source bridge="'virbr0'/">
  <target dev="'vnet0'/">
The last step is to configure the network settings inside the virtual machine (VM1). This is an example /etc/network/interfaces. Notice the pointtopoint statement and the subnet mask:
# The primary network interface
auto eth0
iface eth0 inet static
To wrap it all up:
  • /etc/network/interfaces on the host takes care of creating the virbr0 bridge
  • libvirt will make a vnetX interface (so don't worry about it)
  • /etc/qemu-ifup adds tap interfaces to the correct bridge when booting the virtual machine
  • thanks to ARP the machines will figure out who is where
  • /etc/network/interfaces inside the virtual machine will force traffic towards the host

I'd like to thank Sten Spans for providing some essential hints.


Fluor said...

A few pointers:

1) There's a typo in the VM 'interfaces' example: 'pointtopoint' is actually 'pointopoint'.

2) When using the GUI 'virt-manager' with this setup, it will not allow you to bind a network interface to the 'virbr0'-bridge on your host. Selecting any 'vnetN' interface *seems* to silently bind it to 'virbr0' though.

3) If your virbr0 is not bridged with your host's uplink network interface, your VMs will not be able to benefit from IPv6 Router Advertisements on your host's network.

4) Of lesser importance, this is more of a headsup: This setup depends on the Linux default values for net.ipv4.conf.all.arp_* in sysctl settings.

Fluor said...

Aparently the above claimed 'typo' (1) was never there. I think i messed that up myself. Sorry. :)

Job Snijders said...

Hi Fluor,

1) yes, this is a common mistake :-)

2) I have no experience with the GUI, thanks for that addition!

3) that's correct. In this case I explicitly wanted the dom0 host to do all routing, for both v4 and v6. I left IPv6 completely out of this picture because it was irrelevant to the 'unnumbered' trick I had to do due to shortage of IPv4 addresses. In this setup I have a /64 between the KVM host and the VM's, and if a VM needs more then 1 IPv6 address I route a /64 to them from the KVM host.

4) Can you specify which sysctl settings exactly? That would be good to know.

Thanks for your comments!

Fluor said...

The sysctl's are net.ipv4.conf.all.{arp_announce,arp_filter,arp_ignore}. But since the Linux defaults work, this is just a tip. See linux/Documentation/networking/ip-sysctl.txt for info.

Also, i want to share how to fix pointopoint on FreeBSD. Put the following in /etc/rc.conf to replicate the Linux example from the blogpost above:

ifconfig_em0="inet netmask 0xffffffff"

static_routes="gwnet defgw"
route_gwnet="-net -interface em0"

Cheers! Setup works like a charm and i no longer have to mess with static ARP-entries!