What is the best way to fix this bridge mode routing issue?

Vagrant
1

Hello,

I hope everyone is doing well.

There is a routing issue I am having I hope someone can help me figure out.

When spinning up libvirt based VMs using Vagrant and “bridge” network mode, I end up up in a situation where the VM successfully obtains an IP from my local LAN via the bridge, but also does not route as expected.

The VM obtains an IP from the DHCPD server on my local LAN fine. And I can then ping this VM from my LAN, and otherwise access it fine.

But not from the host the guest is running on. Conversely, the guest OS can not ping or reach the host machine that is running it either.

So the host machine, can’t ping the guest OS when running under ‘bridged’ networking mode. Or vice versa.

It’s a routing issue certainly, but I’m not sure what the best way for me to fix this or address this is.

Here is the example Vagrantfile I’m running with, while I try to figure this out:

# -*- mode: ruby -*-
# # vi: set ft=ruby :

ENV['VAGRANT_DEFAULT_PROVIDER'] = 'libvirt'

Vagrant.configure("2") do |config|
  config.vm.define "fedora-01" do |config|
    config.vm.hostname = "fedora-01"
    config.vm.box = "generic/fedora36"
    config.vm.box_check_update = false
    config.vm.network "public_network", dev: "lan-bridge", :mac => "5CA1AB1E0001"
    config.vm.provider :libvirt do |v|
      v.memory = 1024
    end
  end
end

The routing table of the host as seen by ‘ip r’

default via 10.1.0.1 dev lan-bridge proto dhcp src 10.1.0.10 metric 425 
10.1.0.0/24 dev lan-bridge proto kernel scope link src 10.1.0.10 metric 425 
10.1.2.0/24 dev dmz-bridge proto kernel scope link src 10.1.2.2 metric 426 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

Here is the routing table on the guest OS as seen by ‘ip r’:

default via 192.168.121.1 dev eth0 proto dhcp src 192.168.121.66 metric 100 
default via 10.1.0.1 dev eth1 proto dhcp src 10.1.0.11 metric 101 
10.1.0.0/24 dev eth1 proto kernel scope link src 10.1.0.11 metric 101 
192.168.121.0/24 dev eth0 proto kernel scope link src 192.168.121.66 metric 100

And for a little extra detail, here is the host machines current interface and address assignments as seen by ‘ip a’:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: lan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master lan-bridge state UP group default qlen 1000
    link/ether 80:61:5f:06:a0:13 brd ff:ff:ff:ff:ff:ff
    altname enp65s0
3: dmz0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master dmz-bridge state UP group default qlen 1000
    link/ether d0:50:99:d9:5d:51 brd ff:ff:ff:ff:ff:ff
    altname enp67s0
4: dmz-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether c2:d1:c2:af:d2:33 brd ff:ff:ff:ff:ff:ff
    inet 10.1.2.2/24 brd 10.1.2.255 scope global dynamic noprefixroute dmz-bridge
       valid_lft 595sec preferred_lft 595sec
5: lan-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b6:23:72:fe:3b:78 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.10/24 brd 10.1.0.255 scope global dynamic noprefixroute lan-bridge
       valid_lft 591sec preferred_lft 591sec
    inet6 fe80::a04f:452a:65fb:4690/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
6: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:8b:68:b9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
57: virbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:1a:73:4d brd ff:ff:ff:ff:ff:ff
    inet 192.168.121.1/24 brd 192.168.121.255 scope global virbr1
       valid_lft forever preferred_lft forever
58: vnet6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:ae:0c:83 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:feae:c83/64 scope link 
       valid_lft forever preferred_lft forever
59: macvtap6@lan-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 500
    link/ether 5c:a1:ab:1e:00:01 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5ea1:abff:fe1e:1/64 scope link 
       valid_lft forever preferred_lft forever

The goal for me to be able to able to have the VM spin up with an IP address assigned to it from my LAN’s DHCP server. And then to have that guest machine be reachable by other machines on that LAN.

Seems to be mostly working, except for the host not being able to participate.

Thank you for any help, and for any time spent looking my issue over.

2

I’ve made some progress on this, and think I found a tidy way to force the routes to be in place, on both the host an the guest.

Vagrant has a trigger system, which seems to work well when combined with the ‘shell’ provisioning method.

This is an example Vagrantfile I came up with. It uses bridged devices, and assigns routes to each individual guest.

I could probably improve this with some better use of variables. I’m new to Vagrant, and Ansible as well in general though.

Any thoughts on improvements on how I’m doing this, or other approaches entirely, would be welcome to here.

I really get the impression I should not have to be doing these kinds of manual route creations. It is my understanding this was usually done under the hood automatically for me. But I have something functional, and that makes me happy.

The following excerpt is mostly to help others who might stumble across any problems related to this issue, and might be interested to know what solution I settled on to get moving ahead with my project.

# -*- mode: ruby -*-
# # vi: set ft=ruby :
#
# Guest IP addresses are obtained from a DHCP server on the network. We provide
# MAC address in a Vagrant declaration, and the network provides the IP. IPs
# are assigned this way to get past a bug in Fedora, preventing it from being
# assigned static IP addresses by Vagrant nicely.
# Related issue, different distro:
# https://github.com/vagrant-libvirt/vagrant-libvirt/issues/867

ENV['VAGRANT_DEFAULT_PROVIDER'] = 'libvirt'

Vagrant.configure(2) do |config|
	config.vm.box = "generic/fedora36"
	config.ssh.insert_key = false
	config.vm.synced_folder ".", "/vagrant", disabled: true
	config.vm.provider :libvirt do |v|
		v.memory = 1024
	end

  ####
	# Application Server 1.
  ####
	config.vm.define "app1" do |app1|
		app1.vm.hostname = "app1.test"
    app1.vm.network "public_network", dev: "lan-bridge", :mac => "5CA1AB1E0001"

    # Host Side Route: Bring up every 'up'.
    app1.trigger.after :up do |trigger|
      trigger.info = "Trigger: Adding Host Side route."
      trigger.run = {inline: "sudo ip route add 10.1.0.11 dev virbr0"}
    end

    # Guest Side Route: Provision route on guest creation.
    app1.vm.provision "shell",
      run: "always",
      inline: "ip route add 10.1.0.10 dev eth0"

    # Host Side Route: Remove route when halting guest.
    app1.trigger.after :halt do |trigger|
      trigger.info = "Trigger: Removing Host Side route."
      trigger.run = {inline: "sudo ip route del 10.1.0.11 dev virbr0"}
    end
	end


  ####
	# Application Server 2.
  ####
	config.vm.define "app2" do |app2|
		app2.vm.hostname = "app2.test"

    app2.vm.network "public_network", dev: "lan-bridge", :mac => "5CA1AB1E0002"

    # Host Side Route: Bring up every 'up'.
    app2.trigger.after :up do |trigger|
      trigger.info = "Trigger: Adding Host Side route."
      trigger.run = {inline: "sudo ip route add 10.1.0.12 dev virbr0"}
    end

    # Guest Side Route: Provision route on guest creation.
    app2.vm.provision "shell",
      run: "always",
      inline: "ip route add 10.1.0.10 dev eth0"

    # Host Side Route: Remove route when halting guest.
    app2.trigger.after :halt do |trigger|
      trigger.info = "Trigger: Removing Host Side route."
      trigger.run = {inline: "sudo ip route del 10.1.0.12 dev virbr0"}
    end
	end


  ####
	# Database Server
  ####
	config.vm.define "db" do |db|
		db.vm.hostname = "db.test"
    db.vm.network "public_network", dev: "lan-bridge", :mac => "5CA1AB1E0003"

    # Host Side Route: Bring up every 'up'.
    db.trigger.after :up do |trigger|
      trigger.info = "Trigger: Adding Host Side route."
      trigger.run = {inline: "sudo ip route add 10.1.0.13 dev virbr0"}
    end

    # Guest Side Route: Provision route on guest creation.
    db.vm.provision "shell",
      run: "always",
      inline: "ip route add 10.1.0.10 dev eth0"

    # Host Side Route: Remove route when halting guest.
    db.trigger.after :halt do |trigger|
      trigger.info = "Trigger: Removing Host Side route."
      trigger.run = {inline: "sudo ip route del 10.1.0.13 dev virbr0"}
    end
	end
end