Page MenuHomeVyOS Platform

Enabling VRF breaks connectivity
Closed, ResolvedPublicBUG

Description

Hi,

I'm running VyOS 1.4.0-epa2. I'm using DMVPN and BGP to interconnect multiple sites. My plan would be to separate my interfaces and networks with VRFs, so DMVPN routes as well as BGP routes do not interfere with routes from the management interface. When creating a VRF (and not assigning), I am loosing any connectivity to the router.

The following command breaks connectivity. It "returns" after the commit, but then does not do anything anymore. No connections or other things are working.
Only a reboot of the (hardware) machine brings it back (without the vrf, as the config has not been saved).

vyos@hostname# set vrf name test table 500
[edit]
vyos@hostname# commit
[edit]
vyos@hostname#

See my sample/censored config below:

vyos@myhostname:~$ show configuration 
firewall {
    group {
        ipv6-network-group BGP-INCOMING {
        }
        ipv6-network-group SSH-INCOMING {
		<snip>
        }
        ipv6-network-group TUNNEL-INCOMING {
        }
        network-group BGP-INCOMING {
		<snip>
        }
        network-group SSH-INCOMING {
		<snip>
        }
        network-group TUNNEL-INCOMING {
        }
    }
    ipv4 {
        input {
            filter {
                rule 1 {
                    action accept
                    state established
                    state related
                }
                rule 2 {
                    action accept
                    protocol icmp
                }
                rule 10 {
                    action accept
                    destination {
                        port 22
                    }
                    protocol tcp
                    source {
                        group {
                            network-group SSH-INCOMING
                        }
                    }
                }
                rule 20 {
                    action accept
                    destination {
                        port bgp
                    }
                    protocol tcp
                    source {
                        group {
                            network-group BGP-INCOMING
                        }
                    }
                }
                rule 30 {
                    action accept
                    source {
                        group {
                            network-group TUNNEL-INCOMING
                        }
                    }
                }
                rule 50 {
                    action accept
                    protocol gre
                }
                rule 51 {
                    action accept
                    protocol esp
                }
                rule 52 {
                    action accept
                    protocol ah
                }
                rule 53 {
                    action accept
                    destination {
                        port isakmp
                    }
                    protocol udp
                }
                rule 54 {
                    action accept
                    destination {
                        port ipsec-nat-t
                    }
                    protocol udp
                }
                rule 55 {
                    action accept
                    protocol udp
                    source {
                        port isakmp
                    }
                }
                rule 56 {
                    action accept
                    protocol udp
                    source {
                        port ipsec-nat-t
                    }
                }
                rule 999 {
                    action drop
                }
            }
        }
    }
    ipv6 {
        input {
            filter {
                rule 1 {
                    action accept
                    state established
                    state related
                }
                rule 2 {
                    action accept
                    protocol icmpv6
                }
                rule 10 {
                    action accept
                    destination {
                        port 22
                    }
                    protocol tcp
                    source {
                        group {
                            network-group SSH-INCOMING
                        }
                    }
                }
                rule 20 {
                    action accept
                    destination {
                        port bgp
                    }
                    protocol tcp
                    source {
                        group {
                            network-group BGP-INCOMING
                        }
                    }
                }
                rule 30 {
                    action accept
                    source {
                        group {
                            network-group TUNNEL-INCOMING
                        }
                    }
                }
                rule 999 {
                    action drop
                }
            }
        }
    }
}
interfaces {
    ethernet eth0 {
        address 192.168.1.253/24
        hw-id ec:a8:6b:fe:ad:c7
        vif 10 {
            address 10.10.81.129/27
        }
    }
    loopback lo {
    }
    tunnel tun100 {
        address 10.10.81.243/28
        enable-multicast
        encapsulation gre
        parameters {
            ip {
                key ****************
            }
        }
        source-address 0.0.0.0
    }
    tunnel tun241 {
        address my:prefix::241:243::2/96
        encapsulation sit
        remote 10.10.81.241
        source-address 10.10.81.243
    }
    tunnel tun242 {
        address my:prefix::242:243::2/96
        encapsulation sit
        remote 10.10.81.242
        source-address 10.10.81.243
    }
}
policy {
    local-route {
        rule 50 {
            protocol gre
            set {
                table 100
            }
        }
        rule 51 {
            protocol esp
            set {
                table 100
            }
        }
        rule 52 {
            protocol ah
            set {
                table 100
            }
        }
        rule 53 {
            destination {
                port 500
            }
            protocol udp
            set {
                table 100
            }
        }
        rule 54 {
            destination {
                port 4500
            }
            protocol udp
            set {
                table 100
            }
        }
        rule 55 {
            protocol udp
            set {
                table 100
            }
            source {
                port 500
            }
        }
        rule 56 {
            protocol udp
            set {
                table 100
            }
            source {
                port 4500
            }
        }
        rule 100 {
            set {
                table 100
            }
            source {
                address 192.168.1.253
            }
        }
        rule 150 {
            destination {
                address 192.168.1.0/24
            }
            set {
                table 100
            }
        }
        rule 160 {
            set {
                table 100
            }
            source {
                address 192.168.1.0/24
            }
        }
    }
    prefix-list DMVPN4-HUB {
        rule 1 {
            action permit
            ge 24
            le 32
            prefix 10.10.80.0/23
        }
        rule 2 {
            action permit
            prefix 0.0.0.0/0
        }
    }
    prefix-list DMVPN4-SPOKE {
        rule 1 {
            action permit
            ge 24
            le 32
            prefix 10.10.80.0/23
        }
    }
    prefix-list EBGP-PREPEND4 {
    }
    prefix-list EBGP4 {
    }
    prefix-list6 DMVPN6-HUB {
        rule 1 {
            action permit
            ge 48
            prefix my:other:refix::/44
        }
        rule 2 {
            action permit
            prefix ::/0
        }
    }
    prefix-list6 DMVPN6-SPOKE {
        rule 1 {
            action permit
            ge 48
            prefix my:other:refix::/44
        }
    }
    prefix-list6 EBGP-PREPEND6 {
    }
    prefix-list6 EBGP6 {
    }
    route-map DMVPN-HUB {
        rule 1 {
            action permit
            match {
                ip {
                    address {
                        prefix-list DMVPN4-HUB
                    }
                }
            }
        }
        rule 2 {
            action permit
            match {
                ipv6 {
                    address {
                        prefix-list DMVPN6-HUB
                    }
                }
            }
        }
        rule 999 {
            action deny
        }
    }
    route-map DMVPN-SPOKE {
        rule 1 {
            action permit
            match {
                ip {
                    address {
                        prefix-list DMVPN4-SPOKE
                    }
                }
            }
        }
        rule 2 {
            action permit
            match {
                ipv6 {
                    address {
                        prefix-list DMVPN6-SPOKE
                    }
                }
            }
        }
        rule 999 {
            action deny
        }
    }
    route-map UPSTREAM {
        rule 1 {
            action permit
            match {
                ipv6 {
                    address {
                        prefix-list EBGP-PREPEND6
                    }
                }
            }
            set {
                as-path {
                    prepend "MYASN MYASN MYASN"
                }
            }
        }
        rule 2 {
            action permit
            match {
                ip {
                    address {
                        prefix-list EBGP-PREPEND4
                    }
                }
            }
            set {
                as-path {
                    prepend "MYASN MYASN MYASN"
                }
            }
        }
        rule 3 {
            action permit
            match {
                ipv6 {
                    address {
                        prefix-list EBGP6
                    }
                }
            }
        }
        rule 4 {
            action permit
            match {
                ip {
                    address {
                        prefix-list EBGP4
                    }
                }
            }
        }
        rule 999 {
            action deny
        }
    }
}
protocols {
    bgp {
        address-family {
            ipv4-unicast {
                network 10.10.81.128/27 {
                }
            }
        }
        neighbor 10.10.81.241 {
            peer-group DMVPN4-HUB
        }
        neighbor 10.10.81.242 {
            peer-group DMVPN4-HUB
        }
        peer-group DMVPN4-HUB {
            address-family {
                ipv4-unicast {
                    route-map {
                        export DMVPN-SPOKE
                        import DMVPN-HUB
                    }
                    soft-reconfiguration {
                        inbound
                    }
                }
            }
            remote-as MYASN
        }
        peer-group DMVPN4-SPOKE {
            address-family {
                ipv4-unicast {
                    route-map {
                        export DMVPN-SPOKE
                        import DMVPN-SPOKE
                    }
                    soft-reconfiguration {
                        inbound
                    }
                }
            }
            remote-as MYASN
        }
        system-as MYASN
    }
    nhrp {
        tunnel tun100 {
            cisco-authentication secret
            holding-time 300
            map 10.10.81.241/28 {
                nbma-address other-address
                register
            }
            map 10.10.81.242/28 {
                nbma-address another-address
                register
            }
            multicast nhs
            redirect
            shortcut
        }
    }
    static {
        route 0.0.0.0/0 {
            next-hop 192.168.1.1 {
            }
        }
        /* Originating traffic from the router */
        table 100 {
            route 0.0.0.0/0 {
                next-hop 192.168.1.1 {
                }
            }
        }
    }
}
service {
    ntp {
        allow-client {
            address 0.0.0.0/0
            address ::/0
        }
        server time1.vyos.net {
        }
        server time2.vyos.net {
        }
        server time3.vyos.net {
        }
    }
    ssh {
        disable-password-authentication
        port 22
    }
}
system {
    host-name somehostname
    login {
        user vyos {
            authentication {
                encrypted-password ****************
                public-keys root@root {
                    key ****************
                    type ssh-rsa
                }
                public-keys root@work {
                    key ****************
                    type ssh-rsa
                }
            }
        }
    }
    name-server 8.8.4.4
    name-server 8.8.8.8
    static-host-mapping {
        host-name myhostname {
            inet 127.0.0.1
        }
    }
}
vpn {
    ipsec {
        esp-group ESP-HUB {
            lifetime 1800
            mode transport
            pfs dh-group2
            proposal 1 {
                encryption aes256
                hash sha1
            }
            proposal 2 {
                encryption 3des
                hash md5
            }
        }
        ike-group IKE-HUB {
            key-exchange ikev1
            lifetime 3600
            proposal 1 {
                dh-group 2
                encryption aes256
                hash sha1
            }
            proposal 2 {
                dh-group 2
                encryption aes128
                hash sha1
            }
        }
        interface eth0
        profile NHRPVPN {
            authentication {
                mode pre-shared-secret
                pre-shared-secret ****************
            }
            bind {
                tunnel tun100
            }
            esp-group ESP-HUB
            ike-group IKE-HUB
        }
    }
}


vyos@myhostname:~$ show version
Version:          VyOS 1.4.0-epa2
Release train:    sagitta

Built by:         Sentrium S.L.
Built on:         Tue 12 Mar 2024 11:58 UTC
Build UUID:       7b60be54-0b8f-4337-aa9e-b6e675942946
Build commit ID:  48f7d41a607707

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  
Hardware model:   
Hardware S/N:     
Hardware UUID:    c7587600-34d4-11e1-a8f9-eca86bfeadc7

Copyright:        VyOS maintainers and contributors

Details

Difficulty level
Unknown (require assessment)
Version
1.4.0-epa2
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

fetzerms triaged this task as Normal priority.Wed, Apr 10, 9:50 AM
fetzerms created this task.
fetzerms created this object in space S1 VyOS Public.

Thats common with other vendors aswell.

The workaround in your case would be to create the config on a device which you have physical access to and then copy that config to the device you were originally trying to change and reboot it.

Another workaround is to simply have physical access to the device you wish to convert into using VRF's (through console).

I only created a vrf (but did not assign it to anything else). Is that intend to break connectivity?

Ok. Just to clarify, as T6097 talks about ipv6: This seems to break both ipv6 and ipv4 connectivity for me

Could you try the latest rolling?

Unfortunately this also breaks after the commit (even tho the "commit" command finalizes). If I recall correctly, a commit-confirm won't reboot the box either - but I'll double check that.

Welcome to VyOS!

   ┌── ┐
   . VyOS 1.5-rolling-202404090019
   └ ──┘  current

 * Documentation:  https://docs.vyos.io/en/latest
 * Project news:   https://blog.vyos.io
 * Bug reports:    https://vyos.dev

You can change this banner using "set system login banner post-login" command.

VyOS is a free software distribution that includes multiple components,
you can check individual component licenses under /usr/share/doc/*/copyright
vyos@myhostname:~$ configure 
vyos@myhostname# set vrf name test table 150
[edit]
vyos@myhostname# commit
[edit]
vyos@myhostname#

Yup... it does not even come back after commit-confirm - so I assume something more severe crashes.

Welcome to VyOS!

   ┌── ┐
   . VyOS 1.5-rolling-202404090019
   └ ──┘  current

 * Documentation:  https://docs.vyos.io/en/latest
 * Project news:   https://blog.vyos.io
 * Bug reports:    https://vyos.dev

You can change this banner using "set system login banner post-login" command.

VyOS is a free software distribution that includes multiple components,
you can check individual component licenses under /usr/share/doc/*/copyright
Last login: Wed Apr 10 11:59:18 2024 from 192.168.1.10
vyos@myhostname:~$ configure 
[edit]
vyos@myhostname# set vrf name test table 150
[edit]
vyos@myhostname# commit-confirm 2

commit-confirm will automatically reboot in 2 minutes unless changes
are confirmed.

Proceed ? [Y/n] Y
Initialized commit-confirm; 2 minutes to confirm before reboot
[edit]
vyos@myhostname#

I don't have any issues with your config, but my addresses (of course, I don't have all BGP VPN connections, etc.)

vyos@r4# set vrf name foo table 10101
[edit]
vyos@r4# commit
[edit]
vyos@r4# 
[edit]
vyos@r4# run show ver
Version:          VyOS 1.5-rolling-202404090019
Release train:    current





$ ssh [email protected]
Warning: Permanently added '192.168.122.14' (ED25519) to the list of known hosts.
Welcome to VyOS!

   ┌── ┐
   . VyOS 1.5-rolling-202404090019
   └ ──┘  current

 * Documentation:  https://docs.vyos.io/en/latest
 * Project news:   https://blog.vyos.io
 * Bug reports:    https://vyos.dev

You can change this banner using "set system login banner post-login" command.

VyOS is a free software distribution that includes multiple components,
you can check individual component licenses under /usr/share/doc/*/copyright
Last login: Wed Apr 10 15:36:08 2024 from 192.168.122.1
vyos@r4:~$

You probably should add a lab in the GNS3/Eve-NG with the same topology and re-check what is going with routing/firewall in your specific case.
If you can reproduce it in the lab, we need configs from all routers (/config/config.boot) from the test lab. Without xxx/other-asn/our-asn. Just use the some fake data, if should be reproducible as simple as possible

I don't see any issue with connectivity (over IPv4)

I am trying to narrow down how that happens, as my other experiments with vrf look fine.

I rebuilt my router as (xcp-ng) VM, with the following configuration:

set interfaces ethernet eth0 address '192.168.1.253/24'
set interfaces loopback lo
set policy local-route rule 50 protocol 'gre'
set policy local-route rule 50 set table '100'
set policy local-route rule 51 protocol 'esp'
set policy local-route rule 51 set table '100'
set policy local-route rule 52 protocol 'ah'
set policy local-route rule 52 set table '100'
set policy local-route rule 53 destination port '500'
set policy local-route rule 53 protocol 'udp'
set policy local-route rule 53 set table '100'
set policy local-route rule 54 destination port '4500'
set policy local-route rule 54 protocol 'udp'
set policy local-route rule 54 set table '100'
set policy local-route rule 55 protocol 'udp'
set policy local-route rule 55 set table '100'
set policy local-route rule 55 source port '500'
set policy local-route rule 56 protocol 'udp'
set policy local-route rule 56 set table '100'
set policy local-route rule 56 source port '4500'
set policy local-route rule 100 set table '100'
set policy local-route rule 100 source address '192.168.1.253'
set policy local-route rule 150 destination address '192.168.1.0/24'
set policy local-route rule 150 set table '100'
set policy local-route rule 160 set table '100'
set policy local-route rule 160 source address '192.168.1.0/24'
set protocols static route 0.0.0.0/0 next-hop 192.168.1.1
set protocols static table 100 route 0.0.0.0/0 next-hop 192.168.1.1
set service ssh

After creating a vrf, it looses all connection again.

vyos@myhostname# set vrf name test table 150
[edit]
vyos@myhostname# commit
[edit]
vyos@myhostname#

Starting from the config above:

Adding a vrf

set vrf name test table 150
commit  # now, no more connectivity
del vrf 
commit # still, no connectivity

vrf before policy

del policy
commit
set vrf name test table 150
commit # connectivity, works.
set policy local-route rule 50 protocol 'gre'
set policy local-route rule 50 set table '100'
set policy local-route rule 51 protocol 'esp'
set policy local-route rule 51 set table '100'
set policy local-route rule 52 protocol 'ah'
set policy local-route rule 52 set table '100'
set policy local-route rule 53 destination port '500'
set policy local-route rule 53 protocol 'udp'
set policy local-route rule 53 set table '100'
set policy local-route rule 54 destination port '4500'
set policy local-route rule 54 protocol 'udp'
set policy local-route rule 54 set table '100'
set policy local-route rule 55 protocol 'udp'
set policy local-route rule 55 set table '100'
set policy local-route rule 55 source port '500'
set policy local-route rule 56 protocol 'udp'
set policy local-route rule 56 set table '100'
set policy local-route rule 56 source port '4500'
set policy local-route rule 100 set table '100'
set policy local-route rule 100 source address '192.168.1.253'
set policy local-route rule 150 destination address '192.168.1.0/24'
set policy local-route rule 150 set table '100'
set policy local-route rule 160 set table '100'
set policy local-route rule 160 source address '192.168.1.0/24'
commit # no connectivity
delete policy
commit # now has connectivity

Deleting policy, adding vrf, deleting vrf and re-adding policy

del policy
commit
set vrf name test table 150
commit # connectivity, works.
delete vrf
set policy local-route rule 50 protocol 'gre'
set policy local-route rule 50 set table '100'
set policy local-route rule 51 protocol 'esp'
set policy local-route rule 51 set table '100'
set policy local-route rule 52 protocol 'ah'
set policy local-route rule 52 set table '100'
set policy local-route rule 53 destination port '500'
set policy local-route rule 53 protocol 'udp'
set policy local-route rule 53 set table '100'
set policy local-route rule 54 destination port '4500'
set policy local-route rule 54 protocol 'udp'
set policy local-route rule 54 set table '100'
set policy local-route rule 55 protocol 'udp'
set policy local-route rule 55 set table '100'
set policy local-route rule 55 source port '500'
set policy local-route rule 56 protocol 'udp'
set policy local-route rule 56 set table '100'
set policy local-route rule 56 source port '4500'
set policy local-route rule 100 set table '100'
set policy local-route rule 100 source address '192.168.1.253'
set policy local-route rule 150 destination address '192.168.1.0/24'
set policy local-route rule 150 set table '100'
set policy local-route rule 160 set table '100'
set policy local-route rule 160 source address '192.168.1.0/24'
commit # no connectivity --> After a reboot it has.
delete policy
commit # now has connectivity (same session, no reboot)
Viacheslav changed the task status from Open to Confirmed.Wed, Apr 10, 4:32 PM

The minimal configuration to reproduce:

set interfaces ethernet eth0 address '192.168.122.14/24'

set policy local-route rule 160 set table '100'
set policy local-route rule 160 source address '192.168.122.0/24'

set protocols static route 0.0.0.0/0 next-hop 192.168.122.1
set protocols static table 100 route 0.0.0.0/0 next-hop 192.168.122.1
commit

set vrf name red table 1234
commit
Viacheslav raised the priority of this task from Normal to High.Wed, Apr 10, 5:01 PM

I only created a vrf (but did not assign it to anything else). Is that intend to break connectivity?

Thats my experience from other vendors.

As soon as you create the first VRF (or "VPN-instance") its like the device forgets about IP-addresses etc so you must put them in again (along with defining which VRF the interface belongs to).

Which is why the workaround for this for a remote device is to either have a config already made and upload it to this device and reboot it or make sure you have physical access to the device through console (could of course be remote through some consoleserver if you got a proper out-of-band management).

I think one possible workaround for this in VyOS would be if VyOS always used VRF's to begin with and call it "Default" or such because then the VRF's are already created and the device wont "forget" about its IP-configuration. Or even better have a "secure" default config where 1st interface is for MGMT put in VRF MGMT and the others are put in VRF PROD or such.

syncer changed the subtype of this task from "Task" to "Bug".

It is not related to VRF at all and is related to the policy routing logic:
Reproduced even on 1.3.2

set interfaces ethernet eth1 address '192.168.122.14/24'

set policy local-route rule 160 set table '100'
set policy local-route rule 160 source '192.168.122.0/24'
set protocols static table 100 route 0.0.0.0/0 next-hop 192.168.122.1

After this, the host 192.168.122.15 (or any other host) cannot ping/communicate with the host 192.168.122.14

If we check the rules we see that all lookups check routing table 100

vyos@r-left# ip rule show
160:	from 192.168.122.0/24 lookup 100
32765:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default
[edit]
vyos@r-left#

This way there are no routers for route 192.168.122.0/24 (only via default)
To fix communication for the same network you should add a rule to check the main table route before adding any tables

sudo ip rule add prio 100 from 192.168.122.0/24 to 192.168.122.0/24 lookup main

or related command for 1.4/1.5 in the CLI

Close the task as invalid

Hi @Viacheslav,

thank you very much for your analysis. I am still wondering, why it breaks with adding the vrf and why it works before.
Also, why it starts to work again, after rebooting when removing the vrf again (but not before rebooting)

set interfaces ethernet eth0 address '192.168.1.252/24'
set protocols static route 0.0.0.0/0 next-hop 192.168.1.1
set protocols static table 100 route 0.0.0.0/0 next-hop 192.168.1.1
set service ssh

# Please see the complementing rules for the source/dest address. 
set policy local-route rule 150 destination address '192.168.1.0/24'
set policy local-route rule 150 set table '100'
set policy local-route rule 160 set table '100'
set policy local-route rule 160 source address '192.168.1.0/24'

commit

# Here we have proper connectivity, everything works.
# ip rule:
0:	from all lookup local
150:	from all to 192.168.1.0/24 lookup 100
160:	from 192.168.1.0/24 lookup 100
32766:	from all lookup main
32767:	from all lookup default


set vrf name test table 150
commit # no connectivity anymore

# ip rule (bear with me, I typed it manually ;-) )
150:	from all to 192.168.1.0/24 lookup 100
160:	from 192.168.1.0/24 lookup 100
1000:	from all lookup [l3mdev-table]
2000:	from all lookup [l3mdev-table] unreachable
32765:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default

delete vrf 
commit 
# Still no connectivity. After deleting the vrf, some rules are still in place.
# ip rule
150:	from all to 192.168.1.0/24 lookup 100
160:	from 192.168.1.0/24 lookup 100
1000:	from all lookup [l3mdev-table]
2000:	from all lookup [l3mdev-table] unreachable
32765:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default

reboot # Now we have connectivity again.

# ip rule
0:	from all lookup local
150:	from all to 192.168.1.0/24 lookup 100
160:	from 192.168.1.0/24 lookup 100
32766:	from all lookup main
32767:	from all lookup default

After before adding the vrf, connectivity works fine. After adding it, it breaks.

PR https://github.com/vyos/vyos-1x/pull/3326

vyos@r-left# delete vrf
[edit]
vyos@r-left# commit
ip [ vrf ]
{'no_vni': '', 'policy': {'route-map': {}}, 'vrf_remove': {'red': {}}}

rule[edit]
vyos@r-left# ip rule
0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default
[edit]
vyos@r-left#
Viacheslav changed the task status from Open to Needs testing.Thu, Apr 18, 2:25 PM