VRRPv3 support (VRRP for IPv6)
Open, HighPublic

Details

Difficulty level
Normal (likely a few hours)
afics created this task.Jul 17 2016, 11:12 AM

TODO:

  • check if our keepalived version supports VRRPv3, if not, upgrade to a newer version.
  • implement cli support + config generation

A few notes:

  • We need extra VRRP instances for IPv4 and IPv6, keepalived can only have virtual_addresses of the instances native address family.
  • VRRPv3 supports IPv4 and IPv6
  • Maybe we should still keep support for VRRPv2 for backwards compatibility?
syncer triaged this task as High priority.Jul 17 2016, 1:25 PM
syncer added a project: VyOS 1.1.x (1.1.8).
jbrown added a subscriber: jbrown.Jul 23 2016, 10:07 AM

We need extra VRRP instances for IPv4 and IPv6, keepalived can only have virtual_addresses of the instances native address family.

AFAIK, as long as vrrp_strict isn't set, keepalived happily supports mixed v4 and v6 addresses in the same group (and even under VRRPv2). I've run that configuration under keepalived for several years.

I tested with keepalived version 1.2.22 on Fedora and it didn't seem to work. I'll test again.

[root@test ~]# cat /etc/keepalived/keepalived.conf 
vrrp_instance VI_1 {
    state MASTER
    interface ens3
    virtual_router_id 51
    priority 200
    advert_int 1
    vrrp_version 3
    native_ipv6
    authentication {
        auth_type ah
        auth_pass 1111
    }
    virtual_ipaddress {
        3ffa::1/64
        192.168.100.200/24 
    }
}

Does only work for the v6 address with this configuration.


[root@test ~]# cat /etc/keepalived/keepalived.conf 
vrrp_instance VI_1 {
    state MASTER
    interface ens3
    virtual_router_id 51
    priority 200
    advert_int 1
    vrrp_version 3
    #native_ipv6
    authentication {
        auth_type ah
        auth_pass 1111
    }
    virtual_ipaddress {
        3ffa::1/64
        192.168.100.200/24 
    }
}

Does only work for the v6 address with this configuration.

[root@test ~]# cat /etc/keepalived/keepalived.conf 
vrrp_instance VI_1 {
    state MASTER
    interface ens3
    virtual_router_id 51
    priority 200
    advert_int 1
    vrrp_version 3
    #native_ipv6
    authentication {
        auth_type ah
        auth_pass 1111
    }
    virtual_ipaddress {
        #3ffa::1/64
        192.168.100.200/24 
    }
}

Works for v4 only.

[root@test ~]# cat /etc/keepalived/keepalived.conf 
vrrp_instance VI_1 {
    state MASTER
    interface ens3
    virtual_router_id 51
    priority 200
    advert_int 1
    vrrp_version 3
    native_ipv6
    authentication {
        auth_type ah
        auth_pass 1111
    }
    virtual_ipaddress {
        3ffa::1/64
        #192.168.100.200/24 
    }
}

Works for v6 only.


Separate instances for v4 and v6 do work.

@jbrown Did I get something obvious wrong?

Try it with vrrp_version 2; I believe that will work. It seems to default to standards-compliant behavior (that is to say, no mixed-virtual-addresses) in v3. Not sure if there's a way to turn that off or not. Here's what the config at my current employer looks like

vrrp_instance lb_hi {
    state BACKUP
    interface eth0
    virtual_router_id 186
    priority 50
    advert_int 5
    authentication {
        auth_type PASS
        auth_pass XXX
    }
    virtual_ipaddress {
        ipv6::address::1/128 scope global dev eth1
        ipv6::address::1/128 scope global dev eth1
        ipv4.address.1/32 scope global dev eth1
        ipv4.address.2/32 scope global dev eth1
        ipv4.address.3/32 scope global dev eth0
    }
    virtual_ipaddress_excluded {
    }
}

vrrp_sync_group easypost_lb_hi {
    group {
        lb_hi
    }
    notify /srv/lbng/notify-state.sh
}

That's with 1.2.16 (CentOS).

I can run some more tests when I'm in the office on Monday.

Also, here's the config that I'm running on my VyOS test VMs (with R33:5bccce5948cc44e76f8c9da93f1a1cf8f1212bca ):

global_defs {
	enable_traps
}
vrrp_instance vyatta-eth1-1 {
	state BACKUP
	interface eth1
	virtual_router_id 1
	priority 100
	advert_int 1
	virtual_ipaddress {
		172.16.0.1/16
		fd00:ea51:d000::1/64
	}
}

This is vyos 1.1.7, so keepalived 1.2.2

afics added a comment.EditedJul 24 2016, 7:30 PM

@jbrown This only works for you because your keepalived versions are old enough.
This got "fixed" (well, at least they're standards compliant now ;)) in 1.2.20 I believe.
See https://github.com/acassen/keepalived/issues/375#issuecomment-230148110 for more information.

So I'm afraid we'll have to find another solution. As I see it, we have two options:

  • Either use separate vrrp_instances for v4 and v6 (autogenerated), ugly.
  • Script it our selves

Oh man, that's going to be a disaster for all the companies out there relying on mixed v4/v6 blocks (which is extremely common in v6 setups).

What about something where, in the short term to support ipv6, we *required* both v4 and v6 addresses on any VRRP group and put all the v6 addresses in virtual_ipaddress_excluded?

If I make a patch to use virtual_ipaddress_excluded, do you prefer an arcanist diff or a github pull request?

afics added a comment.Jul 24 2016, 8:05 PM

Does it work, if you use virtual_ipaddress_excluded? Also I don't really understand how this would solve the problem? Could you please explain it?

I believe a github pull request is fine.

virtual_ipaddress_excluded is basically a list of lines that keepalived passes to ip addr add / ip addr del when the main VRRP state changes. Typically, it's used to manage more than 20 addresses with keepalived (only the first 20 go in virtual_ipaddress, which is in the actual VRRP packet; the rest are failed over through this out-of-band configuration). According to that thread (and past experience), it should work for this case.

I changed my test VM to use virtual_ipaddress_excluded, and failover worked correctly.

https://github.com/vyos/vyatta-vrrp/pull/6 is a rough estimate of what implementation would look like.

This may be getting a little off-track from this ticket, I guess; the subject (VRRPv3) will allow v6-only configurations, which is probably something people want even if we get mixed configs working somehow.

afics added a comment.Jul 24 2016, 8:43 PM

Ah, good to know. So if we add a switch like transport ipv4/ipv6 to the cli which is only valid for VRRPv3 (add a switch for that too) and then exclude either all v4 or all v6 addresses, would that work?

If it does, that way we would have implemented at least full functionality.

rps awarded a token.Sep 15 2016, 10:16 AM
rps added a subscriber: rps.
syncer edited subscribers, added: VyOS 1.2.x; removed: VyOS 1.1.x (1.1.8).
syncer mentioned this in Unknown Object (Ponder Answer).May 12 2017, 3:33 PM
aopdal added a subscriber: aopdal.May 26 2017, 10:09 AM

Is there any progress on this? Is there any design documents in progress?

I assume there will be VRRPv2 and VRRPv3 in 1.2 to be able to upgrade existing infrastructure. So if we want to use IPv6 in VRRP we need to switch from VRRPv2 to VRRPv3 with some downtime and when running VRRPv3 with IPv4 it should be possible to add IPv6.

Change from VRRPv2 only to VRRPv3 only is not an option due to the fact that this will break the possibility to perform upgrades.

News about this ?

Any chance to get this fixed in 1.1.8?
We absolutely need ipv6 failover

Even the virtual_ipaddress_excluded workaround would be great as long vyos supports ipv6 in vrrp ASAP

oddboy added a subscriber: oddboy.Nov 17 2017, 6:56 AM

Hi all,

I wouldn't mind an update on this too. it would be very useful.

Using two debian VM i have played around with this today.
I have been using debian 9.2 and keepalived v1.3.2

This is just a my PoC or a proposal to how we may solve this task.

I'm using just one interface for this test, and the feature in the router must be specified.

  • We must have a solution which are possible to upgrade
  • The user should be able to select which version of VRRP to run
  • We must provide a solution which may interop with other devices

When using this configurations it looks nice and the vrrp_instance running version 2 take care of the upgrade
the vrrp instance running version 3 is taking care of IPv6

If this approach is feasible we could start the specification details.

Device1

global_defs {
        enable_traps
}
vrrp_sync_group test {
        group {
                ens224-100
                ens224-100-v3
        }
}
vrrp_instance ens224-100 {
        state BACKUP
        interface ens224
        virtual_router_id 100
        priority 150
        nopreempt
        advert_int 1
        virtual_ipaddress {
                10.1.1.20
        }
}
vrrp_instance ens224-100-v3 {
        vrrp_version 3
		state BACKUP
        interface ens224
        virtual_router_id 100
        priority 150
        nopreempt
        advert_int 1
        virtual_ipaddress {
                2001:4642:3a8e:fff0::20
        }
}

Device2

global_defs {
        enable_traps
}
vrrp_sync_group test {
        group {
                ens224-100
                ens224-100-v3
        }
}
vrrp_instance ens224-100 {
        state BACKUP
        interface ens224
        virtual_router_id 100
        priority 50
        nopreempt
        advert_int 1
        virtual_ipaddress {
                10.1.1.20
        }
}
vrrp_instance ens224-100-v3 {
        vrrp_version 3
		state BACKUP
        interface ens224
        virtual_router_id 100
        priority 50
        nopreempt
        advert_int 1
        virtual_ipaddress {
                2001:4642:3a8e:fff0::20
        }
}

Does anyone have any ideas how to get VRRPv3 in 1.2?
If we could conclude on the approach we could go further by describing cli commands, make out how the upgrade should be done, create documentation and so on.

If anyone would like to implement it I may also test it.

syncer added a subscriber: syncer.Thu, Dec 21, 1:23 PM

@aopdal try latest nightly as we pushed changes related to vrrp

Testing on

vyos@vyos-vrrp1# run sh ver
Version:          VyOS 999.201712220337
Built by:         autobuild@vyos.net
Built on:         Fri 22 Dec 2017 03:37 UTC
Build ID:         92b365a2-b8fc-4f12-9209-bee33323d53f

Architecture:     x86_64
Boot via:         installed image
System type:      VMware guest

Hardware vendor:  VMware, Inc.
Hardware model:   VMware Virtual Platform
Hardware S/N:     VMware-56 4d fc ad 58 82 3c aa-17 8a 66 57 cc 32 59 cc
Hardware UUID:    ADFC4D56-8258-AA3C-178A-6657CC3259CC

Copyright:        VyOS maintainers and contributors

With configuration:

set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 hw-id '00:0c:29:32:59:cc'
set interfaces ethernet eth1 address '10.1.1.31/24'
set interfaces ethernet eth1 hw-id '00:0c:29:32:59:d6'
set interfaces ethernet eth1 vrrp vrrp-group 30 preempt 'false'
set interfaces ethernet eth1 vrrp vrrp-group 30 priority '150'
set interfaces ethernet eth1 vrrp vrrp-group 30 virtual-address '10.1.1.30'
set interfaces loopback 'lo'
set system host-name 'vyos-vrrp1'

and

set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 hw-id '00:0c:29:32:59:cc'
set interfaces ethernet eth1 address '10.1.1.31/24'
set interfaces ethernet eth1 hw-id '00:0c:29:32:59:d6'
set interfaces ethernet eth1 vrrp vrrp-group 30 preempt 'false'
set interfaces ethernet eth1 vrrp vrrp-group 30 priority '150'
set interfaces ethernet eth1 vrrp vrrp-group 30 virtual-address '10.1.1.30'
set interfaces loopback 'lo'
set system host-name 'vyos-vrrp2'

And now VRRP is totally broken!

Please read the documentation of the protocols, and then we can decide how it should work, design and implement the feature according to the design.

VRRPv2 : https://tools.ietf.org/html/rfc3768
VRRPv3 : https://tools.ietf.org/html/rfc5798

The current implementation s using VRRPv3 group address, but are running VRRPv2.
The current implementation will break all upgrade.
The current implementation is not according to any standards.

The current implementation is working on keepalived 1.2.19 (from 2015.07.07). In 1.2.20 (from 2016-04-02) a lot of bugs are fixed and the possibility to use IPv6 in VRRPv2 is gone.
When implementing IPv6 / VRRPv3 we should probably base the implementation on a newer version of keepalived.

@aopdal can you please provide relevant information and not just bunch of already known info?
We need description of problem and how to reproduce it, not comments from captain obvious

On two debian 8 test VM I compiled keepalived 1.3.9 without any errors. It may be a good thing to get this latest version for our new implementation.

@syncer
Use the configurations I provided and observe the packets the router is sending out.
In the nightly build the router is sending out using the IPv6 group address
Up to 1.1.8 the router is sending out using the IPv4 group address
This makes upgrades impossible
Using VRRPv2 with both IPv4 and IPv6 virtual addresses in the same VRRP instance is only possible due to a bug in the 1.2.19 keepalived

I'm sorry if I don't express myself clear enough, but I still think this issue need some specification on how to implement the VRRPv3 in addition to VRRPv2 and not with a "dirty trick".