Page MenuHomeVyOS Platform

Add VRF support
Closed, ResolvedPublicFEATURE REQUEST

Tokens
"Like" token, awarded by Lillecarl."Like" token, awarded by Raeven."Like" token, awarded by xzjt."Like" token, awarded by hard."Like" token, awarded by Maltahl."Like" token, awarded by rherold."Like" token, awarded by isithran."Like" token, awarded by m.cremers."100" token, awarded by mrciarano.
Assigned To
Authored By
afics, Mar 31 2016

Description

Details

Difficulty level
Hard (possibly days)
Version
-
Is it a breaking change?
Unspecified (possibly destroys the router)

Related Objects

StatusSubtypeAssignedTask
ResolvedFEATURE REQUESTc-po
ResolvedFEATURE REQUESTc-po

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
pete added a subscriber: pete.Sep 14 2019, 9:54 AM
maznu added a subscriber: maznu.Sep 23 2019, 3:39 PM
hard awarded a token.Dec 18 2019, 10:10 PM
xzjt awarded a token.Dec 19 2019, 2:02 PM
Raeven added a subscriber: Raeven.

I have a use case for VRF / Linux namespace with VyOS and was wondering if there was a way to break this task down, first allowing the manual creation of VRF/namespace using the CLI (without linking the work to BGP). I will be able to contribute the required changes to the vyos-1x repository should I be given enough direction on how you would like to proceed.

Identified configuration format change:

set vrf <number> description <foo>
set interfaces ethernet eth0 vrf <name>
set service ssh vrf <name>
c-po added a subscriber: c-po.EditedFeb 5 2020, 5:27 PM

Okay, just to give you some guidance, this is what I hacked up in 5 minutes... don't take it as complete

vyos-1x/interface-definitions/vrf.xml

<?xml version="1.0"?>
<interfaceDefinition>
  <tagNode name="vrf" owner="${vyos_conf_scripts_dir}/vrf.py">
    <properties>
      <help>Virtual Routing and Forwarding</help>
      <constraint>
        <!-- maximum suported name by Linux Kernel must be checked -->
        <regex>.{1,32}$</regex>
      </constraint>
      <constraintErrorMessage>VRF name not allowed or to long</constraintErrorMessage>
    </properties>  
    <children>
      <leafNode name="description">
        <properties>
          <help>IP address</help>
          <constraint>
            <regex>.{1,254}$</regex>
          </constraint>
        </properties>
      </leafNode>
    </children>
  </tagNode>
</interfaceDefinition>

Then create a file which can be included by e.g. ethernet or bond interfaceDefinition vyos-1x/interface-definitions/include/interface-vrf.xml.i:

<leafNode name="vrf">
  <properties>
    <help>VRF instance name</help>
    <completionHelp>
      <path>vrf</path>
    </completionHelp>
    <constraint>
      <!-- maximum suported name by Linux Kernel must be checked -->
      <regex>.{1,32}$</regex>
    </constraint>
  <constraintErrorMessage>VRF name not allowed or to long</constraintErrorMessage>
  </properties>
</leafNode>

and vyos-1x/src/conf_mode/vrf.py should be used to create global VRFs, which will create the tables ip link add red type vrf table 1 and bring them up ip link set dev red up

At a later point class VLANIf at https://github.com/vyos/vyos-1x/blob/current/python/vyos/ifconfig.py#L1020 should be made VRF aware so an interface can be placed in a VRF.

Please also see https://www.kernel.org/doc/Documentation/networking/vrf.txt

while it would be quite nice to have a Cumulus 4.0+ like default management VLAN and make all services management aware (which would quite considerably increase the among of work to get something out), I am proposing instead make sure that all services running on the default VRF are available on VRF as a first step.

Could be away. But from my experience most people use vrf to seperate managment from production, and as second prio seperate customers and so on.
But the managment vrf must not be the "default" vrf.

Here is a patch to implement VRF. Binding is set to work on all VRF for daemon so that BGP and other protocols will work on all VRF.
https://gist.github.com/thomas-mangin/7704c538d905190bd05cfe613bd9f4f5

It is working for ethernet interface as far as I could test, other interface types were not tested yet.

This patch implements a work-around for T2027

thomas@vyos:~$ configure
[edit]
thomas@vyos# show vrf
 name blue {
     table 100
 }
 name red {
     table 200
 }
[edit]
thomas@vyos# show interfaces ethernet eth2
 hw-id 08:00:27:68:d0:b1
 vrf blue
[edit]
thomas@vyos:~$ show vrf
interface         state    mac                flags
---------         -----    ---                -----
blue              up       36:7b:15:47:9e:df  noarp,master,up,lower_up
red               up       16:66:7f:42:a0:45  noarp,master,up,lower_up
thomas@vyos:~$ show vrf name blue
interface         state    mac                flags
---------         -----    ---                -----
blue              up       36:7b:15:47:9e:df  noarp,master,up,lower_up
thomas@vyos:~$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:fa:12:53 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:0d:25:dc brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master blue state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:68:d0:b1 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master blue state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:f0:17:c5 brd ff:ff:ff:ff:ff:ff
6: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 36:7b:15:47:9e:df brd ff:ff:ff:ff:ff:ff
7: red: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 16:66:7f:42:a0:45 brd ff:ff:ff:ff:ff:ff

show interfaces was not made vrf aware as it is implemented the "old way" and I am not sure how to perform the change (the policy seems to be to only accept patched for vyos-1x)

At this point, I would need some help with the review to move it forward.

In T31#53189, @rherold wrote:

Could be away. But from my experience most people use vrf to seperate managment from production, and as second prio seperate customers and so on.
But the managment vrf must not be the "default" vrf.

Implementing a management VRF and working all the binding would make this work much more complex. I am not opposed to look into it later on.

rherold added a comment.EditedFeb 12 2020, 12:43 PM

I think we should make somewhere a list of services and which level of vrf support they have.
Openssh for example has build in support for vrf

And we should declare which services should be available in the management vrf.
From my point of view:

  • ssh
  • ntp
  • syslog
  • snmp
  • API
c-po added a comment.Feb 19 2020, 8:26 AM

Why do we need to explicitly create the routing table? why not name the routing table entry like the VRFs name? We should try to keep the CLI as minimal as possible. More CLI nodes, more headache.

When I think about the the Cisco VRF implementation I ise I just create it with a name and refer to it. It creates a routing table for me and I do not care. When adding routes to the table I refer to the VRF name and not an arbitrary number.

Downside of the explicit table number is a user now needs to remember the VRF and table mapping.

We could indeed create the VRF as we parse interfaces, and auto-allocate the VRF number, removing this control from the user.

I would have to move the VRF creation code within the Interface class and any typo on names would create two VRF as there would be no sanitisation, but auto-completion could be done via /etc/iproutes/rt_tables

I am not aware of any "end of configuration check" hook which could be used to parse all the interfaces and notice that a VRF is not used anymore and remove it.
This would have to be run when ALL the configuration has been parsed and acted upon as VRF can be used in different interfaces, which have different handlers.
If we do not, we would end up leaking resources.

c-po added a comment.Feb 19 2020, 7:37 PM

Auto completion should be done on a per CLI path:

<completionHelp>
  <path>vrf name</path>
</completionHelp>

Assuming the VRFs name will be set using set vrf name 'red'

c-po added a comment.Mar 4 2020, 8:00 PM

Adding interfaces to VRFs seems to be a bit of a hazzle:

  • set vrf name <name> member <interface>
    • + has the advantage of having everything in one place
    • - routes could leak during system boot between VRFs until interfaces are properly enslaved to a VRF
  • set interfaces <type> <interface> vrf <name>
    • + has the advantage of knowing the right per interface priority
    • - vrf configuration is split accross different sections

Pondering pro and con I go with option #2 now

c-po added a comment.Mar 4 2020, 8:51 PM

Adding a dummy interface to VRF:

VRF foo

vyos@vyos# show interfaces dummy dum1
 address 1.1.1.1/32
 vrf foo
vyos@vyos# sudo ip vrf exec foo ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=0.073 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=64 time=0.082 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=64 time=0.070 ms
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 30ms
rtt min/avg/max/mdev = 0.070/0.075/0.082/0.005 ms

VRF default

vyos@vyos# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=59 time=7.92 ms

see the latency

c-po changed the task status from Open to In progress.Mar 4 2020, 9:15 PM
c-po claimed this task.
c-po set Is it a breaking change? to Unspecified (possibly destroys the router).
c-po changed the task status from In progress to Needs testing.Mar 6 2020, 6:38 PM
c-po changed Difficulty level from Easy (less than an hour) to Hard (possibly days).
c-po moved this task from Need Triage to In Progress on the VyOS 1.3 Equuleus board.
c-po added a comment.Mar 8 2020, 6:09 PM

VRF route leaking needs to be added to CLI. Routes can be leaked using:

FRR vtysh: ip route 192.0.2.0/24 172.18.204.254 vrf red nexthop-vrf default

This will add a routing table entry into VRF red for destination 192.0.2.0/24 going via the default VRF

vyos# show ip route vrf red
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route


VRF red:
K   0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 00:33:44
S>* 192.0.2.0/24 [1/0] via 172.18.204.254, eth0.204(vrf default), 00:01:01
c-po closed this task as Resolved.Mar 9 2020, 7:28 PM
c-po changed the status of subtask T2111: VRF add route leaking support from Open to In progress.
c-po moved this task from In Progress to Finished on the VyOS 1.3 Equuleus board.Mar 22 2020, 8:52 AM