Page MenuHomePhabricator

L2TPv3 interface gets not loaded after reboot with a configuration error
Closed, InvalidPublicBUG

Description

We have the following configuration with seems to be correct and runs on VyOS 1.2.0-RC3 / RC4
(Its running on HW Protectli FW6A)

interfaces {
  bridge br0 {
      aging 300
      description "L2TPV3 Bridge Grp Mgt"
      hello-time 2
      max-age 20
      priority 32768
      stp false
  }
  ethernet eth0 {
      address 10.52.193.14/29
      duplex auto
      hw-id 00:e0:67:0a:6b:6c
      mtu 9000
      smp-affinity auto
      speed auto
  }
  ethernet eth1 {
      address dhcp
      duplex auto
      hw-id 00:e0:67:0a:6b:6d
      smp-affinity auto
      speed auto
  }
  ethernet eth2 {
      duplex auto
      hw-id 00:e0:67:0a:6b:6e
      smp-affinity auto
      speed auto
  }
  ethernet eth3 {
      duplex auto
      hw-id 00:e0:67:0a:6b:6f
      smp-affinity auto
      speed auto
  }
  ethernet eth4 {
      duplex auto
      hw-id 00:e0:67:0a:6b:70
      smp-affinity auto
      speed auto
  }
  ethernet eth5 {
      bridge-group {
          bridge br0
      }
      description "L2 OPT4 Grp Mgt"
      duplex auto
      hw-id 00:e0:67:0a:6b:71
      smp-affinity auto
      speed auto
  }
  l2tpv3 l2tpeth0 {
      bridge-group {
          bridge br0
      }
      description "L2 Tunnel Grp Mgt"
      destination-port 5000
      encapsulation ip
      local-ip 10.52.193.14
      mtu 8958
      peer-session-id 110
      peer-tunnel-id 10
      remote-ip 10.52.192.174
      session-id 110
      source-port 5000
      tunnel-id 10
  }
  loopback lo {
  }
}
protocols {
  static {
      route 0.0.0.0/0 {
          next-hop 10.52.193.9 {
          }
      }
  }
}
service {
  ssh {
  }
}
system {
  config-management {
      commit-revisions 100
  }
  console {
      device ttyS0 {
          speed 9600
      }
  }
  host-name BRF-R309-RT02
  login {
      user vyos {
          authentication {
              encrypted-password $6$LPWjXZ.50b/LcPZz$oEnBDH8oWD0Y0FQ1C7E1SHfjBRzkf1bOpuwC4XfSWJL16i6JWiL/aJ/2NmgMsZMsrENsXRGhhe7vrSQJEaKlG0
              plaintext-password ""
          }
          level admin
      }
  }
  ntp {
      server 0.pool.ntp.org {
      }
      server 1.pool.ntp.org {
      }
      server 2.pool.ntp.org {
      }
  }
  syslog {
      global {
          facility all {
              level info
          }
          facility protocols {
              level debug
          }
      }
  }
  time-zone Europe/Zurich
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "broadcast-relay@1:cluster@1:config-management@1:conntrack-  sync@1:conntrack@1:dhcp-relay@1:dhcp-server@5:firewall@5:ipsec@4:mdns@1:nat@4:qos@1:quagga@3:system@9:vrrp@2:wanloadbalance@3:webgui@1:webproxy@1:zone-policy@1" === */
/* Release version: 1.2.0-rc3 */

After a reboot the configuration is not loaded completely. The following part is missing and can be loaded with a commit:

  vyos@BRF-R309-RT02:~$ configure
  [edit]
  vyos@BRF-R309-RT02# load
  Loading configuration from '/config/config.boot'...

  Load complete.  Use 'commit' to make changes active.
  [edit]
  vyos@BRF-R309-RT02# compare
  [edit interfaces]
  +l2tpv3 l2tpeth0 {
  +    bridge-group {
  +        bridge br0
  +    }
  +    description "L2 Tunnel Grp Mgt"
  +    destination-port 5000
  +    encapsulation ip
  +    local-ip 10.52.193.14
  +    mtu 8958
  +    peer-session-id 110
  +    peer-tunnel-id 10
  +    remote-ip 10.52.192.174
  +    session-id 110
  +    source-port 5000
  +    tunnel-id 10
  +}
   [edit]

  vyos@BRF-R309-RT02# commit
  Warning: priority inversion [interfaces l2tpv3 l2tpeth0 mtu](461) <= [interfaces l2tpv3 l2tpeth0](800)
         changing [interfaces l2tpv3 l2tpeth0 mtu] to (801)
  [ interfaces l2tpv3 l2tpeth0 bridge-group ]
  Adding interface l2tpeth0 to bridge br0
 
  [edit]
  vyos@BRF-R309-RT02#


  vyos@BRF-R309-RT02:~$ show interfaces
  Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
  Interface        IP Address                        S/L  Description
  ---------        ----------                        ---  -----------
  br0              -                                 u/u  L2TPV3 Bridge Grp Mgt
  eth0             10.52.193.14/29                   u/u
  eth1             172.17.0.100/24                   u/u
  eth2             -                                 u/D
  eth3             -                                 u/D
  eth4             -                                 u/D
  eth5             -                                 u/u  L2 OPT4 Grp Mgt
  l2tpeth0         -                                 u/u  L2 Tunnel Grp Mgt
  lo               127.0.0.1/8                       u/u
                 ::1/128

				 
  vyos@BRF-R309-RT02:~$ show interfaces l2tpv3 l2tpeth0
  l2tpeth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8958 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether 0a:a1:c0:12:18:a6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8a1:c0ff:fe12:18a6/64 scope link
       valid_lft forever preferred_lft forever
    Description: L2 Tunnel Grp Mgt

    RX:  bytes    packets     errors    dropped    overrun      mcast
         62606         44          0          0          0          0
    TX:  bytes    packets     errors    dropped    carrier collisions
         63000         51          0          0          0          0
  vyos@BRF-R309-RT02:~$

Are we have an error in the configuration, are we doing something wrong? It looks like it has something to do with the MTU Settings on the ethernet interfaces and the l2tp interface.

Regards Mätthi

Details

Difficulty level
Unknown (require assessment)
Version
VyOS-1.2.0-RC3, VyOS-1.2.0-RC4
Why the issue appeared?
Will be filled on close

Event Timeline

Maetthi created this task.Oct 27 2018, 9:31 AM
syncer triaged this task as Normal priority.

We did some tests with RC5. Sometimes the complet configuration gets loaded after a reboot, but most of the time not. Maybe a timing issue?

pasik added a subscriber: pasik.Nov 4 2018, 11:21 AM

I have tested, the reason may be that your interface and the switch have not been UP. The route leading to the default route or L2TPV3 PEER address does not take effect. The ip l2tp command does not work.
This can be configured without any network cable. To configure L2TPV3, you will find that it is unable to commit.

My current method is:
    begin:

[ -d /sys/module/l2tp_eth ] || sudo modprobe l2tp_eth
[ -d /sys/module/l2tp_netlink ] || sudo modprobe l2tp_netlink
if [ "$VAR(./encapsulation/@)" = "ip" ]; then
  if [ ! -d /sys/module/l2tp_ip ]; then
    sudo modprobe l2tp_ip
  fi

  if [ ! -d /sys/module/l2tp_ip6 ]; then
    sudo modprobe l2tp_ip6
  fi
fi

if [ ! -d /sys/class/net/dum-l2tpv3 ]; then
  [ -d /sys/module/dummy ] || sudo modprobe dummy
  ip link add name dum-l2tpv3 type dummy
  ip link set dum-l2tpv3 up
  vtysh -c "configure terminal" \
            -c "ip route 0.0.0.0/0 dum-l2tpv3 254";
fi
Maetthi added a comment.EditedDec 10 2018, 1:47 PM

We did some testing with the RC10 Version as requested by the blog.

The issue is still there. After a reboot the settings for the l2tpv3 interfaces are not loaded and we have a difference between the running and the saved configuration. The l2tpv3 interface is bound to eth0 with has a static configuration.

Just for a test, we changed our management interfaces (eth1) from dhcp to static. This means we also changed the default static route setting from dchp to a static address.

In this case, the whole configuration got loaded. We testet that 3 times, no failer. As soon as we set the management interface back to dhcp and set also the default routing back to use the default route provided by dhcp, the issue was here again.

I can provide some logfiles, if needed. Btw. Is there something like a debug mode to get more information what is happening during boot and during applying the stored configuration?

commo added a subscriber: commo.Dec 10 2018, 5:40 PM

I suspect this is the same issue as T1080. if the destination ip of the l2tpv3 tunnel is unreachable(no default route) it wont load. I suspect its loading the l2tpv3 config before DHCP has installed a default route. My workaround is to install a very low metric static default route, that will get overwritten by OSPF(in my case).

syncer assigned this task to hagbard.Feb 8 2019, 12:16 AM

@Maetthi You reach 10.52.192.174 via router 10.52.192.9, is that correct in your config example above? If so, did you try to capture the traffic on 10.52.192.9 for .14 and 174?
I've used your example above, however on vm's only and changed your /29 to a /24 and didn't find any problems. So the question is if you can see on .9 the SCC* messages?

hagbard changed the task status from Open to On hold.May 6 2019, 9:21 PM

@hagbard Yes, we reach the other tunnel endpoint via router 10.52.192.9. This is also running without any issue. Only after a reboot not the whole configuration gets sometimes loaded. If this fails, you are able to do a load and a commit. This will setup the missing part without any issue.

We are on hardware (Protectli FW6A). The behavier also changed, when we canged the default route from dhcp to a static one. I have also not been able to reproduce it on a VM. This might be a timing issue during boot up.

The setup is placed on a customers side. I have just asked, if we still have the issue. I also gave them the version 1.2.1.

As soon as I have some feedback, I will get back to you. Let me know, if I can help with providing logs or where I could get more information what is happening during boot.

We have to consider a situation. If all physical interfaces do not have a link UP, the configuration of l2tpv3 cannot be configured. This part seems to be a problem with IP l2tp.

@rodge.liu I think that might be the issue. I have seen similar behavior on DHCP set interface when the DHCP take quite some time to reply.

@Maetthi Is that still an issue? You need an uplink first and the interface needs to be able to reach (route) the other side.

@hagbard
Yes, it is still an issue. I just did some test with my hardware. If all interfaces are disconnected, I can't create the l2tpv3 interface. If I have the interface up and a default route set, I can create a l2tpv3 interface on the system.

I I restart the system, the configuration gets loaded, if the interface with the l2tpv3 tunel interface on it is connected to the network (means interface will be up).
If the interface is not connected, the loading of the configuration will fail during a reboot.

Used version was vyos 1.2.0. It behaves as you have described it.

commo added a comment.Jun 3 2019, 5:44 PM

I have ran into this as well where we have a router plugged into a juniper switch, the router loads faster then the juniper so at the moment the router comes up all the interfaces are down, then the switch comes up.

@commo That's an issue I can't solve. l2tp needs an uplink and needs to be able to reach the destination to exchange protocol information such as the tunnel id.

hagbard closed this task as Invalid.Fri, Jun 28, 9:35 PM