Page MenuHomeVyOS Platform

Ethernet interface is automatically disabled when removing it from bond
Closed, ResolvedPublicBUG

Description

Ethernet interface is automatically disabled when removing it from bond

vyos@vyos# ip a show dev eth1
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 0c:e6:b1:a1:bd:01 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ee6:b1ff:fea1:bd01/64 scope link tentative 
       valid_lft forever preferred_lft forever

-------------------------------------------------------------------------------

[edit]
vyos@vyos# set interfaces bonding bond0 member interface eth1

-------------------------------------------------------------------------------

[edit]
vyos@vyos# compare 
[edit interfaces bonding bond0 member]
+interface eth1
[edit]

-------------------------------------------------------------------------------

vyos@vyos# commit
[edit]

-------------------------------------------------------------------------------

vyos@vyos# ip a show dev eth1
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc pfifo_fast master bond0 state DOWN group default qlen 1000
    link/ether 0c:e6:b1:a1:bd:00 brd ff:ff:ff:ff:ff:ff

-------------------------------------------------------------------------------

[edit]
vyos@vyos# delete interfaces bonding bond0 member interface eth1

-------------------------------------------------------------------------------

[edit]
vyos@vyos# compare 
[edit interfaces bonding bond0 member]
-interface eth1

-------------------------------------------------------------------------------

[edit]
vyos@vyos# commit

-------------------------------------------------------------------------------

[edit]
vyos@vyos# ip a show dev eth1
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 0c:e6:b1:a1:bd:01 brd ff:ff:ff:ff:ff:ff

Details

Difficulty level
Normal (likely a few hours)
Version
1.3
Why the issue appeared?
Implementation mistake
Is it a breaking change?
Perfectly compatible

Event Timeline

g.skupien created this task.
pasik added a subscriber: pasik.May 27 2020, 5:14 PM
g.skupien changed the task status from Open to Confirmed.May 27 2020, 8:53 PM
g.skupien updated the task description. (Show Details)
jjakob added a subscriber: jjakob.May 28 2020, 8:30 AM

This is difficult to solve with the current config syntax where the bond and bridge members are under the bonding and bridge nodes. When modifying bond or bridge members, only interfaces-bonding.py or interfaces-bridge.py is called, which can't modify the interfaces themselves, as all the interface logic (adding and deleting addresses) is in the interface script itself (e.g. interfaces-ethernet.py). The thing that says which script to run is the owner attribute of the interface node, which is ran by vyatta-cfg scripts on commit.

The old syntax has bond and bridge member under the interface that is the member itself, so that would be easy to solve in that case, as the interface would remove itself from the bond or bridge and reassign configured addresses in one go.

The first and quickest solution I can think of in the current config syntax is to detect which interface script the member interface being removed is, and calling that script after removal to reconfigure it. This workaround wouldn't work if commit staging were implemented in the future.
The 2nd solution would be, if it's possible, to set the owner of the "member interfaces" node to a new script, that would detect the type of member interfaces and call the corresponding interface code. If the syntax were "member interfaces (ethernet|...)" that would be easy, as the owner of "member interfaces ethernet" could simply be set to interfaces-ethernet.py (if it's possible to set a different owner for a subset of nodes of a parent node with a different owner).
The 3rd solution would be to migrate back to the previous syntax where bond/bridge members are configured under the interface itself, That would require minimal changes, just moving some methods in BondIf/BridgeIf/Interfaces around, but the behavior would be correct (a removal of membership would be a change on the interface being removed, so the same script removing itself from the bond/bridge would reconfigure its addresses in one go without calling other scripts).

Personally, under the current limitations, I think the 3rd solution would be the cleanest and most correct, with a negative of required syntax migration to a previous syntax. The other 2 are more hacks IMO.

@jjakob, thanks for your opinion. What if we just add a couple of lines into the apply() function in the interfaces-bonding.py:
https://github.com/vyos/vyos-1x/compare/current...L6NqLW:T2515?expand=1

Seems to work.

I just did a few more tests and the reason why IPv6 is gone is that the interface is being automatically disabled after it leaves bond membership. I will change the description accordingly.

g.skupien renamed this task from IPv6 link-local address is gone after removing the interfaces from bond to Ethernet interface is automatically disabled when removing it from bond.May 28 2020, 8:37 PM
g.skupien updated the task description. (Show Details)

That's basically re-implementing and duplicating code from the ethernet script. It would work for bonding and for the link-local, but I'm thinking there may be other attributes that enslaving it to a bond (or bridge) may have changed (MTU?) that I don't know if they're changed back by the kernel after unenslaving it. It would quickly become a kludge.
You'd also need to do the same in the bridge interface, but there there can be any interface type enslaved, so you'd need to first get the interfaces config path (via Section). You'd end up with 2 pieces of code that are slightly different that duplicate code from the interface scripts (rather I think it's been moved into configdict.intf_from_dict).
It is possible, but I don't like it at all.

One more question/proposition.
Before ethX is added to the bond its IP addresses are being removed (that's good). Then, when ethX is removed from the bond it is being left as disabled. If we manually re-enable that interface the interfaces-ethernet.py script is run and it configures all its settings (including IPv6 link-local address). Therefore, maybe it is not that bad idea to leave the interface disabled when it leaves the bond. So, what if we just add a warning message to inform users that bond leaving interface will stay disabled until it is manually re-enabled.

c-po added a subscriber: c-po.Jun 4 2020, 5:55 PM

I do not like this behavior as the config states the interface is "not disabled" but from an OS point of view it is - this is inconsistent and thus simply wrong.

We should "obey" the pattern that CLI and OS have the same state representation.

A workaround would be to simply call interfaces-ethernet.py from the bond script itself after it removes it from the bond.
The consequence of migrating bond member config from the interface node to the bond node, the same problem exists for bridges as well.

c-po added a comment.Aug 16 2020, 7:06 PM

Please retest with the latest rolling as bond/ethernet went through a major refactoring.

Tested in the latest rolling release and observed that after deleting the member interface, the assigned interface is remained in the admin down state.

vyos@vyos:~$ date
Sun 13 Sep 2020 12:19:52 PM UTC
vyos@vyos:~$ sh ver

Version:          VyOS 1.3-rolling-202009130713
Release Train:    equuleus

Built by:         autobuild@vyos.net
Built on:         Sun 13 Sep 2020 07:13 UTC
Build UUID:       ba95bd3f-a5b8-4970-985b-6c1729055ceb

Added the interface as member:

vyos@vyos# set int bonding bond0 member int eth1
[edit]
vyos@vyos# commit
[edit]
vyos@vyos# ip a sh dev eth1
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast **master bond0 state UP** group default qlen 1000
    link/ether 50:0c:00:0b:00:01 brd ff:ff:ff:ff:ff:ff

Deleted the interface member

vyos@vyos# del int bonding bond0 member int eth1
[edit]
vyos@vyos# commit
[edit]

vyos@vyos# ip a sh dev eth1
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 50:0c:00:0b:00:01 brd ff:ff:ff:ff:ff:ff
[edit]

vyos@vyos:~$ sh int
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
bond0            -                                 A/D
eth0             192.168.255.92/24                 u/u
eth1             -                                 A/D
eth2             -                                 u/u
eth3             -                                 u/u
lo               127.0.0.1/8                       u/u
                 ::1/128
				 
				
vyos@vyos# run sh conf
interfaces {
    bonding bond0 {
        member {
        }
    }

Even after deleting the entire bond configuration, the interface remains down:

vyos@vyos# del int bonding bond0
[edit]
vyos@vyos# comit

  Invalid command: [comit]

[edit]
vyos@vyos# commit
[edit]
vyos@vyos# ip a sh dev eth1
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 50:0c:00:0b:00:01 brd ff:ff:ff:ff:ff:ff

But if I delete the bond interface initially, then the interface comes back as "UP":

vyos@vyos# set int bonding bond0 member interface eth2
[edit]
vyos@vyos# commit
[edit]

vyos@vyos# ip a sh dev bond0
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 50:0c:00:0b:00:02 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::74:fff:fe6c:1ed9/64 scope link
       valid_lft forever preferred_lft forever
[edit]
vyos@vyos# ip a sh dev eth2
4: eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP group default qlen 1000
    link/ether 50:0c:00:0b:00:02 brd ff:ff:ff:ff:ff:ff
[edit]
vyos@vyos# del int bonding bond0
[edit]
vyos@vyos# commit
[edit]
vyos@vyos# ip a sh dev eth2
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 50:0c:00:0b:00:02 brd ff:ff:ff:ff:ff:ff
[edit]
vyos@vyos# date
Sun 13 Sep 2020 01:04:06 PM UTC
[edit]
c-po claimed this task.Tue, Sep 15, 3:59 PM
c-po triaged this task as Normal priority.
c-po changed Difficulty level from Unknown (require assessment) to Normal (likely a few hours).
c-po changed Why the issue appeared? from Will be filled on close to Implementation mistake.
c-po changed Is it a breaking change? from Unspecified (possibly destroys the router) to Perfectly compatible.
c-po closed this task as Resolved.Tue, Sep 15, 4:56 PM