Page MenuHomeVyOS Platform
Create Task
Maniphest T1899

Unionfs metadata folder is copied to the active configuration directory
Closed, ResolvedPublicBUG
Actions

Description

Original task description:

Upgraded to VyOS 1.3-rolling-201912222230 rolling and noticed I still have a vyos user floating around:

kroy@route2:# show system login user vyos
 authentication {
     encrypted-password xxxxxxxxx
     plaintext-password ""
 }
 level admin

[edit]
kroy@route2:# delete system login user vyos
[edit]
kroy@route2:# commit
Failed to generate committed config

On a different server running the same version, I didn't have an issue.

This was the only logging I could find:

cp w->tw failed[boost::filesystem::copy_file: Permission denied: "/opt/vyatta/config/tmp/new_config_5542/.unionfs-fuse/system/login/user/kroy/level/node.val_HIDDEN~", "/opt/vyatta/config/tmp/tmp_5542/work/.unionfs-fuse/system/login/user/kroy/level/node.val_HIDDEN~"]
cp[/opt/vyatta/config/tmp/new_config_5874]->[/opt/vyatta/config/tmp/tmp_5874/work]
cp w->tw failed[boost::filesystem::copy_file: Permission denied: "/opt/vyatta/config/tmp/new_config_5874/.unionfs-fuse/system/login/user/kroy/level/node.val_HIDDEN~", "/opt/vyatta/config/tmp/tmp_5874/work/.unionfs-fuse/system/login/user/kroy/level/node.val_HIDDEN~"]
cp[/opt/vyatta/config/tmp/new_config_6357]->[/opt/vyatta/config/tmp/tmp_6357/work]
cp w->tw failed[boost::filesystem::copy_file: Permission denied: "/opt/vyatta/config/tmp/new_config_6357/.unionfs-fuse/system/login/user/kroy/level/node.val_HIDDEN~", "/opt/vyatta/config/tmp/tmp_6357/work/.unionfs-fuse/system/login/user/kroy/level/node.val_HIDDEN~"]

Updated description:
During commit process .unionfs-fuse directory copied from an updated configuration to the /opt/vyatta/config/active/. During a new configuration session this metadata leaks to a temporary config directory, which blocks changes, if they are trying to be done by other users. So, the issue is wider, that it seems - actually, the ability to configure a router by multiple users is almost broken. To trigger issue you need to add something into a configuration, commit, then delete previously added part of config and commit again during the same config session.

Example:
Under vyos user:

set interfaces ethernet eth0 address dhcp
commit
delete interfaces ethernet eth0 address
commit

And we have metadata folder in active config:

vyos@vyos# ls -la /opt/vyatta/config/active/
total 0
drwxrwxr-x 5 root vyattacfg 100 Apr 27 12:17 .
drwxrwxr-x 4 root vyattacfg 100 Apr 27 12:16 ..
drwxrwxr-x 4 vyos vyattacfg  80 Apr 27 12:17 interfaces
drwxrwxr-x 8 vyos vyattacfg 160 Apr 27 12:17 system
drwxrwxr-x 3 vyos vyattacfg  60 Apr 27 12:17 .unionfs-fuse

After this, changing configuration by any others users, except vyos will be impossible till the next reboot.
Unser test user:

test@vyos# set system name-server 8.8.8.8
[edit]
test@vyos# commit
Failed to generate committed config

I have not seen into commit algorithm deeply, but maybe mounting uinionfs with the -o hide_meta_files option will help us here.

Details

Difficulty level
Unknown (require assessment)
Version
1.3
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Related Objects
Search...

Event Timeline

kroy created this task.Dec 23 2019, 5:02 PM
kroy updated the task description. (Show Details)
kroy added a comment.Dec 23 2019, 5:05 PM

A reboot seems to have fixed it.

I'm wondering if it's due to on this server, I've added and deleted the kroy user a few times in testing some RADIUS stuff. Meaning maybe that stuff was owned by a user ID that doesn't exist anymore?

ddiguru added a subscriber: ddiguru.Dec 27 2019, 9:00 PM
This comment was removed by ddiguru.
ddiguru removed a subscriber: ddiguru.Dec 27 2019, 9:13 PM
hagbard changed the task status from Open to Confirmed.Jan 17 2020, 8:04 PM
hagbard triaged this task as Normal priority.
hagbard changed the task status from Confirmed to In progress.Jan 17 2020, 8:10 PM
hagbard claimed this task.
hagbard added a subscriber: hagbard.

@kroy Can you please test with the latest rolling? I can't reproduce the issue.

hagbard moved this task from Need Triage to In Progress on the VyOS 1.3 Equuleus board.Jan 17 2020, 8:11 PM
kroy added a comment.Jan 17 2020, 9:07 PM

Yep.

I hacked through how to reproduce.

Start with a fresh system.

  1. log in as default vyos user
  2. Do something set interfaces ethernet eth0 description test, commit and save
  3. create new user, set system login user vyos2 authentication plaintest-password vyos and set system login user vyos3 authentication plaintest-password vyos
  4. commit and save.
  5. login as vyos2 user. Do this again, set interfaces ethernet eth0 description test2, commit and save
  6. login as vyos3 user.
  7. delete vyos2 user, commit/save
  8. create vyos2 user, commit save
  9. Login as vyos2 user. Do this again, set interfaces ethernet eth0 description test55, commit and save

Last commit should generate the message. Probably don't need two extra users here, but it was easiest to pick out what was wrong.

It's happening because there's something leftover with the original vyos2 user's numeric ID, which the new vyos2 user doesn't match anymore. So originally vyos2 was 1001. Now it's 1003

I've actually run into a number of permissions issues with stuff under /config, especially as you migrate configs to a new host, if you've created any additional users.

hagbard added a comment.Jan 17 2020, 10:49 PM

Ack, I have already after step 5 an issue. The uids shouldn't be an issue, since the users should be all in the same group and the group has r/w permissions.

hagbard changed the task status from In progress to Confirmed.Jan 17 2020, 10:49 PM
hagbard removed hagbard as the assignee of this task.Feb 8 2020, 4:16 PM
c-po added a subscriber: c-po.Feb 9 2020, 8:50 AM

The system login CLI interface has been rewritten to XML/Python. Please try again with a newer rolling relese

Unknown Object (User) added a subscriber: Unknown Object (User).Apr 8 2020, 10:14 AM
zsdc assigned this task to Unknown Object (User).Apr 13 2020, 11:30 AM
zsdc renamed this task from Unable to delete vyos user to Unionfs metadata folder is copied to the active configuration directory.Apr 27 2020, 12:29 PM
zsdc reassigned this task from Unknown Object (User) to dmbaturin.
zsdc raised the priority of this task from Normal to High.
zsdc updated the task description. (Show Details)
pasik added a subscriber: pasik.Apr 27 2020, 3:00 PM
Viacheslav added a subscriber: Viacheslav.EditedApr 27 2020, 3:11 PM

How I reproduce it.

login as user vyos:

set interfaces ethernet eth1 vif 100 description Auto-vlan-100
set interfaces ethernet eth1 vif 105 description Auto-vlan-105
set interfaces ethernet eth1 vif 105 address 10.2.2.1/24
set interfaces ethernet eth1 vif 111 description Auto-vlan-111
commit
exit

login as user second:

delete interfaces ethernet eth1 vif
commit
exit

Login as user vyos:

vyos@r11-ansible# set interfaces ethernet eth1 vif 100 description Auto-vlan-100
[edit]
vyos@r11-ansible# set interfaces ethernet eth1 vif 105 description Auto-vlan-105
[edit]
vyos@r11-ansible# set interfaces ethernet eth1 vif 105 address 10.2.2.1/24
[edit]
vyos@r11-ansible# set interfaces ethernet eth1 vif 111 description Auto-vlan-111
[edit]
vyos@r11-ansible# commit
[ interfaces ethernet eth1 ]
Cannot get device pause settings: Operation not supported
Cannot change udp-fragmentation-offload

Failed to generate committed config

Commit failed, compare and try commit again.

vyos@r11-ansible# compare 
[edit interfaces ethernet eth1]
+vif 100 {
+    description Auto-vlan-100
+}
+vif 105 {
+    address 10.2.2.1/24
+    description Auto-vlan-105
+}
+vif 111 {
+    description Auto-vlan-111
+}
[edit]
vyos@r11-ansible# 

vyos@r11-ansible# commit
[ interfaces ethernet eth1 ]
Cannot get device pause settings: Operation not supported
Cannot change udp-fragmentation-offload

Failed to generate committed config
[edit]
vyos@r11-ansible#

Logs

root@r11-ansible:/var/log/vyatta# cat cfg-stdout.log | tail -n 10
no tap[/opt/vyatta/config/tmp/tmp_1897/active]
cp[/opt/vyatta/config/tmp/new_config_1897]->[/opt/vyatta/config/tmp/tmp_1897/active]
cp[/opt/vyatta/config/tmp/new_config_2294]->[/opt/vyatta/config/tmp/tmp_2294/work]
cp w->tw failed[boost::filesystem::copy_file: Permission denied: "/opt/vyatta/config/tmp/new_config_2294/.unionfs-fuse/interfaces/ethernet/eth1/vif/100/description/node.val_HIDDEN~", "/opt/vyatta/config/tmp/tmp_2294/work/.unionfs-fuse/interfaces/ethernet/eth1/vif/100/description/node.val_HIDDEN~"]
rm[/opt/vyatta/config/tmp/tmp_2294/work]
cp[/opt/vyatta/config/tmp/new_config_2294]->[/opt/vyatta/config/tmp/tmp_2294/work]
cp w->tw failed[boost::filesystem::copy_file: Permission denied: "/opt/vyatta/config/tmp/new_config_2294/.unionfs-fuse/interfaces/ethernet/eth1/vif/100/description/node.val_HIDDEN~", "/opt/vyatta/config/tmp/tmp_2294/work/.unionfs-fuse/interfaces/ethernet/eth1/vif/100/description/node.val_HIDDEN~"]
rm[/opt/vyatta/config/tmp/tmp_2294/work]
cp[/opt/vyatta/config/tmp/new_config_2294]->[/opt/vyatta/config/tmp/tmp_2294/work]
cp w->tw failed[boost::filesystem::copy_file: Permission denied: "/opt/vyatta/config/tmp/new_config_2294/.unionfs-fuse/interfaces/ethernet/eth1/vif/100/description/node.val_HIDDEN~", "/opt/vyatta/config/tmp/tmp_2294/work/.unionfs-fuse/interfaces/ethernet/eth1/vif/100/description/node.val_HIDDEN~"]

Ok exit discard and check interfaces

vyos@r11-ansible# exit discard
vyos@r11-ansible:~$ show interfaces 
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             192.168.122.11/24                 u/u  
eth1             10.0.0.1/24                       u/u  
eth1.100         -                                 u/u  Auto-vlan-100 
eth1.105         10.2.2.1/24                       u/u  Auto-vlan-105 
eth1.111         -                                 u/u  Auto-vlan-111

Check config again

vyos@r11-ansible# show interfaces 
 ethernet eth0 {
     address 192.168.122.11/24
     hw-id 52:54:00:8d:00:59
 }
 ethernet eth1 {
     address 10.0.0.1/24
     hw-id 52:54:00:1e:ba:76
 }
 loopback lo {
 }
[edit]
vyos@r11-ansible
jestabro claimed this task.EditedMay 4 2020, 1:53 PM
jestabro added subscribers: dmbaturin, jestabro.

Confirmed with the simplest 'login as vyos; set interfaces ... description; login as other; delete interfaces .. description; login as vyos; set ...". T2203 is likely a result of this issue. Will reassign to me.

jestabro added a comment.EditedMay 4 2020, 9:03 PM

This is a result of a regression in unionfs-fuse, fixed in commit 93c2f9b, but not contained in the debian buster package v1.0. The naming convention for directories was changed from unionfs to unionfs-fuse, and then reverted, forgetting to revert the meta directory name. Consequently, bad things can happen for packages that had defined the meta directory consistent with the pre-v1.0 naming, for example, vyatta-cfg. Two fixes have been tested with initial success: (1) patch unionfs-fuse (2) change the meta directory name in vyatta-cfg. Either fix passes initial tests, although closer scrutiny is warranted, as it is a critical issue. The downside of (1) is the need to maintain an upstream or patched version of unionfs-fuse; the downside of (2) is that we are accommodating a regression, which will break again when/if the upstream package is available. Likely the quickest solution is preferred, as this issue is implicated in several recent bug reports.

jestabro changed the task status from Confirmed to In progress.May 4 2020, 9:03 PM
jestabro added a subtask: T2203: http api: "Failed to generate committed config" .May 4 2020, 9:12 PM
jestabro mentioned this in T2203: http api: "Failed to generate committed config" .May 5 2020, 2:59 AM
jestabro changed the status of subtask T2203: http api: "Failed to generate committed config" from Open to Needs testing.
jestabro closed this task as Resolved.May 5 2020, 3:04 AM
c-po moved this task from In Progress to Finished on the VyOS 1.3 Equuleus board.May 22 2020, 10:37 AM
jestabro closed subtask T2203: http api: "Failed to generate committed config" as Resolved.May 23 2020, 7:01 PM
erkin set Issue type to Bug (incorrect behavior).Aug 31 2021, 6:02 PM
syncer edited projects, added VyOS 1.3 Equuleus (1.3.0); removed VyOS 1.3 Equuleus.Aug 30 2022, 2:01 PM
syncer moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus (1.3.0) board.