THis shows up in the logs:

Unfortunately that seems to have made the problem worse. Before, at least each host was seeing one other host. Now most of them see no hosts.

Rebuilding iso, once it finished it will have the correct version.
[...]
Get:152 http://dev.packages.vyos.net/repositories/current/vyos/ current/main libvyosconfig0 amd64 0.0.6 [841 kB]
[...]
Will test it from the iso, just for peace of mind.

Dev.packages has 0.0.06, so something goes sideways during build process, I will work on that and test. I'll take the task back and close it when resolved in ci (looking into it right now). I manually installed the package and everything works as expected.

Still same issue on 1.2.0-rolling+201901250337.

Ok, I've re-tested everything one more time to be sure. I can confirm the somewhat strange (though technically correct) behaviour of blackhole routes: they become ECMP routes if there's another route of the same distance. I would expect normal routes to override them in that case, but the kernel is following its hard and fast rule that two routes with the same distance automatically become ECMP routes, that behaviour is counter-intuitive but consistent.

Until we redesign the firewall CLI, I'm making the rules match eth0+ instead. I hope the performance impact will not be too high.

@jmlccdmd I've been testing it with EPA3 and friday's nightly build.

Nice! I will test it tomorrow for sure.

Sure. I'll set a reminder to check it out tomorrow when I have a free minute. Thanks

I've built lldpd 1.0.3, much newer than that from jessie. Luckily, the maintainer keep Debian packaging in the official repository, so it wasn't much effort to do.

I'm very interested, what is the latest image number? I will test it.

Ok, more interesting than that. In the latest image, the setup just works as described with RFC-compliant VRRP:

This problem is specific to RFC-compliant VRRP setups. Firewall design in VyOS is rather unfortunate in that rulesets are bound to interfaces. If you assign it to eth0, a rule with -i eth0 -j MyRuleset is created. RFC-compliant (shared MAC) VRRP uses those eth0v1 etc. interfaces, but since from netfilter's point of view eth0 and eth0v1 are different interfaces, those rules are never reached.

This issue uncovered more. The parser was written under an assumption that "bare" leaf and tag nodes at the top level are not allowed. In fact, while VyOS config never have them, it's by convention rather than by design, if you create a command definition for one (set foo bar), it works just fine. So, to be consistent with the original parser, the new parser should allow them.
I've removed the restriction on the grammar to allow them (see https://github.com/vyos/libvyosconfig/commit/1dd05b330f3adfe828ba3ca4c71db2e12b8968f6), and now run show configuration commands seems to work with "partial" configs just fine. The change appears to have no ill effects on parsing "normal" config, according to my testing.

Sorry. Spent the week restoring almost half a petabyte of data from backups due to a ZFS crash.

Anyone?

@jmlccdmd
I added a second router and configured conntrack-sync.
Failover and preempt failback works correct.
Both routers show statistics for the firewall rules

The problem discussed here sounds remarkably similar to what I'm seeing: https://github.com/FRRouting/frr/issues/2230

I've tried several variations on the VRRP configuration, and it doesn't seem to make any difference. As far as I can tell, nothing is wrong with VRRP. It is only relevant as a source for change in the routing table. I can demonstrate the problem on a single instance with no VRRP.

Found the bug, https://github.com/hagbard-01/vyos-1x/releases/download/1.2.0-10/vyos-1x_1.2.0-10_all.deb should fix it. As soon as You guys can confirm, I push it upstream.

@c-po All right, found it. Try it without arguments, then it ends up just as */5 * * * * root /usr/bin/logger which causes the issue. That shouldn't be too hard to fix, the existence of the cronjobfile after a reboot without the save command however is a longer journey.

Thanks that helps, I gotta review. Remote authenticated users would act like local ones by the way, pam would resolve it or if it can't be resolved, con exits with 1.

No, it does not work. The problem persist.

Nope, I used:

@c-po
*/5 * * * * cpo sg vyattacfg "/usr/bin/logger foo"

@hagbard I replaced vyos user with another one. Also image corporate setups where RADIUS is used for authing and there are no local users.

I had to pass on libvyos and OCAML, just reading and understanding a few lines took me forever. What would be the fix?

I suppose I can port the fix for |commands to it.

All right, can you please test: https://github.com/hagbard-01/vyos-1x/releases/download/1.2.0-10/vyos-1x_1.2.0-10_all.deb

OK, so the issue happens only if a) the cronjobs was executed by root and b) it modifies the config (which gets then rewritten via union-fs). I created another user called test01, the user vyos has a cron job in his name, regardless what user (test01 or vyos) the script runs, all stays healthy. As soon as the script is triggered via root, you can't set anything in your running config due to the permission changes I wrote yesterday.

But if you run only on the first router, including the VRRP setup it does not work?

With the exact same setup, I diabled vrrp on my second routers, the one in standby, with these commands :

Thanks for confirming. With 2 users, you may encounter always the issue that a cronjob locks up your ability to change the config afterwards. For now the manual workaround should help you, I'm going to revert my changes from yesterday and return to the drawing board.

Yeah. I remove the initial vyos user and add an admin and an ansible user. The admin is just for consistency across different devices.

Thx for testing.

Fixed via T1181

I wouldn't execute a scheduled script. Thats all. Do you recreate then a different user? Since all users have admin privs, the probem with the change permissions will persist. Actually makes it works, one user can block the other. So, I have to find something else out.

@hagbard I remove/change the vyos user too. So it's definitely a breaking change.

@cpo it would just exit 1. I gotta look into the possibility to see the commit user, I was under the assumption that the vyos user always exists. If there are multiple (at least 2 different) and the cron runs a root or the user (the one which did not setup the job), it will disable any config for all other users, since the filesystem permissions change. ACL's would be something which can solve it, but I have to verify it. I'll keep this task open to track it. Do you just replace the vyos user, or are you using root only in your config?

vyos-1.2.0-rolling+201901181924-amd64.iso fixed it for me.

Depending on the task which needs to be executed a script might need to be run as root.

@yun can you please test with the latest rolling?

@kroy install http://dev.packages.vyos.net/repositories/current/vyos/pool/main/v/vyos-1x/vyos-1x_1.2.0-10_all.deb and try again, I have the changes in that package and tonights rolling will have it too. I couldn't find anywhere a requirement that the cronjobs need root, so I switched it to always run as vyos which keeps the file system permissions intact. Test it on a test machine first, but it should now do what you want, I used your script code from above, but didn't have any real ospf adjacency with any other route, but that shouldn't matter at all. Let me know the results please.

OK, I think I found it, however so far I can only give you a quick workaround rather than solving it.
Short explanation, if you setup cron, your script is executed as root which changes the permissions for the configs on union-fs and the directories, that's why already a set fails, it can't simply write as user vyos to the directory.
To get your stuff working, try the following (preferably on a test box, I used the rolling from tonight but any 1.2 image should work if it's not older than 3 months or so)

The 'commit' causes the issue, but right now I'm not sure why.

The latest rolling did seem to correct the base problem. That being cron scripts running breaking the ability to edit config afterwards.

In T1178#30992, @kroy wrote:

@hagbard Note that a reboot does fix the ability to edit configuration again until the next time the cron script runs.

@hagbard Note that a reboot does fix the ability to edit configuration again until the next time the cron script runs.

I'm going to implement it into the configuration, which will assure that is it going to be the last step executed after a reboot.

OK, things is more clearly now.
If you don't have any L2-filters between eth1 interfaces of VyOS instances I could recommend you first to change configuration to something like this (based on your configuration from first message):
Router 1:

@jmlccdmd
I have recreated your setup with Vyos 1.2.0-rc10 and it seems to be working correctly

@bjtangseng This is definitely a NAT issue, if i change the local_ts = dynamic[gre] in /etc/swanctl/swanctl.conf to local_ts = *.*.*.*/32[gre] i can replicate the error you get.

If you can see issue "T1100: Spoke site dynamic IP over NAT connect to Hub site."