I took a look, but was unable to figure out how to finagle VyOS to fix it.

Attached are the pcap and debug logs from a simple setup as outlined above, two hosts. "Master" distributing the route.

Yep. Can confirm issue is fixed with the latest hot fix.

Strange. I’ve seen that error a lot. Every time it’s been when I’ve forgotten to checkout current after cloning the repo.

@Merijn make sure you git checkout currenton everything.

Can you describe the system and disks involved?

@jmlccdmd See the fix in T1120. Does changing the rootdelay to something longer fix it for you? Or turning off acpi as suggested?

Yeah, it wasn't really a workable solution for me either and I too had to roll back. But it would be good to confirm that is the problem.

@lbv2rus There might actually be a few problems here. We might have hacked out that it's the interface-route with the custom routing table that's causing the problem.

I’ll add that I think this is happening because of the .252 and .254 addresses. The 254 address connects, but the 252 address is marked in a constant state of connecting.

There might actually be a bit of a deeper problem here, somewhat conditional on some static interface routing. On an broken system, it does say something about staticd starting

And more info:

I tracked down what is causing this.

Too add, routes are present in FRR

THis shows up in the logs:

Unfortunately that seems to have made the problem worse. Before, at least each host was seeing one other host. Now most of them see no hosts.

Sure. I'll set a reminder to check it out tomorrow when I have a free minute. Thanks

Sorry. Spent the week restoring almost half a petabyte of data from backups due to a ZFS crash.

Yeah. I remove the initial vyos user and add an admin and an ansible user. The admin is just for consistency across different devices.

@hagbard I remove/change the vyos user too. So it's definitely a breaking change.

The latest rolling did seem to correct the base problem. That being cron scripts running breaking the ability to edit config afterwards.

@hagbard Note that a reboot does fix the ability to edit configuration again until the next time the cron script runs.

Okay, spent the whole day messing with this and I've tracked it down so it's reproducible.

Superseded by T1178

Seems to be a problem with just that build. I'll install a newer rolling when I get a chance and see if that corrects it.

Edit... actually I can't update anything:

I was mistaken. Seems to have lost the route again.

As of (at least) VyOS 1.2.0-rolling+201901090739 I believe this problem to be corrected.

@Line2 Thanks for the update.

With VyOS as the edge:

@syncer Just to confirm, the above pull request integrates this and can be merged. I'm not sure if there's a status change I should make to this or just leave it be. Thanks

This is probably a duplicate of T1120, which should be fixed now.

Unfortunately still a problem in the EPA2 release. 30 minutes hits, and I cease getting a default route on my entire OSPF infrastructure

I can confirm that this is broken everywhere "get_response" is used, where the default should be "no". GPU signatures are ignored, disks are deleted, etc. I'll work on making up a sane replacement.

Completed. PR: https://github.com/vyos/vyatta-cfg-system/pull/95

I've been using my branch for a few daily upgrades now, and it seems flawlessly, minus one thing.

@danhusan I'd be curious if an alternative of adding rootdelay=10 instead of the acpi=off works. That may or may not do anything depending on how the kernel is built though.

Yeah, I'm not familiar enough with things to understand why it would be trying to mount the bare RAID partitions, not to mention the actual bare drives

While I'm not sure if it's related, it looks like your system has a buggy ACPI implementation. Sometimes that can cause some weird behaviour.

I’ve been trying to research this a little, and I can’t duplicate. But I suspect it’s because I have fast disk. Your first output says it’s going to take over two hours for a resync.

I pulled down vyos-frr submodule and the above-mentioned commit is present.

Maybe this isn't the same issue? Still a problem in RC11 unfortunately.

Here's a branch that makes this work on an upgrade. For now, it wouldn't cover the initial install, but only subsequent upgrades. This covers a MAJOR pain point for me where I lose my bash history on an upgrade.

I’ve been using ospfv3 on all the RCs and definitely haven’t seen this behavior. OSPFv3 seems immune to the problem I’m seeing in T1020

Just to confirm that RC10 made this work for me too

Would it make sense to create this as a separate partition during installation? Instead of trying to preserve? Given my recent work on the EFI stuff, I've got an idea where this might happen.

@c-po Thanks. Between the legit problem this task revealed early-on, some bad timing with the vyatta-config-migrate from earlier today, and a PEBKAC error, it looks like this might be resolved now.

This seems to be working now. It's working both on my normal build tree and a brand new one, so it looks like I was just attempting when some cache somewhere hadn't updated

I was building yesterday fine except for the above-mentioned vyatta-cfg-firewall problem.

Now a simple build is failing with:

I just tested it on my 8.11 VM and it fails to build there as well for the reason you mentioned (the lack of initrd)

@max1e6 Is your build directory on NFS? I had that problem when my build directory was being stored on NFS

@max1e6 I don't think so. I was building on a Jessie VM for a while as well as Docker, and both fail

Just to confirm, still a problem with RC10

There are definitely strange happenings.

Okay, I think I've nailed down what's going on. A fresh build with no modules (git clone, configure, make iso), works fine.

Actually I spoke too soon. The next run resulted in this, on a 100% fresh build tree.

@hagbard Thanks! I guess I was just running make clean. Removing the build directory and rerunning configure seems to have corrected it.

It's actually failing somewhere else now:

I'll add here that I've got a reasonably complex OSPF setup with around 10 hosts. I converted it over to VyOS when the first RC came out and I haven't seen this issue at all, and I'm constantly rebooting hosts. Currently upgraded the whole setup to RC10 and not a single host crashed. It's worth adding that I've had a bunch of Mikrotiks in the mix at a time and no problem there either.

This might be related to T1079

Pull Request to add the option

Unfortunately not corrected in RC9. After 30 minutes the default route fails to distribute.

This makes sense. As I understand it, it just installs another copy of the .efi file to a more “universal” location.

@bswinnerton Ha, no problem. I just figured with a bit more insight maybe it would be worth breaking this out into two tasks

@bswinnerton I think the "in certain situations" should be defined "when the config on disk is invalid for whatever reason"

Confirmed that the posted RC9 works now too

confirmed. This rolling release has a working wireguard.

As another question, is that a bug that ipoe.ko and vlan_mon.ko are ending up under the old kernel modules?

That didn't do anything.

Umm, that's not what happened. It looks like it needs to be in the 4.19.0 directory to work

Linux route-extern.xxx.xx.xx 4.19.4-amd64-vyos #37 SMP Mon Nov 26 10:03:40 CET 2018 x86_64 GNU/Linux

This was resolved by @UnicronNL when the dual-mode ISO was built. As of RC7, the installer now integrates this and can successfully do a UEFI install

Looks like this might be upstream and has been corrected there.

Just ran it on a few different machines, VMs, set up RAID/non-RAID. It worked great. Thanks!