I have prepared the change
https://github.com/mevertse/vyatta-cfg-quagga/commit/d3685f504e0c22a8c88899aebc1e8705637259c4

This router is receiving BGP from several internal BGP routers each with full table peers or couple of peerings.

This one is running on Hyper-V 2016 and is not pushing any traffic. It is my test router and experimenting with RPKI.
The routers doing traffic are on hardware and not running 1.2.x yet.

I am running 1.2.1 compiled on 17-04-2019, uptime is 6 days without issue.
RIB entries 1366663, using 209 MiB of memory
Peers 16, using 330 KiB of memory
Peer groups 4, using 256 bytes of memory

Relevant config:

With 1.2.0-H4 this issue seems to be fixed on my router.

Do you have an iso to test? I tried latest rolling and also my own iso built from current and i continue to see this issue.
It makes transitioning to 1.2.0 impossible at this moment. Still at 1.1.8 on the routers.

Well that is interesting, the package versions are still the same but now it works.

I tested the current release and the issue still exists.
After adding staticd=yes on a reboot everything seemed to work.
More specific routes work even when a larger blackhole route is present.
However adding a new blackhole route while more specific routes exist (and work fine) stops them from working. The new blackhole routes get loaded and supress the more specific routes.
Removed a blackhole route allow other routes to work even when they are not part of the blackhole route.

The only packages i manually build are vyatta-cfg-firewall and vyatta-nat, the rest of the packages directories are empty so the packages get downloaded.

@kroy by using chroot and trying to install the vyatta-cfg-quagga package i found out what is causing my build iso error:

@Maltahl just re-tested this, with the staticd=yes added, and a reboot done.
When i add two static routes i would expect the /24 route to work because it is more specific. But is does not and show ip route shows only the /23 blackhole.

@Maltahl for me it was not fixed with that addition, and i read above that others had this as well.

@kroy everything is at current, except 'frr' because then i get 7.1dev and i would like 6.0.2 to test if this solved it.
I used debian/master branch from FRR.

I am experiencing the same issues with a router i tested with 1.2.0 current.
Can we create a test release going back to FRR 6.0.2?

@zsdc i meant test with 1.2.0 :-)

We are seeing this issue mostly on BGP routers with Internet Exchange connections because at a reboot we are hitting max-prefix limits with a lot of peers.
At this moment it is not possible to upgrade to latest 1.2.0, still running 1.1.8.

@hagbard Did you merge the second PR also? For vyos/vyatta-nat?

@hagbard created an iso image and loaded it in a VM. I can add the configuration and at commit the right ip6tables rules are created.

@hagbard the changes are created with the patch files mentioned earlier.
I am in the process of creating packages and an iso with it.

@hagbard PRs created, first time so hope its done right.

@jmlccdmd
I added a second router and configured conntrack-sync.
Failover and preempt failback works correct.
Both routers show statistics for the firewall rules

But if you run only on the first router, including the VRRP setup it does not work?

@jmlccdmd
I have recreated your setup with Vyos 1.2.0-rc10 and it seems to be working correctly

After some searching i found that with the following commands it works:

@hagbard
Test setup created

I can probably create a test setup this week and test the normal implementation in NPTv6 te see what is not working in my production setups.
As i do not have a build server at this moment creating a PR would be a bit difficult, the patch files are included in the comments above.

Then we do not have the same setup :-)
set interfaces ethernet eth0 address 'x.x.x.225/24'
set interfaces ethernet eth0 address 'x:x:x:x::225/64'
set interfaces ethernet eth1 address '10.0.201.1/24'
set interfaces ethernet eth1 address 'fd00:10:0:201::1/64'
set nat nptv6 rule 10 outbound-interface 'eth0'
set nat nptv6 rule 10 source prefix 'fd00:10:0:201::/64'
set nat nptv6 rule 10 translation prefix 'x:x:x:x::/64'
set protocols static route 0.0.0.0/0 next-hop x.x.x.1
set protocols static route6 ::/0 next-hop x:x:x:x::1

@hagbard What do you want me to try. I downloaded and loaded that rolling image and i do not see the proposed patches in it.
I rebooted the router and it did not work.

According to FRR this is normal behavior.
https://github.com/FRRouting/frr/pull/3044

Do you mean the 31 and 32 also couldn’t ping eachother?

https://www.shrubbery.net/mibs/BGP4-MIB.txt
We can start with this like @rherold suggests since FRR supports BGP4 MIB.

Seems duplicate with https://phabricator.vyos.net/T366
If we use the Cisco BGP MIBv2 we solve both issues.

I see in the config that you do not have an interface IP on the VRRP members.
This works in 1.1.8 most of the time. But can you test if 1.2.0 works with those added. The hello source address is not needed then and the chances are the kernel wil load the connected route this way.

Solved by disabling engine ID when the version is 9, not sure if this is enough but on my router it works.

Until version 1.7.0 it was possible to (mistakenly) configure the
NetFlow v9 SourceID field/IPFIX Observation Domain ID with the old
NetFlow v5 jargon, ie. '1:1'. This is now threated as invalid and
a positive 32-bit number, ie. '100000', is expected. If exporting
NetFlow v5, nothing changed: the Engine ID/Engine Type input, ie.
'1:1', is still valid and expected.

This behavior was already present in the old Quagga implementation in Vyos 1.1.7.
As a workaround we always shutdown the peers when doing a planned reboot.

Hi, yeah i was planning on adding it later. Thanks.

@dmbaturin I believe you forgot to create the Sync-Group. The following configuration is working, and it is really nice to see how this got created during migration from Vyos 1.1.8, and to finally have IPv6 in the VRRP configuration.

One extra patch needed

SInce this is a 1-to-1 mapping it also opens up all incoming ports to the hosts behind the router.

It was at 2GB because it is a test router running in the production network.
I increased memory and will check to see if this resolves it.

@syncer
It is a virtual machine on Hyper-V 2016 with 2 cores, 2GB memory.
I compared the output from top over te past week and the only changes are in the uacctd processes.

With the following patch the statements are migrated. Needs some work to also migrate the metric and route-map settings. Will try to do that later.

The idea for 'helper ftp' is a bit harder to implement, because it seems it requires iptables -t raw and currently we only have filter and mangle.

Since during parsing we cannot detect if it is FTP traffic or not, because you can choose whatever port you want, i think the only solution would be to add something like

Probable cause is:
https://unix.stackexchange.com/questions/385693/iptables-nf-conntrack-ftp-not-working-under-debian

Upgrade to 1.2.0-rc10 and BGP is still working fine. It starts at boot and loads all BGP peers and several full tables.

With some small changes i changed the scripts to create the proposed NETMAP rules when editing the NPTv6 rules in the configuration. For me this is working and surviving reboot.
I can add the diff files if anyone wants them.

Re-tested with 1.2.0-rc9 with the same result.
I found that i can enable debug in /opt/vyatta/sbin/vyos-update-nptv6.pl to see what is added and deleted.

This seems to be easy.
sudo chmod 776 /opt/vyatta/etc/config/archive/config.boot

Upgrade from 1.1.7 with a couple of changes in the rollback list to the current 1.2.0-rc9 also shows thi same error:
Couldn't open /opt/vyatta/etc/config/archive/config.boot - Permission denied at /opt/vyatta/share/perl5/Vyatta/ConfigMgmt.pm line 108.

@syncer In the 1.2.0-rc9 image this seems to be fixed. I upgraded to this release and the config is working flawlessly as far as i can tell now.
BGPD is started at boot, and no config parts are missing.
I am also running several BGP neighbors, and several of them with a full BGP table.

You don't need this FEAT for others enforcing your ROAs. You need this FEAT to enforce received-routes on VyOS.

This is still a problem with vyos 1.2.0-rc2.

After a reboot now the config misses the BGP config. Loading it and trying commit shows the following:
Warning: connecting to bgpd...failed!
bgpd is not running

@c-po i saved the working complete config to a file, and when loading this after reboot with a lot missing, not a single error message is shown besides: boot-config-loader: Commit failed at boot

At that location only a couple of pre-migration configs exist. In archive subfolder also only old files exist.
But i loaded the latest pre-migration script, this causes all BGP configuration to be removed by the commit.
The error message is:
% Specify remote-as or peer-group commands first

Trying to reproduce it by doing a reboot with a working BGP config i now have lost during the reboot all interface and firewall configuration.
But the BGP config is now present and loaded.
:-(

That would be very nice to be able to use.
I just tested the rolling image and when i try to start bgpd with this module loaded it cannot find it.

I found this to occur sometimes when you configure the same subnet on multiple interfaces. The IPv4 seems to be OK in your config but the IPv6 is hidden and missing the last digit as well.
Also maybe you have created some strange behaving rule in NAT, if you have NAT configured could you post this config, or run through it yourself?

I believe i found the cause in the following issue on FRRouting
https://github.com/FRRouting/frr/issues/1440

My impression was that is was related to the adding of ipv6-next-hop to a route-map.
But without the IPv6 static routes the config commits fine.
When adding the first IPv6 static route the error is shown.

I also tested it on latest rolling and it is not working. It however does change something because when i enable it i can no longer ping from the Vyos router to the gateway.

RFC compatibility settings prevents VRRP from working on Hyper-V because of promiscuous on NICs
Without the RFC compatibility this is working fine.

Option 2 seems best. VRRP version does not need a setting, use VRRP V2 when no vrrp6 block is present for backward compatibility. Use VRRP v3 when it is.

Tested on 1.2.0-rolling

I am running several Vyos instances on Hyper-V and can confirm that 1.1.7 and 1.1.8 are running without issues.
For memory we use 512MB and never have any issues.

The current Quagga included is rather old, so if it is possible to migrate to FRR that would make a big difference.
I am testing my configs in the alpha release and so far it looks good. To assist i can test setups and configs.

It seems this is a bug in FRR 3.0
https://github.com/FRRouting/frr/issues/509

@syncer @dmbaturin
I found the following discussion on Cisco forum very explainable
https://supportforums.cisco.com/discussion/11476526/when-use-bgp-address-family

I manually added the file from Diffusion and this works, but i do not know how to fix this in the source :-)
https://phabricator.vyos.net/diffusion/2/browse/master/scripts/ssh-server-key;970ff0cfe393ccfa63d1f875483f3df7543d6c62

At this moment i am not running the nightly build so i cannot get this information for you. In a week or so i can change it to nightly build and let it run.

I checked a couple of routers and every router without IPSec configured has this error, but every router with IPSec configured had this file so no errors.

I noticed that both routers having the SNMP issues are the VRRP masters. I switches VRRP to Backup en rebooted the routers. SNMP daemon seems stable at this moment.

@rps this distinction also seems to be easy in the original proposed solution by @dmbaturin because key value pairs are not followed by '{' and the rest is.

@dmbaturin I understand that the discussion is "unit 0" vs "unit { 0", what i meant was that i could be an option to keep following the JunOS style as much as possible to maybe enable more interoperability.

Well plain JSON would also be an option then :-)

In the blog post #7 i liked the address [ 192.168.2.1/24 10.10.10.1/30 ]; part. But since i work most of the time with mixed JunOS and Vyos environments a mostly the same syntax would be very nice :-)
However JunOS would be:

A pro for me would be that i can do 'edit interfaces ethernet eth0 vif' and work with all virtual interfaces.

In JunOS the root user enters in the shell and uses 'cli' to enter show mode followed by configure for config mode.
When i add an extra user without shell access, this user is placed directly into show mode.
The vyos user is not the root user, so the way it currently is makes perfect sense to me.