Page MenuHomePhabricator

NPTv6 is broken
Closed, ResolvedPublicBUG

Description

vyos@router# show nat nptv6
 rule 1 {
     inside-prefix 2001:db8:100:1::0/64
     outside-interface eth0
     outside-prefix 2001:db8:200:1::0/64
 }
[edit]
vyos@router# sudo ip6tables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
[edit]
vyos@router# r
rename    rollback  run
[edit]
vyos@router# run show version
No hypervisor detected
Version:          VyOS 999.201609170235
Built by:         autobuild@vyos.net
Built on:         Sat 17 Sep 2016 02:35 UTC
Build ID:         c2a1c9de-d066-4251-b844-442b836fdc8e

Architecture:     x86_64
Boot via:         installed image
System type:      physical

Hardware vendor:  Supermicro
Hardware model:   X9SRE/X9SRE-3F/X9SRi/X9SRi-3F
Hardware S/N:     0123456789
Hardware UUID:    00000000-0000-0000-0000-0CC47A0463C0

Copyright:        VyOS maintainers and contributors

Details

Difficulty level
Hard (possibly days)
Version
999.201609170235
Why the issue appeared?
Other

Related Objects

Event Timeline

mrjester created this task.Oct 2 2016, 8:27 PM
mrjester updated the task description. (Show Details)
syncer triaged this task as Normal priority.Aug 1 2017, 5:17 AM
syncer changed the edit policy from "Task Author" to "Custom Policy".
syncer added a project: VyOS 1.2 Crux.
syncer set Version to 999.201609170235.
syncer removed syncer as the assignee of this task.Nov 3 2017, 12:36 PM
syncer added a subscriber: syncer.
dmbaturin raised the priority of this task from Normal to High.May 24 2018, 6:24 PM
dmbaturin moved this task from Need Triage to Backlog on the VyOS 1.2 Crux board.
dmbaturin edited projects, added VyOS 1.2 Crux (VyOS 1.2.0-rc1); removed VyOS 1.2 Crux.
dmbaturin added a subscriber: dmbaturin.

Needs to be re-tested in recent images.

@mrjester can you test on latest rolling ?

yes, still an issue in the latest rolling. (VyOS 1.2.0-rolling+201806251824)

Merijn added a subscriber: Merijn.Jun 30 2018, 8:30 AM

I also tested it on latest rolling and it is not working. It however does change something because when i enable it i can no longer ping from the Vyos router to the gateway.

syncer lowered the priority of this task from High to Low.Jul 3 2018, 7:18 AM
syncer changed the subtype of this task from "Task" to "Bug".Oct 18 2018, 5:52 AM
pasik added a subscriber: pasik.Nov 4 2018, 11:25 AM

Re-tested with 1.2.0-rc9 with the same result.
I found that i can enable debug in /opt/vyatta/sbin/vyos-update-nptv6.pl to see what is added and deleted.

My configuration is:

interfaces {
     ethernet eth0 {
         address 2001:1000:10:100::225/64
     }
     ethernet eth1 {
         address fd00:10:0:0::1/64
     }
}
nat {
     nptv6 {
         rule 10 {
             disable
             outbound-interface eth0
             source {
                 prefix fd00:10:0:0::/64
             }
             translation {
                 prefix 2001:1000:10:100::/64
             }
         }
     }
}

This executes the following commands:
/sbin/ip6tables -t mangle -F VYOS_SNPT_HOOK
/sbin/ip6tables -t mangle -F VYOS_DNPT_HOOK
/sbin/ip6tables -t mangle -A VYOS_SNPT_HOOK -j RETURN
/sbin/ip6tables -t mangle -A VYOS_DNPT_HOOK -j RETURN
/sbin/ip6tables -t mangle -I VYOS_SNPT_HOOK -s fd00:10:0:0::/64 -o eth0 -j SNPT --src-pfx fd00:10:0:0::/64 --dst-pfx 2001:1000:10:100::/64
/sbin/ip6tables -t mangle -I VYOS_DNPT_HOOK -d 2001:1000:10:100::/64 -i eth0 -j DNPT --src-pfx 2001:1000:10:100::/64 --dst-pfx fd00:10:0:0::/64

From what i can find this should work, but it doesn't. My internal host is at fd00:10:0:0::10 and it can ping the Vyos router but not hosts outside.
Also this causes the Vyos router to be unable to ping hosts outside, and is not pingable from the outside.

In the 2.6 kernel a new option is available NETMAP instead of SNPT and DNPT.
So i deleted the rules from ip6tables and added these rules:
ip6tables -t nat -A POSTROUTING -o eth0 -j NETMAP --to 2001:1000:10:100::/64 -s fd00:10:0:0::/64
ip6tables -t nat -A PREROUTING -i eth0 -j NETMAP -d 2001:1000:10:100::/64 --to fd00:10:0:0::/64

And this works, the issue with ping from the Vyos router is resolved and internal host can ping also. The mapping from internal to external is now static, so the internal host always gets 2001:1000:10:100::10
What i could find about NETMAP is:
NETMAP is a new implementation of the SNAT and DNAT targets where the host part of the IP address isn't changed. It provides a 1:1 NAT function for whole networks which isn't available in the standard SNAT and DNAT functions.

For a lot of situations that 1:1 NAT is the preferred solution because it makes it predictable. The normal DNPT/SNPT creates a checksum neutral address generated from several parameters.

Merijn added a comment.EditedNov 27 2018, 8:38 PM

With some small changes i changed the scripts to create the proposed NETMAP rules when editing the NPTv6 rules in the configuration. For me this is working and surviving reboot.
I can add the diff files if anyone wants them.

Maybe it would be good to give the choice, like:
set nat nptv6 type netmap enable

I could try to implement that solution of needed.

Merijn added a comment.EditedDec 24 2018, 12:43 PM

SInce this is a 1-to-1 mapping it also opens up all incoming ports to the hosts behind the router.

Here are the patch files to make this work:

--- /opt/vyatta/sbin/firewall.init.orig 2018-11-27 20:33:09.938224133 +0000
+++ /opt/vyatta/sbin/firewall.init      2018-11-27 20:31:31.469873734 +0000
@@ -144,12 +144,12 @@
        ip6tables -A OUTPUT -j VYATTA_POST_FW_OUT_HOOK

         # set up NPTv6 prerouting hook
-        ip6tables -t mangle -N VYOS_DNPT_HOOK
-        ip6tables -t mangle -A VYOS_DNPT_HOOK -j RETURN
-        ip6tables -t mangle -A PREROUTING -j VYOS_DNPT_HOOK
-        ip6tables -t mangle -N VYOS_SNPT_HOOK
-        ip6tables -t mangle -A VYOS_SNPT_HOOK -j RETURN
-        ip6tables -t mangle -A POSTROUTING -j VYOS_SNPT_HOOK
+        ip6tables -t nat -N VYOS_DNPT_HOOK
+        ip6tables -t nat -A VYOS_DNPT_HOOK -j RETURN
+        ip6tables -t nat -A PREROUTING -j VYOS_DNPT_HOOK
+        ip6tables -t nat -N VYOS_SNPT_HOOK
+        ip6tables -t nat -A VYOS_SNPT_HOOK -j RETURN
+        ip6tables -t nat -A POSTROUTING -j VYOS_SNPT_HOOK
         # NOTRACK hook : not needed, since every v6 connection is NOTRACK'ed for now (see a few lines up)
         # ip6tables -t raw -N VYOS_NPT_HOOK
         # ip6tables -t raw -A PREROUTING -j VYOS_NPT_HOOK
--- /opt/vyatta/share/perl5/VyOS/Nptv6Rule.pm.orig      2018-11-27 20:06:21.425339938 +0000
+++ /opt/vyatta/share/perl5/VyOS/Nptv6Rule.pm   2018-11-27 20:22:12.629183123 +0000
@@ -69,7 +69,8 @@

 # Make SNPT ip6tables string
 # POSTROUTING
-# ip6tables -t mangle -I VYOS_NPT_HOOK -s inside-pfx -o outside-if -j SNPT --src-pfx inside-pfx --dst-pfx outside-pfx
+# ORIGINAL ip6tables -t mangle -I VYOS_NPT_HOOK -s inside-pfx -o outside-if -j SNPT --src-pfx inside-pfx --dst-pfx outside-pfx
+# ip6tables -t nat -I VYOS_SNPT_HOOK -s inside-pfx -o outside-if -j NETMAP --to outside-pfx
 sub make_snpt_string {
   my ($self) = @_;
   my $snpt_str = "";
@@ -82,9 +83,8 @@
     $snpt_str .= " -o ";
     $snpt_str .= $self->{_outside_if};
   }
-  $snpt_str .= " -j SNPT --src-pfx ";
-  $snpt_str .= $self->{_inside_pfx};
-  $snpt_str .= " --dst-pfx ";
+  $snpt_str .= " -j NETMAP ";
+  $snpt_str .= " --to ";
   $snpt_str .= $self->{_outside_pfx};

   return $snpt_str;
@@ -92,7 +92,8 @@

 # Make DNPT ip6tables string
 # PREROUTING
-# ip6tables -t mangle -I VYOS_NPT_HOOK -d outside-pfx -i outside-if -j DNPT --src-pfx outside-pfx --dst-pfx inside-pfx
+# ORIGINAL ip6tables -t mangle -I VYOS_NPT_HOOK -d outside-pfx -i outside-if -j DNPT --src-pfx outside-pfx --dst-pfx inside-pfx
+# ip6tables -t nat -I VYOS_DNPT_HOOK -d outside-pfx -i outside-if -j NETMAP --to inside-pfx
 sub make_dnpt_string {
   my ($self) = @_;
   my $dnpt_str = "";
@@ -105,9 +106,8 @@
     $dnpt_str .= " -i ";
     $dnpt_str .= $self->{_outside_if};
   }
-  $dnpt_str .= " -j DNPT --src-pfx ";
-  $dnpt_str .= $self->{_outside_pfx};
-  $dnpt_str .= " --dst-pfx ";
+  $dnpt_str .= " -j NETMAP ";
+  $dnpt_str .= " --to ";
   $dnpt_str .= $self->{_inside_pfx};

   return $dnpt_str;

Since it is working great in my test environment it would be cool to have it included in the project. But to keep it compatible with the current NPTv6 implementation, can someone give any direction on how to get the value of my proposed custom setting available in Nptv6Rule.pm so i can create the rules based on that value being enabled or not? We probably need a new ip6tables chain then also.

Then i will adjust the patch files accordingly.

One extra patch needed

--- /opt/vyatta/sbin/vyos-update-nptv6.pl.orig  2018-12-24 13:57:10.832019975 +0000
+++ /opt/vyatta/sbin/vyos-update-nptv6.pl       2018-11-27 20:24:11.417212476 +0000
@@ -53,7 +53,7 @@
 # Send rule to iptables
 sub send_iptables {
   my @cmds = @_;
-  my $prepend = $IPTABLES . " -t mangle ";
+  my $prepend = $IPTABLES . " -t nat ";
   my $cmd;

   for $cmd (@cmds) {
@@ -84,7 +84,7 @@
 # Loop through all loops, sorted numerically
 for $rule (@rule_keys) {
   print OUT "$rule: $rules{$rule}\n";
-  my $tmp = `ip6tables -L -nv --line -t mangle`;
+  my $tmp = `ip6tables -L -nv --line -t nat`;
   print OUT "iptables before:\n$tmp\n";

   my $nrule = new VyOS::Nptv6Rule;

Also the firewall.init script can also just create the chains both in mangle and nat. Tested the patches now on 1.2.0-rc11.

syncer assigned this task to hagbard.Jan 12 2019, 7:10 PM
syncer moved this task from Needs Triage to In Progress on the VyOS 1.2 Crux (VyOS 1.2.0-EPA3) board.
hagbard changed the task status from Open to Needs testing.EditedJan 15 2019, 6:58 PM

I've tested it without doing anything on the code and everything is working properly.

Version: VyOS 1.2.0-rolling+201901150337
Built by: autobuild@vyos.net
Built on: Tue 15 Jan 2019 03:37 UTC
Build ID: 54eec05d-e830-4feb-b4c9-87f67027aa7e

@Merijn Can you please give it a try?

@hagbard What do you want me to try. I downloaded and loaded that rolling image and i do not see the proposed patches in it.
I rebooted the router and it did not work.

There are probably a lot of people using the NPTv6 implementation in VyOS at this moment, i am proposing a simpeler setup that works a lot better in the situations i encounter.
Maybe we can add a switch to choose between the options?

hagbard added a comment.EditedJan 15 2019, 10:06 PM

@Merijn I haven't added anything. I just tested nptv6 and it was working as expected. I used your setup you have initially posted, I just used a different interface for the outgoing traffic. I confirmed via tcpdump that NAT did work.

set interfaces ethernet eth2 address '2001:db8:100:1::/64'
set interfaces ethernet eth3 address '2001:db8:200:1::/64'
set nat nptv6 rule 1 outbound-interface 'eth2'
set nat nptv6 rule 1 source prefix '2001:db8:200:1::/64'
set nat nptv6 rule 1 translation prefix '2001:db8:100:1::/64'

Merijn added a comment.EditedJan 15 2019, 10:35 PM

Then we do not have the same setup :-)
set interfaces ethernet eth0 address 'x.x.x.225/24'
set interfaces ethernet eth0 address 'x:x:x:x::225/64'
set interfaces ethernet eth1 address '10.0.201.1/24'
set interfaces ethernet eth1 address 'fd00:10:0:201::1/64'
set nat nptv6 rule 10 outbound-interface 'eth0'
set nat nptv6 rule 10 source prefix 'fd00:10:0:201::/64'
set nat nptv6 rule 10 translation prefix 'x:x:x:x::/64'
set protocols static route 0.0.0.0/0 next-hop x.x.x.1
set protocols static route6 ::/0 next-hop x:x:x:x::1

Using the rolling image this results in the following ip6tables rules:
-A VYOS_DNPT_HOOK -d x:x:x:x::/64 -i eth0 -j DNPT--src-pfx x:x:x:x::/64 --dst-pfx fd00:10:0:201::/64
-A VYOS_DNPT_HOOK -j RETURN
-A VYOS_SNPT_HOOK -s fd00:10:0:201::/64 -o eth0 -j SNPT--src-pfx fd00:10:0:201::/64 --dst-pfx x:x:x:x::/64
-A VYOS_SNPT_HOOK -j RETURN

Internal server is at fd00:10:0:201::225 and i woud lik to have x:x:x:x::225 as public IP address but it is a /64 so every IPv6 generated should work.
The moment the ip6tables rules become active i cannot ping the router, and ping from the router anymore. Also the host behind the router does not have IPv6 connectivity to Internet.

At the first quick review it works:

23:43:24.030495 08:00:27:a6:fe:2a > 08:00:27:cd:a0:3d, ethertype IPv6 (0x86dd), length 118: 2001:db8:100:1:d057::225 > 2001:db8:100:1::1: ICMP6, echo request, seq 21, length 64

It generates the prefix and uses the last octet, so that shouldn't be an issue.

Also the chain counters increment:

Chain VYOS_DNPT_HOOK (1 references)
pkts bytes target prot opt in out source destination

 0     0 DNPT       all      eth1   *       ::/0                 2001:db8:100:1::/64 src-pfx 2001:db8:100:1::/64 dst-pfx fd00:10:0:201::/64 
69  4952 RETURN     all      *      *       ::/0                 ::/0

Chain VYOS_SNPT_HOOK (1 references)
pkts bytes target prot opt in out source destination

32  3328 SNPT       all      *      eth1    fd00:10:0:201::/64   ::/0                src-pfx fd00:10:0:201::/64 dst-pfx 2001:db8:100:1::/64 
35  3536 RETURN     all      *      *       ::/0                 ::/0

I think I know what you mean now, it also starts translating the global address on the external interface. Can you send a PR for the changes you've made please?

I can probably create a test setup this week and test the normal implementation in NPTv6 te see what is not working in my production setups.
As i do not have a build server at this moment creating a PR would be a bit difficult, the patch files are included in the comments above.

Merijn added a comment.EditedJan 19 2019, 10:25 PM

@hagbard
Test setup created

public-router
vyos 1.2.0-rc10
eth0 connected to public network
192.168.192.51/24
fd00:10:100:0::51/64

core-router
vyos 1.2.0-epa2
eth0 connected to public network
192.168.192.71/24
fd00:10:100:0::71/64

eth1 connected to internal network
192.168.60.1/24
fd00:10:200:0::1/64

set nat nptv6 rule 5 outbound-interface 'eth0'
set nat nptv6 rule 5 source prefix 'fd00:10:200:0::/64'
set nat nptv6 rule 5 translation prefix 'fd00:10:100:0::/64'

internal-router
vyos 1.2.0-rc10
eth0 connected to internal network
192.168.60.11/24
fd00:10:200:0::11/64

static route 0.0.0.0/0 next-hop 192.168.60.1
static route6 ::/0 next-hop fd00:10:200:0::1

The moment i add the NPTv6 rules the public router cannot ping the core router, and vice versa.
The internal router can ping the core router. But not the public router.

The tcpdump of the ping6 from internal to public routers shows that the translation works, but the reply does not work

21:58:55.202924 IP6 (flowlabel 0x072bf, hlim 63, next-header ICMPv6 (58) payload length: 64) fd00:10:100:0:100::11 > fd00:10:100::51: [icmp6 sum ok] ICMP6, echo request, seq 392
21:58:55.204667 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fd00:10:100::51 > ff02::1:ff00:11: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fd00:10:100:0:100::11
          source link-address option (1), length 8 (1): 08:00:27:18:7f:cd
            0x0000:  0800 2718 7fcd

The core router never responds to the reply from the public router.
The neighbors list of the public router:
show ipv6 neighbors
fd00:10:100:0:100::11 dev eth0 FAILED
fd00:10:100::71 dev eth0 lladdr 08:00:27:a1:c0:ed router REACHABLE

After some searching i found that with the following commands it works:

sudo sysctl -w net.ipv6.conf.all.proxy_ndp=1
sudo ip -6 neigh add proxy fd00:10:100:0:100::11 dev eth0

But since without the tcpdump we did not know the translated address to add this is not really practical.

Also DNPT/SNPT is not that practical in general because it seems to lack connection-tracking and does not do ICMPv6 error handling.

syncer set Why the issue appeared? to Will be filled on close.
syncer moved this task from Need Triage to In Progress on the VyOS 1.3 Equuleus board.

@hagbard PRs created, first time so hope its done right.

@Merijn Have you tested your changes already? I was only bale to find https://github.com/vyos/vyatta-cfg-firewall/pull/12 which only contains the ip6tables targets, did you send PRs for systctl too?

@hagbard the changes are created with the patch files mentioned earlier.
I am in the process of creating packages and an iso with it.

@hagbard created an iso image and loaded it in a VM. I can add the configuration and at commit the right ip6tables rules are created.

@hagbard Did you merge the second PR also? For vyos/vyatta-nat?

dmbaturin renamed this task from NPTv6 Broken - 999.201609170235 to NPTv6 is broken.Mar 26 2019, 1:58 AM
dmbaturin changed Difficulty level from Easy (less than an hour) to Hard (possibly days).
dmbaturin changed Why the issue appeared? from Will be filled on close to Other.
c-po moved this task from In Progress to Finished on the VyOS 1.3 Equuleus board.Apr 21 2019, 10:38 AM