In 1.1 we saw the introduction of initial support for VXLAN. In 1.2 I think we should aim to improve VXLAN support despite being close to release.
- The destination port used for creating a VXLAN interface in Linux defaults to its pre-standard value of 8472 to preserve backwards compatibility. For VyOS the official IANA destination port of 4789 should be used by default and we should provide a configuration directive to support a user-specified destination port to override that behavior. Note that VMware and others have already moved to adopt the new standard port of 4789.
This is set using the "dstport" directive when creating the interface, e.g. "ip link add vxlan100 type vxlan id 100 group 126.96.36.199 dev eth0 dstport 4789"
- An MTU below 1500 for a VXLAN interface is against RFC7348 recommendations and as such most commercial implementations require the use of jumbo frames as a prerequisite for VXLAN.
By default VXLAN interfaces on VyOS are created with an MTU of 1450. Attempting to create or modify the MTU of a VXLAN interface will result in failure if the parent interface MTU does not provide sufficient overhead (e.g. a minimum of 1550 for a VXLAN MTU of 1500).
Changing the default MTU for VXLAN interfaces to 1500 might seem like the best option, but this creates a cascade of validation work that might not be appropriate for the 1.2 release. Validation of the "link" interface MTU would be straight-forward; but as dummy and loopback interfaces are created with an MTU of 65536 by default such validation would fail if following the best practice of using a loopback for VTEP termination. This presents a larger problem of likely traffic issues which would be non-obvious to the operator. Instead, it is recommended that we maintain the default of 1450 if not specified but also supply a warning message upon configuration of VXLAN interfaces without an MTU of 1500 such as:
"WARNING: RFC7348 recommends VXLAN tunnels preserve a 1500 byte MTU. To make use of a 1500 byte MTU, a minimum MTU of 1550 is required end-to-end between all tunnel endpoints. Use of a jumbo frames and a 9000 byte MTU on physical interfaces is strongly recommended."
- We should supply an operational mode command to show the forwarding database for a VXLAN interface, e.g. "show interfaces vxlan <int> forwarding-database" which would call "bridge fdb show dev <int>". This is often the most useful tool for troubleshooting VXLAN connectivity.