Page MenuHomePhabricator

VXLAN support improvements
Closed, ResolvedPublicFEATURE REQUEST


In 1.1 we saw the introduction of initial support for VXLAN. In 1.2 I think we should aim to improve VXLAN support despite being close to release.


  1. The destination port used for creating a VXLAN interface in Linux defaults to its pre-standard value of 8472 to preserve backwards compatibility. For VyOS the official IANA destination port of 4789 should be used by default and we should provide a configuration directive to support a user-specified destination port to override that behavior. Note that VMware and others have already moved to adopt the new standard port of 4789.

This is set using the "dstport" directive when creating the interface, e.g. "ip link add vxlan100 type vxlan id 100 group dev eth0 dstport 4789"

  1. An MTU below 1500 for a VXLAN interface is against RFC7348 recommendations and as such most commercial implementations require the use of jumbo frames as a prerequisite for VXLAN.

By default VXLAN interfaces on VyOS are created with an MTU of 1450. Attempting to create or modify the MTU of a VXLAN interface will result in failure if the parent interface MTU does not provide sufficient overhead (e.g. a minimum of 1550 for a VXLAN MTU of 1500).

Changing the default MTU for VXLAN interfaces to 1500 might seem like the best option, but this creates a cascade of validation work that might not be appropriate for the 1.2 release. Validation of the "link" interface MTU would be straight-forward; but as dummy and loopback interfaces are created with an MTU of 65536 by default such validation would fail if following the best practice of using a loopback for VTEP termination. This presents a larger problem of likely traffic issues which would be non-obvious to the operator. Instead, it is recommended that we maintain the default of 1450 if not specified but also supply a warning message upon configuration of VXLAN interfaces without an MTU of 1500 such as:

"WARNING: RFC7348 recommends VXLAN tunnels preserve a 1500 byte MTU. To make use of a 1500 byte MTU, a minimum MTU of 1550 is required end-to-end between all tunnel endpoints. Use of a jumbo frames and a 9000 byte MTU on physical interfaces is strongly recommended."

  1. We should supply an operational mode command to show the forwarding database for a VXLAN interface, e.g. "show interfaces vxlan <int> forwarding-database" which would call "bridge fdb show dev <int>". This is often the most useful tool for troubleshooting VXLAN connectivity.


Difficulty level
Unknown (require assessment)
Why the issue appeared?
Will be filled on close

Event Timeline

rps created this task.Nov 30 2018, 5:12 PM
syncer assigned this task to c-po.Dec 1 2018, 6:00 PM
syncer edited projects, added VyOS 1.3 Equuleus; removed VyOS 1.2 Crux.
syncer triaged this task as High priority.
syncer added subscribers: hagbard, c-po, dmbaturin, syncer.

@c-po @rps @hagbard @dmbaturin
let's do it in subtasks and in 1.3

vmware ask min MTU 1600 for all vxlan related interfaces

c-po added a comment.EditedDec 4 2018, 5:30 PM

Will set interface vxlan vxlan0 remote-port 12345 be appropriate? Using destination-port would also work but the remote address is called remote and also openvpn utilizes remote-port

rps added a comment.Dec 5 2018, 1:48 AM

Juniper and Cisco use a global configuration of "vxlan port" or "vxlan udp port". A per-interface configuration is more flexible and likely makes sense.

Using "remote-port" to be in line with vtun configuration seems logical based on that and I think it's fine.

Alternatively simply "port" would work as well since the RFC suggests that the source port should be generated to be a hash of src-dst that falls within 49152-65535 to facilitate better load-balancing rather than be user-defined; as such there will not be a need to set "local-port" or "source-port" values.

c-po added a comment.Jan 6 2019, 10:43 AM

@rps what about changing the minimum supported MTU from "68" to "1450" which is the default that is used in VyOS.

In addition a warning can be displayed from within the update: statement as recommended by you?

afics added a subscriber: afics.Feb 6 2019, 10:39 AM

I would consider adding EVPN support via FRR as soon as kernel support for the inter working between vxlan and bridge interfaces is more fleshed out. Maybe I should create a new task for that.

c-po added a comment.Mar 8 2019, 6:16 PM

How should we proceed?

syncer added a comment.Mar 8 2019, 6:19 PM

@c-po i think we should bump it to 1450
also need a recommendation for 1600 for production use
maybe at time you create vxlan it should check upstream interface

c-po moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus board.Mar 8 2019, 6:38 PM
c-po closed this task as Resolved.
syncer reopened this task as In progress.Mar 16 2019, 6:24 PM
syncer moved this task from Needs Triage to Backlog on the VyOS 1.2 Crux (VyOS 1.2.1) board.

Need to backport this to 1.2.1

c-po closed this task as Resolved.