Page MenuHomeVyOS Platform

Support for QoS Policy Propagation via BGP (QPPB)
Open, Requires assessmentPublicFEATURE REQUEST


Difficulty level
Hard (possibly days)
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Perfectly compatible
Issue type
Feature (new functionality)

Event Timeline

Demo QPPB implementation supporting bgp-policy destination mode:

The following comments will include architecture, demo topology, debugging technics

v.huti changed Difficulty level from Unknown (require assessment) to Hard (possibly days).

You can find the latest version of the demo implementation here:

  1. volodymyrhuti/linux/tree/QPPB_DEMO_V1.1
  2. volodymyrhuti/frr/tree/QPPB_DEMO_V1.1

This update includes:

  1. Implemented the bgp-policy source mode
  2. Implemented the sysctl per interface switch to configure the QPPB mode
  3. Minor cleanups & fixes

The implementation is done, for the most part. The following plans are:

  • Make an example with QoS configuration
  • Make a clean demo topology
  • Finalizing and posting the documentation

The latest version of the demo can be found here:

  1. volodymyrhuti/linux/tree/QPPB_DEMO_V1.2
  2. volodymyrhuti/frr/tree/QPPB_DEMO_V1.2

This update includes - FRR:

  1. Reworked DSCP manipulation methods + moved them into a separate library module
  2. Introduced the bgp-policy <source|destination> perf interface configuration


  1. Restructured the marking logic
  2. Hid the logic under IP_ROUTE_QPPB config
  3. Basic in-tree documentation, killed dead code, minor cleanups

The latest version of the demo can be found here:

  1. volodymyrhuti/frr/tree/QPPB_DEMO_V1.3
  2. volodymyrhuti/xdp_qppb

During the discussion with @zdc it became clear that the previous implementation was only partially working.
That solution worked on the skb layer, meaning that resources are already allocated, and the ingress tc engine got skipped.
However, it is expected that marking can be noticed by the tc and used for ingress policing.
I have moved the logic to the XDP level to overcome this limitation.

The latest update includes:

  1. XDP program that should be used with ip link / tc infrastructure
  2. Helpers library for FRR to interact with the XDP

The next update will include:

  1. DSCP to VRF mappings
  2. Permission access fix for helper library Currently loading requires disabling verifications, i.e. /proc/sys/kernel/unprivileged_bpf_disabled
  3. GNS Demo topology. This will be much easier now since there is no need for a custom kernel
  4. Some basic functionality/sanity tests
  1. volodymyrhuti/frr/tree/QPPB_DEMO_V1.4
  2. volodymyrhuti/xdp_qppb

Changes on the FRR side:

  • Convert xdp helper library to an optional plugin + bgp hook
  • Minor fixes + cleanups
  • Figured out most of the permission problems

Changes on the XDP side:

  • Convert mappings from legacy iproute format to the latest libbpf one
  • New mappings improve debugging experience by implementing pretty-printing for XDP map dumping
  • Added an xdp-loader for xdp-tools repo

As well we have performed another review round with @zdc.
The next step is to do an initial review with smbdy from the FRR devs.


To demonstrate the feature let's look at the following topology

The scenario is following:

  • R1 has a management interface (loopback); we want to prioritize the associate traffic over other
  • R2 announces prefix of this interface over BGP with associate 'community' list
  • R3 has a QPPB map that associates the community with a DSCP/COS tag for the following traffic control.
  • C1 and C2 are clients communicating with R1 using low/high priority flows.
1. Load QPPB for interfaces and restart the FRR service

   [R3][email protected]:~$ cat /etc/frr/daemons | grep bgp
   bgpd_options="   -A -M vyos_qppb"
   sudo ./xdp-loader load ens4 xdp_qppb.o  -p /sys/fs/bpf/ 
   sudo ./xdp-loader load ens5 xdp_qppb.o  -p /sys/fs/bpf/ 
   sudo service frr restart

2. Wair for the bgp daemon to announce the prefixes.
   Verify that the XDP map was populated
   Configure perf interface QPPB mode

   [R3][email protected]:~$ journalctl -b | grep XDP
   ... bgpd[1085]: ... XDP mark prefix [| dscp 80, err 0]
   ... bgpd[1085]: ... XDP mark prefix [| dscp 124, err 0]
   ... bgpd[1085]: ... XDP mark prefix [| dscp 72, err 0]
   ... bgpd[1085]: ... XDP mark prefix [| dscp 40, err 0]

   $ sudo bpftool map list
   6: array  name qppb_mode_map  flags 0x0
   	key 4B  value 4B  max_entries 64  memlock 4096B
   	btf_id 7
   7: lpm_trie  name dscp_map  flags 0x1
   	key 8B  value 1B  max_entries 100  memlock 8192B
   	btf_id 7
   8: percpu_array  name xdp_stats_map  flags 0x0
   	key 4B  value 16B  max_entries 5  memlock 4096B
   	btf_id 7

   [R3][email protected]:~$ sudo bpftool map dump id 7
           "key": {
               "prefixlen": 32,
               "src": 16777217
           "value": 80
           "key": {
               "prefixlen": 32,
               "src": 167772190
           "value": 152
   [R3][email protected]:~$ dec_to_ip 16777217

   # == 2 (ens4), mode == 2 (BGP_POLICY_SRC)
   $ sudo bpftool map update id 6 key 2 0 0 0  value 2 0 0 0
   # == 3 (ens5), mode == 1 (BGP_POLICY_DST)
   $ sudo bpftool map update id 6 key 3 0 0 0  value 1 0 0 0

3. Open the Wireshark and verify that the marking is working
   [R1] ping -I -c 3
   [R3] -> [C1] => 80(0x50) TOS
   [R3] -> [R2] => 80(0x50) TOS

4. Configure the `tc` rules to prioritize traffic associated with management iface
   alias tc="sudo tc"
   tc qdisc del dev ens4 root
   tc qdisc add dev ens4 root handle 1:0 htb default 10
   # create the parent qdisc, children will borrow bandwidth from
   tc class add dev ens4 parent 1:0 classid 1:1 htb rate 100mbit
   # create children qdiscs, reference parent
   tc class add dev ens4 parent 1:1 classid 1:10 htb rate 1mbit
   tc class add dev ens4 parent 1:1 classid 1:20 htb rate 10mbit ceil 90mbit

4.1 Using classic u32 rules
   tc filter add dev ens4 parent 1:0 prio 1 protocol ip u32 \
                         match ip tos 0x50 0xff flowid 1:20

4.2 Using custom bpf classifier
   tc filter add dev ens4 protocol ip parent 1:0 \
             bpf obj xdp_tc.o sec tc_mark classid 1: direct-action

4.3 Verify that filter is triggered for (un)prioritized traffic
   [R1] ping -I -c 3      #   privileged
   [R1] ping -I -c 3     # unprivileged
4.4 Setup iperf server on clients and verify that traffic shaping is working

   [R1] iperf3 -n 2Gb -B  -c &
   [R1] iperf3 -n 2Gb -B -c &

   [R3] tc -s -p -g f ls dev ens4
   [R3] tc -s -p -g q ls dev ens4
   [R3] tc -s -p -g c ls dev ens4