vyos:
eth0 172.16.0.1
eth1.40 172.16.40.1
eth1.41 172.16.41.1
eth1.42 172.16.42.1
dhcp-relay 172.16.0.10
none of the options for "set service dhcp-relay relay-options relay-agents-packets [discard]" is working. Replacing the vyos VM with opnsense VM works.
Someone else seems to have discovered this effect quite a while ago: https://www.reddit.com/r/homelab/comments/4a92nw/obscure_problem_with_dhcprelay_over_vlan_when/