After some custom build and POC, here's what I got:

• Filtering works, as shown in this table:

table ip filter {
        chain NAME_GEOIP {
                # Some basic test rules
                ip daddr 1.2.3.4 counter packets 0 bytes 0 return

                # Mark and filter traffic comming from Canada
                meta mark set ip saddr map @geoip4
                meta mark 0x0000007c counter packets 23 bytes 1932 drop

                meta l4proto icmp counter packets 0 bytes 0 drop comment "GEOIP-999"
                counter packets 0 bytes 0 return comment "GEOIP default-action accept"
        }
}

Main problem is including such big files in configuration: geoip-ipv6.nft and geoip-ipv4.nft

root@vyos:/home/vyos/nftables-geoip# du -h geoip-ipv4.nft geoip-ipv6.nft 
11M     geoip-ipv4.nft
18M     geoip-ipv6.nft

These leads to long commit times, for example:

vyos@vyos# set firewall name TESTING rule 77 source address 7.7.7.7
[edit]
vyos@vyos# set firewall name TESTING rule 77 action drop 
[edit]
vyos@vyos# 
[edit]
vyos@vyos# time commit

real    0m18.277s
user    0m7.175s
sys     0m10.961s
[edit]
vyos@vyos#

In previous example, only geoip-ipv4.nft is included. Also, with 512k of RAM it did not work. I have to increment RAM to be able to test this config.

Perhaps only in-use sets can be determined and loaded?

Splitting ipv4 files, and just adding what needed. In my case, I extracted content from geoip-ipv4.nft and create and include file geoip-CA-ipv4.nft (Canada IPs)

root@vyos:/home/vyos/nftables-geoip# du -h geoip-CA-ipv4.nft 
968K    geoip-CA-ipv4.nft

And commit went down, but still high compare to almost 0 without this feature:

vyos@vyos# del firewall name TESTING rule 77
[edit]
vyos@vyos# time commit

real    0m7.535s
user    0m3.398s
sys     0m4.035s

I just tested it on VyOS 1.4-rolling-202206260217, everything seems to work so far!
It would be nice to also have the negate option, something like:

set rule 100 source geoip country-code !nl
set rule 100 action drop

For example on my current setup with OPNsense I have a deny rule on top from WAN incoming which blocks everything not coming from certain countries, as that's a lot easier than including every single country that needs to be blocked in that rule.

I would be happy to test things out if that's needed!
If the documentation still needs to be made on this topic, I would like to try that out as well, but that would be my first time writing documentation and such.

Why don't use action accept for nl and drop all others?

Because with a rule like that I accept everything coming from nl from wan to lan, or I would need to add the source nl to every rule. That's why I did it with a deny not coming from nl on top, and then specific rules for the traffic that I want to accept.

Inverse match PR: https://github.com/vyos/vyos-1x/pull/1386

Example for @diekos with suggested syntax if merged:

set rule 100 source geoip country-code nl
set rule 100 source geoip inverse-match
set rule 100 action drop

I see that the pull request was accepted. I just tested it with the latest rolling and it seems to work as expected.
Thanks a lot!