As of this commit, every port is checked with an exec(ipset -T):
https://github.com/vyos/vyatta-cfg-firewall/commit/835304e5aaa252e8b0bcf4651629cd089e670147
If there is a large port range, this can take many many minutes. A simple example would be
conf set firewall group port-group slowwwwww port '20000-65531' commit
That would run ipset -T 45,531 times.
A better idea is to get the ipset result BEFORE the check, and then iterate over the result to see if anything is missing.