The NTP servers configured by "set system ntp server xxx" got ignored if one of the interfaces has received NTP servers via DHCP.
In fact, in my case it was even a security issue, as the DHCP provided server was several days in a past and SSL certificates (and other SSL time dependent VPN functions) has stopped to work, causing a service outage.
After some investigation, I've found the lines to blame in /etc/init.d/ntp
if [ -e /var/lib/ntp/ntp.conf.dhcp ]; then NTPD_OPTS="$NTPD_OPTS -c /var/lib/ntp/ntp.conf.dhcp" fi
Removing the above lines, corrects the issue.
Please, update, if possible.