FB> Jamnntpd server 1.0 doesn't accept the connection if this is on IPv6,
 FB> any hint? :)

1.0? That version is 15 years old. The latest version is 1.3.

Unfortunately it doesn't look like 1.3 has IPv6 support. Just run a TCP reverse
proxy in front of it, like nginx, haproxy, stunnel, ...