How to route NTP traffic?

I have a new 4507 switch where I created 3 vlans I have a dedicated NTP server and it works fine serving time for clients on same vlan but I can not make it work with computers on different vlan, I can ping all devices on the network even between vlans. Any Ideas to solve this problem would be appreciated. Thank you.