GoogleTV on a Chromecast Ultra gains latency as time goes on so that after as little as 30 minutes you must kill the app and restart the program to use it. It gets better, this is ONLY FOR GoogleTV - not even YouTube falls out of sync. Verizon FiOS for Internet and Google Nest with 5 points. This happpen on multiple TVs and on a RoKu stick as well.
Thanks for reaching out. Let's sort this out.
A few questions: when did the issue start? What device are you using for casting? How far is your Chromecast Ultra from the Wi-Fi router?
Try these steps:
Note: If the issue is also happening with other streaming devices such as your Roku stick, the issue might be with your Wi-Fi network. Please reach out to your router manufacturer for further help.