Thanks for posting here in the Google Nest Community. I assume that you’re setting up a Chromecast with Google TV since that comes with its own remote. This actually depends on the make and model of the TV. I suggest you do a full clean-up by performing a factory reset on the main Chromecast device, then rebooting the TV. Also, try to factory reset the remote (optional). Proceed with the setup and let us know how it goes.
