Thanks for posting here in the Google Nest Community.
This is not the experience that we wanted you to have, let me help you out. You can reinsert the batteries. If the issue persists, do a factory reset on your Chromecast device. This step ensures you can pair the remote. Check the steps below for guidance.
Go to Settings System About Factory Reset. Then select Factory Reset again, then select Erase to confirm the reset.
Check this link to set up your Chromecast remote.
Hope this helps.