It sounds like you have done all the regular steps; resetting and changing batteries.
https://support.google.com/chromecast/answer/10070837
I would only suggest trying a totally different brand of batteries. I recently had trouble with a device that I tracked down to a defective box of batteries. Once I bought some different batteries everything worked great.
You can buy a replacement remote from https://store.google.com
You may also check to see if you can get a free replacement if you are still under warranty.
https://support.google.com/chromecast/answer/14076635
