Thanks for reaching out and sorry to hear that you are having an issue with your Voice Remote.
A few questions: what is the LED light status in your Chromecast Voice Remote? Was anything spilled onto the device? Can you confirm if the metal contacts of the battery terminal look contaminated, blocked, or discolored? Please try to reset your Chromecast Voice Remote and observe the behavior.
If the same issue persists, please fill out this form so we can further check this in our end.