I had to reset to factory the smart TV and most important I stopped the automatic updates in Google Play. The TV is updated to the latest software anyway, although I also deactivated the automatic updates for it. It's working more or less well for several weeks already. Sometimes when I start the TV it doesn't but if I tell google to change the TV volume, then it will lower the volume when listening next time. Not perfect, but better than nothing. Very disappointed anyway with these glitches from Google.
Sorry to hear that you're having issues with the Google Assistant on your TV. To confirm, do you have a Google Home/Nest device? Have you had the chance to try the steps provided by @MikeClark?
Looking forward to your response.