If I understand your setup correctly, the reason you're having issues is that you need an Assistant-enabled device connected to your TV so your voice commands can reach it directly. This would be a Chromecast or a Google TV device. While you can get some interaction to your TV through other connected devices, the voice commands require a Google Assistants connection. Currently, that is done through Chromecast devices if the TV itself does not have a built in Chromecast functionality. Let me know if that clears it up for you or if you need more info.