Hi. That's what I am doing. Switching it off with the Chromecast remote.
This also switches the TV off. After that the Chromecast LED remains
white and the screen time is used after a while. I hope you don't mean
that I have to unplug the Chromecast ...