The problem is the chromecast only has the one output, hdmi. So you need a devic that can receive the audio from the hdmi and then outputs the sound. The only 2 common devices to do that are a TV & a stereo receiver with an hdmi input. But neither of those would make sense to use if trying to play on a google speaker.
And, as youve probably seen, when you try to cast a video stream you cant select an audio only device.
So, Im not sure this is possible, wirelessly. If you speaker has an input (headphone jack type) you could use your phone (if it has a headphone jack) to connect the two devices with a male-to-male audio wire.
