Thanks for posting. This is not the experience that we wanted you to have.
A few things: Which Nest speaker? What steps have you tried so far? Is the issue happening on your phone's Google assistant too? Try to reset your Nest speaker to put it back to its default setting and set it up as a new device then observe if this will fix it.
Let me know how it goes.