This is not the experience we wanted you to have, let me help you. A few things: which Nest speakers are we working with? When did your issue begin? Were there any recent changes made? When you did the factory reset, was it the same steps as what you see in this link?
It would help a lot if you could do a sequential reboot first:
Once done, make sure the Google Home app is on the right account. You can double check when you open your Google Home app, tap the icon on the top right. Make sure its the correct one and tell me how it goes.