That certainly hasn't been easy for you ― let's check it out. A few questions: what Nest speakers do you have? How many devices are in each speaker group? May I know the exact commands you're giving? Are all devices on the same Wi-Fi network? It would help a lot if you could do a sequential reboot first:
Check out this guide for additional information about speaker groups.