Google Nest Community

Have you tried switch off WiFi and mobile data on phone Aset up a hotspot on phone AUse another phone B running Google home and connected via WiFi to phone AReset your speakers and connect them in the home app on phone BIf they work it's your WiFi

I had exactly this problem and it was down to poor WiFi signal in some areas. I bought a couple of Xiaomi ax1800 routers, set them up as a meshed network and since then it's a worked fine.