That certainly hasn't been easy for you ― let’s check this out. What Nest speaker do you have? If swiping to the left didn't work, how about the other functions like swipe to the opposite side or up? Is it working on voice commands?
What you need to do is to do a sequential reboot starting with your router followed by your Nest Audio. Restarting your phone might also help. Doing these steps help the connection and functions between devices refresh itself.
Tell me how it goes.