I have been reading this user forum every day for the last 9 months or so and I can tell you A LOT of people are having the same issue. I seem to remember that it is more of a problem on Android phones rather than iPhones. Also, to the best of my knowledge Google has yet to release a fix.
Thanks for reaching out. I appreciate the help @EmptyNester. Let's get this sorted out. What type of Nest Doorbell are you using? Also, how are you using the talk and listen feature?
I would recommend trying a different phone to isolate and check this article on how to Talk and listen with Nest camera or doorbell for detailed steps.
Just checking in to see if you still need assistance with this issue. We'll leave this topic open for another 24 hrs in case there are any follow up questions or comments. Please let me know if you need further assistance.