Intermittent excessive latency (ping time)

[ballfam]

Product: Google Nest WiFi

Version: 14150.882.9

I have been having problems with intermittent excessive ping time for a while, and I noticed there are several posts on this; this one is now locked:
https://www.googlenestcommunity.com/t5/Nest-Wifi/Large-latency-ping-spikes-with-Nest-Wifi-Pro/td-p/2...

But the problem persists on the version above.  My setup is as follows:

• Fiber gateway running symmetrical Gigabit; this is a Spectrum fiber gateway with no router or AP, it simply provides a public IP and a gateway from the fiber the comes into the house to a copper gigabit connection.
• Main Nest router is plugged directly into the fiber gateway and acts as the only router/firewall, it gets the IP from the ISP and provides NAT on 192.168.86.0 network (default configuration). The outbound port is plugged into a gigabit switch which serves hard-wired ports around the house via cat6e cable
• 4 additional nest routers are hard-wired to cat6e sockets around the house and act as access points for the mesh; there is 8,200 sq ft of area to cover, thus the large number APs
• A number of machines are also hard-wired into the network via the cat6e ports that lead to the main nest router

The usual ping time on my gateway is around 8ms, and the download latency will vary up to about 20ms with high volume. From time-to-time (about once a week right now), the latency will spike; the ping time might go all the way up to 900ms and the download latency will spin out over 1,800ms. This causes all kinds of issues with real-time streaming, like watching Netflix or running a Zoom meeting. The ping time is this bad on a wired system...this has absolutely nothing to do with Wi-Fi reception, walls, or distance from the AP.

I have debugged the issue by unplugging the nest router from the gateway and plugging in a wired system directly to the gateway, as soon as I do this, the latency goes back down to 8ms on that system.  As soon as I plug the nest back in and sit back on the other side of it, the latency goes back up to 900ms. The problem has nothing to do with the ISP, who is still providing an 8ms latency.

After resetting the entire mesh, once it reboots, the latency immediately goes back down to 8ms, and I'm good for about a week or so, and then the problem happens again.

This is absolutely a bug in the Google Nest software, which is present in the above version. It has already been seen by several people and as of the above version has not been fixed. Can someone please confirm that it is a known current issue, and someone is working on fixing it.

[ballfam]

BTW: The problem is also most likely unrelated to DNS. When running a speed test, it will initially take 900ms to ping, and then, ongoing, as data is being downloaded from the same host (after the IP has already been translated), the latency still spikes at over 1,800ms.

Moreover, after translating google.com to a real IP address, and then running a manual "ping" on that IP from a Linux command-line on a wired host, the problem persists, the ping time is way up around 900ms even when pinging a physical IP address. DNS is not the problem, and changing the DNS server on the nest router will not fix the issue.

[orellanaortiz]

Hi ballfam, 

Thanks for the troubleshooting performed and for letting us know the firmware version of your Google Nest WiFi and the set-up information. I am sorry that you are still experiencing issues with latency even after processing a reboot. No worries, help is here! 
 

I understand that you verified that this issue is not related to DNS. Nevertheless, the best suggestion that has worked in these situations is to change your DNS using these steps: Change your DNS. 

I also suggest following the recommendations in this article: Troubleshoot slow internet on Google Nest Wifi or Google Wifi. 

Let me know how it goes.

Regards, 
Angel

[ballfam]

Yup, did that already, and turned off IPv6. I have tried Google DNS, ISP DNS, and Cloudflare DNS, none of them fix the issue. Note that this problem is intermittent, and a restart fixes it temporarily, until it shows up again a week or so later. Once it hits, there is no way to fix it except a restart of the whole mesh.

[orellanaortiz]

Hi ballfam, 

Thanks for your patience and for following the steps provided. 

Since you already tried the DNS options and the issue persists after a week despite the reboot being processed, I also suggest trying these options: 
 

1. Ensure the child points are connected to the parent point (or switch) through the LAN port, not the WAN port.

Since the WAN port emulates a LAN port in child mode, it has more overhead and is unable to process information as quickly as using the LAN port directly.

2. I share with you these steps on how to view the data usage of each device connected to your Nest WiFi so you can verify which devices are using more data since this can impact the download latency and the ping time. Once you confirm that, you have the option to prioritize a device on your Wi-Fi network, This option gives a device of your choosing the fastest connection on your Wi-Fi network. 

In case the issue persists, I do recommend you contact us with this form so we can verify more details of the device on our end and follow up to find the best option to resolve this intermittent situation with the latency.

Hope this helps.

Regards, 
Angel

[ballfam]

OK, thanks.

1. Some of the points (nest running in AP/Bridge mode) were connected view the WAN port. I switched them to the LAN port to see if it helps. I have a feeling this won't help since the wired systems, which also experience the issue, are all connected directly to the main nest router via a gigabit switch, and, in theory, would not be affected by any of the APs, since they do not route through the APs. Let's go through the topology so you know what I'm dealing with:
   1. Fiber comes into the house, this is a true symmetric gigabit connection which drops into a simple fiber gateway, fiber in -> copper gigabit out. There is no router in this equipment
   2. The copper gigabit from the gateway is pugged into the WAN port of the first nest. This device now acts as the main router, it gets the IP from the ISP and provides the firewall and NAT for the rest of the network. This nest also acts as the local DHCP server and will provide IP addresses on the 192.168.86.0 network.
   3. The LAN out from the first nest is plugged into a dumb gigabit switch (in case you don't know network hardware, this is just like a multi-way adapter which allows multiple devices to wire in to the nest port). Anything downstream of this switch is on the local LAN behind the firewall, and it gets it's IP from the main nest router
   4. The other ports on the switch are plugged in to wired cat6e cables which lead to RJ45 sockets distributed around the house.
   5. Each other Nest router is plugged directly into one of those RJ45 ports, which effectively wires it directly downstream of the main router, and turns it into an access point for the mesh. Some of these were plugged into the WAN port and I switched them to the LAN port, although I don't think it will fix the issue
   6. There are also other wired systems plugged into other RJ45 sockets, which will effectively wire them directly to the main nest router. Their path does not route via any of the other nest APs, instead, they are wired directly into the LAN port of the main nest router, yet they do see the latency issue. Conclusion is that the latency problem exists right at the main nest router, anything plugged downstream of this router sees the problem, if the router is unplugged from the gateway, and a machine is wired directly into the gateway, the problem goes away, as soon as the device is plugged back downstream of the main nest router, the problem comes back again.
2. The majority of the data flow comes from one of the wired systems, which is a local server; it serves several purposes, including a file server and a video streaming server. Note that the entire network from the source, all the way through to the ISP is gigabit capable, and at no time does any of the network traffic come anywhere near reaching or exceeding the gigabit limit; prioritization is not an issue, and the network is never overloaded.

After changing the APs from WAN to LAN ports, I'll see if it resolves the issue. I suppose it is possible that using the WAN port is causing the nest APs to do something to the main next router which then triggers it into some really bad latency......My current working theory is that the mesh software is endlessly bouncing packets around between the router and APs (remember I have a total of 5 nest devices on the network), and it has flooded the main router, which now takes a really long time to process the network packets, perform the NAT and firewall processing, and send it out on the main link.

It hasn't happened for a week or so.....fingers crossed. If it happens again, I will post.

[orellanaortiz]

Hi ballfam, 
 

Thanks for your response and all the details provided. Let's keep a close eye on it and let me know if any unexpected situation presents again. I appreciate your efforts to resolve this situation in the best way possible.
 

Keep me posted.

Regards, 
Angel

[ballfam]

It happened again yesterday, ping time was back up to 500ms, and download latency was over 1,000ms. Download speeds  are verry choppy and vary from 500Mbps down to 100Mbps on a Gigabit wired connection. The problem shows up in 2  ways; web sites suddenly take a really long time to load (multiple requests on a page have to wait for the massive latency to get packets), or alternatively, video streaming keeps buffering. Running a speed test will show the issue, where both Ping time and download latency will be really large (up to 100 x slower than normal).

Just to 100% prove it is a software/firmware issue on the Nest device, I fixed the problem without even getting out of my seat. I picked up my phone, launched google home, clicked on network settings and clicked "Restart entire Network". Magically, a few minutes later, everything is back up, and the network is back, the ping time is back down to 8ms and web sites load instantly.

This absolutely is a Google Nest bug, there is no doubt about it. I guess I'm going to have to fill out the "contact us" form and report it, hoping one of the engineers pick it up....I really hate the way these consumer products have no bug tracking system that is visible to the customer; I'm sure they probably use Jira, it would be nice to get a trackable bug number.

Anyway, if anyone else starts to see this issue....Yes, it is a Google  Nest bug, and the workaround is to restart the network, fortunately, there is a button to do that in the Google Home app.

[orellanaortiz]

Hi ballfam, 

Thanks for your efforts towards troubleshooting with us.

I am glad to hear that restarting your entire network resolved the latency issue with your Google Nest Wifi. Thank you for sharing this information, I am going to make sure to document this situation for future reference in order to gather more reports and thus work on a fix.

I also include here a form that you can fill with your feedback:
Share your feedback

Please let us know if you have any other questions.

Regards,

Angel

[orellanaortiz]

Hi ballfam,

I deeply appreciate you sharing your thoughts and experiences. We received your feedback for future reference in order to gather more reports and thus work on a fix as soon as possible.

Let's keep a tight eye on it and let me know if any unexpected situation presents again

Regards,

Angel
 

[ballfam]

Thanks Angel,

I already filled the form out to report it to Google. Hopefully they will ping me if there is ever a fix. In the meantime I'll just keep restarting the network when I see it happening, I have gotten quite good at spotting the clues now, and I'll usually just do a restart at a convenient time before it gets too bad. I have a complex network with point-to-point bridges connecting other sites on the property, systems running VPN connections 24x7, and some devices and VPN connections are sensitive to loosing internet, so there is work involved when I have to restart the network; better to do it when it is convenient for me.