Port forwarding crashing?

[DementedJay]

Hi all,

I'm curious to see if anyone else is having a similar issue to mine. I have several servers / VMs running on my network, and had set up port forwarding in conjunction with nginx for SSL and reverse proxy to safely expose some of these services to the internet so I can access them when I'm traveling.

Everything was pretty good until this last weekend when I was on the road. I was able to access my services, and then...all of a sudden, after several hours, something broke (400 error, connection timeouts, etc). 

When I got home, after some troubleshooting, it looks like the router's port forwarding crashes after several hours of use. I'm not entirely sure why. 

I can reach all the services locally with no stability issues. Likewise, nginx is running just fine and still sends traffic appropriately. But the router is the issue here. Resetting it fixes the issue, but that's not really a fix when I'm on the road and my kids and wife are using the home network for their own stuff. 

Is anyone else seeing this? I'm on software version 14150.43.81 if that helps. I've contacted support and was told someone would get back to me, but...no one has. 

If I can't figure it out, I'm going to have to use a different router upstream and do some major network rearranging to get this working, I guess.

[Geekchik]

I'm not on Gen1 anymore, but maybe check if you have caps set elsewhere in the admin. I did this once in a GN to help detour bots from a gaming network, then forgot to remove the cap and had similar issues. If you are able to raise the limits, then send test through separated VMs and VPNs and see what happens? Is there packet loss or just straight 400 errors? Any other words in your log? What is your TTL and are you using same DNS? I'm thinking that since resetting fixes the issue (and if you are indeed resetting, not rebooting) that one of your mods is causing the conflict. It may not be handling the external traffic through your designated ports if there are caps on each or total. Lmk

[DementedJay]

Where do you set the caps you're mentioning? I don't think that's something I've ever set up. 

Looking in my nginx error logs, there are tons of entries that look like request for non-existent resources, which might be from bots or maybe from malicious exploit attempts, I can't be sure. 

I'm going to try moving my external ports to some random "safer" ports, and redirecting those internally to the servers.

[olavrb]

Why do you port forward if you use a reverse proxy? Isn't the whole idea with reverse proxies that you don't port forward/ shoot a hole in the firewall?

Edit: Never mind, did some reading, makes sense to me now.

Honestly, I'd just not trust Nest Wifi for any advanced use case. Also because troubleshooting and fixing any issues with Google support, especially niche use cases, often takes a lot of time. If it ever get's fixed.

• I've escalated this thread though, so that Google can prove me wrong. 🙂

Maybe look into solutions that does not require port forwarding, like Tailscale?

• https://tailscale.com/pricing/

[DementedJay]

Yes, as I'm sure you're now aware, you still need to open ports in your router / firewall for the reverse proxy to work. It looks like there are external entities that are constantly barraging my ports with requests for resources that don't exist, after looking at my nginx error logs. Those could be bots attempting to access a non-existent sitemap.xmp or robots.txt file; I see those requests coming in and failing. But there are also requests for non-existent local resources that make me think this is more like an attack.

I'm going to move my ports around, see if this has any affect.

[DementedJay]

Running a whois on the IPs requesting these resources shows a lot of abusive IP addresses:

185.196.220.81

66.240.192.138

etc

I'm going to attempt to block some IP ranges and see if that helps as well.

[Geekchik]

That sounds like a plan. Sometimes ISP or firewall can send false alerts in your log, which is essentially a built in security check. But you are on the correct path by whois on the IPs. In the example I gave above, the IP would be an ISP or colo, or with your proxy. If the IPs are malicious, or even unknown, blocking and reassigning ports would be a good first step. Then try running a boatload of traffic reverse full circle to mimic your travel use cases. I.e. endpoint would be ping or route to "home" from "travel location".

  Depending on use case, you could try load balancing if you're running your own (multiple) servers in case that in contributing to time outs. I'm assuming you're using static IPs with designated assignments, yes? Randomizing may cause issue depending on other settings, like if router is automagically switching devices between radios to balance bandwidth. 

 I'm not familiar with your GUI, but caps should be in the same higher level menu as port forwarding, "advanced settings... "do  not touch unless you know what you're doing" 😉 something like that. 

[Geekchik]

To add to my recent comment, are you running your projects plus family through the same network? I would set up a separate network on the unit, or  even an entirely separate box and network to isolate been home and work activities. This way you can config and confirm that the issue is from your activities and not something like a gaming console or smart home vulnerabilities. Best practice IMHO is to have IoT on their own network(s), and then kids and benign devices separate. You can usually, not always, set up sharing between. I've noticed some smart devices and IoT require less secure networks. Some early gen or newer to market companies are still using highly exploitable bluetooth, or even requiring access to the entire network (eg. my washer and dryer aren't connected to WiFi because I discovered this requirement. Basically if the machines have WiFi turned on and even if connected to a secured net, it's a big welcome sign to anyone with a scanner) These scenarios open the door wide open in terms of vulnerability to anything else on that net. Just something else to think about while you're handling potential backdoor exploits

[DementedJay]

They're not on the same same network, but they're running on the same router. I have multiple routers at my disposal, but am currently using the Google / Nest WiFi router as the "main" router. I have switches with multiple VLANs for my home lab vs. the rest of the house's wired network vs. PoE devices, for instance. 

Most of my servers are virtual and running on a TrueNAS Core box I built last year. 

Thank you for your advice; I'm fairly advanced at this stuff, which is why the Google WiFi router is so frustrating. It's fairly dumbed-down, and there's very little user control over what happens on it. But port-forwarding is usually such a rock-solid feature that I've never really considered it crashing on a router.

But yes, people exist who write scripts for bots to run to exploit port forwarding on common ports. 

I've moved my ports to a different random port range and so far everything seems to be stable again.

[Jhonleanmel]

Hi everyone,

@olavrb and @Geekchik, thanks for the help.

@DementedJay, I'm glad that your network is stable. Thanks for sharing the steps you did here in our community ― it's a big help.

If you're still in need of any assistance, feel free to let us know.

Best,
Mel

[DementedJay · Accepted solution]

Adding a few more notes here for anyone else running into issues around port forwarding "crashing."

It's now pretty apparent that having port 80 open at all is probably a bad idea. I've tentatively turned on port 443 again to see if there are attacks against it, and also implemented a project that creates a dynamic block list from known bad actors into nginx (https://github.com/AmplitudeDesignInc/abuseipdb-php-nginx-blacklist-create) which seems to be doing the trick so far. I may not need those higher ports if this eliminates the bot attacks.

[Jhonleanmel]

Hi DementedJay,

We always appreciate your helpful response.

Cheers,

Mel

[Jhonleanmel]

Hey folks,

I wanted to follow up and see if you are still in need of any help. Please let me know if you are still having any trouble from here, as I would be happy to take a closer look and assist you further.

Thanks,
Mel

[DementedJay]

I'm not having any trouble at the moment, but that's because I implemented an IP blacklist. If I hadn't, my router (and presumably every router) would be under constant attack on every commonly used open port.

But I appreciate the follow up. I think Google should consider this in light of how the world has changed since a feature like port forwarding was first introduced. On its own, it's no longer adequate. 

[Jhonleanmel]

Hi DementedJay,

We appreciate your helpful responses. We're always looking for ways to improve and we'll take it as feedback.

I also suggest that you send feedback by following this link. Let me know if you have other questions or concerns ― we'd love to help.

Cheers,

Mel

[Princesss]

Hi there,

We just want to check if you have seen our response posted above. Let us know if you have additional questions, otherwise we'll be locking the thread.
 

Best,

Princess

[Princesss]

Hello there,

Due to inactivity, I'll go ahead and lock this thread in 24 hours. Feel free to start a new thread if you need assistance with your Google Wifi and we'd be glad to assist you further.
 

Best,

Princess