Diagnosing VPN issues

[HCHTech]

I am trying to diagnose a problem with remote access for the owner (of course) of one of my clients.

There is a Sonicwall (TZ500 - latest firmware) in the office, and we're using Sonicwall's NetExtender SSLVPN client. This office has about 35 employees and we have about 32 of them setup for remote access using this same setup. They connect the VPN tunnel with NetExtender, then RDP to their office desktops.

For the owner, there is an intermittent problem where NetExtender will report a successful connection, but RDP will fail to connect. Testing has proven that when the problem occurs, you cannot ping any device on the office network, so I believe that NetExtender is reporting a successful connection, but no traffic is actually being allowed through the tunnel, or there really isn't a successful connection despite the status indicator. If you stop to check whether you can ping the office network, and that is successful, then RDP will work every time. Typically, rebooting the remote machine will make things work, but obviously that is cumbersome.

I have been around the block with Sonicwall tech support about this issue for about a month now. Before the latest session with them this morning, I managed to successfully capture all of the packet traffic from the Boss's home computer to the Sonicwall during a connection failure event. In all, I captured about 60 seconds worth of traffic, which resulted in about 46,000 packets of information. Sonicwall searched through these captured packets and found about 40 RST TCP packets (I've copied one below with the identifying data redacted). Sonicwall then declared that the problem was with the boss's ISP (possibly injecting RST packets to thwart VPN connections, or maybe just not "VPN Friendly"). They suggested we test with a hotspot to see if the problem recurred, and declared the ticket resolved. Ugh.

I am seriously out of my depth, so I didn't have an intelligent response for them. It seems to me by looking at the packet below that the SOURCE is the office IP and the DESTINATION is the boss's home IP, so that looks to me like a problem with the OFFICE ISP, not the home ISP, but I could be misinterpreting things. No one else at the client has reported this problem, to that does lend credence to the Home ISP being on the suspect list.

Here is one of the packets:

*Packet number: 207*
Header Values:
 Bytes captured: 54, Actual Bytes on the wire: 54
Packet Info(Time:09/09/2020 12:09:35.192):
 in:X1*(system-stack), out:X1, Generated (Sent Out), 1:3)
Ethernet Header
 Ether Type: IP(0x800), Src=[MAC Addr Redacted], Dst=[00:11:22:33:44:55]
IP Packet Header
 IP Type: TCP(0x6), Src=[Office IP Redacted], Dst=[Home IP Redacted]
TCP Packet Header
 TCP Flags = [RST,], Src=[4433], Dst=[57822], Checksum=0x7e77
Application Header
 Not Known
Value:[0]
Hex and ASCII dump of the packet:
 00112233 445518b1 REDACTED 08004520 0028ac06 40004006 *.."3DU..i.....E .(..@.@.*
 1d874b97 e6214841 f7281151 e1de61REDACTED000 00005004 REDACTED.(.Q..aD......P.*
 00007e77 0000

Does this make sense to anyone?

 

[SAFCasper]

Yes, it appears as if the source of the RST is your office IP.... but that's how packet injection works. It spoofs the source address to look like a genuine packet.

You need to capture traffic from both ends of the connection to prove whether packets are injected or not. When you receive RST packets at one end there should be a matching record of the RST being sent from the other end. If not, something is injecting them in the middle.

However, I'm siding with tech support here. The easier option is just run a hotspot for a few days to completely rule out any ISP interference. Save yourself the headache of trawling through more packet traces.

 

[Sky-Knight]

Yeah, and BS like this is why I use Untangle. Sonicwall's SSLVPN rides over PPP, Meraki relies on L2TP, both protocols are ancient, and prone to failure due to a horde of reasons. Untangle gives me L2TP, OpenVPN, and soon WireGuard all in one place. So I'm free to deploy one or all of them to the end user so they can swap around until they find one that works. It's essential when working with some hotel networks in particular.

The part that concerns me is the reboot on the remote device, if that is fixing anything it seems to me the ISP is fine but the machine is suspect. I'm more inclined to blame a software firewall on the endpoint.

But there are a horde of terrible ISPs out there... so I can't really refute support's claims fully.

 

[Markverhyden]

Did you test it with a hotspot? That's a check I'll do early on with networking related things like that. When the existing ISP has a VPN problem, just flip it over to hotspot with no other changes to see what happens.

 

[HCHTech]

Thank you for the responses. We have NOT tested with a hotspot yet. Let's assume we do, and the problem goes away, pointing definitely to the ISP. What are my available options? In this case it is Verizon residential FIOS, so I doubt very much that there is any recourse at all. Even if I was a masochist and decided to call their tech support, I'm sure they would tell me "we don't support VPNs". And rightly so, frankly.

I have tried disabling the local Windows firewall as part of our diagnostics, which didn't impact the problem at all.

I can tell you that we have about 80 clients using this setup for their employees, since we are a Sonicwall shop. I don't have a count of the total employees using NetExtender, but it must be a several hundred at least over all the sites. I'm not getting this complaint from anyone else. FIOS and Comcast are the two big ISPs in my area, so I'm thinking that if this whole thing IS a FIOS issue, it's isolated to this guy's connection or node or however they divvy up their customers.

I'm also not going to change my business model away from Sonicwall solutions because of one seemingly isolated issue. They could buy a GoToMyPC license for the owner and probably steer around this problem - I think that will be my suggestion.

 

[Markverhyden]

I did a quick search on

Application Header
 Not Known

and it came up right away with Sonicwall hits. Several comments about possible ARP problems amongst other things.

FIOS? Doubtful it's an issue with their underlying system. Technically it could be a problem with the equipment. The problem is that it's lengthy process to work through the permutations at the client site. I'd give VZ a call to have them reset everything and make sure all firmware is up to date.

 

[Sky-Knight]

Windows Firewall cannot really cause this sort of issue in my experience. The kinds of things that do are malware integrated firewalls, if you don't have any 3rd party AV installed then that's ruled out.

I'm not suggesting you ditch Sonicwall, it's a solid product. I'm just saying this sort of thing is quite normal, and you will have to make a plan for some users to work around weird issues that you may not ever find the real cause of. GoToMyPC isn't my first choice, but it does work.

Mark is correct, it could be an ARP issue. It could also be an MTU issue. If you have other users on the same ISP and they're working, then we're pointing more at gear at the specific site in question and not the area or ISP again. The above is a great suggestion, get all that equipment re-provisioned at least, it'll force an updated firmware and might clean things up.