BitDefender GZ bringing workstation network speeds to a crawl.

[thecomputerguy]

Dental Office has a new Server I recently installed and about 15 workstations. In addition to replacing the server I replaced all of their mixed 10/100 networking equipment with All Ubiquiti (USG, 48P Switch, 16P 150w POE Switch for phones and AP's). Then got them onto MSP using Bitdefender for AV.

The office manager calls and says their program (Eaglesoft) is at a crawl on her station. I login, It's definitely slow. I kick everyone out and reboot the server even though nothing is pointing to the server being the issue. It reboots and she says it feels a 'little better'. I told her to let things stabilize and check in with me later. An hour later she says nothing has changed its still slow.

I go onsite and now I'm thinking maybe the networking equipment or all of the cabling I replaced has a couple bad cables. I do a transfer of a large file about 1GB from the server to her computer and sure enough ... transfer speeds are like 300kb/s and it would take an hour to transfer. I got to another computer and do the same and that one transfers fine ... the GB file transferred fully in about 3 seconds. I go to another one and it's slow... ultimately 4/15 of the computers were afffected by this slow down.

I kick everyone out and reboot the USG, and both switches. No change.

I decide to try uninstalling BD and see if there is a change. I didn't think there would be because I have BD on all 15 computers and only 4/15 are experiencing this slow down. Sure enough, uninstall BD and speeds are back up.

I reinstall BD on the 4 affected computers and let it update and run a full scan so it's stable. Speeds are still fine. I leave. I don't hear from them for the next two days. Today, Monday morning ... Same issue with the same computers (so far).

I don't want to uninstall BD so I created a new policy for them and disabled "Network Protection" and the "Firewall". Rebooted and now speeds are back to normal again.

How can I troubleshoot this? I don't want to remove BD or leave anything disabled but with this issue being intermittent it's driving everyone crazy, and to be clear this isn't an Eaglesoft issue, this issue is dealing with the transfer speed of particular workstations.

 

[YeOldeStonecat]

Unfortunately with todays antivirus....the good ones that check in so many areas, and are good against lots of ransomware....the "stuff" they do can affect certain software. I had a dental client many years ago with Eaglesoft...we don't have any current ones, so I have no magic formula. However I can relate with some accounting software.
I have some profiles setup specific to them.
First...proper exclusions....on both the server, and workstations. Usually the software vendor has a good FAQ section....listing directories and certain files. They'll hopefully list the processes too...which processes to exclude. Specific to the software itself...and if the software runs on a database engine such as SQL...be sure to exclude those.

Specific to Bitdefender...in the policies, under antimalware, make sure to uncheck "Scan net work files" for the workstations. I also disabled hyper-detect and advanced anti exploit on the server...as well as Firewall and Network Protection on the server.

Process of elimination...see if you can use a workstation for a bit...turn BD off..and then enable features one at a time...testing in between.

Had one client that had oddball software that I had to tell BD to only scan applications only. Under On-Access scanning settings. They're ditching that software soon so that'll be a relief to open up the AV again to more normal.

 

[thecomputerguy]

Unfortunately with todays antivirus....the good ones that check in so many areas, and are good against lots of ransomware....the "stuff" they do can affect certain software. I had a dental client many years ago with Eaglesoft...we don't have any current ones, so I have no magic formula. However I can relate with some accounting software.
I have some profiles setup specific to them.
First...proper exclusions....on both the server, and workstations. Usually the software vendor has a good FAQ section....listing directories and certain files. They'll hopefully list the processes too...which processes to exclude. Specific to the software itself...and if the software runs on a database engine such as SQL...be sure to exclude those.

Specific to Bitdefender...in the policies, under antimalware, make sure to uncheck "Scan net work files" for the workstations. I also disabled hyper-detect and advanced anti exploit on the server...as well as Firewall and Network Protection on the server.

Process of elimination...see if you can use a workstation for a bit...turn BD off..and then enable features one at a time...testing in between.

Had one client that had oddball software that I had to tell BD to only scan applications only. Under On-Access scanning settings. They're ditching that software soon so that'll be a relief to open up the AV again to more normal.

That's kind of what I was afraid of ... the problem is it is so intermittent, based on my OP. I can replicate the issue on a computer, I uninstall BD, speeds return to normal, I reinstall BD speeds are fine for a few days then all the sudden back to a crawl. I was however able to get the speeds back up WITHOUT fully uninstalling it this time. Disabled Firewall and Disabled Network protection. Reboot and were good.

It wasn't the reboot that ultimately fixed it ... obviously many reboots have already been had.

I think I'm going to give them a few days hopefully with no issues. Then I'll re-enable the network protection and wait a few more then enable Firewall and see where we are at.

Just very frusterating for the client.

 

[YeOldeStonecat]

I have a profile I used for our default BD install (via Syncro). EVERYTHING is turned off. We use this profile for our defaults...for the push installs, and then I set a profile to each client network (and server). So nothing is turned on upon install...don't want software eaten.
So you should not have to uninstall, and the following reinstall, BD...just apply different profiles, including the "everything is turned off" one when needed.

So the hyper-detect, and the anti exploit...on the server...try killing those yet?
I never was BD managing the firewall..I have that off everywhere.

 

[thecomputerguy]

I have a profile I used for our default BD install (via Syncro). EVERYTHING is turned off. We use this profile for our defaults...for the push installs, and then I set a profile to each client network (and server). So nothing is turned on upon install...don't want software eaten.
So you should not have to uninstall, and the following reinstall, BD...just apply different profiles, including the "everything is turned off" one when needed.

So the hyper-detect, and the anti exploit...on the server...try killing those yet?
I never was BD managing the firewall..I have that off everywhere.

Do you know how long it takes for GZ to push a policy update to the station? Like less than 5 minutes usually right?

 

[YeOldeStonecat]

Oh yeah it's quick...mere minutes at the most...usually see it in under a minute.

 

[thecomputerguy]

Oh yeah it's quick...mere minutes at the most...usually see it in under a minute.

2nd Client now ... Brand new server with a 6SSD RAID10 ... calls because Lacerte and Lacerte DMS are running slow. I login and test the network transfer speeds by copy and pasting a 800MB file from the server to the PC.

The transfer speeds are below 5 MB/s and sometimes dip into the KB/s ... it takes roughly a minute to transfer this 800MB file.

I reboot the system and the issues still persists.

I uninstall BitD and test the transfer speeds after and the network transfer speeds are pinned at 110MB/s and that 800MB file takes less than 4 seconds to transfer, and Lacerte is now instantaneous, so is DMS.

Reinstall BitD and let it update, reboot computer and test speeds and speeds are now back below 5MB/s

Disable The Firewall and Network Protection module ... speeds still slow ... uninstall BitD ... speeds back up to 110MB/s.

I'm LOSING MY FREAKING MIND OVER THIS DAMN PROGRAM

 

[thecomputerguy]

Disabled the Anti-Malware on-access scanner ... so Bitdefender is like literally completely disabled and speeds are still below 5MB/s

 

[thecomputerguy]

Disabled on-access advanced threat control

 

[Markverhyden]

Have you created a ticket with BDGZ?

 

[YeOldeStonecat]

NM got side tracked by a phone call and was replying....and you replied more and answered my questions.

 

[thecomputerguy]

Yes I put in a ticket one through the website and one by phone

 

[YeOldeStonecat]

Hyper Detect disabled?
Advanced Anti Exploit disabled?
Screenshot of exclusions?

 

[thecomputerguy]

Ok I disabled Hyper Detect and Advanced Anti-Exploit

They are an accounting firm so I can't continue troubleshooting until after the 15th.

My other dental office still has the same problem

Currently I have the whole server folder as an exclusion

 

[YeOldeStonecat]

Here's a copy/paste from exclusions for an accounting firms server we have...that runs LaCerte...
The drive letters are how the workstations see the server...so those 'll prolly be diffy for you.
This is under Antimalware...Settings.

������

Folder	T:\*	On-demand, On-Access, ATC/IDS
Folder	S:\*	On-demand, On-Access, ATC/IDS
Folder	Q:\*	On-demand, On-Access, ATC/IDS
Folder	C:\Lacerte\*	On-demand, On-Access, ATC/IDS
Folder	%CommonProgramFiles%\Lacerte Shared\*	On-demand, On-Access, ATC/IDS
Folder	%COMMONPROGRAMFILES(X86)%\Lacerte Shared\*	On-demand, On-Access, ATC/IDS
Folder	%COMMONPROGRAMFILES(X86)%\Intuit Shared\*	On-demand, On-Access, ATC/IDS
Folder	%CommonProgramFiles%\Intuit Shared\*	On-demand, On-Access, ATC/IDS
Folder	C:\ProgramData\Lacerte\*	On-demand, On-Access, ATC/IDS
Folder	%AppData%\Lacerte\*	On-demand, On-Access, ATC/IDS
Folder	%LOCALAPPDATA%\Lacerte\*	On-demand, On-Access, ATC/IDS
Folder	%LOCALAPPDATA%\TempDMSTemp\*	On-demand, On-Access, ATC/IDS
Folder	%PROGRAMFILES(X86)%\Intuit\DMS\*	On-demand, On-Access, ATC/IDS
Folder	C:\Program Files (x86)\Intuit\DMS\*	On-demand, On-Access, ATC/IDS
Folder	C:\USERS\%USERNAME%\Appdata\Roaming\Lacerte\*	On-demand, On-Access, ATC/IDS
File	C:\Lacerte\19Tax\OpIndex.p9	On-demand, On-Access
Folder	C:\Lacerte\19Tax\*	On-demand, On-Access, ATC/IDS

 

[thecomputerguy]

The thing is I'm not even opening lacerte I'm literally just copy pasting a quickbooks download (about 750MB) from their installs directory on the server into a local folder.

Same thing with my dental office, I use a large QB download to test network transfer speeds.

 

[thecomputerguy]

@YeOldeStonecat @Markverhyden

One thing I've noticed is that when I'm logged into it remotely it seems like the NIC is working excessively hard and I see a lot of packets being sent and received even at idle.

Obviously there should be some packets even though it's idle because I'm logged into it... but do either of you have a local network monitor you can recommend so I can see where local network packets are going? I looked in the network monitor to see if that provided any information but it didn't seem too helpful at the time.

 

[Markverhyden]

Wireshark is my goto. I'd put it on one of the laggards, the server and then one of the normal machines.

But if you want to look at the entire network it's more complicated. Since switches don't multicast traffic like a hub you can't just plop a sniffer on the network. You have to mirror all the switch ports to the port where the sniffer it. Fortunately it's available. Log into the controller, click on devices on the left, click on the switch. In the popout panel on the right click on ports (middle) icon. There's lots you can do there without wireshark.

 

[thecomputerguy]

@Markverhyden @YeOldeStonecat

I think I'm making some progress here. I got another email from BitD support and apparently I had been troubleshooting this slightly wrong. I had been going back and disabling components on the POLICY level in BitD GZ. Support said that I needed to create a reconfigure task for the client to remove the modules completely from the client.

Upon install I had all modules enabled including:

Anti-Malware
Advanced Anti-Exploit
Firewall
Network Protection
- Content Control
- Network Attack Defense
Device Control

Which are the default options (for me at least when creating a package). Apparently just leaving these modules installed, even though they are disabled, can cause issues like this. My guess is this may actually fix it and I might just create future packages with only Anti-Malware Enabled.

@YeOldeStonecat

Can you confirm what modules you use for most clients when they are considered finalized and on an MSP package. I know that you said that you have a default policy/package that on install has all this stuff disabled so that important stuff doesn't get yanked.

But as of the end result when things are stable, which modules are you typically using?

 

[Diggs]

I've bit my tongue here but I have a bad taste for BD. I mentioned in another thread how I installed it at a small business and it immediately killed networking. Arrruuu? Isn't that what small business is about? A small network? I had to go in four levels deep in the menus and disable some firewall options and tweak a few other settings just so they could just do some file sharing. You'd think BD would ask a few questions during setup to avoid this kind of failure.