O365 randomly disconnected until reboot. Making me lose my mind!

[thecomputerguy]

I have about 8-10 different businesses, some with 3 people some with 15 people (Probably 40-60 people total). Almost all of these clients have experienced this same issue. These clients are different businesses, different domains, different internet connections, and different brands/types of computers. There is only three things they all have in common, Windows 7, Kabuto+Emsisoft, and Office 365 for email.

I have posted about this before but I have zero resolution to this and I have been dealing with this issue for about a year. Sometimes the issue randomly stops for that client then comes back, sometimes it's fixed for a couple days and then it comes back.

The issue: The client will randomly disconnect from Office 365 Exchange saying 'This folder was last updated xx (2 hours ago) and they will get a credential pop-up. Typing in the credentials still does not allow the client to connect, and the credentials will appear again, and again. The client identifies the issue because they stop receiving mail and anything they send gets stuck in the Outbox. Sometimes they don't know there is an issue for hours. The only way to fix the issue is by rebooting which will fix the issue for up to a couple hours, to possibly a day.

These clients operated perfectly fine for years prior to this including while having Kabuto+Emsisoft now all of the sudden as soon as I wonder why a particular client hasn't had the issue, I get massive phone calls for multiple people having this problem so it is incredibly wide-spread among my clients.

I trying to save face by explaining I don't know what the issue is and I will troubleshoot it obviously without telling my clients that I have already been troubleshooting other clients for months.

I have tried:

Re-creating profiles (and removing all autodiscover/OST's out of the data directory)
Re-installing Office
Repairing Office
Downgrading Office
Redirecting Autodiscover to a local XML (regedit)
Other random Regedits
Calling MS (Who has me re-create the profile and reboot which will temp fix the issue and they consider it resolved)
CTRL+Click the Outlook icon in the tray and testing Autodiscover configuration (Test fails, unable to autodiscover)
Microsoft O365 connection support tool (Fails unable to login until reboot)
Verifying MX and autodiscover CNAME
Deleting credentials out of credential manager
Removing Kabuto+Emsisoft (Plus all of the above) sometimes might fix the issue
I'm still testing whether it's Emsisoft causing this but I FEEL like it is.
Whitelisting Outlook in Emsisoft

Webmail continues to function fine
Phones still receive email
Clients are on Office 365 either through MS, or Godaddy, or Appriver

When the issue goes away long-term for a client theres no exact reason why.

I have clients using Kabuto+Emsisoft that don't have the issue, but clients at that same company do experience the issue.

I have clients who had the issue and then the issue dissappeared while Kabuto+Emsisoft stayed on their system.

One client who brought their own AV in hasn't experienced the issue but they are the only client I have who brought their own AV.

Im literally about to lose my mind over this. I FEEL like it's Emsisoft but I don't have reliable evidence of this. I also have 250+ clients on Emsisoft so I don't want to believe it's them and then have to switch out 250+ clients.

I'm going absolutely nuts pretending like every phone call is a unique call and then just bashing my head against a wall trying to figure this out. I have clients who I've resolved this issue for by doing nothing, it just goes away for a couple months then comes back.

I just removed Kabuto+Emsisoft out of another clients machine and I will pay special attention to how it acts in the upcoming days.

 

[thecomputerguy]

Also, just moved a brand new client from POP to O365 into brand new empty mailboxes. Within one day they experienced this same issue, and yep you guessed it. Windows 7, Kabuto+Emsisoft, Office 365 through MS.

 

[mikeroq]

To me this has to be something with what you are using/setting up their environments. Some issue with Emsisoft or Kabuto? Good idea to remove it and see what happens.

 

[GTP]

Removing Kabuto+Emsisoft (Plus all of the above) sometimes might fix the issue

If you've removed them how does that "sometimes fix the issue?"
If they've been removed how can they cause the issue?

I'm still testing whether it's Emsisoft causing this but I FEEL like it is.

..and I assume you still haven't contacted Emsisoft support?
I guess it's just easier blame the tools?

 

[trevm999]

This communication is happening over HTTPS? Grab mitmproxy, set WinHTTP to use the proxy and then see if the traffic is getting to the server, if there is a response, etc.

Also is it possible to use SysInternal's Process Monitor to see if there is any activity by Emsisoft at that time?

 

[Sky-Knight]

Honestly I blame Windows 7, it's dead Jim... upgrade them. You've got 12.5 months to do it.

 

[YeOldeStonecat]

We have about 90x O365 client domains/tenants in our CSP portal....of various sizes. Not 90x Office 365 installs...but 90x different businesses on O365. So..throwing a wild hunch of how many Office 355 users total, probably hovering 1500 give or take several hundge.

At least 75% of them still Windows 7. And the remaining percent Win10.
I'd say a 50/50 split of Office 2013 and Office 2016.

The only time I've run into issues such as you describe, is on our very small far far few clients who are so small and unmanaged..they use the ISP provided gateway that is doing IPv6. (usually Comcast with that SMC Comcast Business Gateway). And those symptoms that I see really do match what you describe 100%. Can sometimes get it working for a day or two..and then it reappears.

Once I log into the Comcast gateway and turn off DHCP for IPv6 and reboot it..and then reboot the computers...problem gone for good.

The vast majority of our clients will have an edge device we installed and manage (typically Untangle or a some Ubiquiti edge router or unifi gateway, or perhaps if an older basic client from the "prior to Ubiquiti" days a Stinksys LRT224 or Stinksys wrvsomething. And we do not experience the issue you describe with those.

So the only other variable here is you have Kabuto and Emsisoft.

 

[Sky-Knight]

At least 75% of them still Windows 7. And the remaining percent Win10.

So you're saying 2019 is the year of machine replacements? Because support is dead Jan 2020...

 

[YeOldeStonecat]

Yeah this winter/spring/summer should be big in workstation sales for us for replacing rigs.

 

[thecomputerguy]

We have about 90x O365 client domains/tenants in our CSP portal....of various sizes. Not 90x Office 365 installs...but 90x different businesses on O365. So..throwing a wild hunch of how many Office 355 users total, probably hovering 1500 give or take several hundge.

At least 75% of them still Windows 7. And the remaining percent Win10.
I'd say a 50/50 split of Office 2013 and Office 2016.

The only time I've run into issues such as you describe, is on our very small far far few clients who are so small and unmanaged..they use the ISP provided gateway that is doing IPv6. (usually Comcast with that SMC Comcast Business Gateway). And those symptoms that I see really do match what you describe 100%. Can sometimes get it working for a day or two..and then it reappears.

Once I log into the Comcast gateway and turn off DHCP for IPv6 and reboot it..and then reboot the computers...problem gone for good.

The vast majority of our clients will have an edge device we installed and manage (typically Untangle or a some Ubiquiti edge router or unifi gateway, or perhaps if an older basic client from the "prior to Ubiquiti" days a Stinksys LRT224 or Stinksys wrvsomething. And we do not experience the issue you describe with those.

So the only other variable here is you have Kabuto and Emsisoft.

UPDATE: The two people last week who were having this issue have been resolved by removing Kabuto+Emsisoft.

@Barcelona I sent an email to Emsisoft for support on this issue. Part of the reason why I have tried to figure it out on my own is because they don't have a phone number (unless you have one) and their email support takes up to 24 hours to respond. I was hoping and praying it wasn't EAM because I don't want to spend the next week going back and forth with them via email to try and troubleshoot their product for which they will need logs and proof that it is causing the disconnect.

 

[Sky-Knight]

Yeah this winter/spring/summer should be big in workstation sales for us for replacing rigs.

I've got 4 Windows 7 stations left in service, and 4 more Windows 8 units. Either new box or refurb, everything pretty much got 10'd this year.

SSDs just about everywhere now too... It's to the point where I'm starting to grumble at people with platters, I don't want to work on them anymore. I've got one customer that just hasn't done it yet, and it looks like Q1 is finally the time. Honestly, I cannot wait for them to see just how much of their day they've been losing to this stuff. I keep telling them, but they just don't get it. All I need is that first box, that's the way it works. One SSD upgrade and the rest will fall. Because who's got time to wait for a platter anymore?

 

[YeOldeStonecat]

SSDs just about everywhere now too... It's to the point where I'm starting to grumble at people with platters, I don't want to work on them anymore.

YYYYup! Hate sitting down at spinners now.
I started doing all SSDs on new computers about 2 years ago. And most MSP clients of mine..existing computers, I cloned to SSD.

But...lesse....out of just over 3k total computers out there, with ~75% still needed to go to Win10, most of those should be new workstation sales...we'll have a banner year for workstation sales for 2019.