W32.Changeup kickin my butt

[LAconsult]

This virus sucks. On one computer/server maybe it wouldn't be such a big deal, but I've got an uber infection on a client network that started a few days ago, and progress has been very slow. Hopefully something can be learned from this post as I keep trying to find a resolution, try any suggestions short of reinstalling everyone's computers from scratch, and document my attempts. This is an environment I've been trying to convert from time and material to managed services for a while now. They know the benefits of managed services, I've tried out two different kinds (GFI and Kaseya) on this network and they worked great, with no issues and staying on top of automated maintenance/virus notification etc for months, but the cost @ $12/computer and $30/server per month (I was trying to give a great deal because I really wanted the best for my client but needed to make -something- for management of the product), was apparently too high for the owner. Plus he complained that the RMM software made their computers too slow.

So I moved to AVG free on all of the computers, and again owner complained it made the computers too slow! Fed up with the complaints, I caved and hoped that the firewall on the Cisco WRVS4400N, latest Windows updates and encouragement of good security practices would suffice. Well it did for about the last 8 months or so. I would remote in and manually do a cleanup on all of the computers, check the health of the server regularly etc all for a minimal cost. We were skating by with no antivirus, but things seemed to be going well. Until just a few days ago. Wham! My phone rings off the hook, text messages, emails and all. "Our folders are gone!! we can't work!!" I knew within minutes of logging in, this office with 6 computers (4 on Win XP, 2 on Win 7), and a domain controller/file server running Server 2008 has been infected with W32.Changeup. http://www.symantec.com/connect/blogs/w32changeup-keeps-giving

I realized that in the interest of being the flexible tech, I have allowed the client to sacrifice his business so as to better accommodate convenience over security. Bad move!

Since I now live out of state, I had a tech I've worked with a lot go on site, after many attempts to resolve remotely had failed. Not only are all the desktops infected, but the network shares hosted by the server are as well.
The office had a half day yesterday, and we have been trying to remediate ever since. To reimage all the computers, deal with licensing, install/config of proprietary applications etc would literally take about 20 hours of work. Presently we have about 6hrs into it and once we get to about 12, I think we will pull the plug. I'm hoping there is a way other than starting over again to resolve this.

What we've tried so far:
Disable autorun on domain controller and in group policy.
For whatever reason this didn't take when I tried through group policy and ran gpupdate /force on all systems. Also on the XP systems, the policy for autorun was not listed in gpedit so I did it manually on each computer per the Microsoft KB article on this:

Click Start, click Run, type regedit in the Open box, and then click OK.
Locate and then click the following entry in the registry:
HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\policies\Explorer\NoDriveTypeAutorun
Right-click NoDriveTypeAutoRun, and then click Modify.
In the Value data box, type 0xFF to disable all types of drives. Or, to selectively disable specific drives, use a different value as described in the "How to selectively disable specific Autorun features" section.
Click OK, and then exit Registry Editor.
Restart the computer.

Installed/updated/ran MBAM, removing many infected files.
Install SEP 12 and update.
Boot every system (server included) into safe mode and run a full scan.
Boot into regular mode and run another full scan after wiping out quarantine.
At this point, all systems appeared clean, even the file shares.
The very next day, virus was back. Everything was infected again. I restored no backups, the only thing I can think of is that it's getting through the firewall, or someone used an infected external medium. I advised them to get all new flash cards for the company cameras until we can scan in a protected environment to make sure they were OK, and ran the same scans again. It appeared we had once again cleaned the network of the virus, but within a few hours it was back again!

According the all the info I can find via google, I've done everything I can do with the exception of finding/blocking the C&C servers that the virus communicates with and finding/blocking the ports that it uses. However, the firewall is configured to only allow what is specified, and deny anything else. I'm stumped! Here's what I'm considering:

-Running Wireshark or some other sniffer to gain insight into traffic patterns, and trying to block at the network level, however my experience with this is slim.

Hoping someone here has a recommendation. Thanks in advance!

 

[Boston Pro Comp Serv LLC]

I can't offer much but I'd isolate a single machine, clean it. Document what you did then move on to the others with confidence. Isolate I mean from a network and the Internet.

 

[othersteve]

A couple preliminary things:

1. I don't care what the customer wants, running without security of some type is suicide!
2. This stuff normally drops due to Java vulnerabilities I believe.

Here's the MMPC write-up on the latest variant:

http://www.microsoft.com/security/portal/threat/encyclopedia/Entry.aspx?Name=Worm:Win32/Vobfus.NE

I just removed this from a client's network the other day. The plan off attack (using an offline boot disk) was:

1. Disconnect from the network.
2. Clear temp files.
3. Check autostarts by hooking into the registry (ERD can do this, or D7 I believe also) and remove any obviously suspicious entries.
4. Clear all executables from the %USERPROFILE% directories (most are obviously-named and identically sized).
5. Check AppData Local/Roaming directories.
6. Check ProgramData directories (for good measure).
7. Reboot and perform final cleanup scan.
8. Check all removable drives.

Scanning from offline (as usual) will not completely kill this thanks to the quickly-changing nature of its variants.

I hope this helps!

 

[LAconsult]

Steve/Boston,
Thanks a lot for the input. You're right Steve. I've done my best actually to weed out customers who don't like to invest in technology because just as I was afraid of learning the hard way, it has become a liability for me. Once I get this fixed, I will kindly propose that either we run technology according to best practices or a waiver be signed stating I am in no way responsible for anything that may happen if the customer chooses not to do so. If neither one of these are possible, it's just not worth the risk. Had this been a more devastating virus or a trojan allowing a malicious cracker access who deleted/sabotaged everything, this could have been very bad, even with working backups. Thanks again.

 

[LAconsult]

W32.Changeup infecting network Neutralized!

After about 15 hours over two days this was finally resolved.
Not in exact order but the same nonetheless.

Disable Autoplay throughout domain.
http://www.symantec.com/connect/art...-autoplay-feature-prevent-virus-spreading-way
http://support.microsoft.com/kb/2328787

Test on all desktops. despite running gpupdate /force, rebooting and logging back in, this did not take on the desktops so I did it manually on each one. Happy this is a small network. Wish we had RMM, could have saved time.

Set OpenDNS settings to their highest.

Disabled java at the firewall since it's not used in this environment anyway (management wants no browsing unless work related so makes my job all the better).

Disable system restore on all computers since virus infected c:\system volume info

XP:
Steps to turn off System Restore

Click Start, right-click My Computer, and then click Properties.
In the System Properties dialog box, click the System Restore tab.
Click to select the Turn off System Restore check box. Or, click to select the Turn off System Restore on all drives check box.
Click OK.
When you receive the following message, click Yes to confirm that you want to turn off System Restore:

7:
http://www.howtogeek.com/howto/windows-vista/disable-system-restore-in-windows-vista/

Delete restore points - changeup infected these with trojans and other nasties.

Get a local tech on-site in case something goes wrong and for general assistance (I'm 100% remote from this location).
Boot every computer and server on the network into safe mode, install trial versions of SEP, update and scan. Run MBAM as well(thankfully a small network which became infected on a Thursday so I had the whole weekend to play).
Scan again in safe mode until little or no viruses are discovered. Have tech normal boot once infection is minimized. Scan again. delete infected backups.

Eat something, then grab a coffee and some of that Chicago cheesy caramel popcorn that Costco sells.

Inform customer of semi-containment. Tell customer this probably could have been avoided. Take note customer said "do whatever it takes to prevent from now on".

Update and scan SEP/MBAM again in normal mode.

At this point things were looking a lot better, but the next day the virus spread again. As it turns out, one of the consultants were using their personal cell phone to upload some photos, we think it spread from here.

Have customer replace all thumb drives and all camera memory sticks. Send a strong recommendation (finally with some owner cooperation) to everyone - never plug their personal cell phones or personal anything into any computer in this building ever again.

More coffee.

Test logins and proprietary applications, printers etc. Demote all users, even the owner, from Power Users to "Users". This company can afford leniency, but my nerves cannot so everyone gets bare bone access from now forward, at least until we have AV and/or RMM installed. Virus infects user with local admin rights- virus gets admin permissions through that user account to roam like crazy throughout the computer.

Here's where a lot of labor came in - Upon testing access to ensure everything truly was back to normal, every login was faced with a temporary profile!!

So on each desktop, I had to do the following:
reboot system and login with local admin account.
backup user data (favorites, desktop, .pst if applicable, desktop)
to c:\backup (mydocs stored on individual server shares so OK here).
regedit over to local machine, software, microsoft, windows nt, current profile, profilelist and delete the user key. delete the old directory.
log out and login as user, creating a fresh profile and permanent fix. run a batch script to re-map common drive shares for proprietary applications, quicbooks etc and test. restore printers. do for each user experiencing a temp profile.

Noticed there were missing files/directories. Dredding a restore of the data due to the chance that not restoring to an early enough version would re-infect, did more research on the virus and thankfully it does not delete, but hides files/directories in the worst way. Browse to root of database hosting files shares and use

attrib -h -r -s /s /d *.*

Sip coffee impatiently and wait. This is a big share and will take a while.
Presto! Files are back and luckily permissions are manageable.

I was not particularly impressed with the latest reviews on SEP and the statements on their forums as well as around the net stating the virus/malware got past it, so I'm looking into alternatives.

Things are pretty much back to normal. I should note that during the turmoil the customer asked me to remote in and check things out on his system, and upon remote login client was looking at another local consulting company. I brushed it off, but am firing off an email with my work log (as well as my invoice) letting them know of this story:

I took my car into the shop and paid for a mechanic to do an oil change and requested a look over as I was experiencing some weird sounds/performance issues. The mechanic gave me a list of recommendations, including the 120,000 mile service and advising I change out the radiator. I did the radiator myself, but since receiving this news I hadn't had time for the 120k service or money for it. I get a second opinion who concurred. If I flat out refuse and never consider getting this work done because I don't have the time, don't want to spend the money (let's pretend I'm cheap), and simply don't think anything bad will happen to me by not doing the work, and the car breaks down leaving me on the side of the road, do I then look for a new mechanic? Of course not! I realize I have done things according to my schedule and my desire to not make it a priority and therefore put myself at risk for inconvenience and loss of money due to not being able to go to work etc. I would estimate this business, in addition to paying me for 12 hours (discounted from 15) of hard labor has lost not only man hours, requiring potential overtime and lots of stress on everyone, but maybe even 1 potential customer, which would amount to at least $5k.

I've learned from this. From hereon, if a client refuses antivirus, refuses a decent firewall and refuses a decent backup solution they will sign a waiver which describes the worst case scenario and releases me of all liability. Had this customer purchased the BDR solution I had proposed (approx $1200 hardware + approx $200/mo for off-site data), kept the RMM solution we had which was working great but according to him slowing down the computers a bit at approx. $100 per month for the whole site, be it Kaseya or even GFI, the labor would have taken maybe 3 hours for this whole ordeal- including testing. Unfortunately even with full Windows backups locally, it would have taken even more time to reimage/redeploy every computer and the server, reconfigure etc. just to ensure the environment was not infected.

I should also note that we were once trying a thinclient solution, which also worked quite well for what the users do on a daily basis, but because of the extra cost for RDP licenses (approx $450) this didn't work out. I think it could have helped in this situation, but with virus coming from autoplay who knows. Hindsight really is 20/20 I guess.