It all started around 2pm on the 29th. One of our developers pointed out that one of our installers didn’t look ‘quite’ right.

A directory which should have simply contained “Product (build).exe” had both that file and one named: “Product (buildE.exe”. “Product (buildE.exe” still had it’s digital signature, where as the one that was properly named didn’t.

Hrm, let’s take a look. Scan the files with Trend Micro’s Office Scan, nothing, scan files with AVG, nothing. Jump on Trend’s website, using house call, still nothing.

Do a quick Google search for Virus software ratings which point to “BitDefender” being the best. Run that and *blam* we have an un-identified Win32/File Injector.

Nice….

These files so happened to be on the same server as all of our software builds, legacy and what not.

We’ve been a big believer in Trend. Office Scan is rolled out in the organization. We have Trend on our Exchange servers and McAfee on our Mail gateway.

Of course none of this protects us when certain QA machines and other machines which change their OS more often than I change my underwear go unprotected. They are supposed to be used in our LAB, our lab which allows these machines internet access and shouldn’t.

We blow images of all shapes and sizes on lots of hardware. It’s meant to be used for testing not surfing. That’s our best guess as to how we contracted this virus but it’s not conclusive at this point.

To make matters worse, a whole LOT worse, the server that holds all of our final builds and legacy builds shares two other services. Our source code change management product and our defect tracking database. A year ago this was fine, but we’ve outgrown a single server for all of these purposes and as Murphy would have it, the replacement servers are on order to arrive early next week.

< insert screaming and banging head on desk here >

We desperately look for a product that will locate and eradicate this virus. As I said previously BitDefender was so far the only thing we could find. However running their web-based *Freebie* was too slow and too limited. We could only delete the infections. Not quarantine them. Did I mention how slow it was? I tried like hell to download a real live trial edition, BitDefenders site wouldn’t give it up. It would take my info but not send me the email I needed to download. Frustrations grew.

Looking at that list of what’s supposed to be good we go for Kaspersky. I download the file server edition for small business and queue it up to be installed. It want’s to reboot, but I can’t reboot that server right now. It’s only 5pm and there’s still work to be done.

Having settled on the fact that we don’t know where this originated but assume we got the most likely targets with the web scan we call it a day. We make sure we have the absolute latest version of Trend and kick off a scan on that server.

Two hours later we’re getting emails about our precious server not allowing files to be copied to/from it. Our change management software is tossing up errors too. A developer suggest we run Check Disk on that volume based upon the error we’re seeing.

This has the potential to be really bad. (We do have backups of this box, we back it up every day, but the question is, are those backups infected?)

Throwing caution into the wind, I shutdown the services, and proceed with the Check Disk. 2 and a half hours later it’s done. No major errors to report. Hrm….

Well, perhaps it and Trend just don’t get along. We need to reboot this box anyway and the services are stopped so let’s go for it. Of course I’m working remotely and cross my fingers, say a prayer and do a sacrificial dance to the IT Gods that it will come back up. 10 minutes later it appears to be so.

I get on remotely, start to poke around with Kaspersky (which was installed previously). Then the server just freezes. Locks up, cold.

F#$%

I contact my right hand man and we’re both enroute. 20 minutes later we’re discussing our options. Are we going to build a new box? Repurpose a box? That’s so wasteful, as new machines are on order. But we have to get this up. We’re able to get a management console hooked up and kill off the Kaspersky service. The server springs back to life, sort of…

We deinstall it. It’s much happier now.

We’re finally able to procure a copy of BitDefender, and get that installed. It’s scanning, but seems like it’s deleting a lot of stuff, and it’s not much faster than the web version.

Sh*t

We go looking again and decide to try NOD32. This software works the best thus far. It too doesn’t really know what it is but it isn’t messing around.

We start doing some investigating with our little infection off the network with a loaner laptop. We can’t figure out what this thing does other than sometimes ruin your original .exe and propagate itself.

The potential for mas wreckage though is pretty high, considering the infected product installers. Not externally, but internally.

We immediately start scanning the main server, and all points of contact with it. From those that reported a problem to those that may have had a casual opportunity to brush up against it.

It’s now about 4:30am, this scanning business is still too slow, but it is progressing. People will be here in two hours and things need to be as back to normal as possible.

Of course most of the organization has Trend installed, and it and NOD32 don’t mix too well.

We start deinstalling Trend and scanning what we can. At 6 am we break for breakfast.

People start showing up and things get even more interesting. We put out an email explaining our situation, asking for patience and giving guidance on what to do and what not to do.

For the most part our users were very helpful. I’d say from the user level, their cooperation was at an all time high and that’s a good thing. All in all, I’d say they exceeded my expectations by quite a bit. Of course we’re very, very tired having stayed up all night battling this so maybe we’re just numb. Nah, they did good.

We still have a few hurdles in front of us. Products that we paid good money for that let us down. Servers that are still acting a little unhappy about the fact that they need this extra resource hogging process on them. But we’ll have most of that taken care of by the middle of next week when the new hardware arrives.

I’ll report back if things turn for the worst.