A directory which should have simply contained “Product (build).exe” had both that file and one named: “Product (buildE.exe”. “Product (buildE.exe” still had it’s digital signature, where as the one that was properly named didn’t.

Hrm, let’s take a look. Scan the files with Trend Micro’s Office Scan, nothing, scan files with AVG, nothing. Jump on Trend’s website, using house call, still nothing.

Do a quick Google search for Virus software ratings which point to “BitDefender” being the best. Run that and *blam* we have an un-identified Win32/File Injector.

Nice….

These files so happened to be on the same server as all of our software builds, legacy and what not.

We’ve been a big believer in Trend. Office Scan is rolled out in the organization. We have Trend on our Exchange servers and McAfee on our Mail gateway.

Of course none of this protects us when certain QA machines and other machines which change their OS more often than I change my underwear go unprotected. They are supposed to be used in our LAB, our lab which allows these machines internet access and shouldn’t.

We blow images of all shapes and sizes on lots of hardware. It’s meant to be used for testing not surfing. That’s our best guess as to how we contracted this virus but it’s not conclusive at this point.

To make matters worse, a whole LOT worse, the server that holds all of our final builds and legacy builds shares two other services. Our source code change management product and our defect tracking database. A year ago this was fine, but we’ve outgrown a single server for all of these purposes and as Murphy would have it, the replacement servers are on order to arrive early next week.

< insert screaming and banging head on desk here >

We desperately look for a product that will locate and eradicate this virus. As I said previously BitDefender was so far the only thing we could find. However running their web-based *Freebie* was too slow and too limited. We could only delete the infections. Not quarantine them. Did I mention how slow it was? I tried like hell to download a real live trial edition, BitDefenders site wouldn’t give it up. It would take my info but not send me the email I needed to download. Frustrations grew.

Looking at that list of what’s supposed to be good we go for Kaspersky. I download the file server edition for small business and queue it up to be installed. It want’s to reboot, but I can’t reboot that server right now. It’s only 5pm and there’s still work to be done.

Having settled on the fact that we don’t know where this originated but assume we got the most likely targets with the web scan we call it a day. We make sure we have the absolute latest version of Trend and kick off a scan on that server.

Two hours later we’re getting emails about our precious server not allowing files to be copied to/from it. Our change management software is tossing up errors too. A developer suggest we run Check Disk on that volume based upon the error we’re seeing.

This has the potential to be really bad. (We do have backups of this box, we back it up every day, but the question is, are those backups infected?)

Throwing caution into the wind, I shutdown the services, and proceed with the Check Disk. 2 and a half hours later it’s done. No major errors to report. Hrm….

Well, perhaps it and Trend just don’t get along. We need to reboot this box anyway and the services are stopped so let’s go for it. Of course I’m working remotely and cross my fingers, say a prayer and do a sacrificial dance to the IT Gods that it will come back up. 10 minutes later it appears to be so.

I get on remotely, start to poke around with Kaspersky (which was installed previously). Then the server just freezes. Locks up, cold.

F#$%

I contact my right hand man and we’re both enroute. 20 minutes later we’re discussing our options. Are we going to build a new box? Repurpose a box? That’s so wasteful, as new machines are on order. But we have to get this up. We’re able to get a management console hooked up and kill off the Kaspersky service. The server springs back to life, sort of…

We deinstall it. It’s much happier now.

We’re finally able to procure a copy of BitDefender, and get that installed. It’s scanning, but seems like it’s deleting a lot of stuff, and it’s not much faster than the web version.

Sh*t

We go looking again and decide to try NOD32. This software works the best thus far. It too doesn’t really know what it is but it isn’t messing around.

We start doing some investigating with our little infection off the network with a loaner laptop. We can’t figure out what this thing does other than sometimes ruin your original .exe and propagate itself.

The potential for mas wreckage though is pretty high, considering the infected product installers. Not externally, but internally.

We immediately start scanning the main server, and all points of contact with it. From those that reported a problem to those that may have had a casual opportunity to brush up against it.

It’s now about 4:30am, this scanning business is still too slow, but it is progressing. People will be here in two hours and things need to be as back to normal as possible.

Of course most of the organization has Trend installed, and it and NOD32 don’t mix too well.

We start deinstalling Trend and scanning what we can. At 6 am we break for breakfast.

People start showing up and things get even more interesting. We put out an email explaining our situation, asking for patience and giving guidance on what to do and what not to do.

For the most part our users were very helpful. I’d say from the user level, their cooperation was at an all time high and that’s a good thing. All in all, I’d say they exceeded my expectations by quite a bit. Of course we’re very, very tired having stayed up all night battling this so maybe we’re just numb. Nah, they did good.

We still have a few hurdles in front of us. Products that we paid good money for that let us down. Servers that are still acting a little unhappy about the fact that they need this extra resource hogging process on them. But we’ll have most of that taken care of by the middle of next week when the new hardware arrives.

I’ll report back if things turn for the worst.

I was lucky enough to be at a Krogers last night and experience the chaos that ensued when the scanners didn’t work.

Of course I only needed to pick up three things. Easy in and out for some cold medicine, apple juice and bread. That turned into a 30 minute debacle.

The Self checkouts were closed, and apparently there are only 2 cashiers capable of doing math and making change. (On a Friday night at 8p.m.)

Of the few things I purchased 2 rung up wrong and one wasn’t able to ring. Nice…

So the story above explains what happened. Apparently when they updated their recall info so they wouldn’t sell dog food that is tainted, the system got all fubar’d.

They couldn’t cope and had to shut down the stores for many hours. I wondered why at 8p.m. m 24 hour store was shutting down the lights.

It would have been easier I would imagine to just remove the stuff you can’t/shouldn’t sell from the customers reach. That’s how we did it in the good old days.

This is one downside of too much reliance on technology. Some things are just not all that complicated and aren’t technology problems to solve.

I’m glad I’m not the product/project manager for this. A whole bunch of stores closed. How much revenue was lost?

powered by performancing firefox

$300 buys you, a slim Mac Mini. You provide your own cables to hook it to your TV.

That TV must be a wide-screen tv (HiDef). You then download content via another real computer somewhere via iTunes, then push that to the AppleTV.

(OK so that part should be easy).

It requires HiDef, but you don’t get to watch anything in high-def, as what iTunes delivers isn’t. Yeah, that makes sense. I get it, it looks good sitting in your entertainment center, has pretty menus and all but the actual video looks like crap.

Where do I sign up?

Most things work, and have the right time. That is unless you have a 797x color phone. You know one of the ‘good’ phones, the expensive ones.

If you have one of these, check the time. Is it right?

It probably is because when you did your update you rebooted all of the phones right?

Now reach around back and unplug it (if you are using this powered over Ethernet simply unplug the cable, if you have a power brick, just unplug that).

Let the phone boot up.

What time does your phone say now? Yeah, that’s what I thought, it’s back to the wrong time.

Now simply reset your phone, ‘hit settings, **#**’ and viola!, you’re back to the right time.

So what’s the difference from a ‘powered-off boot’ and a reset/reboot? And why does that screw up the time? Who knows, but it’s stuff like this that drive us absolutely crazy.

(This works so well because as we know, end users just love running tools to fix IT problems that really should be addressed at the server level, and of course they fully understand what has to happen and why things are broken. Yes, delegating IT problems down to the user level is always the best move).

The best they could do from an Exchange administrator perspective is this:

- Create a client fix tool.

- Create absolutely the most convoluted way to take that ‘client’ based tool and script it to run on one machine, mailbox after mailbox. (That is in fact what they are doing).

- Exchange security issues as side, this should be fixable at the server level, without pretending to be a MAPI client for crying out loud.

Remember when Microsoft released the Windows NT domain, and promised us Single point of administration? This is not what I had in mind, if I wanted to script client tools to do things in bulk across my organization, we’d still be running DOS.

“Note Do not confuse the Outlook tool installer package that is named Tzmove.exe with the actual Outlook tool executable file that is also named Tzmove.exe”

When you actually run these tools they can spew error code: 0×80004005

Which apparently can mean about 18 different things, none of which are related. I haven’t written error handling code in some time but even I, as lazy as I am, was more descriptive than that.

We did finally get the tools to run without error, but nothing has changed. We still have dicked up appointments.

I especially like this:

Recurring meetings that are created in Outlook Web Access are not updated by the Exchange Tool

Beh, who uses Outlook Web Access anyway?

Sheesh.

It has treated us well. Fairly easy to configure and pretty dang accurate. For the cost associated with it, we’ve been more than happy.

Leave it to SonicWall to screw up a good thing. When it was announced that SonicWall was purchasing MailFrontier, I was none too happy.

Fast forward to now. We’ve seen multiple updates to the product, most of them haven’t had anything great to add. More and more spam gets through and the overall performance of the product has gone in the shitter.

Note: we do not have the appliance. We have the software running on a Windows 2k3 server that greatly exceeds their minimum requirements.

I’ve been digging for any tid bit of information or tip that will help get performance back where it was, and have so far come up with nada.

The SonicWall website only has this little tidbit in regards to performance:

High Performance

SonicWALL ensures peak system performance through its unique preemptive scanning MTA, which offers breakthrough message analysis that is 40-290% faster than competitor solutions.

We have to call BS here. This product cannot be 40-290% faster than the competition. Unless the competition simply doesn’t work at all.

The poohbah’s are not happy.