It's my fault?

[Your PCMD]

A big company here that I MSP for and have built right at 40 computers for just over a year ago gives me a call yesterday. Their smaller branch (under different name) has an issue with the server. So I told them, no problem, I'll be there tomorrow as I was solidly booked today (this was yesterday btw).

Now then, the server is an HP ProLiant ML110 G5. It's running Windows Server 2008 RC2 and the software they use only runs and is running on Windows XP inside VirtualBox. It's ALL janky.

Now then, just over a year ago I was there on an unrelated issue and was asked to look at the server to see if it needed cleaning. It did, I did it. At that time I noticed their backup drive was an old WD external 80Gb hard drive and recommended that they get rid of that and use a proper 2-bay NAS for backup and redundancy. I also recommended that they put the server on a UPS. All to which they said "not right now, we're not too worried about it".

Now then, back to today. It seems that in the middle of a backup session on Tuesday, the power went out. When it came back on, and they got back into the server, the program they use to run that part of the company inside VB was completely gone.

So I go in to see if I can find it, nope, its not there. I said no worries, we can restore from the previous days backup. I go to click on the backup drive, no response. Windows sees it, but it goes into unresponsive mode.
I'm thinking, hmmmm. Disconnect it from the USB, reconnect it, Windows picks it up but same errors. Except this time I hear the HDD clicking and grinding. Sure enough, drive is dead. And there is no telling how long it has been dead.

When I give the branch manager the bad news, it was "how did it happen, didnt you set stuff up a year ago like you were 'paid' to do?" And this in a berated tone. I snapped back because I aint putting up with no chit from a client - especially when I am right. I told him that no, if you remember a year ago you told me NOT TO DO ANYTHING WITH IT. Oh yeah, thats right I forgot is what I got next.

Now its costing them big bucks to get a NAS, much needed upgrades to the server, a UPS AND for the guy that wrote the program for their company to fly from Connecticut to rebuild the program.

But yeah, it was my fault.

 

[Archon Prime]

Good times. Idiots.

 

[cypress]

That's all it took for them to start listening to you again.

 

[Slaters Kustum Machines]

PITA, but sounds like this might turn out well for you.

 

[coffee]

Typical business approach to a problem. Blame the current tech. This time it didnt work. Its all their fault and I hope you were there to sell them all the improvements this time. Tack on a bit for the belittling chatter from the contact.

I tell you, I went in to a company and they were very proud of the IBM (Yuck) server they bought and the owner installed himself. They let me look it over and I pointed out a message about their current backup log ----> "Verify Failed". They told me to not worry about it and its nothing.

These places set themselves up for failure all the time. Thats where we earn the big bucks because not their nuts are in a vice and they have no choice

Did you get to sell them the stuff?

 

[Your PCMD]

Did you get to sell them the stuff?

Yes I did. About $2.4k worth and thats not including labor either.

 

[glennd]

"not right now, we're not too worried about it".

We're not too worried about losing everything...

 

[Slaters Kustum Machines]

We're not too worried about losing everything...

They never are, until it's gone. Then if and when you save the day they forget the panic they were in and want to take their sweet time to pay......well that's my residential experience anyway.

 

[glennd]

They never are, until it's gone. Then if and when you save the day they forget the panic they were in and want to take their sweet time to pay......well that's my residential experience anyway.

A business customer asked me only this week, "What should I be backing up?". I said, "Only what you're not prepared to lose." And proceeded into the standard rant about *when* the computer dies and all the data is gone, what then? That got them thinking...

 

[Moltuae]

When I give the branch manager the bad news, it was "how did it happen, didnt you set stuff up a year ago like you were 'paid' to do?" And this in a berated tone. I snapped back because I aint putting up with no chit from a client - especially when I am right. I told him that no, if you remember a year ago you told me NOT TO DO ANYTHING WITH IT. Oh yeah, thats right I forgot is what I got next.

I've had customers just like this in the past.

The best way I've found to deal with it is to put it in writing, repeatedly. No matter how many times you mention it verbally, they'll 'forget' you said it when it comes to the crunch.

When I take on a new customer, I audit/evaluate their systems then provide them with documentation that highlights any issues, graded in levels of severity. Stuff relating to backups, UPS, firewalls, AV, etc get a prominent red warning at the very beginning of the document, with lower graded stuff like 'recommendations' (such as greater NAS capacity for more granular/frequent backups) towards the end of the documentation. I present the documentation to them in person and follow up with email warnings about anything critical. If they ignore my initial warnings, a red warning note is added to every invoice in a suggestions/notes area (eg: Level CRITICAL: No server backups are in-place. Component failure, damage or theft could lead to the loss of ALL data). I also reiterate any outstanding critical issues via email (and verbally) at every appropriate opportunity.

 

[HCHTech]

a red warning note is added to every invoice in a suggestions/notes area (eg: Level CRITICAL: No server backups are in-place. Component failure, damage or theft could lead to the loss of ALL data). I also reiterate any outstanding critical issues via email (and verbally) at every appropriate opportunity.

Ooh - well done. I like this idea and may or may not be stealing it. Now I'm wondering how I might do this systems-wise. I could have a zero-cost inventory or service item in Quickbooks, I suppose, but that wouldn't make it stand out, maybe a big red checkmark stamp...

 

[Larry Sabo]

Use the Customer Message area at the bottom of the invoice, and create a generic one you can just select when appropriate?

 

[GTP]

I've had customers just like this in the past.

The best way I've found to deal with it is to put it in writing, repeatedly. No matter how many times you mention it verbally, they'll 'forget' you said it when it comes to the crunch.

When I take on a new customer, I audit/evaluate their systems then provide them with documentation that highlights any issues, graded in levels of severity. Stuff relating to backups, UPS, firewalls, AV, etc get a prominent red warning at the very beginning of the document, with lower graded stuff like 'recommendations' (such as greater NAS capacity for more granular/frequent backups) towards the end of the documentation. I present the documentation to them in person and follow up with email warnings about anything critical. If they ignore my initial warnings, a red warning note is added to every invoice in a suggestions/notes area (eg: Level CRITICAL: No server backups are in-place. Component failure, damage or theft could lead to the loss of ALL data). I also reiterate any outstanding critical issues via email (and verbally) at every appropriate opportunity.

I like this idea. I am so stealing it!

 

[Your PCMD]

I like this idea and may or may not be stealing it.

I like this idea. I am so stealing it!

Unlike these guys here, I won't steal it. But I will borrow it, for however long as I see fit.

 

[brandonkick]

Hopefully you didn't forget to factor in what I like to call the @!#hole tax. I would have.

 

[Altster]

But yeah, it was my fault.

"Shite Happens!"

 

[Tech bud]

My Biggest fear, why I stick with residential stuff, so much more stuff they can blame you for

 

[ComputerRepairTech]

Now then, back to today. It seems that in the middle of a backup session on Tuesday, the power went out. When it came back on, and they got back into the server, the program they use to run that part of the company inside VB was completely gone.

The VB still ran but just that one program disappeared? Thats super weird right? I don't use VB at all so I don't know what kind of oddities it may have.

 

[fencepost]

A single file disappearing particularly during a backup sounds like an antivirus on-access false (or not false?) positive to me.