How to update an old production server

[uktech]

Have another really dodgy server update to perform soon and I would love to know how everyone else would tackle this:

~ Windows Server (I think it is 2012 but yet to be confirmed)
~ It's the only one in the business and it's 'live' - so fix out of hours only - but must work the following day
~ Hasn't had any Windows updates since 2016 !
~ There isn't any backup.

Great job !

Should I increase my Hiscox insurance and crack on with it or walk away?

Your insights would be much appreciated.

 

[YeOldeStonecat]

Get a good full image backup on it and get catching up on updates.
How many users on this network? What is it running/hosting?

 

[Mick]

well - the first thing to do is fix the 'no back-up' issue. A copy of WSUS offline might come in handy too.

 

[uktech]

Thanks for the reply's - it's only a small network of 5 or so users - running dental software which is business critical.

In the past I would have run up a second server, tested everything working, and then re-used the old one elsewhere.

This customer isn't going to pay for that though - and my worry is, if the server crashes in a ball of flame and wont update and won't recover from an image.....i'm then into purchasing / configuring a new server - which isn't going to happen overnight. Joy !

Any preferences for server imaging? (My usual go to is Macrium Reflect.)

 

[YeOldeStonecat]

- running dental software which is business critical...i'm then into purchasing / configuring a new server - which isn't going to happen overnight. Joy !

Any preferences for server imaging? (My usual go to is Macrium Reflect.)

A health care place, "business critical"...and...*cough*....no backup in place?
I would not be using FREE products, I'd get this client on a business continuity product like Datto tout-de-suite!

 

[uktech]

I was referring to Macrium Server (not free), though I know what you mean about Datto - just pretty sure they won't pay the cost.

I've no experience of the Datto offering, but a quick look at the pricing suggests I'll never get them to purchase anything appliance based - not sure if you can get Datto without an appliance?

It's been the pain of running my business - I do more data recovery than backup i sales - been trying to move heaven and earth to get customers to buy into proper backups - they just don't see the value, until they come to me for data recovery !

:-(

 

[YeOldeStonecat]

Yeah Datto is not cheap, but it's really "hands off" for the reseller, their support does everything for you. As the reseller we really spend very little time "supporting" the product. Which is key.
It does a "test bootup" of the prior nights backup each morning, e-mails you a screen shot of the ctrl+a;t+del login each day so you have confidence that the backup is working properly, and more importantly, that the servers backup image is functional and doing what it needs to do. My clients love those emails, a nice "warm 'n fuzzy". And I get one too!

There are certainly other similar products out there...most of which require a bit more "work/support" from the reseller. But I'll point out you really need to take a stance with a potential client and educate them on the whole "recovery process".
Face to face...ask them how many days they mind being without a server. Most people think any old backup will mean they will be back up and running in like an hour or two. They don't really (because they're not educated/consulted on the process) aware of how long they might be down as a business if disaster strikes! Below are some of the key points I make in my pitch:

*Say your server catches fire, melts down, floats away in a flood, big power surge comes in, whatever..
*How long will it take to get a new/replacement server? Let's see...I need to spend time spec'ing one...and getting an estimate.
*I then need to give you a bill, and receive your money, so I can go and order it.
*I will have to rush this as an overnight delivery. Will be expensive because overnighting big server boxes...pricey
*May not have the ideal parts in stock to overnight, so we might have to lower our standards and gets a more basic/cheaper server than we really wanted...because that's all that can be configured and shipped by next day.
*Receive the server, build it, skip the usual burn in
*Start to put the pieces of your dead servers data back in place,

Above steps can be a while. Often you can't get a server "the next day"...could be a day or three...

Let that sink in with your client. And give it more time to sink in.

Versus...having a "business continuity product" (like Datto, RepliBit, etc)...my client server could catch fire and burn to a smoldering pile of soot...and I won't break a sweat, In 15 minutes or less I can remote in and "boot up" the last backup from an hour ago..and make it available on the network...and the client can proceed with business day to day. And then I can take my time ordering the proper server hardware, receive it, and plan a time to "restore" the image to it.

 

[fencepost]

This is a situation of "You go on a backup plan, you pay me more than the cost of the backup plan to do a manual babysat backup, or you go elsewhere."

If you're talking about backing up a live SQL Server with Macrium Reflect Server Plus, they're already talking about a big chunk of change plus your time setting up for the backup, and if they use those workstations on the Internet they're vulnerable to both ransomware that will cost them thousands of dollars more for dodgy recovery options or to ransomware that's completely impossible to recover from because it encrypts but throws away the key. Is the dentist prepared to close the practice with no notice if something goes wrong?

 

[marley1]

It's just Windows updates.

Get a full backup.

Solarwinds Backup is great.
You also have Windows backup for free built in.
Then start running updates.

Does server have iDrac or similar?

 

[nlinecomputers]

Walk away. Cheap clients are bad clients.

 

[Mick]

Second all the above. To really hammer it home, ask them how much it will cost them to have a) no income and b) five users playing Candy Crush all day on their phones on full pay. And how many customers/patients are likely to just wander off to the dentist down the street in future?

 

[kwest]

Test drives first and make sure they are healthy.

Get a two backups.

Update with wsus

If the customer doesn’t want to pay for it to be properly done then walk. Or have him sign a waiver that if it breaks it will cost this.

 

[lan101]

Yeah Datto is not cheap, but it's really "hands off" for the reseller, their support does everything for you. As the reseller we really spend very little time "supporting" the product. Which is key.
It does a "test bootup" of the prior nights backup each morning, e-mails you a screen shot of the ctrl+a;t+del login each day so you have confidence that the backup is working properly, and more importantly, that the servers backup image is functional and doing what it needs to do. My clients love those emails, a nice "warm 'n fuzzy". And I get one too!

There are certainly other similar products out there...most of which require a bit more "work/support" from the reseller. But I'll point out you really need to take a stance with a potential client and educate them on the whole "recovery process".
Face to face...ask them how many days they mind being without a server. Most people think any old backup will mean they will be back up and running in like an hour or two. They don't really (because they're not educated/consulted on the process) aware of how long they might be down as a business if disaster strikes! Below are some of the key points I make in my pitch:

*Say your server catches fire, melts down, floats away in a flood, big power surge comes in, whatever..
*How long will it take to get a new/replacement server? Let's see...I need to spend time spec'ing one...and getting an estimate.
*I then need to give you a bill, and receive your money, so I can go and order it.
*I will have to rush this as an overnight delivery. Will be expensive because overnighting big server boxes...pricey
*May not have the ideal parts in stock to overnight, so we might have to lower our standards and gets a more basic/cheaper server than we really wanted...because that's all that can be configured and shipped by next day.
*Receive the server, build it, skip the usual burn in
*Start to put the pieces of your dead servers data back in place,

Above steps can be a while. Often you can't get a server "the next day"...could be a day or three...

Let that sink in with your client. And give it more time to sink in.

Versus...having a "business continuity product" (like Datto, RepliBit, etc)...my client server could catch fire and burn to a smoldering pile of soot...and I won't break a sweat, In 15 minutes or less I can remote in and "boot up" the last backup from an hour ago..and make it available on the network...and the client can proceed with business day to day. And then I can take my time ordering the proper server hardware, receive it, and plan a time to "restore" the image to it.

Forgive my server knowledge ignorance lol. I only do basic stuff with servers. I am just wondering if they only have 1 physical server and it literally burns to the ground or gets flooded out of the office, where do you restore the latest backup to get them back up and running?

 

[fencepost]

Part of what you're paying with on Datto is generally a "hybrid" backup - both to a local device and to "cloud" storage. That cloud backup gets booted up to a Ctrl-Alt-Del screen after each backup, which is the screenshot that @YeOldeStonecat was talking about.

As I understand it, if there's a hardware failure on the server, the most current "cloud" version can be spun up as a VM with a VPN connection back to that local Datto device so it appears to be on the local network. When the local server is restored, that cloud backup image that became the live server can be restored down to the local server. If the entire office no longer exists this is probably more complicated, but I'm sure it's not a new situation either. Those cloud VMs may not be able to handle the full load of the office depending on how they're being used and local Internet connectivity, but they're likely going to be plenty to at least get 1-2 users up so patients can be called, etc.

You can build a lot of the same capabilities with other backup software (Shadowprotect, Cloudberrylab, probably Macrium Server, etc.) or services (Solarwinds BDR, others?) but part of what you're paying for with Datto is the convenience of not having to do the setup yourself or do much of anything manual to do the image restore/boot, etc. and having experienced folks (at Datto's support) doing that for you.

 

[nerd2u]

I would be questioning why it has not gotten an update since 2016? Did they apply one and it didn't play nice with their software and now they want someone else to do it and take the headache?

I would be asking some questions before jumping in.

Sent from my SM-G870W using Tapatalk

 

[nlinecomputers]

I would be questioning why it has not gotten an update since 2016? Did they apply one and it didn't play nice with their software and now they want someone else to do it and take the headache?

I would be asking some questions before jumping in.

Sent from my SM-G870W using Tapatalk

Probably something just broke windows update....

 

[YeOldeStonecat]

Forgive my server knowledge ignorance lol. I only do basic stuff with servers. I am just wondering if they only have 1 physical server and it literally burns to the ground or gets flooded out of the office, where do you restore the latest backup to get them back up and running?

Fence started answering...
So, in a real quick birds eye view, Datto offers 2x different product lines. Both keep data locally, and offsite in data centers in geographically redundant data centers around the globe.
*Alto...the budget service, a small appliance that sit onsite, basically it runs a very fancy version of StorageCraft ImageManger...which syncs data both locally on the appliance, and offsite. If the clients server burns to the ground, you can "mount and spin up" the most recent backup in a cloud instance from a data center...and provide multiple ways to connect to it, such as RDP, or VPN. If you choose VPN, the server that is running in the cloud data center of course becomes available to the workstations at the clients network...through the VPN connection. The Alto appliance acts as a VPN proxy to this for the network. Running slow...yes...but...running. Decent affordable for small businesses that have 1 maybe 2 servers.
*Siris...the bigger devices. Larger 1U or larger appliances. Lots of CPU and RAM. In addition to the features of the Alto mentioned above, the Siris appliances can also mount and boot up (virtualize) the servers most recent backup locally, on themselves. So, performance is good. For a client with 6 or 8 or more servers, you just get 1x huge Siris appliance to cover them all.

Dattos can mount a servers backup, make it available as a mounted drive from across the network, make file/folder restore easy and quick. They can also take a backup and export as a virtual hard drive file...so you can quickly and easily restore to a hypervisor. They also allow direct restore to new bare metal.

 

[YeOldeStonecat]

I would be questioning why it has not gotten an update since 2016? Did they apply one and it didn't play nice with their software and now they want someone else to do it and take the headache?

Depending on what software tech stack they run, probably got sick of Windows updates breaking their older practice management software and/or credit card processing services like digital highway. I know some places that really hate allowing Microsoft updates on servers because it breaks things with their LOB apps so frequently. Pretty soon some people get tired of sticking their hands in the fire.

 

[HCHTech]

I know some places that really hate allowing Microsoft updates on servers because it breaks things with their LOB apps so frequently. Pretty soon some people get tired of sticking their hands in the fire.

Oh boy, THIS. I have clients that have pleaded with me to stop updating their servers because of this. We have a 6-month hold on most updates at the worst offenders...

 

[lan101]

Fence started answering...
So, in a real quick birds eye view, Datto offers 2x different product lines. Both keep data locally, and offsite in data centers in geographically redundant data centers around the globe.
*Alto...the budget service, a small appliance that sit onsite, basically it runs a very fancy version of StorageCraft ImageManger...which syncs data both locally on the appliance, and offsite. If the clients server burns to the ground, you can "mount and spin up" the most recent backup in a cloud instance from a data center...and provide multiple ways to connect to it, such as RDP, or VPN. If you choose VPN, the server that is running in the cloud data center of course becomes available to the workstations at the clients network...through the VPN connection. The Alto appliance acts as a VPN proxy to this for the network. Running slow...yes...but...running. Decent affordable for small businesses that have 1 maybe 2 servers.
*Siris...the bigger devices. Larger 1U or larger appliances. Lots of CPU and RAM. In addition to the features of the Alto mentioned above, the Siris appliances can also mount and boot up (virtualize) the servers most recent backup locally, on themselves. So, performance is good. For a client with 6 or 8 or more servers, you just get 1x huge Siris appliance to cover them all.

Dattos can mount a servers backup, make it available as a mounted drive from across the network, make file/folder restore easy and quick. They can also take a backup and export as a virtual hard drive file...so you can quickly and easily restore to a hypervisor. They also allow direct restore to new bare metal.

Awesome thank you for explaining that. Datto services certainly seem worth it for any business that relies on the availability of their server, but trying to get the client to see that and spend the money is probably another thing for a lot of them lol. But like you said if you get a good sales pitch down about why they need it, it probably sells itself.