Testing Backups

[Velvis]

A client of mine is filling out a cyber insurance questionnaire and one of the questions is if testing of successful restoration of data from backups is routinely performed. I was wondering how to test restoration of a cloud service (in this case Google Workspace) from a backup (in this case an offsite Synology NAS).

I know that Synology has an integrity test but the insurance form specifically asks if testing of successful restoration of data is performed.

Is there a way of doing something like that short of deleting the users data on the computer and then attempting a recovery?

 

[britechguy]

You could also advise an honest answer, "No."

Any given backup could fail to restore. I've had exactly that happen. Having several backups of progressively older ages or multiple backups is the best workaround.

 

[Markverhyden]

I know I'm stating the obvious but emphasize to them the importance of being truthful. Insurance companies aren't here to help us. They're here to provide a service while minimizing variable expenses. Like paying claims. You can guarantee that the underwriter will audit the claim. Part of that audit will be finding proof that the backups were actually tested. No proof, claim denied.

That question has been around for ages is really is focused on OS plus LoB/database type apps. But it does matter for discrete files as well. A single spreadsheet or report could easily represent if lost or corrupted.

Is this a backup in the sense of grabbing files, compressing them all into a big tar ball? Or is it just grabbing all the files every now and then, maintaining original file/folder structure? How many and what size we are talking about? Are they looking for bare metal restores?

*nix's have a command called diff which will report the differences between files/folders. I'm sure there's something similar in PowerShell.

 

[britechguy]

*nix's have a command called diff which will report the differences between files/folders. I'm sure there's something similar in PowerShell.

There is, but it's probably easier to install one of the many packages that give you access to a number of common *nix commands directly in either Command Prompt or PowerShell.

Even though this stackoverflow topic is old (or at least the start of it is) it covers the issue pretty well and reasonably briefly: https://stackoverflow.com/questions/30889161/native-powershell-equivalent-of-unix-diff-and-patch

 

[britechguy]

Is this a backup in the sense of grabbing files, compressing them all into a big tar ball? Or is it just grabbing all the files every now and then, maintaining original file/folder structure? How many and what size we are talking about? Are they looking for bare metal restores?

An excellent question, as "backup" is an incredibly broad catch all.

And as you say, the idea is, ultimately, to catch you rather than make your life easier. The phrase, "Insurance is a racket," is, in my opinion, unquestionably true.

 

[SAFCasper]

Best I can think is performing testing on a small subset of files.

Come up with some logic to randomly select 20-30 files from anywhere within Google Workspace.
Your verification is restoring these files. New list generated every week/month or however often you want to test.


PS.
By restore I obviously don't mean overwrite. Restore them to a different location or subfolder etc.

 

[YeOldeStonecat]

So, part of your MSP services should factor this in...and if your clients compliance suggests to do it....either use a service that does this, or a backup monitoring service that at least checks a couple of boxes and/or produces some daily reporting.

For servers and/or important workstations, there are really good services such as Datto Alto/Siris....which do FULL IMAGE backups, and..importantly...will do a daily "boot up verification screen shot" that does a test of the backup image. It goes the job of testing that backup image for you. Since they're meant to virtualize the backup image...it will mount the latest image...boot it up in a virtual environment..and send a picture of that login screen to designed email addresses. We of course include ourselves, and...if the client wants is..them too. Gives them a warm 'n fuzzy. Because it was able to boot up that image in a virtual environment, you know it works.

If the screen shot verification test fails, you'll see that too..and know to go into action to fix it.

With SaaS backup of cloud services, such as services that back up 365 tenants, or Google Workplace tenants...obviously that's not something you can "boot up to test". But what we do is utilize a service called "Backup Radar".

Backup Radar

Backup Radar is the global leader in backup monitoring and compliance. We proudly protect 5,000+ businesses around the world.

us.backupradar.com

This service will "plug into" the backup services you use....and monitor all of the accounts you have in there, and send you daily reporting. Opening up a ticket for you in your ticketing system if anything failed. And it has some auto remediation processes...for certain backup services, that will push a few buttons and dials to fix a lot of issues that might cause errors. And...it can close that ticket for your helpdesk if it fixed it. Pretty slick.

 

[HCHTech]

Backup Radar looks interesting, but almost every backup software has notification capabilities - that's what we are using, So backup failures or suspends or other issues create tickets in our ticketing system. There is the inevitable noise from spurious tickets, but it's not overwhelming. We have a bunch of Synologys (ies?) out there as backup targets, and we'll do periodic directory restores to check readability. It's manual work, but billable, so just part of the daily tasks. Since we're a T&M shop and not an MSP, we don't have the pressure of having to build the expense of this kind of stuff into an already agreed-upon pricing.

 

[YeOldeStonecat]

Yeah we went as long as we could with just having each backup service report to our helpdesk directly. But it got overwhelming. We have..Datto...the Alto and Siris devices...as well as Datto Continuity ...the workstation backup, as well as Datto SaaS (for backing up 365). And then we still have some budget clients with servers on Solarwinds Cove (although we're shifting away from that). And then we have Veeam backup (moving budget on prem servers to this, away from Cove). And then we have Dropsuite...for most of our 365 backups. And then we have some Synology setups....for their active business backup for servers.

So having 1x dashboard is...a nice relief.

 

[HCHTech]

So having 1x dashboard is...a nice relief.

For sure - although I look at our CRM as "one dashboard" for all of that stuff as well, but I will freely admit that there is a difference between "setting up notifications" and "getting efficient, non-noisy notifications". There is quite a bit of tweaking to be done to keep the noise at bay, and not all softwares let you do the obvious thing. I think I've separately yelled at every vendor we have about "Only notify me if I have to DO something!!". Backup successful? Great! -- but I don't need to know that with a ticket that needs to be cleared. I swear, half the vendors out there think you only have them and only have one client on the system.....they never think about managing an entire group of clients on their stuff efficiently.