Fab's AutoBackup 7 Pro - a must have tool for techs

[sapphirescales]

Another possibility that could save some time : I could also do something like passing "C:\Users\CrazyDude\Documents" AND "C:\Users\CrazyDude\Documents\Taxes\2021" BUT I would use an option that will not look inside sub folders.

That would be GREAT!

He had this while running a portable win10 install on a USB SSD, so errors came from (I guess) lack of permissions on source drive

Actually I just did another transfer today. About 500GB worth of data. I got a bunch of errors. I did this one in the native running operating system because I figured I might have better luck that way.

So what happens when bitlocker is turned on?

It doesn't matter what the computer is in for. I decrypt the drive and convert it to a local account. If I'm doing a nuke n' pave, I decrypt the drive before extracting it and using Fabs. Then I install a fresh copy of Windows with a local account and transfer their files over again using Fabs.

So does that have to be done from a system with bitlocker (TPM) capable or enabled? Can an old (say gen 4) system do that?

My main office machine is a 4th gen i7 and I decrypt drives with that machine all the time.

we run it from a Windows bench machine which has the customer's drive in a hot-swap bay.

I do it this way sometimes too but it doesn't seem to make a difference. There are always errors. Usually only a few dozen on small transfers, but I've yet to have one completely successful transfer without errors.

 

[Metanis]

I've never tried running Fabs as administrator before.

Can't blame the application when it has to skip items for which it has no permissions.

 

[sapphirescales]

Can't blame the application when it has to skip items for which it has no permissions.

Yeah, it's definitely not a permissions issue. When it backs up 733/748 files under the Documents directory, it definitely has permission to read content in the Documents directory.

 

[fabs]

Yeah, it's definitely not a permissions issue. When it backs up 733/748 files under the Documents directory, it definitely has permission to read content in the Documents directory.

I would be so sure about that. What about a sub folder with special permissions on it ? The fix might be just some kind of permission check and adjustment when it hits a sub folder.
It could also be some special permissions on the file itself... If that's the case, I'll have to make the program check if it has permissions on EVERY file it finds. Listing processes will take a while...
Edit : idea => If file fails then check and adjust permissions on it and try again. If still fails => pass its folder to Unstoppable Copier. That should solve most of issues while not slowing down listing process. (brain is smoking ATM...)

 

[fabs]

@sapphirescales : I've not implemented Unstoppable Copier (yet ?). However, since you seem to have that problem very often (always?), would you mind testing some development versions ? So far, I've added an instruction that will check permissions (generic read or full permissions) on file if copy fails at first attempt. If it has not needed permissions, then it will take ownership of it and set required permissions before the second copy attempt.
I would like to know if this is enough to fix that problem or if I really need to go further with Unstoppable copier implementation. Let me know !

 

[fabs]

Can't blame the application when it has to skip items for which it has no permissions.

Fab's AutoBackup Pro prompts for admin rights at startup so it always runs as administrator

 

[Metanis]

Fab's AutoBackup Pro prompts for admin rights at startup so it always runs as administrator

I always run important tools by right-clicking and using the "Run As Administrator" option so I didn't know you hard-coded it! Thanks!

 

[sapphirescales]

would you mind testing some development versions

Sure. I'd be happy to see if it's a permissions issue, but I highly doubt it. On my main bench system I've turned the permissions BS off on an account-wide level so my user account completely ignores permissions. I haven't done this on my portable version of Windows 10. When I run Unstoppable Copier I don't run it any differently than I do Fabs and it has no problems at all whatsoever accessing any file.

 

[fabs]

Sure. I'd be happy to see if it's a permissions issue, but I highly doubt it. On my main bench system I've turned the permissions BS off on an account-wide level so my user account completely ignores permissions. I haven't done this on my portable version of Windows 10. When I run Unstoppable Copier I don't run it any differently than I do Fabs and it has no problems at all whatsoever accessing any file.

I've sent a link for a test version in a private message.
Let me know if that addresses the problem.

 

[sapphirescales]

Let me know if that addresses the problem.

Okay I'll get back to you. I'm going to re-run a backup I recently did that had a lot of failed files. There's about 500GB worth of data in this backup though so it will probably take a long time to finish. I figured this would be a good test since I've already backed this system up before. Unfortunately I don't have the original log file but I'll pass on the new log file to you once I make sure there's no personally identifiable information in it (unless of course there are no failed files this time, in which case I'll be very happy).

 

[fabs]

Okay I'll get back to you. I'm going to re-run a backup I recently did that had a lot of failed files. There's about 500GB worth of data in this backup though so it will probably take a long time to finish. I figured this would be a good test since I've already backed this system up before. Unfortunately I don't have the original log file but I'll pass on the new log file to you once I make sure there's no personally identifiable information in it (unless of course there are no failed files this time, in which case I'll be very happy).

Thanks for your time. It did not work in this case. Hopefully, I'll be able to reproduce the problem with the log you provided in private.

 

[fabs]

First look at the log and it appears that except Dropbox files, all the ones in error seems to have a very long path (first one is 268 chars). I suspect that the method I use to set permissions may not work with such long paths. Unstoppable copie works since it does NOT need to adjust ANY permission since it uses local "system" account. This one can access anything without ownership or things like that.
At least I can try to make the worst possible test case with long paths and very restrictive permissions. This way I will not have to disturb you with tests. I'll post here as soon as I have something new

 

[fabs]

@sapphirescales
I've been able to reproduce the issue : File on the desktop, path is 282 chars long and no user except owner and local system user can access it.
Let's start working on the fix!

Edit : in fact, I have not really been able to reproduce the problem.
Here's what I have :
- A 282 chard deep path .docx file inside several sub folders on the desktop with restricted permissions on it (only owner and local system account is granted on it, all other permissions it initially got by inheritance have been removed, so, that's the worst possible case).
- I have no restricted permissions on folders, just on this file.
- Source drive (a VHD) is mounted on my W11 development rig

If I run Fab's against this hooked drive, it fails on this file because it thinks it just does not exists (that's a first weird thing since it has been able to list it for copy). Then it trows an error saying that it is "Unable to find source file" and it skips to next file. So far, it exactly looks like the issue you have. Then Why do I say that I've not really been able to reproduce the problem ? Simple : If I fire up Unstoppable Copier and set "D:\Users\RestrictedUser\Desktop" as source folder and "E:\Test\2022-01-16\DESKTOP-1BKDB50\RestrictedUser\archive\Desktop" as target, it fails on it too, saying that it could not access the file. If Unstoppable Copier fails on my end and succeeds on yours, then we are not in the same configuration, so, problem is not really reproduced. By the way, even Windows command line tools designed to take ownership and set permissions fail on this file.

If I set permissions back to their defaults on this file (it just added full control on it for administrators group), then Fab's can copy it without a hitch. So far, I just do not understand what's really going on here. Still searching

By the way, how do you set Unstoppable Copier on your end ? I mean source folder, target folder and options. Thanks!

 

[fabs]

I'm making some progress here.
In the worst scenario described upper (the one when even Unstoppable Copier has failed), I've found a tool (setacl) that can change ownership and reset permissions to inherited ones on such a long path file. Once this was done, Fab's was able to copy the file flawlessly.
It's a bummer that integrated command line tools that ship with Windows (takeown.exe and icacls.exe) do not support long paths like this one.
I'm still looking at ways to do this on my own but I have asked them a distribution license. That could be a very good replacement for native Windows tools. Wait and see!

 

[fabs]

When official Fab's version is dealing with a long path restricted file, it throws errors but it creates target file anyway. Problem : this file is 0 kb
In current Development version, it calls SetACL tool, changes ownership and sets inherited permissions to the file, then it tries again and this time, it succeeds. However, I still have to polish the thing as there's still an error message that says source file could not be found (despite the fact it was copied ).
So, things are going better but there is still work to do !

Update : I've narrowed that residual false error. When checking paths, there's a needed instruction that adds a special prefix to the path so it can be used if it exceeds 255 characters. I forgot to add this instruction when checking source file path. Because of that, Fab's did not find a file named only with first 255 chars and threw an error saying that is dos not exist. Now, I've still a warning about source file's date (12/30/1899) that does not match copied file's date (01/16/2022) but that's a minor issue.

If I get the distribution agreement for SetACL tool, this will be a big win and it will be included in next update.

@sapphirescales I'm about to upload another test build. If you're OK to test it against your terrible drive that would be great.

 

[HCHTech]

This diagnosis / solution-development is such an interesting process to watch. Thanks @sapphirescales for sticking with it and thanks @fabs for being so transparent (as always - IMO)!

 

[fabs]

Well, last test build was just a disaster. Let's start it all over again...

 

[sapphirescales]

Well, last test build was just a disaster. Let's start it all over again...

LOL, yeah in your first test build it took 3 hours to back up my test machine and had 500 failed files. In the second, it only had 50 failed files, but it took 17 freaking hours! If it's going to take 17 hours to do a backup I'm gonna have to like quadruple my prices for data backup LOL.

Let me know when you want me to run another test @fabs. It's just set it and forget it really so don't feel bad about "wasting" my time.

 

[nlinecomputers]

So what happens when bitlocker is turned on?

It would have to run in Windows and be decrypted on the fly, like everything is. FABS would be unaware that the file was encrypted and the file would be decrypted on the destination.

If run outside of Windows then FABS wouldn’t see the drive until WinPE decrypted the drive. Just like every other program in existence. BitLocker isn’t magic. It’s either encrypted or it isn’t and programs are totally isolated from the encryption process. Doesn’t matter if it’s Excel or a file level backup program.

 

[seashore]

@fabs On another topic, could you give a list of what can only be backed up from within a live system (and unavailable from an offline drive)? Not talking about encrypted drives here, just generally.