And people wonder why I hate Crowdstrike...

Sky-Knight said:
this specific CEO should personally know better, I feel he should face criminal charges for the lives he took, and the injuries incurred by all the flights, and medical procedures cancelled due to his "mistake." Furthermore, the company should be subject to legal liability for all of the repairs required to bring systems back online.
Click to expand...

I'll cop to the last sentence, but not the first.

It has always been ludicrous, regardless of the company, to believe that any CEO has that level of knowledge of what's happening "in the trenches." That is not a CEO's job, and shouldn't be a CEOs job.

The buck may stop with the CEO, and his head, too, may roll, but it's way further down the management and development chain where the actual responsibility for this incident lies.

I can't count the number of CEOs who've lost jobs over stuff like this and, yet, nothing changes. And there's no shock there, because CEOs are not the people who actually control what needs to be changed in most cases. They are "flying at 30,000 feet," as is necessary, but all the changes need to occur less than "30 feet off the ground."

I don't expect the CEO of any company to micromanage to that extent. The actual people who should have known better at the level of controlling the creation, testing, and distribution of this update are where the laser focus needs to be. The CEO is a figurehead, and if focus only stays there, nothing changes, generally speaking.
 
Can we also acknowledge that Microsoft needs a more robust "recovery toolset" and a more resilient boot process.

It doesn't seem out of the realm of possibility to have a way to disable drivers at boot time - and/or have Windows report "what failed" and give a set of options to get past it. And for gosh sakes - gotta do something about Bitlocker. Give the recovery environment some basic online ability to login to a MS account and login to get the key seamlessly - as if you were logging into windows itself.

Turn back on Restore Points!
 
phaZed said:
Turn back on Restore Points!
Click to expand...

I agree with this, but I also tell everyone to do so but NOT to rely on them. They're wonderful, when they work, but they don't always work.

There certainly could be ways to "force ignore" things like this that would keep the machine from booting entirely. More fault tolerance needs to be built in along with the ability to get the machine booting on its own, even if only into safe mode, when stuff like this happens.

If something like the Offline Scan for Windows Defender can be done, then the same sort of thing to allow programmatic cleanup in a situation just like this one should be possible, too. It's an override situation where very specific actions can be taken before Windows itself loads, and can be taken sans direct human intervention by hand.
 
I wonder how many systems were able to use a snapshot and just roll back to a previously saved restore point? To me it's just more evidence that bare metal recovery is still a good thing in an era of cloud everywhere hype.
 
Metanis said:
I wonder how many systems were able to use a snapshot and just roll back to a previously saved restore point?
Click to expand...

While I get, and agree with, your overarching point about full system images/snapshots for restoration, you still can't automate that process on machines that will not boot into Windows.

The ugliest thing, by far, about this debacle is that it leaves everything in a state where human intervention is an absolute requirement to restore things, no matter what restoration option is being used.

If someone knows of a way to make computers crippled by this update able to be repaired programmatically, I have yet to hear about it, here or elsewhere.
 
britechguy said:
While I get, and agree with, your overarching point about full system images/snapshots for restoration, you still can't automate that process on machines that will not boot into Windows.

The ugliest thing, by far, about this debacle is that it leaves everything in a state where human intervention is an absolute requirement to restore things, no matter what restoration option is being used.

If someone knows of a way to make computers crippled by this update able to be repaired programmatically, I have yet to hear about it, here or elsewhere.
Click to expand...

Windows VMs running on a non-Windows hypervisor should have been able to do this easily if properly configured. And even if not configured to do it automatically it could have been done remotely using the hypervisor's management utility.
 
Metanis said:
Windows VMs running on a non-Windows hypervisor should have been able to do this easily if properly configured.
Click to expand...

I do not have any desire to get into a pissing match, but these would be edge cases if ever there were one. The vast majority, by far, of what's dead out there are plain old Windows boxes.

"Prevailing conditions" at the center of the bell curve of possible tech combinations are what I'm discussing, because they are just that.
 
phaZed said:
Can we also acknowledge that Microsoft needs a more robust "recovery toolset" and a more resilient boot process.

It doesn't seem out of the realm of possibility to have a way to disable drivers at boot time - and/or have Windows report "what failed" and give a set of options to get past it. And for gosh sakes - gotta do something about Bitlocker. Give the recovery environment some basic online ability to login to a MS account and login to get the key seamlessly - as if you were logging into windows itself.

Turn back on Restore Points!
Click to expand...
I'm at a loss for so much of this.

1.) Drivers are exactly that, and while there should be a more graceful failure mode... when we had that in the past it was used by threat actors to maintain persistence.
2.) Bitlocker isn't a problem here, because the drive unlocks with the attached TPM module, booting into safe mode doesn't trigger it... and if it did you have OTHER problems.
3.) Last Known good is still there... the problem is we can't mash F8 to get the boot menu anymore. AND in this case it WOULD NOT HAVE HELPED ANYWAY because the fault was in a file outside the control of the OS.

What do you want MS to do? Stop letting people install drivers?
 
Sky-Knight said:
I'm at a loss for so much of this.
Click to expand...
1. I hear you. However, why not lock the ability behind Password/Bitlocker, like it often does already? Right now there is virtually no facility to do anything besides CMD. Threat actors would have always needed physical access for "maintaining persistence" in this way, anyway.

2.Bitlocker was a huge problem because it blocked the ability of users to remediate the issue. No BL key, no ability to remove the bad driver file. Because there is no online functionality to retrieve the Bitlocker key using an online account/password - it required the IT staff to retrieve each BL key manually - if they could even do that at all. Not to mention, "Device Encryption" (Not Bitlocker full, but same tech) - there is no BL key to retrieve from MS servers - many of us have seen this problem in customer machines. If a User/Pass can be "perfectly suitable", as when logging in.. it should be suitable during the PE failsafe.
3. Restore Points are turned off by default. Turn them back on by default. F8 still works (?). In the case you can't get F8 to work because of Fast Boot or whatever... it's a simple act of turning the PC off during the boot process to trigger the PE upon next boot. It would have helped! While the file may be "Outside the OS" - the driver is loaded by virtue of the Registry, which would be reverted causing the driver not to load, or revert to the "old" driver that was there to begin with. Also, System Restore does revert 3rd party apps and programs/drivers.. it's not limited to MS 1st party software.

I don't want MS to stop letting people install drivers, I want them to make it possible to restore from a major screwup such as what just happened.
 
Sky-Knight said:
I'm at a loss for so much of this.

1.) Drivers are exactly that, and while there should be a more graceful failure mode... when we had that in the past it was used by threat actors to maintain persistence.
2.) Bitlocker isn't a problem here, because the drive unlocks with the attached TPM module, booting into safe mode doesn't trigger it... and if it did you have OTHER problems.
3.) Last Known good is still there... the problem is we can't mash F8 to get the boot menu anymore. AND in this case it WOULD NOT HAVE HELPED ANYWAY because the fault was in a file outside the control of the OS.

What do you want MS to do? Stop letting people install drivers?
Click to expand...
The could build the windows boot process to work more like Linux. The driver is a kernel level driver. Linux compiles that into a single file and most distros keep the older kernels. So if something borks that current one you can select a previous one from a menu.

As you mentioned Microsoft has stupidly disabled the F8 menu making what was already difficult even harder. Windows sucks and I wish the world could move away from it. Or M$ had the guts to do what Mac did when they changed from OS9 to OSX.
 
callthatgirl said:
a client who I do supplemental support for just emailed, none of them can access SharePoint.
Click to expand...

But that's still not a CrowdStrike "strike" on their computers, but somewhere between them and SharePoint.

There are definitely still ripple effects going on, as not every machine hit is back online yet.

If you are working in Windows your machine either has not been hit or has already been remediated.
 
nlinecomputers said:
The could build the windows boot process to work more like Linux. The driver is a kernel level driver. Linux compiles that into a single file and most distros keep the older kernels. So if something borks that current one you can select a previous one from a menu.

As you mentioned Microsoft has stupidly disabled the F8 menu making what was already difficult even harder. Windows sucks and I wish the world could move away from it. Or M$ had the guts to do what Mac did when they changed from OS9 to OSX.
Click to expand...

And again, confused on my end... because this is blatant ignorance.

For the anti-malware software to do its job it needs deep hooks into the kernel, and those hooks when malfunctioning cause kernel related issues.

CrowdStrike managed to break the RedHat kernel JUST LAST MONTH!

access.redhat.com

Kernel panic observed after booting 5.14.0-427.13.1.el9_4.x86_64 by falcon-sensor process. - Red Hat Customer Portal

eBPF program causes kernel panic on kernels 5.14.0-410+ . Below is an example of a kernel panic on the falcon-sensor process, observed after booting on kernel version 5.14.0-427.13.1.el9_4.x86_64. [ 462.396258] BUG: unable to handle page fault for address: ffff9a4bdb0f2d88 [ 462.396291] #PF...
access.redhat.com access.redhat.com

"being Linux" doesn't solve this problem.

Bitlocker making things more complicated by preventing external OSs from accessing the drive is also working as intended. There is no way to automatically fill the recovery key into the box because the platform has no Internet connectivity at that point. It's the world's largest chicken and egg problem.
 
Last edited:
@Sky-Knight

What it boils down to is if the fix is what they say it is, and I have no reason to doubt it, there could be a way that this could be remedied programmatically if Microsoft wanted to build it. Deleting a file from a non-bootable-into-Windows-proper machine should be possible.

As I said earlier, if they can do something like the Offline Scan for Windows Defender, which runs without Windows itself at all, there is a way to allow this kind of programmatic access for other purposes. It just doesn't exist, now. It could.
 
Sky-Knight said:
And again, confused on my end... because this is blatant ignorance.

For the anti-malware software to do its job it needs deep hooks into the kernel, and those hooks when malfunctioning cause kernel related issues.

CrowdStrike managed to break the RedHat kernel JUST LAST MONTH!

access.redhat.com

Kernel panic observed after booting 5.14.0-427.13.1.el9_4.x86_64 by falcon-sensor process. - Red Hat Customer Portal

eBPF program causes kernel panic on kernels 5.14.0-410+ . Below is an example of a kernel panic on the falcon-sensor process, observed after booting on kernel version 5.14.0-427.13.1.el9_4.x86_64. [ 462.396258] BUG: unable to handle page fault for address: ffff9a4bdb0f2d88 [ 462.396291] #PF...
access.redhat.com access.redhat.com

"being Linux" doesn't solve this problem.

Bitlocker making things more complicated by preventing external OSs from accessing the drive is also working as intended. There is no way to automatically fill the recovery key into the box because the platform has no Internet connectivity at that point. It's the world's largest chicken and egg problem.
Click to expand...
I’m aware of that. You missed my point. To get those systems booted all they had to do was load the previous kernel. (This as assuming that you retain copies of your previous kernel when you patch it. Not everyone does). It’s an easier fix than booting into safe mode than Windows.
 
nlinecomputers said:
It’s an easier fix than booting into safe mode than Windows.
Click to expand...

There are multiple ways this could be accomplished, and if Microsoft wanted to implement one or more of them, they could.

They've taken a "many roads to Rome" approach to almost everything, but not this. An incident like this one should push them, even though it's not their fault, to make remediation efforts easier and, as a direct result, more rapid.
 
You must log in or register to reply here.
Back
Top