The July 19th Crowdstrike Windows outage has impacted about 8.5 million devices globally, causing entire business sectors to grind to a halt. A faulty software update led to a cascade of systems failures among airlines, financial institutions, healthcare organizations, and more. 

Here’s what happened 

On July 19th 2024, Crowdstrike released a configuration update to its Falcon platform. This routine update was supposed to improve sensor performance, but it contained a critical logic error causing affected systems to crash. 

The outage is not a cyberattack, and has no connection to threat actor activity. However—threat actors are actively exploiting the situation, creating spoof Crowdstrike domains and distributing malware with misleading filenames like “crowdstrike-hotfix.zip”. 

The critical logic error is contained in a series of configuration files called Channel Files. Each file has a unique identifier. The file that caused the outage is C-00000291.sys. Even though it ends with a .sys extension, the file is not a kernel driver. 

Who is affected? 

All devices running Falcon sensor for Windows version 7.11 and above that were online on Friday, July 19th 2024 between 04:09 UTC and 05:27 UTC may be affected. Windows systems that downloaded and ran the update during this time may have crashed and become inoperable.  

Crowdstrike released a fixed update at 05:27 UTC, so any device not already impacted by this time received the updated configuration file. Devices using Max, Linux, and other operating systems are not affected. 

The Importance of the SOC Visibility Triad 

Imagine relying solely on CrowdStrike (EDR) for your security visibility. At Lumifi, we advocate for the SOC Visibility Triad, which includes EDR, NDR, and SIEM. We believe that most companies should implement all three for robust protection. Losing one of these tools temporarily is a manageable disruption, but losing your only security tool, even briefly, can leave you vulnerable. Diversify your security tools—don't let a single vendor blind you! 

Here’s what you need to do 

Crowdstrike has already published a workaround for the outage, which involves manually rebooting each machine into safe mode. Here are some things Crowdstrike customers can do right now to help remediate the issue: 

• Temporarily set your Sensor Update Policy to the previous stable version. Access Host Setup and Management and navigate to Sensor Update Policies.  

• • Change the policy that sets the Build from “Auto - Latest” to “Auto - N-1” 

• If your machine crashes on startup, they can’t receive the new policy update. Try rebooting into Safe Mode or the Windows Recovery Environment. 

• • Navigate to %WINDIR%\System32\drivers\CrowdStrike 

• • Locate C-00000291*.sys and delete it 

• • Boot the host normally and allow it to receive the new update. 

• If your machine is encrypted with Bitlocker, you will need a recovery key.  

• • Microsoft has released an updated version of its Recovery Tool to address this scenario. 

• If your machine is on a public cloud instance, you have two options. 

• • Option 1: 

• • • Detach the operating system disk volume from the impacted virtual server. 

• • • Create a snapshot or backup of the disk volume before proceeding. 

• • • Mount the volume to to a new virtual server. 

• • • Navigate to %WINDIR%\\System32\drivers\CrowdStrike  

• • • Locate “C-00000291*.sys”, and delete it. 

• • • Detach the volume from the new virtual server. 

• • • Reattach the fixed volume to the impacted virtual server. 

• • Option 2: 

• • • Roll back to a snapshot before 0409 UTC.  

• If your Virtual Machine (VM) is on Azure, you can access it using the serial console. 

• • Login to Azure console and go to Virtual Machines.  Select the VM. 

• • Click "Connect" on the upper left, then select “More ways to Connect" and click on "Serial Console". 

• • Once SAC has loaded, type in 'cmd' and press enter. 

• • • Type in the 'cmd' command 

• • • Type in: ch -si 1 

• • Press any key. You will be prompted to enter Administrator credentials 

• • Type the following: 

• • • bcdedit /set {current} safeboot minimal 

• • • bcdedit /set {current} safeboot network 

• • Restart the VM. 

• • Optional: Try confirming the boot state. 

• Run this command: wmic COMPUTERSYSTEM GET BootupState 

Immediate, mid-term, and long-term remediation programs 

In a complex enterprise IT environment, recovering from a large-scale outage is easier said than done. IT leaders must create action plans that address immediate, mid-term, and long-term risks associated with the event:  

• Immediate actions include engaging the incident response team, notifying employees and clients, and preventing users from attempting self-service remediation actions. It also includes using attack surface management tools to identify machines that contain the corrupt Channel File and prioritizing remediation for the most critical assets. 

• Mid-term actions include reviewing anomalies surrounding Crowdstrike assets and connections. SOC analysts must be prepared for threat actors to leverage the event for phishing and social engineering attacks. Automating update settings for endpoint protection tools should be systematically reviewed across the organization. 

• Long-term actions include reviewing incident response playbooks for large-scale outages and providing training in disaster recovery and business continuity to key employees. Many organizations report being unable to handle the sudden influx of support requests, so finding scalable on-demand support resources is vital when preparing for future events. 

Each of these steps relies on having unlimited visibility and control over your IT environment. When security teams run into issues accessing IT assets or conducting discovery of the environment, it makes remediating complex issues that much harder.  

 Contact Lumifi for on-demand product expertise 

Managing a disruptive, large-scale event is no easy task. Even well-prepared enterprise IT teams can be quickly overwhelmed by the sheer volume of issues that arise in these scenarios. Security leaders who partner with reputable MDR vendors like Lumifi gain access to scalable product expertise they can rely on when it matters most.