As the world slowly reboots from yesterday’s colossal IT outage, the cybersecurity company responsible – CrowdStrike – has just released a technical overview of the incident.
Earlier the CEO issued an apology and a warning.
“Valued Customers and Partners, I want to sincerely apologise directly to all of you for today’s outage,” said George Kurtz, Founder and CEO.
“We know that adversaries and bad actors will try to exploit events like this. I encourage everyone to remain vigilant and ensure that you’re engaging with official CrowdStrike representatives.
“All of CrowdStrike understands the gravity and impact of the situation.”
The ‘situation’ was a global shutdown of Windows computer systems using Falcon sensor that caused chaos at airports, supermarkets, banks, hospitals and broadcast media.
“We understand the gravity of the situation and are deeply sorry for the inconvenience and disruption,” added Kurtz.
“We are working with all impacted customers to ensure that systems are back up and they can deliver the services their customers are counting on.”
The problem started mid-afternoon Friday when people started reporting BSOD (Blue Screen of Death) in huge numbers. PCs around the world were stuck in a boot loop and their users didn’t know why.
The reason soon became clear.
“The outage was caused by a defect found in a Falcon content update for Windows hosts,” said Kurtz.
“Mac and Linux hosts are not impacted. This was not a cyberattack.”
The outage impacted corporations or businesses with large IT systems that use Falcon. Home Windows users were not affected. Although little to no information came from CrowdStrike for several hours, a quick fix was posted online.
Workaround steps for individual hosts:
- Reboot the host to give it an opportunity to download the reverted channel file. We strongly recommend putting the host on a wired network (as opposed to WiFi) prior to rebooting as the host will acquire internet connectivity considerably faster via ethernet.
- If the host crashes again, then:
- Boot Windows into Safe Mode or the Windows Recovery Environment
- NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.
- Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
- Note: On WinRE/WinPE, navigate to the Windows\System32\drivers\CrowdStrike directory of the OS volume
- Locate the file matching “C-00000291*.sys” and delete it.
- Boot the host normally.
- Note: BitLocker-encrypted hosts may require a recovery key.
- Boot Windows into Safe Mode or the Windows Recovery Environment
This worked.
I watched the Sky News Australia IT team going one-by-one to each Windows machine to implement the above quick fix. This was to stop the blue screen of death so that machines could download the reverted channel file.
“The issue has been identified, isolated and a fix has been deployed,” said Kurtz.
The first statement came from CrowdStrike around 8pm Sydney time. I saw it appear on Linkedin first.
The apology arrived in the early hours of the morning, with the promise of transparency. The updates are now coming more freely.
“Nothing is more important to me than the trust and confidence that our customers and partners have put into CrowdStrike,” Kurtz stated.
“As we resolve this incident, you have my commitment to provide full transparency on how this occurred and steps we’re taking to prevent anything like this from happening again.”
So what about Microsoft?
“Earlier today, a CrowdStrike update was responsible for bringing down a number of IT systems globally.” said a company spokesperson.
“We are actively supporting customers to assist in their recovery.”
The company also wanted to make it clear the current issue with CrowdStrike, is unrelated a previous outage in the Central US Azure region on July 18 – impacting Azure customers using that region as well as some Microsoft 365 services. That issue has fully recovered.
The CrowdStrike incident shows just how vulnerable our modern life is to a large outage. Lessons must be learned about this and similar types of disruptions so we continue to function as a civilised society.
CROWDSTRIKE TECHNICAL OVERVIEW
What Happened?
On July 19, 2024 at 04:09 UTC, as part of ongoing operations, CrowdStrike released a sensor configuration update to Windows systems. Sensor configuration updates are an ongoing part of the protection mechanisms of the Falcon platform. This configuration update triggered a logic error resulting in a system crash and blue screen (BSOD) on impacted systems.
The sensor configuration update that caused the system crash was remediated on Friday, July 19, 2024 05:27 UTC.
This issue is not the result of or related to a cyberattack.
Impact
Customers running Falcon sensor for Windows version 7.11 and above, that were online between Friday, July 19, 2024 04:09 UTC and Friday, July 19, 2024 05:27 UTC, may be impacted.
Systems running Falcon sensor for Windows 7.11 and above that downloaded the updated configuration from 04:09 UTC to 05:27 UTC – were susceptible to a system crash.
Configuration File Primer
The configuration files mentioned above are referred to as “Channel Files” and are part of the behavioral protection mechanisms used by the Falcon sensor. Updates to Channel Files are a normal part of the sensor’s operation and occur several times a day in response to novel tactics, techniques, and procedures discovered by CrowdStrike. This is not a new process; the architecture has been in place since Falcon’s inception.
Technical Details
On Windows systems, Channel Files reside in the following directory:
C:\Windows\System32\drivers\CrowdStrike\
and have a file name that starts with “C-”. Each channel file is assigned a number as a unique identifier. The impacted Channel File in this event is 291 and will have a filename that starts with “C-00000291-” and ends with a .sys extension. Although Channel Files end with the SYS extension, they are not kernel drivers.
Channel File 291 controls how Falcon evaluates named pipe1 execution on Windows systems. Named pipes are used for normal, interprocess or intersystem communication in Windows.
The update that occurred at 04:09 UTC was designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks. The configuration update triggered a logic error that resulted in an operating system crash.
Channel File 291
CrowdStrike has corrected the logic error by updating the content in Channel File 291. No additional changes to Channel File 291 beyond the updated logic will be deployed. Falcon is still evaluating and protecting against the abuse of named pipes.
This is not related to null bytes contained within Channel File 291 or any other Channel File.
Remediation
The most up-to-date remediation recommendations and information can be found on our blog or in the Support Portal.
We understand that some customers may have specific support needs and we ask them to contact us directly.
Systems that are not currently impacted will continue to operate as expected, continue to provide protection, and have no risk of experiencing this event in the future.
Systems running Linux or macOS do not use Channel File 291 and were not impacted.
Root Cause Analysis
We understand how this issue occurred and we are doing a thorough root cause analysis to determine how this logic flaw occurred. This effort will be ongoing. We are committed to identifying any foundational or workflow improvements that we can make to strengthen our process. We will update our findings in the root cause analysis as the investigation progresses.