Root Cause Analysis （RCA）

CS has published a Root Cause Analysis (RCA).
Root Cause Analysis PDF
For the executive summary, please see the “Channel File 291 RCA Exec Summary” section of this article.

Preliminary Post Incident Review (PIR)

CS has published a Preliminary Post Incident Review (PIR).
Please see the "Preliminary Post Incident Review" section of this article.
​The Root Cause Analysis (RCA) will be made public in the future.

A summary version of the PIR (PDF) is available here.

Summary

- Windows host crashes (BSOD) related to the Falcon Sensor have been confirmed.

・Technical details of this issue (CS blog post)

・CS's Falcon platform system is operating normally, so if your system is operating normally, even if the Falcon Sensor is installed, there will be no impact on the system's protection. Falcon Complete and Overwatch services will not be interrupted by this incident.

Technical Overview

CS identified the trigger for this issue to be Windows Sensor related content deployment and reverted these changes. The content in question was the channel file located in the %WINDIR%\System32\drivers\CrowdStrike directory.

・If the timestamp of the causative channel file "C-00000291*.sys" is between 13:09 and 14:27 Japan time on July 19, 2024, you may be affected. (If the channel file is after 14:27 Japan time on July 19, 2024, it has been corrected and no recovery action is required.)
Note: It is normal for there to be multiple C-00000291*.sys files in the CrowdStrike directory. If any of the files in the folder have a timestamp later than 2024/7/19 14:27 Japan time, the file will be considered active.

-From 1:30 pm on July 23, 2024, CS used existing quarantine technology to isolate the channel file that was causing the BSOD issue from the device that had it, and provided recovery support to the device that was unable to boot normally.
As a result, the file name "C-00000291*.sys" may be displayed multiple times on the Quarantine Dashboard (Endpoint Security > Quarantined Files), but this is not a problem as the quarantine is due to processing.
For devices that continue to be unable to start up normally, we ask that you try to restore your device using the methods described on this page.

File Classification Status

The channel file that was causing system crashes beginning at 04:09 UTC on Friday, July 19, 2024 has been identified and deprecated on production systems. Upon deprecation, a new file will be deployed, but the old file will remain in the sensor's directory.

As a precaution, the affected versions of the channel files have been added to the known bad list for Falcon in CrowdStrike Cloud to prevent further disruption to Windows systems.

No sensor updates, new channel files, or code were deployed from the CrowdStrike Cloud as a hygiene measure for production machines.

For affected systems with a strong network connection (assuming a wired network connection), this action may also lead to automatic recovery of systems that are in a boot loop. This was scheduled for Tuesday, July 23, 2024 UTC in US-1, US-2, and EU.

Unaffected devices

・Windows devices that went online after 14:27 on July 19, 2024 (Japan time)

- Windows devices installed and provisioned after 14:27 on July 19, 2024 (Japan time)

- Mac or Linux device

Step 1: Identify affected devices

How to identify devices using Advanced Event Search

*Note: An Insight subscription is required to use this feature.

Please see this KB article​

How to identify a device using the dashboard

The dashboard shows the affected channels and CIDs, as well as the affected sensors, and depending on your subscription, is available in one of the console menus below.
*Note: An Insight subscription is required to use this feature.

・Next-Gen SIEM > Log management > Dashboard
・Investigate > Dashboards
Dashboard name: hosts_possibly_impacted_by_windows_crashes_granular_status

*Note: Dashboard cannot be used in conjunction with the Live button

Step 2: Recovery

If your host continues to crash and no fixes are being applied from the cloud, you can recover by following the steps below:
Although there are procedures other than those listed below that are published on external websites, we ask that you please follow the procedures published by our company and CS Inc.

・How to recover using bootable media:
- For detailed recovery instructions, please refer to this KB Building CrowdStrike Bootable Recovery Images.
In addition, CS has released the following video showing the work procedure.
Youtube: CrowdStrike Host Remediation with Bootable USB Drive

Manual recovery method without bootable media:
- If you have a recovery key for a BitLocker encrypted device (it can also be used on non-encrypted devices)
・Option 1 - Youtube: Manually remediating a host
Option 2 -Microsoft KB

・ Repairing the Falcon Windows Sensor:
・ Repairing Falcon Windows Sensors– Link to article on repairing sensors when renaming or deleting CrowdStrike folder
・ Removing locked memory.dmp files after successful remediation- Link to article on deleting memory.dmp file if it is locked after sensor repair

BitLocker Recovery in a Microsoft Environment

　BitLocker Recovery in Microsoft Azure
　BitLocker Recovery in a Microsoft Environment using SCCM
　BitLocker Recovery in a Microsoft Environment using Active Directory and GPO 
　BitLocker recovery in a Microsoft environment using Ivanti Endpoint Manager 
　BitLocker Recovery in Microsoft Environments using ManageEngine Desktop Central 
　BitLocker recovery in Microsoft environments using IBM BigFix

BitLocker recovery without recovery key

　BitLocker recovery without recovery key

BitLocker Recovery in the Workspace ONE Portal

　User Access to Recovery Key in the Workspace ONE Portal

BitLocker Recovery with Tanium

　Reference: Windows encryption management

Bitlocker Recovery with Citrix

　BitLocker recovery key

AWS Recovery

　How do I recover AWS resources that were affected by the CrowdStrike Falcon agent?

Azure Recovery

　Azure status

Google Cloud Platform (GCP) Recovery

Manually recover from a blue screen on a GCP Windows instance
GCP CrowdStrike File Remediation Script - A Python script that customers can use to remediate affected hosts residing in GCP

Recovery procedures for public cloud or similar environments, including virtual

[Option 1]
1. Detach the operating system disk volume from the affected virtual server
2. Create a snapshot or backup of your disk volume before proceeding (precaution against unintentional changes).
3. Attach/mount the volume to the new virtual server
4. Go to the “%WINDIR%\\System32\drivers\CrowdStrike” directory
5. Delete the files that start with “C-00000291” and end with “.sys” (“C-00000291*.sys”)
6. Detach the volume from the new virtual server
7. Attach/mount the fixed volume to the affected virtual server

[Option 2]

Rollback to a snapshot before 13:09, 19 July 2024 (UTC)​

Intel vPro Technology Repair Guide

　Remediate CrowdStrike Falcon® update issue on Windows systems with Intel vPro® technology

Absolute Software Support

　Steps to Repair BSOD Devices and Run Corrupted File Detection Procedure

Rubrik Recovery

　CrowdStrike & Rubrik Customer Content Update Recovery For Windows Hosts

Cohesity Support

　Cohesity’s support for CrowdStrike’s Falcon Sensor updates

Contact information

Regarding this issue, we have published the following article on our support site and an article from CS Inc. Please refer to the article on our support site, which will be updated from time to time along with this page.
*Our support site also provides information on how to operate the dashboard and case studies related to this issue.

Our support article: https://support.mnc.macnica.co.jp/hc/ja/articles/35269019637657　
CS article: https://supportportal.crowdstrike.com/s/article/Tech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19

Macnica
CrowdStrike Product Support: crowdstrike@macnica.co.jp

Revision history

2024/07/19 21:00 First edition published

2024/07/19 23:00 Updated the following items
・Addition of details
-Addition of current actions (workarounds)
Workaround steps for individual hosts
Workarounds for public cloud or similar environments
-Add link information
・AWS article on workarounds for virtual machines on AWS
・CS article on Bitlocker recovery procedures

2024/07/20 2:45 Updated the following items
Update details
・The following text has been deleted:
　　Windows 7 and Windows Server 2008 R2 hosts are not affected by this issue.
-Update current action (workaround)
-Added information about queries
Use Advanced Event Search to query to see affected hosts
-Add link information
- User access recovery key in Workspace ONE portal
-CrowdStrike Holdings、Inc. article on Bitlocker recovery procedures

2024/07/20 7:30
・Updated article title and summary

2024/07/20 10:30
-Updated information about queries
-Add link information
Automatic recovery of Windows instances on GCP
・Windows encryption management using Tanium
-Bitlocker recovery using Citrix
- Updated link information
・AWS article on workarounds for virtual machines on AWS

2024/07/20 12:30
-Added a dashboard to view affected hosts
Updated workaround instructions for individual hosts

2024/07/20 19:00
・ Added manufacturer article (Bitlocker recovery without recovery keys) on what to do if the Bitlocker recovery key is not available.

2024/07/20 21:00
Updated the link to the article on automatic recovery of Windows instances on GCP and the CS article on Bitlocker recovery procedures.

2024/07/21 11:00
- Technical Overview, added non-affected devices
- Identifying affected devices and organizing recovery procedures

2024/07/21 20:30
- Added removal procedure using recovery tool

2024/07/22 19:15
- Added information about unaffected devices
・Updated the content of Step 1: Identifying affected devices
- Updated the method for identifying devices using Advanced Event Search

2024/7/23 11:00
・Updated the contents of Step 2: Recovery
- Updated the Google Cloud Platform (GCP) Recovery section

2024/7/23 20:50
・Updated the contents of Step 2: Recovery

2024/7/24 08:40
- Added the contents of Update File Classification Status

2024/7/24 14:08
-Added Preliminary Post Incident Review (PIR)

2024/7/24 16:00
- Updated the contents of File Classification Status Update
- Added Falcon Windows sensor repair

2024/7/25 9:15
-Added information about the summary (PDF) version of the PIR

2024/07/26 2:30
- Added a video link on how to recover using bootable media

2024/07/27 6:30
・Added information to Falcon Windows Sensor Repair
-Added Absolute Software support

2024/08/07 9:45
-Added information on Root Cause Analysis (RCA)