Rackspace Technology response to Crowdstrike July Incident

Starting on 18 July 2024, at 23:49 CDT, Rackspace became aware of an issue following a recent CrowdStrike update impacting systems running various CrowdStrike sensor versions. Impacted users were getting a BSOD (Blue Screen of Death) error and systems were rebooting automatically. 

CrowdStrike identified issues related to the Falcon sensor and provided recommended workarounds on the affected Windows-hosted systems. CrowdStrike has since identified and isolated the issue and deployed a fix. 

Rackspace engineers continue to implement the solution on remaining impacted devices. 

Details 
Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor. 

Current action 
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes. 
If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue: 

Workaround Steps: 
1. Boot Windows into Safe Mode or the Windows Recovery Environment 
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory 
3. Locate the file matching "C-00000291*.sys", and delete it. 
4. Boot the host normally. 

If you have any questions, please contact a member of your support team. 

Please refer to the following vendor pages for the latest updates and workarounds: 

https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/ 

https://status.cloud.google.com/incidents/DK3LfKowzJPpZq4Q9YqP#FxgErUaBEZS6pEeX6yiz 

https://azure.status.microsoft/en-us/status 

https://health.aws.amazon.com/health/status