IT Infrastructure Issues - Windows OS Service Impact
Incident Report for Duke IT
Resolved
If you are still experiencing Windows machine issues please contact the OIT Service Desk: https://oit.duke.edu/help/
Posted Jul 24, 2024 - 11:40 EDT
Update
Duke University IT staff are continuing to restore systems following the recent global IT outages caused by a CrowdStrike update.

Though the outage itself is not the result of a cyber-attack, there is a growing risk of hackers exploiting the situation and the U.S. Cybersecurity and Infrastructure Security Agency (CISA) has warned of increased phishing and malicious activity. It is urged to use extreme caution as fake websites and phishing attempts are on the rise, pretending to be CrowdStrike support.

Examples of fake sites include: crowdstrikebluescreen[.]com and crowdstrikefix[.]com

Please remember the following:

CrowdStrike will NOT contact you directly.
Only Duke University IT staff will apply patches or fixes.
Duke University IT staff will NEVER charge you to restore systems.
Duke University IT staff will NOT ask for your password or sensitive information.
All official incident communication will come from an "@duke.edu" email address.

If you suspect a phishing attempt or have visited a suspicious site, contact the Duke Information Security Office (ISO) at security@duke.edu. Additionally, report suspicious emails using the “Report” button in your Duke email.

If you have any questions about this information please contact the OIT Service Desk: https://oit.duke.edu/help/
Posted Jul 23, 2024 - 07:04 EDT
Update
Duke University IT staff have updated the remediation instructions for University Windows devices affected by the CrowdStrike outage: https://security.duke.edu/crowdstrike-windows-issue/

If you need assistance with the steps provided on the Duke Information Security web site please contact the OIT Service Desk: https://oit.duke.edu/help/
Posted Jul 22, 2024 - 07:45 EDT
Update
Duke University IT staff have deployed a fix to address University Windows devices affected by the CrowdStrike outage. The fix is being implemented for Duke Health shortly.

Machines must be on the Duke network and rebooted (may take 3-5 reboots). This fix is only available currently for devices that are connected to the Duke network (Wifi or Ethernet). IT staff are still testing the recovery process for off-campus/remote devices.

For more information, visit the following website: https://duke.is/crowdstrike-windows-issues
Posted Jul 19, 2024 - 08:06 EDT
Update
If your computer is impacted by this issue, OIT is recommending that you reboot your machine several times.

If several reboots does not resolve the issue, please visit the following web site for additional remediation instructions: https://duke.is/crowdstrike-windows-issues

Additionally, OIT is working on an automated solution and we will provide an update shortly.

If you need assistance with the above steps please contact the OIT Service Desk: https://oit.duke.edu/help/
Posted Jul 19, 2024 - 07:15 EDT
Update
OIT continues to work on remediation instructions for this issue and we will posted those shortly.

In the meantime if you need immediate assistance please contact the OIT Service Desk: https://oit.duke.edu/help/
Posted Jul 19, 2024 - 06:17 EDT
Update
We are continuing to work on a fix for this issue.
Posted Jul 19, 2024 - 05:59 EDT
Update
We are continuing to work on a fix for this issue.
Posted Jul 19, 2024 - 04:46 EDT
Update
OIT has identified the issue as being caused by the Crowdstrike Falcon Sensor client installed on Windows servers and laptops/desktops. This issue has widespread impact beyond just Duke University.

Please see the below work around to address the issue:

-Boot Windows into Safe Mode or the Windows Recovery Environment
-Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
-Locate the file matching “C-00000291*.sys” and delete it. 
-Boot the host normally. 

If you need assistance with the above steps please contact the OIT Service Desk: https://oit.duke.edu/help/
Posted Jul 19, 2024 - 03:46 EDT
Identified
OIT has confirmed a service issue with the operating system on Windows servers and laptops/desktops. Multiple services that rely on Windows servers are impacted. Additionally, users may experience 'blue screen of deaths' and restarts of their computer.

OIT staff are working to restore service but there is no estimated time to resolution.

If you have questions about this outage or would like to report additional service impacts please contact the OIT Service Desk (https://oit.duke.edu/help) or join the conversation on the 'University IT Outage Communication' MS Teams site.
Posted Jul 19, 2024 - 03:14 EDT
Investigating
OIT is investigating possible infrastructure issues that are causing multiple service outages across campus. If you are experiencing a service issue please report it to the OIT Service Desk (https://oit.duke.edu/help) or join the conversation on the 'University IT Outage Communication' MS Teams site. Your reports will help OIT determine the possible extend of the outage.
Posted Jul 19, 2024 - 02:52 EDT
This incident affected: Accounts and Email (Active Directory, Alumni Email, Directory Services, Email Alias, Group Manager, Kerberos, Mail Transport and Routing, NetID, OneLink, Self Service Tools, Service Desk Utilities, Shibboleth, Sponsored Guest Tool), Collaboration and Communication (911 Calls, Call Center Services, Call Recording, CampusVision, Cellular Infrastructure (DAS), Cvent, Digital Signage, Duke Box, Duke Notify, DukeWiki, Electronic Fax, Emergency Notifications, Emma, EMS, Jabber, Jira, Microsoft Teams, Microsoft 365, OneDrive For Business, Phone Services, Phone, Web, and Video Conferencing, Qualtrics, Scholars@Duke, SharePoint, Sympa, Voice Mail, Zoom Meetings), Information Systems and Automation (API Healthcare, DukeHub (PeopleSoft), Duke@Work Portal, Gift Records (gifts.duke.edu), Gitlab, Kuali Build, Perceptive Content (ImageNow), Research Computing, SAP, Tableau), Network (Core Network, Data Center Network, DukeBlue, eduRoam, Network Registration, Public Internet, VPN), Other (Other), Printing and Labs (ePrint, 3DprinterOS), Web and Applications (Duke Today, DukeMobile, Main Duke Website (www.duke.edu), MyDuke, Sites@Duke Express (WordPress), Sites@Duke Pro), Endpoint Management (BigFix, Jamf Pro, Microsoft Endpoint Configuration Manager), Wireless and Wired Connectivity (West Campus Buildings, Central Campus Buildings, East Campus Buildings, Remote/Off Campus Buildings, West Campus Residence Halls, Central Campus Residence Halls, East Campus Residence Halls), Security (Certificate Issuance, 1Password, Multi-Factor Authentication, Network Firewall, Phishing Alert, Security Alert, Splunk), Virtualization and Storage (Clockworks, Database Hosting and Administration, Data Storage, Load Balancing, Log Analysis, NAS, OpenShift, Rubrik Backup, Virtual Server Hosting), Teaching and Learning Tools (Canvas, Container Manager (CMGR Coursework Containers), Coursera, Kits, LinkedIn Learning, Panopto, Sakai, Software License Management, Virtual Computing Manager (VCM Coursework VMs), Warpwire), and DukeCard (DukeCard).