[Editor’s Note: Follow Khaleej Times’ live blog as we report on the Microsoft outage affecting Outlook and other services globally]
On Thursday, a Microsoft outage hit businesses worldwide, causing widespread disruption to workplaces and customers. The outage which has now lasted for over 18 hours has affected essential services like Outlook, and the tech giant is working to resume service.
Downdetector, which tracks outages by collating status reports from a number of sources, showed more than 25,000 reports of Outlook problems in the US at 9.30 pm ET (6.30am UAE time) on November 12. At its peak, Downdetector showed more than 130,000 reports of Outlook problems in the US around 1 pm ET (10pm UAE time) on Thursday.
Stay up to date with the latest news. Follow KT on WhatsApp Channels.
Microsoft said there is no evidence of “malicious activity”, but investigations are ongoing and a full post-incident report will be released after mitigation is complete.
The tech giant said it is seeing “encouraging recovery progress in Europe; Japan remains comparatively more stable, while North America is also improving”, while admitting that users may face intermittent issues.
What has Microsoft said so far? Here’s what we know:
-
At around 7.40pm UAE time on November 12, the firm first said users were experiencing issues when accessing or using Microsoft Teams functionality in North America.
-
Less than three hours later, Microsoft said it was “investigating reports of users experiencing issues accessing Exchange Online mailboxes”, and confirmed the issue may affect Microsoft 365 service connectivity for users in any region.
-
The firm said the connectivity errors may be due to a portion of infrastructure experiencing “elevated load” across multiple regions, and to balance the load, connection traffic was redistributed across broader worldwide infrastructure.
-
It was later determined that the “elevated load” was not driven by an increase in traffic, but by servers being “unexpectedly available,” which strained remaining infrastructure.
-
The server failure was due to a specific component that is responsible for processing server health, as servers were becoming unavailable after failing automated health checks, the Microsoft investigation found.
-
An alternative service health check method was validated, which allowed a “subset of affected servers” to return to service, Microsoft said, along with automated recovery systems being prepared to continually detect and restart unresponsive servers.
-
Around 2am UAE time, nearly seven hours after an outage was first reported, the tech giant said the actions taken so far “have not provided the anticipated relief,” and that they were opening all available pathways to restore service.
-
Nearly two hours later, Microsoft said a portion of affected servers were returned to service, and remained stable; however, they could become unresponsive again after receiving traffic. Diagnostic data analysis was ongoing to determine what the trigger is, and types of traffic that can reach the affected infrastructure was limited.
-
As of 8am today, Microsoft confirmed “partial recovery” across regions where they applied traffic adjustments, but added the actions weren’t resulting in sustained recovery.
-
Microsoft extended email timeout to 48 hours to help preserve delayed mail, and said recovery actions are ongoing, but “service health remains variable and capacity has not yet fully returned to baseline.”
-
As of 3pm, improvements were seen in mailbox access although intermittent issues may continue, the firm said.
-
In its latest update, at 4.09pm, Microsoft said it is “seeing encouraging recovery progress, with service availability and email delivery performance improving across a growing portion of affected Microsoft 365 infrastructure”. Email queue and backlog reduction remain a priority, it added.
Source: Khaleej Times