Medchat - Application Service Interruption – Incident details

While Microsoft is keeping their incident tickets open, they have confirmed that majority of their impacted services have now recovered. They have an updated dashboard where all services on the affected region are now back online.

Root cause per Microsoft: The underlying cause was due to a backend cluster management workflow deployed a configuration change that caused backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region. This resulted in the compute resources automatically restarting when connectivity was lost to virtual disks.

The Medchat team also conducted some health checks to ensure the system is back up and fully functional. Closing this incident.

While the Medchat application is back online and seems to be back in a fully functional state, Microsoft is continuing to send status updates (last at 9:10 pm PT) that the incident is still active, and that customers should continue to see increasing recovery at this time as residual and downstream impact mitigation progresses.

Medchat team will continue to monitor system health and status updates.

Microsoft update @7:33 pm PT

• Status: Incident is now mitigated

• Next steps: Engineers will continue to investigate to establish the full root cause and prevent future occurrences.

We are currently performing checks on the Medchat Application to ensure overall system health; we will continue monitoring for another hour or so.

Microsoft update at 6:31 pm PT:

Current Status: ... We’ve determined the underlying cause and are currently working towards mitigation. We will start to see incremental recovery in next 90 minutes. The next update will be provided in 60 minutes, or as events warrant.

Update from Microsoft @4:56 pm PT:

Current status: We have determined this issue was impacted by an underlying storage outage in the Central US region that the services were dependent upon. Once the underlying storage outage is mitigated, this impact will be resolved. The next update will be in 2 hours, or as events warrants.

Latest update from Microsoft @4:45 pm PT

Impact Statement: Starting at 21:56 UTC on 18 Jul 2024, a subset of customers may experience issues with multiple Azure services in the Central US region including failures with service management operations and connectivity or availability of services.

Current Status: We are aware of this issue and have engaged multiple teams to investigate. As part of the investigation, we are reviewing previous deployments, and are running other workstreams to investigate for an underlying cause. The next update will be provided in 60 minutes, or as events warrant. 

This message was last updated at 23:45 UTC on 18 July 2024

Confirmed that the issue is due to Microsoft Azure. From their website:

Investigating issues in the Central US region

Impact Statement: Starting at approximately 21:56 UTC on 18 Jul 2024, a subset of customers may experience issues with multiple Azure services in the Central US region including failures with service management operations and connectivity or availability of services.

Current Status: We are aware of this issue and are actively investigating. The next update will be provided in 60 minutes, or as events warrant.

This message was last updated at 23:17 UTC on 18 July 2024