Medchat - 504 Errors During Authentication – Incident details

All systems operational

Show current status

504 Errors During Authentication

ResolvedMajor outage2 hours 16 minutes

Affected

Authentication

Major outage from 7:14 PM to 9:30 PM

Medchat Auth Application

Major outage from 7:14 PM to 9:30 PM

Google SSO

Major outage from 7:14 PM to 9:30 PM

Custom OIDC SSO

Major outage from 7:14 PM to 9:30 PM

Custom SAML SSO

Major outage from 7:14 PM to 9:30 PM

  • ResolvedUTC

    When today's outage began, connection limit exceptions started appearing in our telemetry data. By scaling the DB resources, we were able to increase our maximum number of connections, allowing the system to recover. Production support personnel are continuing to investigate automations, metrics, and proactive alerts that may be used to avoid similar problems in the future.

  • MonitoringUTC

    The DB scaling operations are complete and the system appears to be returning to normal operations. We will continue to monitor and investigate the root cause of the unusual activity.

  • InvestigatingUTC

    Around 1:14 PM CST, MedChat customers began experiencing 504 errors when attempting to authenticate with MedChat. The outage is due to a exceptional and prolonged spike of database activity. The root cause is still under investigation. We have initiated scaling of DB resources to alleviate the outage.