Medchat - 504 Errors During Authentication – Incident details

All systems operational

504 Errors During Authentication

Resolved
Major outage
Started over 2 years agoLasted about 2 hours

Affected

Authentication

Major outage from 7:14 PM to 9:30 PM

Medchat Auth Application

Major outage from 7:14 PM to 9:30 PM

Google SSO

Major outage from 7:14 PM to 9:30 PM

Custom OIDC SSO

Major outage from 7:14 PM to 9:30 PM

Custom SAML SSO

Major outage from 7:14 PM to 9:30 PM

Updates
  • Resolved
    UTC
    Resolved

    When today's outage began, connection limit exceptions started appearing in our telemetry data. By scaling the DB resources, we were able to increase our maximum number of connections, allowing the system to recover. Production support personnel are continuing to investigate automations, metrics, and proactive alerts that may be used to avoid similar problems in the future.

  • Monitoring
    UTC
    Monitoring

    The DB scaling operations are complete and the system appears to be returning to normal operations. We will continue to monitor and investigate the root cause of the unusual activity.

  • Investigating
    UTC
    Investigating

    Around 1:14 PM CST, MedChat customers began experiencing 504 errors when attempting to authenticate with MedChat. The outage is due to a exceptional and prolonged spike of database activity. The root cause is still under investigation. We have initiated scaling of DB resources to alleviate the outage.