Medchat - Notice history

All operations have returned to normal.

Affected Azure resources are coming back online. The keyword detection components are operational once again. We will continue monitoring the system for further disruptions.

All of MedChat's Azure app services (development through production) experienced issues between 9:54 AM CST to 10:51 AM CST where service availability was intermittent. The root cause is still unknown at this time, but all app services have returned to normal operation.

Throughout the morning of 10/3/23 and in the weeks leading up to this incident, the support team received automated alerts from the system regarding elevated (though not critical) DB resource consumption. The elevated DB consumption would quickly return to acceptable levels without affecting any users.

After observing the pattern over several weeks, the support team applied a request to scale out the DB, an operation which was not expected to cause any downtime. The scaling operation took approximately 18 minutes, during which the system was largely inaccessible to MedChat users. While the databases were still available during the scaling operation, they appear to have had a reduced capacity (further investigation still in progress). Once scaling completed, the system returned to normal operations.

In the future, all DB scaling will be performed off-hours to avoid the potential for service disruptions.

The root cause appears to be a downtime from scaling out DB resources. The team is continuing to investigate the outage, which was not expected from the scaling operation.

The scaling operation is now complete and the system appears to be back to normal operations. Continuing to monitor.