Medchat Status - Incident history https://status.medchatapp.com Medchat Mon, 2 Mar 2026 17:47:09 +0000 Issues logging into Medchat Type: Incident Duration: 15 minutes Affected Components: Cloud Infrastructure, SMS Provider, Admin UI, Custom OIDC SSO, Admin UI, Web Application, Agent UI, Admin UI, Webhooks, Google SSO, Widget, Medchat Auth Application, Agent UI, Agent UI, Custom SAML SSO Mar 2, 17:47:09 GMT+0 - Investigating - We are currently investigating this incident. Mar 2, 17:52:38 GMT+0 - Identified - Users should be able to login successfully now. We are continuing to monitor the issue. Mar 2, 18:02:29 GMT+0 - Resolved - This incident has been resolved. Mar 2, 18:09:22 GMT+0 - Postmortem - **Date:** 3/2/26 **Service:** Production Application (Azure App Service) **Duration of Impact:** 9:34 AM PST – 9:49 AM PST **Impact:** Intermittent connection loss and service instability We experienced service instability affecting our production environment. The issue was traced to one of our Azure App Service resources within the active production environment. To restore stability quickly, we executed a blue/green environment swap, removing the impacted App Service from the production mix. Following the swap, services returned to normal operation and stability was restored. We are continuing to investigate the underlying cause of the App Service degradation. Type: Incident

Duration: 15 minutes

Affected Components: , , , , , , , , , , , , , ,

<p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>17:47:09</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>17:52:38</var> GMT+0</small><br><strong>Identified</strong> - Users should be able to login successfully now. We are continuing to monitor the issue..</p> <p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>18:02:29</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>18:09:22</var> GMT+0</small><br><strong>Postmortem</strong> - **Date:** 3/2/26 **Service:** Production Application (Azure App Service) **Duration of Impact:** 9:34 AM PST – 9:49 AM PST **Impact:** Intermittent connection loss and service instability We experienced service instability affecting our production environment. The issue was traced to one of our Azure App Service resources within the active production environment. To restore stability quickly, we executed a blue/green environment swap, removing the impacted App Service from the production mix. Following the swap, services returned to normal operation and stability was restored. We are continuing to investigate the underlying cause of the App Service degradation..</p> ]]> Mon, 2 Mar 2026 17:47:09 +0000 https://status.medchatapp.com/incident/cmm9h19ml07wz14nj76j4ch36 https://status.medchatapp.com/incident/cmm9h19ml07wz14nj76j4ch36 System login and performance issue Type: Incident Duration: 1 hour and 2 minutes Affected Components: , , Cloud Infrastructure, SMS Provider, Admin UI, Custom OIDC SSO, , , , Admin UI, , Web Application, Agent UI, Admin UI, Webhooks, Google SSO, Widget, Medchat Auth Application, Agent UI, Agent UI, Custom SAML SSO, Team Chat → Journeys → Live Chat → Text Chat → General → Authentication → Nov 13, 21:56:57 GMT+0 - Investigating - We are currently investigating this incident; sporadic system login and slowness -- likely caused by an outage in Azure. Nov 13, 22:17:57 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Nov 13, 22:58:28 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 2 minutes

Affected Components: , , , , , , , , , , , , , , , , , , , ,

<p><small>Nov <var data-var='date'> 13</var>, <var data-var='time'>21:56:57</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident; sporadic system login and slowness -- likely caused by an outage in Azure. .</p> <p><small>Nov <var data-var='date'> 13</var>, <var data-var='time'>22:17:57</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Nov <var data-var='date'> 13</var>, <var data-var='time'>22:58:28</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Thu, 13 Nov 2025 21:56:57 +0000 https://status.medchatapp.com/incident/cmhxyyp93085bvuvu6q5zx5zp https://status.medchatapp.com/incident/cmhxyyp93085bvuvu6q5zx5zp System performance Type: Incident Duration: 2 hours and 12 minutes Affected Components: Custom OIDC SSO, Cloud Infrastructure, Admin UI, SMS Provider, Google SSO, Medchat Auth Application, Custom SAML SSO, Agent UI, Admin UI, Widget, Admin UI, Agent UI, Web Application, Agent UI, Webhooks Jul 28, 17:22:00 GMT+0 - Identified - We are continuing to work on a fix for this incident. Jul 28, 16:00:23 GMT+0 - Investigating - We are currently investigating this incident. Jul 28, 17:45:00 GMT+0 - Monitoring - We have implemented mitigations to return to normal operations. We will continue to monitor the system over the next few hours to ensure all components are fully functional. Jul 28, 18:11:55 GMT+0 - Resolved - This incident has been resolved. Jul 29, 16:15:28 GMT+0 - Postmortem - Root cause was due to the application's capacity limits as a result of a recent increase in traffic. The system was operating as expected but required additional resources to support the elevated load. Resolution was to scale out the app's Azure infrastructure to accommodate the increased demand. Services have since stabilized, and team is continuing to monitor performance closely to ensure continued reliability. Type: Incident

Duration: 2 hours and 12 minutes

Affected Components: , , , , , , , , , , , , , ,

<p><small>Jul <var data-var='date'> 28</var>, <var data-var='time'>17:22:00</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> <p><small>Jul <var data-var='date'> 28</var>, <var data-var='time'>16:00:23</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jul <var data-var='date'> 28</var>, <var data-var='time'>17:45:00</var> GMT+0</small><br><strong>Monitoring</strong> - We have implemented mitigations to return to normal operations. We will continue to monitor the system over the next few hours to ensure all components are fully functional..</p> <p><small>Jul <var data-var='date'> 28</var>, <var data-var='time'>18:11:55</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jul <var data-var='date'> 29</var>, <var data-var='time'>16:15:28</var> GMT+0</small><br><strong>Postmortem</strong> - Root cause was due to the application's capacity limits as a result of a recent increase in traffic. The system was operating as expected but required additional resources to support the elevated load. Resolution was to scale out the app's Azure infrastructure to accommodate the increased demand. Services have since stabilized, and team is continuing to monitor performance closely to ensure continued reliability..</p> ]]> Mon, 28 Jul 2025 16:00:23 +0000 https://status.medchatapp.com/incident/cmdndm2jk00b613yp9atbjp05 https://status.medchatapp.com/incident/cmdndm2jk00b613yp9atbjp05 System outage Type: Incident Duration: 1 hour and 44 minutes Affected Components: Webhooks, Admin UI, Agent UI, Custom OIDC SSO, Web Application, Google SSO, Medchat Auth Application, Cloud Infrastructure, Agent UI, Admin UI, SMS Provider, Custom SAML SSO, Admin UI, Widget, Agent UI Feb 13, 22:11:00 GMT+0 - Identified - Around 4:11 PM CST, Medchat's authentication service became unresponsive after a planned release. The team identified widespread database connection issues in one application. After initial triage, Medchat reverted to the previous application versions, which resulted in the system becoming responsive again around 4:39 PM CST. Feb 13, 22:39:00 GMT+0 - Monitoring - Around 4:11 PM CST, Medchat's authentication service became unresponsive after a planned release. The team identified widespread database connection issues in one application. After initial triage, Medchat reverted to the previous application versions, which resulted in the system becoming responsive again around 4:39 PM CST. Feb 13, 23:55:29 GMT+0 - Resolved - We will continue to monitor the system to ensure operational status. Medchat will perform a more detailed analysis of the root cause within the next 24 hours. Type: Incident

Duration: 1 hour and 44 minutes

Affected Components: , , , , , , , , , , , , , ,

<p><small>Feb <var data-var='date'> 13</var>, <var data-var='time'>22:11:00</var> GMT+0</small><br><strong>Identified</strong> - Around 4:11 PM CST, Medchat's authentication service became unresponsive after a planned release. The team identified widespread database connection issues in one application. After initial triage, Medchat reverted to the previous application versions, which resulted in the system becoming responsive again around 4:39 PM CST..</p> <p><small>Feb <var data-var='date'> 13</var>, <var data-var='time'>22:39:00</var> GMT+0</small><br><strong>Monitoring</strong> - Around 4:11 PM CST, Medchat's authentication service became unresponsive after a planned release. The team identified widespread database connection issues in one application. After initial triage, Medchat reverted to the previous application versions, which resulted in the system becoming responsive again around 4:39 PM CST..</p> <p><small>Feb <var data-var='date'> 13</var>, <var data-var='time'>23:55:29</var> GMT+0</small><br><strong>Resolved</strong> - We will continue to monitor the system to ensure operational status. Medchat will perform a more detailed analysis of the root cause within the next 24 hours..</p> ]]> Thu, 13 Feb 2025 22:11:00 +0000 https://status.medchatapp.com/incident/cm73xs7sz0013en9kx5spxab5 https://status.medchatapp.com/incident/cm73xs7sz0013en9kx5spxab5 Application Service Interruption Type: Incident Duration: 6 hours and 13 minutes Affected Components: Webhooks, Admin UI, Agent UI, Custom OIDC SSO, Web Application, Google SSO, Medchat Auth Application, Cloud Infrastructure, Agent UI, Admin UI, SMS Provider, Custom SAML SSO, Admin UI, Widget, Agent UI Jul 18, 23:06:53 GMT+0 - Investigating - We are currently investigating this incident. At the moment, we're seeing widespread outage with Microsoft Azure, Medchat's Cloud Services provider. Microsoft has just sent out a status update that they are investigating issues in their Central US region. Jul 18, 23:20:47 GMT+0 - Identified - Confirmed that the issue is due to Microsoft Azure. From their website: --- **Investigating issues in the Central US region** **Impact Statement:** Starting at approximately 21:56 UTC on 18 Jul 2024, a subset of customers may experience issues with multiple Azure services in the Central US region including failures with service management operations and connectivity or availability of services. **Current Status:** We are aware of this issue and are actively investigating. The next update will be provided in 60 minutes, or as events warrant. This message was last updated at 23:17 UTC on 18 July 2024 Jul 18, 23:49:47 GMT+0 - Identified - Latest update from Microsoft @4:45 pm PT --- **Impact Statement:** Starting at 21:56 UTC on 18 Jul 2024, a subset of customers may experience issues with multiple Azure services in the Central US region including failures with service management operations and connectivity or availability of services. **Current Status:** We are aware of this issue and have engaged multiple teams to investigate. As part of the investigation, we are reviewing previous deployments, and are running other workstreams to investigate for an underlying cause. The next update will be provided in 60 minutes, or as events warrant. This message was last updated at 23:45 UTC on 18 July 2024 Jul 18, 23:57:05 GMT+0 - Identified - Update from Microsoft @4:56 pm PT: --- **Current status:** We have determined this issue was impacted by an underlying storage outage in the Central US region that the services were dependent upon. Once the underlying storage outage is mitigated, this impact will be resolved. The next update will be in 2 hours, or as events warrants. Jul 19, 01:32:58 GMT+0 - Monitoring - Microsoft update at 6:31 pm PT: --- **Current Status:** ... We’ve determined the underlying cause and are currently working towards mitigation. We will start to see incremental recovery in next 90 minutes. The next update will be provided in 60 minutes, or as events warrant. Jul 19, 02:51:14 GMT+0 - Monitoring - Microsoft update @7:33 pm PT * **Status:** Incident is now mitigated * **Next steps:** Engineers will continue to investigate to establish the full root cause and prevent future occurrences. --- We are currently performing checks on the Medchat Application to ensure overall system health; we will continue monitoring for another hour or so. Jul 19, 04:25:16 GMT+0 - Monitoring - While the Medchat application is back online and seems to be back in a fully functional state, Microsoft is continuing to send status updates (last at 9:10 pm PT) that the incident is still active, and that customers should continue to see increasing recovery at this time as residual and downstream impact mitigation progresses. Medchat team will continue to monitor system health and status updates. Jul 19, 05:19:34 GMT+0 - Resolved - While Microsoft is keeping their incident tickets open, they have confirmed that majority of their impacted services have now recovered. They have an updated dashboard where all services on the affected region are now back online. Root cause per Microsoft: The underlying cause was due to a backend cluster management workflow deployed a configuration change that caused backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region. This resulted in the compute resources automatically restarting when connectivity was lost to virtual disks. The Medchat team also conducted some health checks to ensure the system is back up and fully functional. Closing this incident. --- Type: Incident

Duration: 6 hours and 13 minutes

Affected Components: , , , , , , , , , , , , , ,

<p><small>Jul <var data-var='date'> 18</var>, <var data-var='time'>23:06:53</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. At the moment, we're seeing widespread outage with Microsoft Azure, Medchat's Cloud Services provider. Microsoft has just sent out a status update that they are investigating issues in their Central US region..</p> <p><small>Jul <var data-var='date'> 18</var>, <var data-var='time'>23:20:47</var> GMT+0</small><br><strong>Identified</strong> - Confirmed that the issue is due to Microsoft Azure. From their website: --- **Investigating issues in the Central US region** **Impact Statement:** Starting at approximately 21:56 UTC on 18 Jul 2024, a subset of customers may experience issues with multiple Azure services in the Central US region including failures with service management operations and connectivity or availability of services. **Current Status:** We are aware of this issue and are actively investigating. The next update will be provided in 60 minutes, or as events warrant. This message was last updated at 23:17 UTC on 18 July 2024.</p> <p><small>Jul <var data-var='date'> 18</var>, <var data-var='time'>23:49:47</var> GMT+0</small><br><strong>Identified</strong> - Latest update from Microsoft @4:45 pm PT --- **Impact Statement:** Starting at 21:56 UTC on 18 Jul 2024, a subset of customers may experience issues with multiple Azure services in the Central US region including failures with service management operations and connectivity or availability of services. **Current Status:** We are aware of this issue and have engaged multiple teams to investigate. As part of the investigation, we are reviewing previous deployments, and are running other workstreams to investigate for an underlying cause. The next update will be provided in 60 minutes, or as events warrant. This message was last updated at 23:45 UTC on 18 July 2024.</p> <p><small>Jul <var data-var='date'> 18</var>, <var data-var='time'>23:57:05</var> GMT+0</small><br><strong>Identified</strong> - Update from Microsoft @4:56 pm PT: --- **Current status:** We have determined this issue was impacted by an underlying storage outage in the Central US region that the services were dependent upon. Once the underlying storage outage is mitigated, this impact will be resolved. The next update will be in 2 hours, or as events warrants..</p> <p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>01:32:58</var> GMT+0</small><br><strong>Monitoring</strong> - Microsoft update at 6:31 pm PT: --- **Current Status:** ... We’ve determined the underlying cause and are currently working towards mitigation. We will start to see incremental recovery in next 90 minutes. The next update will be provided in 60 minutes, or as events warrant..</p> <p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>02:51:14</var> GMT+0</small><br><strong>Monitoring</strong> - Microsoft update @7:33 pm PT * **Status:** Incident is now mitigated * **Next steps:** Engineers will continue to investigate to establish the full root cause and prevent future occurrences. --- We are currently performing checks on the Medchat Application to ensure overall system health; we will continue monitoring for another hour or so..</p> <p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>04:25:16</var> GMT+0</small><br><strong>Monitoring</strong> - While the Medchat application is back online and seems to be back in a fully functional state, Microsoft is continuing to send status updates (last at 9:10 pm PT) that the incident is still active, and that customers should continue to see increasing recovery at this time as residual and downstream impact mitigation progresses. Medchat team will continue to monitor system health and status updates..</p> <p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>05:19:34</var> GMT+0</small><br><strong>Resolved</strong> - While Microsoft is keeping their incident tickets open, they have confirmed that majority of their impacted services have now recovered. They have an updated dashboard where all services on the affected region are now back online. Root cause per Microsoft: The underlying cause was due to a backend cluster management workflow deployed a configuration change that caused backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region. This resulted in the compute resources automatically restarting when connectivity was lost to virtual disks. The Medchat team also conducted some health checks to ensure the system is back up and fully functional. Closing this incident. ---.</p> ]]> Thu, 18 Jul 2024 23:06:53 +0000 https://status.medchatapp.com/incident/clyrvs6du57078hcofvy6bpklb https://status.medchatapp.com/incident/clyrvs6du57078hcofvy6bpklb 504 Errors During Authentication Type: Incident Duration: 2 hours and 16 minutes Affected Components: Custom SAML SSO, Custom OIDC SSO, Google SSO, Medchat Auth Application Jan 23, 19:14:00 GMT+0 - Investigating - Around 1:14 PM CST, MedChat customers began experiencing 504 errors when attempting to authenticate with MedChat. The outage is due to a exceptional and prolonged spike of database activity. The root cause is still under investigation. We have initiated scaling of DB resources to alleviate the outage. Jan 23, 19:56:00 GMT+0 - Monitoring - The DB scaling operations are complete and the system appears to be returning to normal operations. We will continue to monitor and investigate the root cause of the unusual activity. Jan 23, 21:30:08 GMT+0 - Resolved - When today's outage began, connection limit exceptions started appearing in our telemetry data. By scaling the DB resources, we were able to increase our maximum number of connections, allowing the system to recover. Production support personnel are continuing to investigate automations, metrics, and proactive alerts that may be used to avoid similar problems in the future. Type: Incident

Duration: 2 hours and 16 minutes

Affected Components: , , ,

<p><small>Jan <var data-var='date'> 23</var>, <var data-var='time'>19:14:00</var> GMT+0</small><br><strong>Investigating</strong> - Around 1:14 PM CST, MedChat customers began experiencing 504 errors when attempting to authenticate with MedChat. The outage is due to a exceptional and prolonged spike of database activity. The root cause is still under investigation. We have initiated scaling of DB resources to alleviate the outage..</p> <p><small>Jan <var data-var='date'> 23</var>, <var data-var='time'>19:56:00</var> GMT+0</small><br><strong>Monitoring</strong> - The DB scaling operations are complete and the system appears to be returning to normal operations. We will continue to monitor and investigate the root cause of the unusual activity..</p> <p><small>Jan <var data-var='date'> 23</var>, <var data-var='time'>21:30:08</var> GMT+0</small><br><strong>Resolved</strong> - When today's outage began, connection limit exceptions started appearing in our telemetry data. By scaling the DB resources, we were able to increase our maximum number of connections, allowing the system to recover. Production support personnel are continuing to investigate automations, metrics, and proactive alerts that may be used to avoid similar problems in the future. .</p> ]]> Tue, 23 Jan 2024 19:14:00 +0000 https://status.medchatapp.com/incident/clrqrxy51141929b8oi7lb3b5rl https://status.medchatapp.com/incident/clrqrxy51141929b8oi7lb3b5rl App Service Interruption Type: Incident Duration: 1 hour Affected Components: Webhooks, Admin UI, Widget, Custom SAML SSO, Agent UI, Custom OIDC SSO, Agent UI, Web Application, Google SSO, Medchat Auth Application, Cloud Infrastructure, Agent UI, Admin UI, SMS Provider, Admin UI Jan 13, 00:00:00 GMT+0 - Resolved - This incident has been resolved. Jan 12, 23:00:00 GMT+0 - Investigating - MedChat's App Services in the Azure East US region appear to be experiencing an outage resulting in 502 responses. We are preparing to swap services to West US to restore functionality. Jan 12, 23:30:00 GMT+0 - Monitoring - We have swapped our app services to the West US region. Initial checks show service has been restored. We will continue to monitor for lingering issues. Type: Incident

Duration: 1 hour

Affected Components: , , , , , , , , , , , , , ,

<p><small>Jan <var data-var='date'> 13</var>, <var data-var='time'>00:00:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jan <var data-var='date'> 12</var>, <var data-var='time'>23:00:00</var> GMT+0</small><br><strong>Investigating</strong> - MedChat's App Services in the Azure East US region appear to be experiencing an outage resulting in 502 responses. We are preparing to swap services to West US to restore functionality..</p> <p><small>Jan <var data-var='date'> 12</var>, <var data-var='time'>23:30:00</var> GMT+0</small><br><strong>Monitoring</strong> - We have swapped our app services to the West US region. Initial checks show service has been restored. We will continue to monitor for lingering issues..</p> ]]> Fri, 12 Jan 2024 23:00:00 +0000 https://status.medchatapp.com/incident/clrb9hcy26617ben4gtb0n1li https://status.medchatapp.com/incident/clrb9hcy26617ben4gtb0n1li Degraded keyword detection Type: Incident Duration: 4 hours and 33 minutes Affected Components: Agent UI Nov 8, 21:22:00 GMT+0 - Identified - An Azure hardware outage is affecting the keyword detection component of Live Chat. Azure is aware of the issue and is working to restore service. Nov 9, 00:59:54 GMT+0 - Monitoring - Affected Azure resources are coming back online. The keyword detection components are operational once again. We will continue monitoring the system for further disruptions. Nov 9, 01:55:02 GMT+0 - Resolved - All operations have returned to normal. Type: Incident

Duration: 4 hours and 33 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>21:22:00</var> GMT+0</small><br><strong>Identified</strong> - An Azure hardware outage is affecting the keyword detection component of Live Chat. Azure is aware of the issue and is working to restore service..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>00:59:54</var> GMT+0</small><br><strong>Monitoring</strong> - Affected Azure resources are coming back online. The keyword detection components are operational once again. We will continue monitoring the system for further disruptions..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>01:55:02</var> GMT+0</small><br><strong>Resolved</strong> - All operations have returned to normal..</p> ]]> Wed, 8 Nov 2023 21:22:00 +0000 https://status.medchatapp.com/incident/cloqcvbor178578beollzd392fn https://status.medchatapp.com/incident/cloqcvbor178578beollzd392fn Sporadic 502 errors Type: Incident Duration: 57 minutes Affected Components: Webhooks, Admin UI, Admin UI, Widget, Custom SAML SSO, Agent UI, Custom OIDC SSO, Agent UI, Web Application, Google SSO, Medchat Auth Application, Cloud Infrastructure, Agent UI, Admin UI, SMS Provider Oct 10, 14:54:00 GMT+0 - Investigating - The site is experiencing sporadic 502 errors on UI app services Oct 10, 15:51:00 GMT+0 - Resolved - All of MedChat's Azure app services (development through production) experienced issues between 9:54 AM CST to 10:51 AM CST where service availability was intermittent. The root cause is still unknown at this time, but all app services have returned to normal operation. Type: Incident

Duration: 57 minutes

Affected Components: , , , , , , , , , , , , , ,

<p><small>Oct <var data-var='date'> 10</var>, <var data-var='time'>14:54:00</var> GMT+0</small><br><strong>Investigating</strong> - The site is experiencing sporadic 502 errors on UI app services.</p> <p><small>Oct <var data-var='date'> 10</var>, <var data-var='time'>15:51:00</var> GMT+0</small><br><strong>Resolved</strong> - All of MedChat's Azure app services (development through production) experienced issues between 9:54 AM CST to 10:51 AM CST where service availability was intermittent. The root cause is still unknown at this time, but all app services have returned to normal operation..</p> ]]> Tue, 10 Oct 2023 14:54:00 +0000 https://status.medchatapp.com/incident/clnkils6789474c2n4fsb5iyrx https://status.medchatapp.com/incident/clnkils6789474c2n4fsb5iyrx System Inaccessible Type: Incident Duration: 1 day, 22 hours and 52 minutes Affected Components: Webhooks, Admin UI, Admin UI, Widget, Custom SAML SSO, Agent UI, Custom OIDC SSO, Agent UI, Web Application, Google SSO, Medchat Auth Application, Cloud Infrastructure, Agent UI, Admin UI, SMS Provider Oct 3, 16:35:00 GMT+0 - Investigating - We are currently investigating this incident. Users have been logged out of the application and are no longer able to sign in. Oct 3, 16:53:00 GMT+0 - Monitoring - The root cause appears to be a downtime from scaling out DB resources. The team is continuing to investigate the outage, which was not expected from the scaling operation. The scaling operation is now complete and the system appears to be back to normal operations. Continuing to monitor. Oct 3, 17:21:51 GMT+0 - Resolved - Throughout the morning of 10/3/23 and in the weeks leading up to this incident, the support team received automated alerts from the system regarding elevated (though not critical) DB resource consumption. The elevated DB consumption would quickly return to acceptable levels without affecting any users. After observing the pattern over several weeks, the support team applied a request to scale out the DB, an operation which was not expected to cause any downtime. The scaling operation took approximately 18 minutes, during which the system was largely inaccessible to MedChat users. While the databases were still available during the scaling operation, they appear to have had a reduced capacity (further investigation still in progress). Once scaling completed, the system returned to normal operations. In the future, all DB scaling will be performed off-hours to avoid the potential for service disruptions. Type: Incident

Duration: 1 day, 22 hours and 52 minutes

Affected Components: , , , , , , , , , , , , , ,

<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>16:35:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. Users have been logged out of the application and are no longer able to sign in..</p> <p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>16:53:00</var> GMT+0</small><br><strong>Monitoring</strong> - The root cause appears to be a downtime from scaling out DB resources. The team is continuing to investigate the outage, which was not expected from the scaling operation. The scaling operation is now complete and the system appears to be back to normal operations. Continuing to monitor..</p> <p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>17:21:51</var> GMT+0</small><br><strong>Resolved</strong> - Throughout the morning of 10/3/23 and in the weeks leading up to this incident, the support team received automated alerts from the system regarding elevated (though not critical) DB resource consumption. The elevated DB consumption would quickly return to acceptable levels without affecting any users. After observing the pattern over several weeks, the support team applied a request to scale out the DB, an operation which was not expected to cause any downtime. The scaling operation took approximately 18 minutes, during which the system was largely inaccessible to MedChat users. While the databases were still available during the scaling operation, they appear to have had a reduced capacity (further investigation still in progress). Once scaling completed, the system returned to normal operations. In the future, all DB scaling will be performed off-hours to avoid the potential for service disruptions..</p> ]]> Tue, 3 Oct 2023 16:35:00 +0000 https://status.medchatapp.com/incident/clnakbhr460680bdoeciduaivm https://status.medchatapp.com/incident/clnakbhr460680bdoeciduaivm