Observation Group, Inc. Status - Incident history https://obs.instatus.com Observation Group, Inc. Fri, 24 Jan 2025 14:16:51 +0000 Degraded Performance for OBS Pay and OBS Pay APIs & Partial Outage for OBS SecuRA Runtime Environment and Encryption APIs Type: Incident Duration: 3 hours and 18 minutes Affected Components: OBS Pay , OBS Pay APIs, OBS SecuRA Runtime Environment , OBS SecuRA Encryption APIs, OBS MIRD Research Cloud Jan 24, 14:16:51 GMT+0 - Investigating - Incident Report Date: January 24, 2025 Time: 11:30 AM UTC Reported By: Incident Response Team Incident ID: IR-20250124-001 \--- Degraded Performance for OBS Pay and OBS Pay APIs & Partial Outage for OBS SecuRA Runtime Environment and Encryption APIs \--- Incident Timeline Detection: January 24, 2025, 9:00 AM UTC First User Report: January 24, 2025, 9:15 AM UTC Mitigation Initiated: January 24, 2025, 9:30 AM UTC Partial Restoration: January 24, 2025, 10:30 AM UTC Full Resolution (Estimated): January 24, 2025, 1:00 PM UTC \--- Affected Services 1\. Degraded Performance: OBS Pay OBS Pay APIs 2\. Partial Outage: OBS SecuRA Runtime Environment OBS SecuRA Encryption APIs \--- Incident Description OBS Pay and OBS Pay APIs: Between 9:00 AM and 11:00 AM UTC, users experienced significant delays in payment processing and intermittent API failures. The latency for processing payments increased by over 67.3%, and some API requests timed out. Initial diagnostics pointed to high CPU utilization on database servers due to an unexpected spike in transaction volume. OBS SecuRA Runtime Environment and Encryption APIs: A partial outage was detected for the SecuRA Runtime Environment and associated encryption APIs. Approximately 40% of API requests failed due to a service node experiencing connectivity issues with the central orchestration layer. This impacted encryption and decryption processes critical to secure transactions. \--- Impact Assessment 1\. OBS Pay and APIs: Users Affected: Estimated 25,000 transactions delayed or failed globally. Severity: Medium Financial Impact: Pending calculation based on transaction delays. 2\. OBS SecuRA Services: Scope: Approx. 40% of encryption requests affected for users relying on SecuRA APIs. Severity: High Security Impact: No evidence of data compromise; issue limited to service availability. \--- Root Cause Analysis OBS Pay and APIs: Primary Cause: Increased transaction volume caused database contention, leading to slow query responses and API timeouts. Contributing Factors: Insufficient auto-scaling thresholds for peak load management. OBS SecuRA Services: Primary Cause: A failed service node caused connectivity disruptions with the orchestration layer. Contributing Factors: Lack of failover readiness for the affected node and delayed health-check responses. Jan 24, 17:31:21 GMT+0 - Identified - Issues Identified in OBS Pay, OBS Pay APIs, OBS SecuRA Runtime Environment, and Encryption APIs --- ### **Summary of Issues Identified** **OBS Pay and OBS Pay APIs** 1. **Degraded Performance:** * Significant delays in transaction processing. * API requests experiencing timeouts and intermittent failures. 2. **Root Cause Identified:** * High database contention caused by an unexpected surge in transaction volume. * Inadequate auto-scaling thresholds for managing peak loads. **OBS SecuRA Runtime Environment and Encryption APIs** 1. **Partial Outage:** * Approximately 40% of encryption and decryption requests failed. * Service node connectivity issues disrupted secure operations. 2. **Root Cause Identified:** * Failure in a service node connecting to the central orchestration layer. * Insufficient failover readiness for the affected node. --- ### **Current Status** 1. **OBS Pay and APIs:** * Mitigations applied, and performance has improved as of 11:00 AM UTC. * Monitoring continues to ensure stability. 2. **OBS SecuRA Services:** * Partial restoration achieved by 10:30 AM UTC. * Full resolution is expected by 1:00 PM UTC. --- ### **Next Steps** 1. Enhance auto-scaling and load-balancing mechanisms for OBS Pay systems. 2. Implement improved failover and recovery mechanisms for OBS SecuRA nodes. 3. Conduct a post-incident analysis to address root causes and prevent recurrence. --- **We apologize for the inconvenience caused and appreciate your patience as we work towards a full resolution. For updates, please contact the Incident Response Team at incident\_response@engineering.obsgroup.tech** Jan 24, 17:34:47 GMT+0 - Resolved - Resolution of Issues Affecting OBS Pay, OBS Pay APIs, OBS SecuRA Runtime Environment, and Encryption APIs --- ### **Summary of Resolution** **OBS Pay and OBS Pay APIs** 1. **Issues Identified:** * Degraded performance, including transaction delays and API timeouts. 2. **Resolution Steps:** * Scaled up database and application server resources. * Applied temporary traffic throttling to stabilize performance. * Implemented enhanced monitoring to detect future spikes in real time. 3. **Status:** * Services restored and stable as of 11:00 AM UTC. **OBS SecuRA Runtime Environment and Encryption APIs** 1. **Issues Identified:** * Partial outage with approximately 40% of encryption and decryption requests failing. 2. **Resolution Steps:** * Replaced and restarted the failed service node. * Rerouted traffic to healthy nodes. * Increased monitoring granularity for critical nodes. 3. **Status:** * Full service restoration achieved at 1:00 PM UTC. --- ### **Impact Summary** 1. **OBS Pay and APIs:** * Approximately 25,000 transactions delayed or failed globally during the incident. * No data loss or security impact. 2. **OBS SecuRA Services:** * Affected approximately 40% of encryption requests for a subset of users. * No data compromise occurred; the issue was limited to availability. --- ### **Next Steps** 1. Conduct a detailed post-incident review and publish findings by January 26, 2025. 2. Enhance auto-scaling and failover mechanisms across all affected systems. 3. Perform stress testing to ensure systems handle peak loads without service degradation. 4. Roll out a robust communication plan to inform users about service improvements. --- ### **Acknowledgment** We sincerely apologize for the inconvenience caused during this incident and thank you for your patience and understanding as we worked towards a resolution. For any further concerns or support, please contact the Incident Response Team at incident\_response@engineering.obsgroup.tech Type: Incident

Duration: 3 hours and 18 minutes

Affected Components: , , , ,

<p><small>Jan <var data-var='date'> 24</var>, <var data-var='time'>14:16:51</var> GMT+0</small><br><strong>Investigating</strong> - Incident Report Date: January 24, 2025 Time: 11:30 AM UTC Reported By: Incident Response Team Incident ID: IR-20250124-001 \--- Degraded Performance for OBS Pay and OBS Pay APIs & Partial Outage for OBS SecuRA Runtime Environment and Encryption APIs \--- Incident Timeline Detection: January 24, 2025, 9:00 AM UTC First User Report: January 24, 2025, 9:15 AM UTC Mitigation Initiated: January 24, 2025, 9:30 AM UTC Partial Restoration: January 24, 2025, 10:30 AM UTC Full Resolution (Estimated): January 24, 2025, 1:00 PM UTC \--- Affected Services 1\. Degraded Performance: OBS Pay OBS Pay APIs 2\. Partial Outage: OBS SecuRA Runtime Environment OBS SecuRA Encryption APIs \--- Incident Description OBS Pay and OBS Pay APIs: Between 9:00 AM and 11:00 AM UTC, users experienced significant delays in payment processing and intermittent API failures. The latency for processing payments increased by over 67.3%, and some API requests timed out. Initial diagnostics pointed to high CPU utilization on database servers due to an unexpected spike in transaction volume. OBS SecuRA Runtime Environment and Encryption APIs: A partial outage was detected for the SecuRA Runtime Environment and associated encryption APIs. Approximately 40% of API requests failed due to a service node experiencing connectivity issues with the central orchestration layer. This impacted encryption and decryption processes critical to secure transactions. \--- Impact Assessment 1\. OBS Pay and APIs: Users Affected: Estimated 25,000 transactions delayed or failed globally. Severity: Medium Financial Impact: Pending calculation based on transaction delays. 2\. OBS SecuRA Services: Scope: Approx. 40% of encryption requests affected for users relying on SecuRA APIs. Severity: High Security Impact: No evidence of data compromise; issue limited to service availability. \--- Root Cause Analysis OBS Pay and APIs: Primary Cause: Increased transaction volume caused database contention, leading to slow query responses and API timeouts. Contributing Factors: Insufficient auto-scaling thresholds for peak load management. OBS SecuRA Services: Primary Cause: A failed service node caused connectivity disruptions with the orchestration layer. Contributing Factors: Lack of failover readiness for the affected node and delayed health-check responses..</p> <p><small>Jan <var data-var='date'> 24</var>, <var data-var='time'>17:31:21</var> GMT+0</small><br><strong>Identified</strong> - Issues Identified in OBS Pay, OBS Pay APIs, OBS SecuRA Runtime Environment, and Encryption APIs --- ### **Summary of Issues Identified** **OBS Pay and OBS Pay APIs** 1. **Degraded Performance:** * Significant delays in transaction processing. * API requests experiencing timeouts and intermittent failures. 2. **Root Cause Identified:** * High database contention caused by an unexpected surge in transaction volume. * Inadequate auto-scaling thresholds for managing peak loads. **OBS SecuRA Runtime Environment and Encryption APIs** 1. **Partial Outage:** * Approximately 40% of encryption and decryption requests failed. * Service node connectivity issues disrupted secure operations. 2. **Root Cause Identified:** * Failure in a service node connecting to the central orchestration layer. * Insufficient failover readiness for the affected node. --- ### **Current Status** 1. **OBS Pay and APIs:** * Mitigations applied, and performance has improved as of 11:00 AM UTC. * Monitoring continues to ensure stability. 2. **OBS SecuRA Services:** * Partial restoration achieved by 10:30 AM UTC. * Full resolution is expected by 1:00 PM UTC. --- ### **Next Steps** 1. Enhance auto-scaling and load-balancing mechanisms for OBS Pay systems. 2. Implement improved failover and recovery mechanisms for OBS SecuRA nodes. 3. Conduct a post-incident analysis to address root causes and prevent recurrence. --- **We apologize for the inconvenience caused and appreciate your patience as we work towards a full resolution. For updates, please contact the Incident Response Team at incident\_response@engineering.obsgroup.tech**.</p> <p><small>Jan <var data-var='date'> 24</var>, <var data-var='time'>17:34:47</var> GMT+0</small><br><strong>Resolved</strong> - Resolution of Issues Affecting OBS Pay, OBS Pay APIs, OBS SecuRA Runtime Environment, and Encryption APIs --- ### **Summary of Resolution** **OBS Pay and OBS Pay APIs** 1. **Issues Identified:** * Degraded performance, including transaction delays and API timeouts. 2. **Resolution Steps:** * Scaled up database and application server resources. * Applied temporary traffic throttling to stabilize performance. * Implemented enhanced monitoring to detect future spikes in real time. 3. **Status:** * Services restored and stable as of 11:00 AM UTC. **OBS SecuRA Runtime Environment and Encryption APIs** 1. **Issues Identified:** * Partial outage with approximately 40% of encryption and decryption requests failing. 2. **Resolution Steps:** * Replaced and restarted the failed service node. * Rerouted traffic to healthy nodes. * Increased monitoring granularity for critical nodes. 3. **Status:** * Full service restoration achieved at 1:00 PM UTC. --- ### **Impact Summary** 1. **OBS Pay and APIs:** * Approximately 25,000 transactions delayed or failed globally during the incident. * No data loss or security impact. 2. **OBS SecuRA Services:** * Affected approximately 40% of encryption requests for a subset of users. * No data compromise occurred; the issue was limited to availability. --- ### **Next Steps** 1. Conduct a detailed post-incident review and publish findings by January 26, 2025. 2. Enhance auto-scaling and failover mechanisms across all affected systems. 3. Perform stress testing to ensure systems handle peak loads without service degradation. 4. Roll out a robust communication plan to inform users about service improvements. --- ### **Acknowledgment** We sincerely apologize for the inconvenience caused during this incident and thank you for your patience and understanding as we worked towards a resolution. For any further concerns or support, please contact the Incident Response Team at incident\_response@engineering.obsgroup.tech.</p> ]]> Fri, 24 Jan 2025 14:16:51 +0000 https://obs.instatus.com/incident/cm6auig4m004iy11hlaifji94 https://obs.instatus.com/incident/cm6auig4m004iy11hlaifji94 Quarterly Scheduled Maintenance Type: Maintenance Duration: 17 days and 12 hours Affected Components: OBS ONE Alpha , OBS HSZ A HYD, OBS HSZ B BLR, OBS HSZ D LON, OBS HSZ E NYC, OBS Pay , OBS Pay APIs, OBS Dreamer ID Services, OBS Global CDN, OBS Anugraha Alpha, OBS SecuRA ZTE , OBS SecuRA Runtime Environment , OBS Serenity Workspace, OBS MIRD Research Cloud Jan 23, 23:20:00 GMT+0 - Identified - Dear Valued Users, We would like to inform you of a scheduled maintenance activity to enhance the performance and reliability of our systems. Affected Services: OBS MIRD Research Cloud OBS HyperScalar Zones A, B, and D Maintenance Schedule: Date: Friday, January 24, 2025 Time: 12:00 AM to 6:00 AM (UTC) Duration: 6 Hours During this maintenance window, access to the listed services may be temporarily unavailable or limited. We recommend saving your work and logging out of these systems prior to the maintenance period to avoid any disruptions. Purpose: This maintenance is aimed at upgrading infrastructure components, applying critical updates, and optimizing system performance to ensure seamless and secure operations. We apologize for any inconvenience caused and appreciate your understanding. If you have any questions or concerns, please contact our support team at support@theobservation.tech. Thank you for your cooperation. Reverence, OBS Engineering Jan 24, 06:20:00 GMT+0 - Completed - Maintenance has completed successfully Jan 23, 23:20:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 17 days and 12 hours

Affected Components: , , , , , , , , , , , , ,

<p><small>Jan <var data-var='date'> 23</var>, <var data-var='time'>23:20:00</var> GMT+0</small><br><strong>Identified</strong> - Dear Valued Users, We would like to inform you of a scheduled maintenance activity to enhance the performance and reliability of our systems. Affected Services: OBS MIRD Research Cloud OBS HyperScalar Zones A, B, and D Maintenance Schedule: Date: Friday, January 24, 2025 Time: 12:00 AM to 6:00 AM (UTC) Duration: 6 Hours During this maintenance window, access to the listed services may be temporarily unavailable or limited. We recommend saving your work and logging out of these systems prior to the maintenance period to avoid any disruptions. Purpose: This maintenance is aimed at upgrading infrastructure components, applying critical updates, and optimizing system performance to ensure seamless and secure operations. We apologize for any inconvenience caused and appreciate your understanding. If you have any questions or concerns, please contact our support team at support@theobservation.tech. Thank you for your cooperation. Reverence, OBS Engineering .</p> <p><small>Jan <var data-var='date'> 24</var>, <var data-var='time'>06:20:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Jan <var data-var='date'> 23</var>, <var data-var='time'>23:20:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Thu, 23 Jan 2025 23:20:00 +0000 https://obs.instatus.com/maintenance/cm69y692n001qgazioff937wl https://obs.instatus.com/maintenance/cm69y692n001qgazioff937wl OBS Official Webspace outage Type: Incident Duration: 2 minutes Affected Components: OBS Official Webspace Dec 11, 03:10:18 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Dec 11, 03:08:46 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Type: Incident

Duration: 2 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>03:10:18</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> <p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>03:08:46</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> ]]> Wed, 11 Dec 2024 03:08:46 +0000 https://obs.instatus.com/incident/cm4jb9qlu00q31051ty7gqz2w https://obs.instatus.com/incident/cm4jb9qlu00q31051ty7gqz2w OBS Official Webspace outage Type: Incident Duration: 1 minute Affected Components: OBS Official Webspace Dec 9, 20:07:43 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 9, 20:09:01 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Type: Incident

Duration: 1 minute

Affected Components:

<p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>20:07:43</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> <p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>20:09:01</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> ]]> Mon, 9 Dec 2024 20:07:43 +0000 https://obs.instatus.com/incident/cm4hgsef9005acf6v5mg0co45 https://obs.instatus.com/incident/cm4hgsef9005acf6v5mg0co45 OBS Official Webspace outage Type: Incident Duration: 1 minute Affected Components: OBS Official Webspace Dec 9, 13:58:29 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 9, 13:59:20 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Type: Incident

Duration: 1 minute

Affected Components:

<p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>13:58:29</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> <p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>13:59:20</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> ]]> Mon, 9 Dec 2024 13:58:29 +0000 https://obs.instatus.com/incident/cm4h3lki9000y10gm8384pv65 https://obs.instatus.com/incident/cm4h3lki9000y10gm8384pv65 OBS Official Webspace outage Type: Incident Duration: 1 minute Affected Components: OBS Official Webspace Dec 9, 01:39:55 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Dec 9, 01:39:05 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Type: Incident

Duration: 1 minute

Affected Components:

<p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>01:39:55</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> <p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>01:39:05</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> ]]> Mon, 9 Dec 2024 01:39:05 +0000 https://obs.instatus.com/incident/cm4gd6omk01h8hgqdehuodyya https://obs.instatus.com/incident/cm4gd6omk01h8hgqdehuodyya OBS Official Webspace outage Type: Incident Duration: 1 minute Affected Components: OBS Official Webspace Dec 8, 19:31:48 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 8, 19:32:35 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Type: Incident

Duration: 1 minute

Affected Components:

<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>19:31:48</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> <p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>19:32:35</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> ]]> Sun, 8 Dec 2024 19:31:48 +0000 https://obs.instatus.com/incident/cm4g02cqn0021100ugn81y6yr https://obs.instatus.com/incident/cm4g02cqn0021100ugn81y6yr OBS Official Webspace outage Type: Incident Duration: 1 hour and 49 minutes Affected Components: OBS Official Webspace Dec 7, 00:24:36 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 7, 00:26:25 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Type: Incident

Duration: 1 hour and 49 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>00:24:36</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> <p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>00:26:25</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> ]]> Sat, 7 Dec 2024 00:24:36 +0000 https://obs.instatus.com/incident/cm4dfn7n100bpiurz9fx21hov https://obs.instatus.com/incident/cm4dfn7n100bpiurz9fx21hov OBS Official Webspace outage Type: Incident Duration: 52 minutes Affected Components: OBS Official Webspace Dec 6, 18:23:49 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 6, 18:24:41 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Type: Incident

Duration: 52 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>18:23:49</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> <p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>18:24:41</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> ]]> Fri, 6 Dec 2024 18:23:49 +0000 https://obs.instatus.com/incident/cm4d2r8fa000kaf3jrbuxjayo https://obs.instatus.com/incident/cm4d2r8fa000kaf3jrbuxjayo OBS Official Webspace outage Type: Incident Duration: 1 hour and 37 minutes Affected Components: OBS Official Webspace Dec 6, 12:17:50 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 6, 12:19:27 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Type: Incident

Duration: 1 hour and 37 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>12:17:50</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> <p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>12:19:27</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> ]]> Fri, 6 Dec 2024 12:17:50 +0000 https://obs.instatus.com/incident/cm4cpokti000o1i0dv013294a https://obs.instatus.com/incident/cm4cpokti000o1i0dv013294a OBS Official Webspace outage Type: Incident Duration: 1 hour and 17 minutes Affected Components: OBS Official Webspace Dec 5, 18:06:49 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Dec 5, 18:05:33 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Type: Incident

Duration: 1 hour and 17 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>18:06:49</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> <p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>18:05:33</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> ]]> Thu, 5 Dec 2024 18:05:33 +0000 https://obs.instatus.com/incident/cm4bmnvm500a9my63ggwfyxt2 https://obs.instatus.com/incident/cm4bmnvm500a9my63ggwfyxt2 OBS Official Webspace outage Type: Incident Duration: 50 minutes Affected Components: OBS Official Webspace Dec 5, 05:48:46 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Dec 5, 05:47:56 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Type: Incident

Duration: 50 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>05:48:46</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> <p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>05:47:56</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> ]]> Thu, 5 Dec 2024 05:47:56 +0000 https://obs.instatus.com/incident/cm4awbaz000dq13k78lsk7kwz https://obs.instatus.com/incident/cm4awbaz000dq13k78lsk7kwz OBS Official Webspace outage Type: Incident Duration: 50 minutes Affected Components: OBS Official Webspace Dec 4, 17:33:25 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 4, 17:34:15 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Type: Incident

Duration: 50 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>17:33:25</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> <p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>17:34:15</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> ]]> Wed, 4 Dec 2024 17:33:25 +0000 https://obs.instatus.com/incident/cm4a62pzz00uxiopwlaoiht5e https://obs.instatus.com/incident/cm4a62pzz00uxiopwlaoiht5e OBS Official Webspace outage Type: Incident Duration: 2 hours and 6 minutes Affected Components: OBS Official Webspace Dec 4, 11:32:32 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Dec 4, 11:30:26 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Type: Incident

Duration: 2 hours and 6 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>11:32:32</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> <p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>11:30:26</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> ]]> Wed, 4 Dec 2024 11:30:26 +0000 https://obs.instatus.com/incident/cm49t3x13007ybcpnqpnivec3 https://obs.instatus.com/incident/cm49t3x13007ybcpnqpnivec3 OBS Official Webspace outage Type: Incident Duration: 1 hour and 36 minutes Affected Components: OBS Official Webspace Dec 3, 17:05:51 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 3, 17:07:26 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Type: Incident

Duration: 1 hour and 36 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>17:05:51</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> <p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>17:07:26</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> ]]> Tue, 3 Dec 2024 17:05:51 +0000 https://obs.instatus.com/incident/cm48pneoe005uin1z3l8avtnj https://obs.instatus.com/incident/cm48pneoe005uin1z3l8avtnj OBS Official Webspace outage Type: Incident Duration: 1 hour and 52 minutes Affected Components: OBS Official Webspace Dec 3, 11:03:59 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 3, 11:05:51 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Type: Incident

Duration: 1 hour and 52 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>11:03:59</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> <p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>11:05:51</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> ]]> Tue, 3 Dec 2024 11:03:59 +0000 https://obs.instatus.com/incident/cm48cq21k0001d1mkvpro82rf https://obs.instatus.com/incident/cm48cq21k0001d1mkvpro82rf OBS Official Webspace outage Type: Incident Duration: 2 hours and 47 minutes Affected Components: OBS Official Webspace Dec 3, 05:00:57 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 3, 05:03:44 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Type: Incident

Duration: 2 hours and 47 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>05:00:57</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> <p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>05:03:44</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> ]]> Tue, 3 Dec 2024 05:00:57 +0000 https://obs.instatus.com/incident/cm47zr6t700hx1zf1wngor8cs https://obs.instatus.com/incident/cm47zr6t700hx1zf1wngor8cs OBS Official Webspace outage Type: Incident Duration: 2 hours and 20 minutes Affected Components: OBS Official Webspace Dec 2, 22:53:55 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 2, 22:56:14 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Type: Incident

Duration: 2 hours and 20 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>22:53:55</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> <p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>22:56:14</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> ]]> Mon, 2 Dec 2024 22:53:55 +0000 https://obs.instatus.com/incident/cm47mn68s000csi67g86w2256 https://obs.instatus.com/incident/cm47mn68s000csi67g86w2256 OBS Official Webspace outage Type: Incident Duration: 1 hour and 20 minutes Affected Components: OBS Official Webspace Dec 2, 16:52:10 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 2, 16:53:30 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Type: Incident

Duration: 1 hour and 20 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>16:52:10</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> <p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>16:53:30</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> ]]> Mon, 2 Dec 2024 16:52:10 +0000 https://obs.instatus.com/incident/cm479pynq000c1429vhovkb31 https://obs.instatus.com/incident/cm479pynq000c1429vhovkb31 OBS Official Webspace outage Type: Incident Duration: 49 minutes Affected Components: OBS Official Webspace Dec 2, 10:50:56 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 2, 10:51:44 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Type: Incident

Duration: 49 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>10:50:56</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> <p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>10:51:44</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> ]]> Mon, 2 Dec 2024 10:50:56 +0000 https://obs.instatus.com/incident/cm46wteil00nc3nuyhotiebqx https://obs.instatus.com/incident/cm46wteil00nc3nuyhotiebqx OBS Official Webspace outage Type: Incident Duration: 1 hour and 37 minutes Affected Components: OBS Official Webspace Dec 1, 22:40:11 GMT+0 - Investigating - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 1, 22:41:48 GMT+0 - Resolved - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS. Type: Incident

Duration: 1 hour and 37 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>22:40:11</var> GMT+0</small><br><strong>Investigating</strong> - OBS Official Webspace cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> <p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>22:41:48</var> GMT+0</small><br><strong>Resolved</strong> - OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS..</p> ]]> Sun, 1 Dec 2024 22:40:11 +0000 https://obs.instatus.com/incident/cm466po2v00aensuep7031qwh https://obs.instatus.com/incident/cm466po2v00aensuep7031qwh Global Service Disruption: Partial Outages in OBS MIRD Research Cloud & APIs, Major Outages in OBS HyperScalar Zone D and Dreamer ID Services Type: Incident Duration: 6 hours and 14 minutes Affected Components: OBS HSZ D LON, OBS MIRD Research Cloud , OBS Dreamer ID Services, OBS MIRD Research Cloud APIs Nov 30, 11:21:52 GMT+0 - Investigating - A series of service disruptions and outages occurred across several OBS Group Inc. platforms, impacting users globally. The incidents are currently under investigation, and updates will be provided as they become available. Below is a summary of the affected services and regions: 1. **OBS MIRD Research Cloud** * **Impact**: Partial outage. * **Affected Regions**: Europe, the United States, and India. * **Description**: Users in the mentioned regions experienced degraded performance and intermittent access issues. 2. **OBS MIRD Research Cloud APIs** * **Impact**: Partial outage. * **Affected Regions**: Europe, the United States, and India. * **Description**: APIs associated with MIRD Research Cloud are reporting latency issues and partial failures, impacting integrations with external systems. 3. **OBS HyperScalar Zone D (London)** * **Impact**: Major global outage. * **Affected Region**: Zone D (London). * **Description**: All services in this zone are offline, resulting in significant disruptions for hosted workloads and applications. 4. **OBS Dreamer ID Services** * **Impact**: Major global outage. * **Affected Region**: Global. * **Description**: Dreamer ID authentication services are entirely unavailable, causing login and access issues for users worldwide. --- ### **Initial Investigation** 1. **Observations** * Service degradation in OBS MIRD Research Cloud and APIs began at 10PM -13:30UTC. * The global outage of OBS HyperScalar Zone D (London) and Dreamer ID Services was detected shortly after, at approximately 2 AM -13:30UTC. 2. **Potential Causes** * Network connectivity issues across multiple regions. * Possible hardware failure or power issues at the London data center (Zone D). * A systemic failure in Dreamer ID's authentication infrastructure. * Dependencies between services may have propagated the impact. 3. **Current Status** * Teams are actively investigating root causes for each impacted service. * Mitigation steps are being planned and implemented for services where partial functionality can be restored. --- ### **Impact Assessment** * **Business Impact** * Disrupted access to key research and cloud services, affecting academic and enterprise users. * Global inability to log into services using Dreamer ID. * Critical operations hosted in Zone D are offline, leading to delays and potential financial loss for clients. * **User Impact** * Limited or no access to cloud-based resources. * Interruptions in data processing and collaboration workflows. * Complete loss of authentication functionality, preventing access to all dependent services. --- ### **Next Steps** 1. **Ongoing Investigation** * Incident response teams are analyzing logs and conducting a root cause analysis. * Collaboration with regional data centers and network providers is underway. 2. **Service Restoration** * Teams are prioritizing partial recovery for the MIRD Research Cloud and APIs. * HyperScalar Zone D is under review for potential hardware fixes or system restarts. * Engineering teams are working to restore Dreamer ID Services globally. 3. **Communications** * Regular updates will be shared with affected users and stakeholders. * An incident retrospective will be conducted after service restoration to identify and implement preventative measures. --- ### **Contacts** * **Incident Manager**: James Dupont * **Technical Lead**: Sarthak Videet * **Customer Support**: support@obsgroup.tech --- ### **Final Note** OBS Group Inc. is committed to resolving these issues as quickly as possible. We apologize for the inconvenience caused and appreciate your patience during this time. Nov 30, 14:50:20 GMT+0 - Identified - ### **Root Cause Analysis** **Cause** * A misconfiguration in the cloud auto-scaling settings inadvertently limited resource allocation, preventing the system from scaling to meet demand during peak usage. * The issue propagated through dependent systems, leading to widespread disruptions. **Key Contributing Factors** 1. **MIRD Research Cloud and APIs** * Insufficient compute and storage scaling in the affected regions (Europe, US, and India) caused performance degradation and partial outages. 2. **HyperScalar Zone D (London)** * The scaling misconfiguration resulted in resource starvation, triggering a complete outage in Zone D. 3. **Dreamer ID Services** * Due to automatic log-outs from authenticated devices lead to High authentication traffic which overwhelmed the misconfigured system, leading to a total global outage. --- ### **Impact Assessment** **Business Impact** * Interrupted access to OBS services for research institutions, enterprises, and global customers. * Affected user workflows, leading to potential financial losses for clients reliant on hosted services. * Damage to OBS Group Inc.'s reputation and customer trust. **User Impact** * **MIRD Research Cloud & APIs**: Delayed or failed operations in data-intensive tasks across Europe, US, and India. * **HyperScalar Zone D**: Complete downtime for workloads hosted in Zone D, affecting global operations. * **Dreamer ID Services**: Inability to authenticate, locking users out of multiple OBS services worldwide. --- ### **Resolution Steps** **Immediate Actions** 1. Identified the misconfigured auto-scaling parameters in the affected services. 2. Adjusted scaling thresholds to allow for proper allocation of resources during high demand. 3. Restarted critical services in OBS HyperScalar Zone D and Dreamer ID authentication systems. 4. Conducted validation tests to ensure stability and restored access. **Long-Term Mitigation Measures** 1. Implement automated monitoring and alerts for misconfigured scaling policies. 2. Conduct a comprehensive review of all cloud scaling configurations across OBS services. 3. Enhance load-testing protocols to simulate peak demand scenarios. 4. Provide additional training for engineering teams on cloud scaling best practices. Nov 30, 17:36:03 GMT+0 - Resolved - OBS Group Inc. experienced a global service disruption affecting multiple key platforms due to **misconfigured cloud scaling resources**. The incident impacted the following services: 1. **OBS MIRD Research Cloud & APIs** (Partial outage across Europe, US, and India) 2. **OBS HyperScalar Zone D** (Major global outage in London data center) 3. **OBS Dreamer ID Services** (Global outage of authentication systems) The issues have been successfully resolved, and all services have been restored to normal operation. --- ### **Resolution Actions Taken** **Immediate Remediation** 1. **Scaling Configuration Adjustment** * Corrected misconfigured auto-scaling parameters across the affected platforms to restore proper resource allocation. 2. **Service Restarts** * Restarted critical components in OBS HyperScalar Zone D and Dreamer ID authentication services to resume functionality. 3. **Performance Validation** * Conducted comprehensive tests to ensure all services were operating as expected without residual issues. **Monitoring and Stabilization** * Deployed enhanced monitoring tools to track resource usage and performance metrics in real-time. * Implemented temporary safeguards to prevent similar scaling misconfigurations while a long-term solution is developed. --- ### **Root Cause Summary** The incident was caused by a misconfiguration in cloud scaling policies, which: 1. Limited the system’s ability to allocate additional resources during peak demand. 2. Propagated resource shortages across dependent systems, leading to widespread outages and degraded performance. --- ### **Impact Overview** **Service** **Impact** **Resolution** **OBS MIRD Research Cloud** Partial outages and performance degradation across Europe, US, and India. Restored resource allocation. **OBS MIRD APIs** Latency and partial failures. Corrected scaling and validated APIs. **HyperScalar Zone D** Complete outage in London Zone D, affecting hosted workloads. Restarted services after scaling fix. **Dreamer ID Services** Global authentication failure, locking users out. Resolved configuration; resumed access. --- ### **Conclusion** OBS Group Inc. has fully resolved the service disruptions caused by the misconfigured cloud scaling resources. The organization has implemented immediate fixes and initiated long-term improvements to ensure service reliability and prevent similar issues in the future. OBS Group Inc. apologizes for the inconvenience caused and appreciates the patience and understanding of our customers during this incident. --- ### Type: Incident

Duration: 6 hours and 14 minutes

Affected Components: , , ,

<p><small>Nov <var data-var='date'> 30</var>, <var data-var='time'>11:21:52</var> GMT+0</small><br><strong>Investigating</strong> - A series of service disruptions and outages occurred across several OBS Group Inc. platforms, impacting users globally. The incidents are currently under investigation, and updates will be provided as they become available. Below is a summary of the affected services and regions: 1. **OBS MIRD Research Cloud** * **Impact**: Partial outage. * **Affected Regions**: Europe, the United States, and India. * **Description**: Users in the mentioned regions experienced degraded performance and intermittent access issues. 2. **OBS MIRD Research Cloud APIs** * **Impact**: Partial outage. * **Affected Regions**: Europe, the United States, and India. * **Description**: APIs associated with MIRD Research Cloud are reporting latency issues and partial failures, impacting integrations with external systems. 3. **OBS HyperScalar Zone D (London)** * **Impact**: Major global outage. * **Affected Region**: Zone D (London). * **Description**: All services in this zone are offline, resulting in significant disruptions for hosted workloads and applications. 4. **OBS Dreamer ID Services** * **Impact**: Major global outage. * **Affected Region**: Global. * **Description**: Dreamer ID authentication services are entirely unavailable, causing login and access issues for users worldwide. --- ### **Initial Investigation** 1. **Observations** * Service degradation in OBS MIRD Research Cloud and APIs began at 10PM -13:30UTC. * The global outage of OBS HyperScalar Zone D (London) and Dreamer ID Services was detected shortly after, at approximately 2 AM -13:30UTC. 2. **Potential Causes** * Network connectivity issues across multiple regions. * Possible hardware failure or power issues at the London data center (Zone D). * A systemic failure in Dreamer ID's authentication infrastructure. * Dependencies between services may have propagated the impact. 3. **Current Status** * Teams are actively investigating root causes for each impacted service. * Mitigation steps are being planned and implemented for services where partial functionality can be restored. --- ### **Impact Assessment** * **Business Impact** * Disrupted access to key research and cloud services, affecting academic and enterprise users. * Global inability to log into services using Dreamer ID. * Critical operations hosted in Zone D are offline, leading to delays and potential financial loss for clients. * **User Impact** * Limited or no access to cloud-based resources. * Interruptions in data processing and collaboration workflows. * Complete loss of authentication functionality, preventing access to all dependent services. --- ### **Next Steps** 1. **Ongoing Investigation** * Incident response teams are analyzing logs and conducting a root cause analysis. * Collaboration with regional data centers and network providers is underway. 2. **Service Restoration** * Teams are prioritizing partial recovery for the MIRD Research Cloud and APIs. * HyperScalar Zone D is under review for potential hardware fixes or system restarts. * Engineering teams are working to restore Dreamer ID Services globally. 3. **Communications** * Regular updates will be shared with affected users and stakeholders. * An incident retrospective will be conducted after service restoration to identify and implement preventative measures. --- ### **Contacts** * **Incident Manager**: James Dupont * **Technical Lead**: Sarthak Videet * **Customer Support**: support@obsgroup.tech --- ### **Final Note** OBS Group Inc. is committed to resolving these issues as quickly as possible. We apologize for the inconvenience caused and appreciate your patience during this time..</p> <p><small>Nov <var data-var='date'> 30</var>, <var data-var='time'>14:50:20</var> GMT+0</small><br><strong>Identified</strong> - ### **Root Cause Analysis** **Cause** * A misconfiguration in the cloud auto-scaling settings inadvertently limited resource allocation, preventing the system from scaling to meet demand during peak usage. * The issue propagated through dependent systems, leading to widespread disruptions. **Key Contributing Factors** 1. **MIRD Research Cloud and APIs** * Insufficient compute and storage scaling in the affected regions (Europe, US, and India) caused performance degradation and partial outages. 2. **HyperScalar Zone D (London)** * The scaling misconfiguration resulted in resource starvation, triggering a complete outage in Zone D. 3. **Dreamer ID Services** * Due to automatic log-outs from authenticated devices lead to High authentication traffic which overwhelmed the misconfigured system, leading to a total global outage. --- ### **Impact Assessment** **Business Impact** * Interrupted access to OBS services for research institutions, enterprises, and global customers. * Affected user workflows, leading to potential financial losses for clients reliant on hosted services. * Damage to OBS Group Inc.'s reputation and customer trust. **User Impact** * **MIRD Research Cloud & APIs**: Delayed or failed operations in data-intensive tasks across Europe, US, and India. * **HyperScalar Zone D**: Complete downtime for workloads hosted in Zone D, affecting global operations. * **Dreamer ID Services**: Inability to authenticate, locking users out of multiple OBS services worldwide. --- ### **Resolution Steps** **Immediate Actions** 1. Identified the misconfigured auto-scaling parameters in the affected services. 2. Adjusted scaling thresholds to allow for proper allocation of resources during high demand. 3. Restarted critical services in OBS HyperScalar Zone D and Dreamer ID authentication systems. 4. Conducted validation tests to ensure stability and restored access. **Long-Term Mitigation Measures** 1. Implement automated monitoring and alerts for misconfigured scaling policies. 2. Conduct a comprehensive review of all cloud scaling configurations across OBS services. 3. Enhance load-testing protocols to simulate peak demand scenarios. 4. Provide additional training for engineering teams on cloud scaling best practices..</p> <p><small>Nov <var data-var='date'> 30</var>, <var data-var='time'>17:36:03</var> GMT+0</small><br><strong>Resolved</strong> - OBS Group Inc. experienced a global service disruption affecting multiple key platforms due to **misconfigured cloud scaling resources**. The incident impacted the following services: 1. **OBS MIRD Research Cloud & APIs** (Partial outage across Europe, US, and India) 2. **OBS HyperScalar Zone D** (Major global outage in London data center) 3. **OBS Dreamer ID Services** (Global outage of authentication systems) The issues have been successfully resolved, and all services have been restored to normal operation. --- ### **Resolution Actions Taken** **Immediate Remediation** 1. **Scaling Configuration Adjustment** * Corrected misconfigured auto-scaling parameters across the affected platforms to restore proper resource allocation. 2. **Service Restarts** * Restarted critical components in OBS HyperScalar Zone D and Dreamer ID authentication services to resume functionality. 3. **Performance Validation** * Conducted comprehensive tests to ensure all services were operating as expected without residual issues. **Monitoring and Stabilization** * Deployed enhanced monitoring tools to track resource usage and performance metrics in real-time. * Implemented temporary safeguards to prevent similar scaling misconfigurations while a long-term solution is developed. --- ### **Root Cause Summary** The incident was caused by a misconfiguration in cloud scaling policies, which: 1. Limited the system’s ability to allocate additional resources during peak demand. 2. Propagated resource shortages across dependent systems, leading to widespread outages and degraded performance. --- ### **Impact Overview** **Service** **Impact** **Resolution** **OBS MIRD Research Cloud** Partial outages and performance degradation across Europe, US, and India. Restored resource allocation. **OBS MIRD APIs** Latency and partial failures. Corrected scaling and validated APIs. **HyperScalar Zone D** Complete outage in London Zone D, affecting hosted workloads. Restarted services after scaling fix. **Dreamer ID Services** Global authentication failure, locking users out. Resolved configuration; resumed access. --- ### **Conclusion** OBS Group Inc. has fully resolved the service disruptions caused by the misconfigured cloud scaling resources. The organization has implemented immediate fixes and initiated long-term improvements to ensure service reliability and prevent similar issues in the future. OBS Group Inc. apologizes for the inconvenience caused and appreciates the patience and understanding of our customers during this incident. --- ### .</p> ]]> Sat, 30 Nov 2024 11:21:52 +0000 https://obs.instatus.com/incident/cm4431irm004nbkqkckh6of42 https://obs.instatus.com/incident/cm4431irm004nbkqkckh6of42 We are aware of delays in syncing development information on OBS Serenity Workspace, selected Datacenter OBS HZD-LON where information may take up to an hour to appear in other devices. We are scaling up our services to deal with the load and reduce processing times. Type: Incident Duration: 4 hours and 30 minutes Affected Components: OBS HSZ D LON, OBS Security Motion APIs , OBS HSZ C BOM, OBS Serenity Workspace Oct 2, 18:01:00 GMT+0 - Investigating - We are currently investigating this incident and are identifying areas to improve the synchronization consensus algorithm. Oct 2, 20:28:00 GMT+0 - Identified - We have identified the error caused by a few obsolete config.yaml files trying to override the service account credentials. Oct 2, 22:30:41 GMT+0 - Resolved - We have issued a resolution for this issue. Type: Incident

Duration: 4 hours and 30 minutes

Affected Components: , , ,

<p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>18:01:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident and are identifying areas to improve the synchronization consensus algorithm. .</p> <p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>20:28:00</var> GMT+0</small><br><strong>Identified</strong> - We have identified the error caused by a few obsolete config.yaml files trying to override the service account credentials. .</p> <p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>22:30:41</var> GMT+0</small><br><strong>Resolved</strong> - We have issued a resolution for this issue. .</p> ]]> Wed, 2 Oct 2024 18:01:00 +0000 https://obs.instatus.com/incident/cm1sf38g4000t1lffuoksl8er https://obs.instatus.com/incident/cm1sf38g4000t1lffuoksl8er