OBS Group Inc. - Notice history

OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS.

OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS.

OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS.

OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS.

OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS.

OBS Group Inc. experienced a global service disruption affecting multiple key platforms due to misconfigured cloud scaling resources. The incident impacted the following services:

1. OBS MIRD Research Cloud & APIs (Partial outage across Europe, US, and India)

2. OBS HyperScalar Zone D (Major global outage in London data center)

3. OBS Dreamer ID Services (Global outage of authentication systems)

The issues have been successfully resolved, and all services have been restored to normal operation.

Resolution Actions Taken

Immediate Remediation

1. Scaling Configuration Adjustment

   • Corrected misconfigured auto-scaling parameters across the affected platforms to restore proper resource allocation.

2. Service Restarts

   • Restarted critical components in OBS HyperScalar Zone D and Dreamer ID authentication services to resume functionality.

3. Performance Validation

   • Conducted comprehensive tests to ensure all services were operating as expected without residual issues.

Monitoring and Stabilization

• Deployed enhanced monitoring tools to track resource usage and performance metrics in real-time.

• Implemented temporary safeguards to prevent similar scaling misconfigurations while a long-term solution is developed.

Root Cause Summary

The incident was caused by a misconfiguration in cloud scaling policies, which:

1. Limited the system’s ability to allocate additional resources during peak demand.

2. Propagated resource shortages across dependent systems, leading to widespread outages and degraded performance.

Impact Overview

Service Impact Resolution OBS MIRD Research Cloud Partial outages and performance degradation across Europe, US, and India. Restored resource allocation. OBS MIRD APIs Latency and partial failures. Corrected scaling and validated APIs. HyperScalar Zone D Complete outage in London Zone D, affecting hosted workloads. Restarted services after scaling fix. Dreamer ID Services Global authentication failure, locking users out. Resolved configuration; resumed access.

Conclusion

OBS Group Inc. has fully resolved the service disruptions caused by the misconfigured cloud scaling resources. The organization has implemented immediate fixes and initiated long-term improvements to ensure service reliability and prevent similar issues in the future.

OBS Group Inc. apologizes for the inconvenience caused and appreciates the patience and understanding of our customers during this incident.

Root Cause Analysis

Cause

• A misconfiguration in the cloud auto-scaling settings inadvertently limited resource allocation, preventing the system from scaling to meet demand during peak usage.

• The issue propagated through dependent systems, leading to widespread disruptions.

Key Contributing Factors

1. MIRD Research Cloud and APIs

   • Insufficient compute and storage scaling in the affected regions (Europe, US, and India) caused performance degradation and partial outages.

2. HyperScalar Zone D (London)

   • The scaling misconfiguration resulted in resource starvation, triggering a complete outage in Zone D.

3. Dreamer ID Services

   • Due to automatic log-outs from authenticated devices lead to High authentication traffic which overwhelmed the misconfigured system, leading to a total global outage.

Impact Assessment

Business Impact

• Interrupted access to OBS services for research institutions, enterprises, and global customers.

• Affected user workflows, leading to potential financial losses for clients reliant on hosted services.

• Damage to OBS Group Inc.'s reputation and customer trust.

User Impact

• MIRD Research Cloud & APIs: Delayed or failed operations in data-intensive tasks across Europe, US, and India.

• HyperScalar Zone D: Complete downtime for workloads hosted in Zone D, affecting global operations.

• Dreamer ID Services: Inability to authenticate, locking users out of multiple OBS services worldwide.

Resolution Steps

Immediate Actions

1. Identified the misconfigured auto-scaling parameters in the affected services.

2. Adjusted scaling thresholds to allow for proper allocation of resources during high demand.

3. Restarted critical services in OBS HyperScalar Zone D and Dreamer ID authentication systems.

4. Conducted validation tests to ensure stability and restored access.

Long-Term Mitigation Measures

1. Implement automated monitoring and alerts for misconfigured scaling policies.

2. Conduct a comprehensive review of all cloud scaling configurations across OBS services.

3. Enhance load-testing protocols to simulate peak demand scenarios.

4. Provide additional training for engineering teams on cloud scaling best practices.

We have issued a resolution for this issue.

We have identified the error caused by a few obsolete config.yaml files trying to override the service account credentials.