OBS Group Inc. - Notice history

Resolution of Issues Affecting OBS Pay, OBS Pay APIs, OBS SecuRA Runtime Environment, and Encryption APIs

Summary of Resolution

OBS Pay and OBS Pay APIs

1. Issues Identified:

   • Degraded performance, including transaction delays and API timeouts.

2. Resolution Steps:

   • Scaled up database and application server resources.

   • Applied temporary traffic throttling to stabilize performance.

   • Implemented enhanced monitoring to detect future spikes in real time.

3. Status:

   • Services restored and stable as of 11:00 AM UTC.

OBS SecuRA Runtime Environment and Encryption APIs

1. Issues Identified:

   • Partial outage with approximately 40% of encryption and decryption requests failing.

2. Resolution Steps:

   • Replaced and restarted the failed service node.

   • Rerouted traffic to healthy nodes.

   • Increased monitoring granularity for critical nodes.

3. Status:

   • Full service restoration achieved at 1:00 PM UTC.

Impact Summary

1. OBS Pay and APIs:

   • Approximately 25,000 transactions delayed or failed globally during the incident.

   • No data loss or security impact.

2. OBS SecuRA Services:

   • Affected approximately 40% of encryption requests for a subset of users.

   • No data compromise occurred; the issue was limited to availability.

Next Steps

1. Conduct a detailed post-incident review and publish findings by January 26, 2025.

2. Enhance auto-scaling and failover mechanisms across all affected systems.

3. Perform stress testing to ensure systems handle peak loads without service degradation.

4. Roll out a robust communication plan to inform users about service improvements.

Acknowledgment

We sincerely apologize for the inconvenience caused during this incident and thank you for your patience and understanding as we worked towards a resolution.

For any further concerns or support, please contact the Incident Response Team at incident_response@engineering.obsgroup.tech

Issues Identified in OBS Pay, OBS Pay APIs, OBS SecuRA Runtime Environment, and Encryption APIs

Summary of Issues Identified

OBS Pay and OBS Pay APIs

1. Degraded Performance:

   • Significant delays in transaction processing.

   • API requests experiencing timeouts and intermittent failures.

2. Root Cause Identified:

   • High database contention caused by an unexpected surge in transaction volume.

   • Inadequate auto-scaling thresholds for managing peak loads.

OBS SecuRA Runtime Environment and Encryption APIs

1. Partial Outage:

   • Approximately 40% of encryption and decryption requests failed.

   • Service node connectivity issues disrupted secure operations.

2. Root Cause Identified:

   • Failure in a service node connecting to the central orchestration layer.

   • Insufficient failover readiness for the affected node.

Current Status

1. OBS Pay and APIs:

   • Mitigations applied, and performance has improved as of 11:00 AM UTC.

   • Monitoring continues to ensure stability.

2. OBS SecuRA Services:

   • Partial restoration achieved by 10:30 AM UTC.

   • Full resolution is expected by 1:00 PM UTC.

Next Steps

1. Enhance auto-scaling and load-balancing mechanisms for OBS Pay systems.

2. Implement improved failover and recovery mechanisms for OBS SecuRA nodes.

3. Conduct a post-incident analysis to address root causes and prevent recurrence.

We apologize for the inconvenience caused and appreciate your patience as we work towards a full resolution. For updates, please contact the Incident Response Team at incident_response@engineering.obsgroup.tech

OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS.

OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS.

OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS.

OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS.

OBS Official Webspace is now operational! This update was created by an automated monitoring service of OBS.

OBS Group Inc. experienced a global service disruption affecting multiple key platforms due to misconfigured cloud scaling resources. The incident impacted the following services:

1. OBS MIRD Research Cloud & APIs (Partial outage across Europe, US, and India)

2. OBS HyperScalar Zone D (Major global outage in London data center)

3. OBS Dreamer ID Services (Global outage of authentication systems)

The issues have been successfully resolved, and all services have been restored to normal operation.

Resolution Actions Taken

Immediate Remediation

1. Scaling Configuration Adjustment

   • Corrected misconfigured auto-scaling parameters across the affected platforms to restore proper resource allocation.

2. Service Restarts

   • Restarted critical components in OBS HyperScalar Zone D and Dreamer ID authentication services to resume functionality.

3. Performance Validation

   • Conducted comprehensive tests to ensure all services were operating as expected without residual issues.

Monitoring and Stabilization

• Deployed enhanced monitoring tools to track resource usage and performance metrics in real-time.

• Implemented temporary safeguards to prevent similar scaling misconfigurations while a long-term solution is developed.

Root Cause Summary

The incident was caused by a misconfiguration in cloud scaling policies, which:

1. Limited the system’s ability to allocate additional resources during peak demand.

2. Propagated resource shortages across dependent systems, leading to widespread outages and degraded performance.

Impact Overview

Service Impact Resolution OBS MIRD Research Cloud Partial outages and performance degradation across Europe, US, and India. Restored resource allocation. OBS MIRD APIs Latency and partial failures. Corrected scaling and validated APIs. HyperScalar Zone D Complete outage in London Zone D, affecting hosted workloads. Restarted services after scaling fix. Dreamer ID Services Global authentication failure, locking users out. Resolved configuration; resumed access.

Conclusion

OBS Group Inc. has fully resolved the service disruptions caused by the misconfigured cloud scaling resources. The organization has implemented immediate fixes and initiated long-term improvements to ensure service reliability and prevent similar issues in the future.

OBS Group Inc. apologizes for the inconvenience caused and appreciates the patience and understanding of our customers during this incident.

Root Cause Analysis

Cause

• A misconfiguration in the cloud auto-scaling settings inadvertently limited resource allocation, preventing the system from scaling to meet demand during peak usage.

• The issue propagated through dependent systems, leading to widespread disruptions.

Key Contributing Factors

1. MIRD Research Cloud and APIs

   • Insufficient compute and storage scaling in the affected regions (Europe, US, and India) caused performance degradation and partial outages.

2. HyperScalar Zone D (London)

   • The scaling misconfiguration resulted in resource starvation, triggering a complete outage in Zone D.

3. Dreamer ID Services

   • Due to automatic log-outs from authenticated devices lead to High authentication traffic which overwhelmed the misconfigured system, leading to a total global outage.

Impact Assessment

Business Impact

• Interrupted access to OBS services for research institutions, enterprises, and global customers.

• Affected user workflows, leading to potential financial losses for clients reliant on hosted services.

• Damage to OBS Group Inc.'s reputation and customer trust.

User Impact

• MIRD Research Cloud & APIs: Delayed or failed operations in data-intensive tasks across Europe, US, and India.

• HyperScalar Zone D: Complete downtime for workloads hosted in Zone D, affecting global operations.

• Dreamer ID Services: Inability to authenticate, locking users out of multiple OBS services worldwide.

Resolution Steps

Immediate Actions

1. Identified the misconfigured auto-scaling parameters in the affected services.

2. Adjusted scaling thresholds to allow for proper allocation of resources during high demand.

3. Restarted critical services in OBS HyperScalar Zone D and Dreamer ID authentication systems.

4. Conducted validation tests to ensure stability and restored access.

Long-Term Mitigation Measures

1. Implement automated monitoring and alerts for misconfigured scaling policies.

2. Conduct a comprehensive review of all cloud scaling configurations across OBS services.

3. Enhance load-testing protocols to simulate peak demand scenarios.

4. Provide additional training for engineering teams on cloud scaling best practices.