OBS Group Inc. - Degraded Performance for OBS Pay and OBS Pay APIs & Partial Outage for OBS SecuRA Runtime Environment and Encryption APIs – Incident details

Resolution of Issues Affecting OBS Pay, OBS Pay APIs, OBS SecuRA Runtime Environment, and Encryption APIs

Summary of Resolution

OBS Pay and OBS Pay APIs

1. Issues Identified:

   • Degraded performance, including transaction delays and API timeouts.

2. Resolution Steps:

   • Scaled up database and application server resources.

   • Applied temporary traffic throttling to stabilize performance.

   • Implemented enhanced monitoring to detect future spikes in real time.

3. Status:

   • Services restored and stable as of 11:00 AM UTC.

OBS SecuRA Runtime Environment and Encryption APIs

1. Issues Identified:

   • Partial outage with approximately 40% of encryption and decryption requests failing.

2. Resolution Steps:

   • Replaced and restarted the failed service node.

   • Rerouted traffic to healthy nodes.

   • Increased monitoring granularity for critical nodes.

3. Status:

   • Full service restoration achieved at 1:00 PM UTC.

Impact Summary

1. OBS Pay and APIs:

   • Approximately 25,000 transactions delayed or failed globally during the incident.

   • No data loss or security impact.

2. OBS SecuRA Services:

   • Affected approximately 40% of encryption requests for a subset of users.

   • No data compromise occurred; the issue was limited to availability.

Next Steps

1. Conduct a detailed post-incident review and publish findings by January 26, 2025.

2. Enhance auto-scaling and failover mechanisms across all affected systems.

3. Perform stress testing to ensure systems handle peak loads without service degradation.

4. Roll out a robust communication plan to inform users about service improvements.

Acknowledgment

We sincerely apologize for the inconvenience caused during this incident and thank you for your patience and understanding as we worked towards a resolution.

For any further concerns or support, please contact the Incident Response Team at incident_response@engineering.obsgroup.tech

Issues Identified in OBS Pay, OBS Pay APIs, OBS SecuRA Runtime Environment, and Encryption APIs

Summary of Issues Identified

OBS Pay and OBS Pay APIs

1. Degraded Performance:

   • Significant delays in transaction processing.

   • API requests experiencing timeouts and intermittent failures.

2. Root Cause Identified:

   • High database contention caused by an unexpected surge in transaction volume.

   • Inadequate auto-scaling thresholds for managing peak loads.

OBS SecuRA Runtime Environment and Encryption APIs

1. Partial Outage:

   • Approximately 40% of encryption and decryption requests failed.

   • Service node connectivity issues disrupted secure operations.

2. Root Cause Identified:

   • Failure in a service node connecting to the central orchestration layer.

   • Insufficient failover readiness for the affected node.

Current Status

1. OBS Pay and APIs:

   • Mitigations applied, and performance has improved as of 11:00 AM UTC.

   • Monitoring continues to ensure stability.

2. OBS SecuRA Services:

   • Partial restoration achieved by 10:30 AM UTC.

   • Full resolution is expected by 1:00 PM UTC.

Next Steps

1. Enhance auto-scaling and load-balancing mechanisms for OBS Pay systems.

2. Implement improved failover and recovery mechanisms for OBS SecuRA nodes.

3. Conduct a post-incident analysis to address root causes and prevent recurrence.

We apologize for the inconvenience caused and appreciate your patience as we work towards a full resolution. For updates, please contact the Incident Response Team at incident_response@engineering.obsgroup.tech