Elevated API Errors
Incident Report for Ulobby
Resolved
We are delighted to inform you that the incident we experienced recently has been fully resolved following our scheduled maintenance.

All of our services are now fully operational, with all features working as expected. The intensive monitoring phase post-maintenance has concluded with no observed anomalies or issues.

We want to assure you that we took this matter very seriously. The necessary upgrades and fixes have been implemented to prevent similar incidents from happening in the future.

In case you encounter any issues or have any queries, please feel free to reach out to us. Our team can be reached via email at [email protected]
Posted Jun 02, 2023 - 12:19 CEST
Update
We are pleased to announce that the scheduled maintenance on our platform has been successfully completed within the planned maintenance window. Our services are now fully restored and operational.

Following the updates, we have immediately commenced a period of intensive system monitoring. This is to ensure that all components are working correctly and that the previously experienced issue has been fully mitigated.

Maintenance Details:
Start Time: Friday, June 2, 2023, at 10:00 AM (CET)
End Time: Friday, June 2, 2023, at 10:25 AM (CET)
We finished the maintenance ahead of schedule, with the service interruption lasting only 25 minutes, less than the anticipated downtime.

Our teams are currently closely watching the performance of all systems to ensure everything is running smoothly. We are making this effort to guarantee the highest level of service availability and reliability following the upgrade.

Should you experience any difficulties or irregularities in your service, please do not hesitate to reach out to us. You can contact our support team via email at [email protected].

We would like to extend our sincerest gratitude for your patience and understanding during this maintenance window. We remain committed to offering you the best service possible.
Posted Jun 02, 2023 - 10:25 CEST
Update
We are reaching out to inform you about some important unscheduled maintenance we plan to conduct on our platform. This maintenance should proactively address the issue that some users experienced yesterday.

To prevent further disruptions and enhance your user experience, we will be carrying out necessary upgrades and fixes on our system. This is a part of our ongoing efforts to provide reliable, robust and high-quality service to all our users.

Maintenance Window Details:

Start Time: Friday, June 2, 2023, at 10:00 AM (CET)
End Time: Friday, June 2, 2023, at 12:00 AM (CET)
During this period, there will be a brief period of downtime which could last up to 30 minutes, but we will aim to minimize this as much as possible.

We understand that the maintenance period may disrupt some of your operations. Please rest assured that we have chosen this time window after extensive consideration, aiming to ensure the least possible impact on your activities.

We sincerely apologize for any inconvenience caused due to the issue experienced yesterday and during this maintenance window. Rest assured, our team is committed to carrying out these upgrades swiftly and efficiently, thereby ensuring service stability in the future.

After the maintenance, if you encounter any issues, please feel free to reach out to our support team. You can contact us via email at [email protected].

Thank you for your understanding and continued support. We appreciate your patience during this time and assure you of our best service always.
Posted Jun 02, 2023 - 10:12 CEST
Monitoring
We are pleased to report that as of 12:30 PM CET, our services have been fully restored and are back online.

The sudden, high load issue causing the service disruption has been resolved. Our team worked diligently to rebalance the load and to add additional resources which helped us manage the unprecedented spike in demand.

We would like to assure you that our engineering and operations teams are now closely monitoring the situation to prevent any reoccurrence. The stability of the platform and the robustness of all services are our top priorities, and we will continue to make improvements to ensure these incidents are avoided in the future.

We understand the inconvenience and disruption this outage may have caused and we extend our sincere apologies. We are committed to learning from this incident and implementing necessary measures to prevent similar issues in the future.

We would like to take this opportunity to thank you for your patience and understanding during this time. We value your trust and are devoted to providing you with the reliable service you expect from us.

For further information, queries, or concerns, please feel free to reach out to our support team via email at [email protected] or through our dedicated help center.

Thank you for your continued support and understanding.
Posted Jun 01, 2023 - 12:53 CEST
Identified
We are currently experiencing an unexpected outage across our platform. The preliminary investigation points to an issue caused by a sudden, exceptionally high load on our servers.

Starting at 3:32 AM CET, our monitoring systems started alerting us of performance degradation. It escalated into an outage at 8:46 AM CET, causing interruptions in our services. The affected services include the core application, API connectivity, and third-party integrations.

Our engineering team is currently working hard to identify the root cause and resolve the issue as quickly as possible. We have initiated load rebalancing procedures and are adding more resources to manage the sudden spike in demand. We anticipate an improvement in the situation over the next couple of hours.

We understand that the service disruption can affect your work, and we apologize sincerely for the inconvenience. Our primary focus at the moment is to restore the full functionality of our services.

We encourage you to check back on this status page for regular updates regarding this issue. We will keep you informed about the progress and when you can expect our services to be fully operational.

Please reach out to our support team via email at [email protected] or through our dedicated help center if you have any questions or concerns related to this outage.

Again, we appreciate your patience and understanding while we work diligently to resolve this issue.
Posted Jun 01, 2023 - 12:29 CEST
Investigating
We're experiencing an elevated level of API errors and are currently looking into the issue.
Posted Jun 01, 2023 - 10:52 CEST
This incident affected: Core services (Web Application, Data processing pipeline, API-v1).