We identified undetected underprovisioning in one of our network components.
We scaled this up now and working on a fix to mitigate this long-term.
Thank you for your patience.
Posted Apr 25, 2025 - 11:03 UTC
Update
Performance and error rate are back to normal levels.
we're still investigating the root cause for this issue.
Posted Apr 25, 2025 - 09:45 UTC
Update
We are continuing to investigate this issue.
Notice about US: this incident never affected the US environment. The "partial outage" status was wrong for that. We will correct this later. apologies for the inconvencience
Posted Apr 25, 2025 - 09:32 UTC
Update
The error rate has gone down, we're still looking for the root cause.
Posted Apr 25, 2025 - 09:28 UTC
Investigating
Elevated error rates are coming up again, we're investigating
Posted Apr 25, 2025 - 08:52 UTC
Monitoring
We identified a surge in memory usage and workload eviction events. We scaled up feature flags and web app to mitigate.
We're monitoring this.
Posted Apr 25, 2025 - 08:20 UTC
Update
Situation has calmed down after scaling up resources. We're still investigating the root cause.
Notice: in an earlier message, it was reported that this was about the US region. This was wrong, this is only about the EU region. Apologies for the initial wrong reporting
Posted Apr 25, 2025 - 08:12 UTC
Update
We are continuing to investigate this issue.
Posted Apr 25, 2025 - 08:03 UTC
Investigating
We're experiencing an elevated level of API errors incl feature flags and are currently looking into the issue.
Posted Apr 25, 2025 - 08:02 UTC
This incident affected: US Cloud 🇺🇸 (App, Feature Flags and Experiments) and EU Cloud 🇪🇺 (App, Feature Flags and Experiments).