Resolved -
We identified undetected underprovisioning in one of our network components.
We scaled this up now and working on a fix to mitigate this long-term.
Thank you for your patience.
Apr 25, 11:03 UTC
Update -
Performance and error rate are back to normal levels.
we're still investigating the root cause for this issue.
Apr 25, 09:45 UTC
Update -
We are continuing to investigate this issue.
Notice about US: this incident never affected the US environment. The "partial outage" status was wrong for that. We will correct this later. apologies for the inconvencience
Apr 25, 09:32 UTC
Update -
The error rate has gone down, we're still looking for the root cause.
Apr 25, 09:28 UTC
Investigating -
Elevated error rates are coming up again, we're investigating
Apr 25, 08:52 UTC
Monitoring -
We identified a surge in memory usage and workload eviction events. We scaled up feature flags and web app to mitigate.
We're monitoring this.
Apr 25, 08:20 UTC
Update -
Situation has calmed down after scaling up resources. We're still investigating the root cause.
Notice: in an earlier message, it was reported that this was about the US region. This was wrong, this is only about the EU region. Apologies for the initial wrong reporting
Apr 25, 08:12 UTC
Update -
We are continuing to investigate this issue.
Apr 25, 08:03 UTC
Investigating -
We're experiencing an elevated level of API errors incl feature flags and are currently looking into the issue.
Apr 25, 08:02 UTC