Resolved -
This incident has been resolved.
Jun 25, 15:23 UTC
Update -
we're monitoring the systems after fixes were applied and they are stable.
We're processing the delayed data which should be completed soon.
Web app and feature flags are fully operational again for a while now.
We update this once the delayed data is fully processed.
Jun 25, 13:59 UTC
Monitoring -
We applied some more fixes and seeing signs of recovery. We're monitoring the situation.
We still need some time to crunch through the data. We will send an update.
No data has been lost in all of this.
Jun 25, 12:09 UTC
Identified -
We identified the cause for the increased latency. The error rates for feature flags have decreased (from 1.5%) and the ingestion lag is coming down.
Jun 25, 11:49 UTC
Update -
We're investigating increased error rates in feature flags and an increase in latency in the ingestion pipeline.
There is no data loss, but we're currently processing events with a delay.
Jun 25, 09:30 UTC
Update -
We're still investigating but also discovered that the web app is showing increased latency.
Jun 25, 08:27 UTC
Investigating -
We've spotted elevated error rates in feature flags. We're currently investigating the issue, and will provide an update soon.
Jun 25, 07:24 UTC