Capture latency and error rates are back to nominal on both regions. We are backfilling through the ingestion backlog now.
Between 8h15 and 11h GMT, the event submission success rate was averaging 91% on the US region. Thanks to client-side retries in posthog-js and most other SDKs, we expect the event loss ratios to be lower than this.
Posted Jun 02, 2023 - 11:13 UTC
Update
The EU region is fully operational. The US is still experiencing an elevated event capture error rate, mitigations have been deployed and should fix this shortly. Events successfully captured are currently processed with a delay >30 minutes.
Posted Jun 02, 2023 - 11:02 UTC
Identified
We are investigating elevated event submission latencies on both regions. Some partial data loss might happen.
Posted Jun 02, 2023 - 10:48 UTC
This incident affected: US Cloud 🇺🇸 (Event and Data Ingestion) and EU Cloud 🇪🇺 (Event and Data Ingestion).