Data Processing Delays - Reporting Tools Affected

Incident Report for PostHog

Resolved

All systems are operating as normal. Happy querying!
Posted Jul 22, 2023 - 05:30 UTC

Identified

The backfilling efforts strained other systems and we are currently assessing the full impact.

- Insights and other queries are currently showing elevated error rates
- Analytics data is currently delayed by up to one hour, but will be safely backfilled
- Exports & webhooks are equally delayed
Posted Jul 21, 2023 - 15:49 UTC

Update

There is some increased delay across insight data and webhooks related to the same issue. We are still monitoring the progress and getting ingestion back up to speed. No data loss has occurred.
Posted Jul 21, 2023 - 14:49 UTC

Update

Insights and webhooks have caught up with exports continuing to slowly backfill. No data loss has occurred.
Posted Jul 21, 2023 - 13:47 UTC

Monitoring

We've identified a bottleneck that was causing a delay in processing and are monitoring it closely. We are catching up with data ingestion and will be back to normal shortly.
Posted Jul 21, 2023 - 13:06 UTC

Investigating

Our data processing infrastructure is running behind which is causing inaccuracies in the reporting tools. No data has been lost and the system should be caught up shortly.
Posted Jul 21, 2023 - 12:47 UTC
This incident affected: EU Cloud 🇪🇺 (App, Event and Data Ingestion Lag).