Data Processing Delays - Reporting Tools Affected

Incident Report for PostHog

Resolved

We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Posted May 30, 2023 - 17:23 UTC

Update

A small portion of the traffic is still experiencing ingestion delay, so we'll keep this status page open until we're full caught up.
Posted May 30, 2023 - 15:38 UTC

Update

We processed all of the events we stashed away this morning, all analytics events should now be processed with nominal latency.

We will monitor the situation for a bit longer before closing this incident.
Posted May 30, 2023 - 13:15 UTC

Monitoring

Events submitted in the last 30 minutes should be ingested without delay or duplication, some events from earlier have been stashed in a separate topic and are slowly being backfilled.
Posted May 30, 2023 - 11:45 UTC

Identified

We detected a significant amount of duplicated events going to ClickHouse. While these events will be deduplicated, they affect insight quality on recent events.

Mitigation work is ongoing, and as part of it are currently delaying part of the traffic. No data is lost.
Posted May 30, 2023 - 10:38 UTC
This incident affected: US Cloud 🇺🇸 (Event and Data Ingestion Lag).