Data Processing Delays - Reporting Tools Affected

Incident Report for PostHog

Resolved

Cluster is back to normal health. Pipeline is up to date. Everything is working nominally!
Posted Nov 30, 2023 - 23:18 UTC

Monitoring

We've increased the speed of merges on our ClickHouse cluster and added settings to prevent us from collecting so many parts in the future. We are now monitoring recovery which should be complete within 2-3 hours. Will update once that is done!
Posted Nov 30, 2023 - 19:33 UTC

Update

The cluster is currently recovering, although data ingestion is still paused. We're continuing identification of the root cause, and working on accelerating recovery.
Posted Nov 30, 2023 - 16:42 UTC

Investigating

Our analytics database cluster is experiencing unusual load, resulting in data ingestion running behind. We're investigating the issue.
Posted Nov 30, 2023 - 15:29 UTC
This incident affected: EU Cloud 🇪🇺 (Event and Data Ingestion Lag).