US: Delayed event ingestion

Incident Report for PostHog

Resolved

We've caught up on our backlog of messages. Ingestion rates look optimal. Parts are being merged as they should. New nodes are fully online. Query latencies are looking great at 100ms avg. Should be smooth sailing from here on out. Enjoy your Friday!

Posted May 16, 2025 - 19:18 UTC

Monitoring

There were some recurring errors in our infrastructure that led us to restart clickhouse nodes.

We are falling behind on events ingestion, as we are replacing some nodes in our ClickHouse cluster. This will increase lag in our ingestion pipeline.
Performance may be impacted during this time too. We are still working on this and monitoring it.

Posted May 15, 2025 - 21:26 UTC

This incident affected: US Cloud 🇺🇸 (Event and Data Ingestion Lag).