Ingestion has caught up and everything is back to normal.
Posted Dec 09, 2025 - 21:32 UTC
Update
The query load issue is resolved and queries should be loading as normal. We're still working through the backlog of events, so some charts might be showing data that is ~30 minutes old. That should be resolved soon.
No data was lost.
Posted Dec 09, 2025 - 18:46 UTC
Monitoring
Cluster is looking stable now and we have been able to resume the ingestion of events.
Query performance should be now back to normal, as we are not hitting the limits anymore.
There is still events lag that we are already consuming, so data shown won't be up to date yet. We'll send an update once it's completely recovered.
No data has been lost during this period.
Posted Dec 09, 2025 - 18:36 UTC
Update
We are continuing to see queries failing and ingestion lag. We are switching on more capacity, which should hopefully resolve the queries failing (though they'll be showing data that is ~30-60 minutes out of date).
This is not impacting workflows or CDP, and no data has been lost.
We will keep you up to date.
Posted Dec 09, 2025 - 17:54 UTC
Update
We are under heavy load, and seeing query outages and delayed ingestion of events. No data is lost.
Posted Dec 09, 2025 - 17:06 UTC
Identified
Our ClickHouse cluster is going under heavy load right now and the ingestion of events is being delayed.
We have found the root cause and are now working to leave the cluster in a stable state so we can catch up on the ingestion.
Posted Dec 09, 2025 - 15:28 UTC
This incident affected: US Cloud 🇺🇸 (App, Event and Data Ingestion Lag).