The issue has been fixed and the ingestion LAG is recovering.
Posted Jul 19, 2024 - 02:06 UTC
Update
We've identified the issue and we are currently fixing it enable the ingestion again.
Posted Jul 18, 2024 - 22:36 UTC
Update
We're monitoring recovery of postgres now. Some tables are very large and this might take several hours - we're investigating whether we can work some magic to speed this up.
Current impact is still that event ingestion is delayed.
Note that this means person updates aren't being processed so any experiments or flags that rely on changes to person profiles won't see those until the event lag is resolved.
Posted Jul 18, 2024 - 18:19 UTC
Identified
We've spotted an issue with our postgres infrastructure and we're working to resolve it right now.
You'll experience ingestion lag since we're delaying event ingestion to reduce load while we fix this. No events have been lost.
We'll update with an expected time to recovery as soon as we have one. Sorry for the interruption.
Posted Jul 18, 2024 - 17:25 UTC
This incident affected: US Cloud 🇺🇸 (Event and Data Ingestion, Feature Flags and Experiments) and EU Cloud 🇪🇺 (Event and Data Ingestion, Feature Flags and Experiments).