Data Processing Delays in US region

Incident Report for PostHog

Resolved

All data ingestion has caught up and our systems are back to normal. Thank you for your patience.
Posted Jan 07, 2025 - 14:28 UTC

Monitoring

We've mitigated the underlying issue in our datastore and are now catching back up. We will be totally caught up in ~8 hours. No events were lost in this incident and we have identified several steps we will take to prevent this from happening again. <3 Posthog
Posted Jan 07, 2025 - 05:20 UTC

Update

We're still monitoring our application. Our lag is at 4 hours right now—i.e., events appear inside PostHog 4 hours after they're sent—while we continue recovering some of the affected infrastructure. We'll update this once we have more information.

No data has been lost as we're still collecting all the events.
Posted Jan 07, 2025 - 03:36 UTC

Update

In an attempt to sort out the problem , we've restarted some of our Clickhouse nodes. For that reason, we're also seeing some delays in batch exports. All batch exports will complete once the nodes are back online.
Posted Jan 06, 2025 - 20:36 UTC

Investigating

We're still seeing degraded performance when ingesting events and session replays. We're still working on tweaking our internal infrastructure to process the backlog.

There will be no data loss as we're still storing all of the events for later processing.
Posted Jan 06, 2025 - 20:23 UTC

Update

We're still monitoring our infrastructure. Ingestion delay has plateaued at around 45 minutes of delay between sending the event vs. you seeing the events inside our platform. It's important to notice that there's no data loss as everything is being captured appropriately.
Posted Jan 06, 2025 - 18:24 UTC

Monitoring

We've identified the likely culprit behind the increased ingestion time. We're monitoring it closely and expect to see a decrease in our ingestion lag soon.
Posted Jan 06, 2025 - 16:22 UTC

Investigating

We're experiencing some delay in ingesting new events. We're investigating.
Posted Jan 06, 2025 - 15:56 UTC
This incident affected: US Cloud 🇺🇸 (Event and Data Ingestion Lag, Session Replay Ingestion).