Data Processing Delays - Legacy Webhooks

Incident Report for PostHog

Resolved

An upstream service (cymbal) adopted a compression scheme not supported by the legacy webhooks and onevent consumers (due to their reliance on kafkajs). The fix deployed was to add support for the compression scheme (LZ4) to kafkajs, using another 3rd party library. All systems have now fully caught up to the real-time data stream.
Posted Nov 28, 2024 - 09:28 UTC

Update

Issue resolved, all systems now recovering.
Posted Nov 27, 2024 - 17:49 UTC

Update

Our legacy webhook processing infrastructure is running behind because of an bad configuration change. Note that this is not affecting CDP based webhooks in any way. No data has been lost and the system should be caught up shortly.
Posted Nov 27, 2024 - 13:31 UTC

Update

Our webhook and legacy plugins processing infrastructure is running behind because of an bad configuration change. Note that this is not affecting CDP in any way. No data has been lost and the system should be caught up shortly.
Posted Nov 27, 2024 - 13:15 UTC

Identified

Our webhook infrastructure is running behind because of a bad configuration change. No data has been lost and the system should be caught up shortly.
Posted Nov 27, 2024 - 13:09 UTC
This incident affected: US Cloud 🇺🇸 (Event and Data Ingestion Lag) and EU Cloud 🇪🇺 (Event and Data Ingestion Lag).