An upstream service (cymbal) adopted a compression scheme not supported by the legacy webhooks and onevent consumers (due to their reliance on kafkajs). The fix deployed was to add support for the compression scheme (LZ4) to kafkajs, using another 3rd party library. All systems have now fully caught up to the real-time data stream.
Posted Nov 28, 2024 - 09:28 UTC
Update
Issue resolved, all systems now recovering.
Posted Nov 27, 2024 - 17:49 UTC
Update
Our legacy webhook processing infrastructure is running behind because of an bad configuration change. Note that this is not affecting CDP based webhooks in any way. No data has been lost and the system should be caught up shortly.
Posted Nov 27, 2024 - 13:31 UTC
Update
Our webhook and legacy plugins processing infrastructure is running behind because of an bad configuration change. Note that this is not affecting CDP in any way. No data has been lost and the system should be caught up shortly.
Posted Nov 27, 2024 - 13:15 UTC
Identified
Our webhook infrastructure is running behind because of a bad configuration change. No data has been lost and the system should be caught up shortly.
Posted Nov 27, 2024 - 13:09 UTC
This incident affected: US Cloud 🇺🇸 (Event and Data Ingestion Lag) and EU Cloud 🇪🇺 (Event and Data Ingestion Lag).