Resolved -
An upstream service (cymbal) adopted a compression scheme not supported by the legacy webhooks and onevent consumers (due to their reliance on kafkajs). The fix deployed was to add support for the compression scheme (LZ4) to kafkajs, using another 3rd party library. All systems have now fully caught up to the real-time data stream.
Nov 28, 09:28 UTC
Update -
Issue resolved, all systems now recovering.
Nov 27, 17:49 UTC
Update -
Our legacy webhook processing infrastructure is running behind because of an bad configuration change. Note that this is not affecting CDP based webhooks in any way. No data has been lost and the system should be caught up shortly.
Nov 27, 13:31 UTC
Update -
Our webhook and legacy plugins processing infrastructure is running behind because of an bad configuration change. Note that this is not affecting CDP in any way. No data has been lost and the system should be caught up shortly.
Nov 27, 13:15 UTC
Identified -
Our webhook infrastructure is running behind because of a bad configuration change. No data has been lost and the system should be caught up shortly.
Nov 27, 13:09 UTC