We have finished recovering all missing events and have resolved the underlying issue. We apologize for any inconvenience this may have caused.
Posted Jun 29, 2024 - 05:42 UTC
Update
Event recovering is still in progress. Due to the volume of data being handled, we expect the process to finish between today and tomorrow EOD.
Posted Jun 27, 2024 - 13:30 UTC
Identified
We have identified the partitions that were not consumed correctly into CH and have sequestered them for re-ingestion. The vast majority of missing events will be restored by EOD tomorrow.
The issue initially started on Thursday ~16:00Z when we brought a new CH replica online. We've removed that replica from the Kafka consumer group and all data is now being consumed successfully.
Posted Jun 25, 2024 - 20:04 UTC
Investigating
There may be fewer events reported than actual for the last few days. We are investigating the root cause. We have already pushed a fix that has fixed the issue going forward, but are working to identify impacted events in the past.