We have narrowed down the problem to a very small set of Snowflake batch exports that we have manually cancelled. If you were affected we will be reaching out.
All other batch exports are fully recovered or on the path to recovery. Performance of ongoing batch exports will soon be on pace with real time once again.
Posted Apr 01, 2025 - 18:49 UTC
Investigating
We were unable to make a full recovery, and the issue seems to persist. We are investigating new potential fixes. In the meantime, batch exports will be delayed.
Posted Apr 01, 2025 - 17:25 UTC
Monitoring
We are monitoring the backfill process for Snowflake batch exports and any pending large batch exports for other destinations. All backfills are progressing normally.
On-going batch exports are operating normally, but users with pending backfills may see us lag behind real time until all the backfilling is done.
Posted Apr 01, 2025 - 14:00 UTC
Update
We have deployed our fixes and have managed to resolve the concurrency issues with Snowflake batch exports.
Most batch exports besides Snowflake should be fully recovered, with the exception of larger batch exports that will still need some time to work through the backlog.
We will shortly begin backfilling any Snowflake batch exports that were cancelled due to this incident.
Posted Apr 01, 2025 - 13:22 UTC
Update
We are continuing to work on a fix for this issue.
Posted Apr 01, 2025 - 12:37 UTC
Identified
We have reasons to believe the cause of the problem is a deadlock happening while connecting to Snowflake. We are attempting to deploy a patch that would deal with the deadlock when it happens, leaving the investigation of what is causing the deadlock for later.
Assuming the patch is successful in addressing the problem we will begin back filling any Snowflake batch exports that were cancelled.
Posted Apr 01, 2025 - 10:42 UTC
Update
We have reason to believe that the problem is related to Snowflake batch exports. In consequence, most Snowflake batch exports are being cancelled to be retried at a later date. We are investigating how to remediate the problem with Snowflake.
Users of other destinations should see batch exports recovering over time. Depending on the size of the data exported, this recovery could take less or more time.
Posted Apr 01, 2025 - 09:20 UTC
Investigating
We have been making slow progress on batch exports and a backlog has built up, particularly on larger batch exports. It is taking us some time to work through the backlog, so users may see batch exports be delayed in delivering data.
No data loss has happened nor is it expected.
Posted Apr 01, 2025 - 07:38 UTC
This incident affected: US Cloud 🇺🇸 (Destinations) and EU Cloud 🇪🇺 (Destinations).