All systems are operating normally. Thank you for bearing with us as we overflow our ints in unexpected places!
Posted Jul 31, 2023 - 22:56 UTC
Monitoring
Mitigations have been successful and we resumed processing of analytic events. Current ingestion lag is currently at 6 hours and going down. We will post updates as the backfill progresses.
Feature flag evaluation and updates are now OK, although their result is affected by the delayed event processing.
Posted Jul 31, 2023 - 14:43 UTC
Update
The first round of mitigations have been unsuccessful, so ingestion is still paused for now.
Feature flags with ‘persist flags across auth steps’ checked are not working for new users, but will continue working for existing users. Rest of feature flags continue to work fine.
Posted Jul 31, 2023 - 12:11 UTC
Identified
We identified the issue and are working towards resuming event ingestion. Our plan is to resume ingestion first by skipping cohort and feature flag updates for newly created persons, then backfilling this data later today. Ingestion lag is currently at 3 hours, data is safe and will be backfilled. Feature flag evalution (/decide) error rate is stable, but results may be outdated for persons created/merged today.
Posted Jul 31, 2023 - 09:52 UTC
Investigating
A database issue is affecting ingestion of new persons. The current ingestion delay is 2 hours and rising. Feature flag and cohort management for these new persons is also affected. Evaluation is succeeding but the values might be outdated.
Posted Jul 31, 2023 - 08:54 UTC
This incident affected: US Cloud 🇺🇸 (Event and Data Ingestion Lag, Feature Flags and Experiments).