Person deletes failing in EU
Identified - We're seeing that person deletes via the API are returning an elevated number of 504's/time outs in the EU region. This appears to be due to degraded query performance in our database, along the path required to delete person profiles. We're working to resolve this now.
Jan 10, 2025 - 16:26 UTC
Uptime over the past 90 days. View historical uptime.
PostHog.com ? Operational
90 days ago
100.0 % uptime
Today
US Cloud 🇺🇸 Operational
90 days ago
99.91 % uptime
Today
App ? Operational
90 days ago
99.96 % uptime
Today
Event and Data Ingestion Operational
90 days ago
99.71 % uptime
Today
Feature Flags and Experiments ? Operational
90 days ago
100.0 % uptime
Today
Session Replay Ingestion Operational
90 days ago
100.0 % uptime
Today
EU Cloud 🇪🇺 Operational
90 days ago
99.92 % uptime
Today
App ? Operational
90 days ago
100.0 % uptime
Today
Event and Data Ingestion Operational
90 days ago
99.71 % uptime
Today
Feature Flags and Experiments ? Operational
90 days ago
100.0 % uptime
Today
Session Replay Ingestion Operational
90 days ago
100.0 % uptime
Today
Support APIs Operational
90 days ago
100.0 % uptime
Today
Update Service Operational
90 days ago
100.0 % uptime
Today
License Server Operational
90 days ago
100.0 % uptime
Today
AWS US 🇺🇸 Operational
AWS ec2-us-east-1 Operational
AWS elb-us-east-1 Operational
AWS rds-us-east-1 Operational
AWS elasticache-us-east-1 Operational
AWS kafka-us-east-1 Operational
AWS EU 🇪🇺 Operational
AWS elb-eu-central-1 Operational
AWS elasticache-eu-central-1 Operational
AWS rds-eu-central-1 Operational
AWS ec2-eu-central-1 Operational
AWS kafka-eu-central-1 Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

System Metrics Month Week Day
US Ingestion End to End Time ?
Fetching
US Decide Endpoint Response Time
Fetching
US App Response Time
Fetching
US Event/Data Ingestion Response Time
Fetching
EU Ingestion End to End Time ?
Fetching
EU App Response Time
Fetching
EU Decide Endpoint Response Time
Fetching
EU Event/Data Ingestion Endpoint Response Time
Fetching
Past Incidents
Jan 14, 2025

No incidents reported today.

Jan 13, 2025
Data Processing Delays - Webhook delivery affected
Resolved - The issue was solved and processing is back to normal
Jan 13, 13:08 UTC
Monitoring - We've identified the cause of the issue and deployed a fix. Processing is catching up and will be back to real-time soon
Jan 13, 12:30 UTC
Investigating - We have found an issue in our webhook delivery system leading webhooks delivery to fail. We are investigating the issue.
Jan 13, 11:07 UTC
New event types not showing up in dropdowns
Resolved - Event and property definitions backfill has been completed, and new definitions are being generated in real time. System stability is being actively monitored.
Jan 13, 08:04 UTC
Monitoring - The issue has been identified and resolved, new event and property definitions are now being created properly. We're re-processing historical data to ensure all definitions are available for use.
Jan 9, 09:25 UTC
Investigating - The database failover was completed successfully, however it did not appear to be the root cause and new event names are still delayed. We are still investigating this
Jan 8, 11:47 UTC
Identified - We have identified the root cause as a database issue, we will shortly be performing a database failover which may result in some ingestion delays. There will not be any data loss
Jan 8, 11:28 UTC
Investigating - New event names (ie posthog.capture('new event name')) are not being created or are severely delayed in being created. That means that you can't select them in the interface
Jan 8, 01:02 UTC
Jan 12, 2025

No incidents reported.

Jan 11, 2025

No incidents reported.

Jan 10, 2025

Unresolved incident: Person deletes failing in EU.

Jan 9, 2025
Increased data duplication for January
Resolved - This incident has been resolved.
Jan 9, 00:45 UTC
Monitoring - We’re seeing some ongoing impact from the "Data Processing Delays in US region" incident reported on January, 6th and a small number of users may see increased data duplication for January as a result.

We’re already in the process of cleaning this up and expect it to be resolved within the next 12 hours.
Jan 8, 15:25 UTC
Jan 8, 2025
Jan 7, 2025
Data Processing Delays in US region
Resolved - All data ingestion has caught up and our systems are back to normal. Thank you for your patience.
Jan 7, 14:28 UTC
Monitoring - We've mitigated the underlying issue in our datastore and are now catching back up. We will be totally caught up in ~8 hours. No events were lost in this incident and we have identified several steps we will take to prevent this from happening again. <3 Posthog
Jan 7, 05:20 UTC
Update - We're still monitoring our application. Our lag is at 4 hours right now—i.e., events appear inside PostHog 4 hours after they're sent—while we continue recovering some of the affected infrastructure. We'll update this once we have more information.

No data has been lost as we're still collecting all the events.
Jan 7, 03:36 UTC
Update - In an attempt to sort out the problem , we've restarted some of our Clickhouse nodes. For that reason, we're also seeing some delays in batch exports. All batch exports will complete once the nodes are back online.
Jan 6, 20:36 UTC
Investigating - We're still seeing degraded performance when ingesting events and session replays. We're still working on tweaking our internal infrastructure to process the backlog.

There will be no data loss as we're still storing all of the events for later processing.
Jan 6, 20:23 UTC
Update - We're still monitoring our infrastructure. Ingestion delay has plateaued at around 45 minutes of delay between sending the event vs. you seeing the events inside our platform. It's important to notice that there's no data loss as everything is being captured appropriately.
Jan 6, 18:24 UTC
Monitoring - We've identified the likely culprit behind the increased ingestion time. We're monitoring it closely and expect to see a decrease in our ingestion lag soon.
Jan 6, 16:22 UTC
Investigating - We're experiencing some delay in ingesting new events. We're investigating.
Jan 6, 15:56 UTC
Elevated API Errors
Resolved - The issue has been resolved.
Jan 7, 03:20 UTC
Monitoring - We've fixed the issue and we are currently monitoring.
Jan 7, 03:09 UTC
Investigating - We're experiencing an elevated level of API errors and are currently looking into the issue.
Jan 7, 02:46 UTC
Jan 6, 2025
Jan 5, 2025

No incidents reported.

Jan 4, 2025

No incidents reported.

Jan 3, 2025
Potential Data Processing Delays in US region
Resolved - We're back to normal now, thank you for your patience!
Jan 3, 14:25 UTC
Update - The maintenance operations are done, ingestion and data processing is catching up.

We're monitoring and keep you updated once the ingestion and data processing returned to a normal state.
Jan 3, 13:43 UTC
Update - We're resuming ingestion, there may be prolonged delays for events to appear in queries.

Maintenance operations are still in progress and we're reporting back once the delay is reducing.
Jan 3, 13:12 UTC
Monitoring - We are performing some internal maintenance on our US systems that could lead to small ingestion delays in the US region.

We will update the progress here. No data will be lost during this operation.
Jan 3, 12:14 UTC
Jan 2, 2025
Potential Data Processing Delays
Resolved - Maintenance operations are completed and data ingestion is looking good.
Thank you for your patience!
Jan 2, 15:29 UTC
Update - Ingestion on EU is catching up and we're monitoring.
Jan 2, 14:46 UTC
Update - Some processes are taking longer than expected. We hope that this is resolved within the next 30 mins.
Jan 2, 14:17 UTC
Monitoring - We are performing some internal maintenance on our EU systems that could lead to small ingestion delays.

No data will be lost during this operation.
Jan 2, 13:24 UTC
Jan 1, 2025
Web app instability
Resolved - Rate limiting has come into effect and appears to be mitigating the issue now.
Jan 1, 12:23 UTC
Identified - The web app is currently experiencing intermittent instability, due to unexpected load along a particular endpoint. Rate limiting appears to be preventing further outages.
Jan 1, 11:12 UTC
Investigating - We're experiencing web app instability. Data capture and feature flags currently appear unaffected.
Jan 1, 10:09 UTC
Dec 31, 2024

No incidents reported.

← Incident History Powered by Atlassian Statuspage