ClickHouse memory issues

Incident Report for PostHog

Resolved

We've identified the root cause causing the OOM issues on our ClickHouse cluster and have mitigated the issue. We've replaced cluster nodes that were lost during the incident. All systems are operating as expected!

Posted Sep 28, 2024 - 03:47 UTC

Investigating

We've spotted that something has gone wrong with memory consumption in ClickHouse. We're currently investigating the issue, and will provide an update soon.

In the meantime queries may return more slowly or not at all.

Posted Sep 27, 2024 - 10:45 UTC

This incident affected: US Cloud 🇺🇸 (App, Event and Data Ingestion).