Elevated latencies and timeouts

Incident Report for Fauna

Resolved

This incident has been resolved.

Posted Dec 09, 2020 - 17:06 UTC

Monitoring

Changes that result in reduced peak memory usage have been deployed. We are monitoring the system.

Posted Dec 09, 2020 - 16:14 UTC

Update

Further investigation has identified this issue also caused a similar event of elevated latencies and timeouts lasting from 1:41-2:24.

Posted Dec 09, 2020 - 10:56 UTC

Identified

The issue has been identified as an excessive memory usage as part of processing user queries. We have a fix that we are deploying.

Posted Dec 09, 2020 - 10:13 UTC

Investigating

We have experienced timeouts and elevated latencies in cloud, starting from 5:10AM-6:10 UTC and with smaller magnitude from 6:30-9:10.

Posted Dec 09, 2020 - 10:12 UTC

This incident affected: Cloud.