The deploy of a new build has addressed the issue. Monitoring continues as does additional work as we follow the steps of our Incident Response plan.
Posted Apr 20, 2020 - 11:08 PDT
Update
Apply latencies remain high and degradation is still evident throughout the system. We are, presently, deploying a new build throughout the cluster and are actively working to return to stability
Posted Apr 20, 2020 - 09:29 PDT
Update
We are continuing to monitor for any further issues and address symptoms of the core issue as they arise.
Posted Apr 20, 2020 - 08:48 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Apr 20, 2020 - 08:29 PDT
Identified
The problem has been identified and remediation is ongoing. The issue manifests in login errors via console, increased write latency, and timeouts. Updates forthcoming as we progress in resolution.
Posted Apr 20, 2020 - 08:13 PDT
Investigating
We are currently investigating high request latencies and timeouts.
Posted Apr 20, 2020 - 07:02 PDT
This incident affected: Global Region Group (FQL API).