On Monday October 19th a Fauna operator deployed a change that fixed the behavior of several new FQL functions for authentication that are not yet publicly available, but will launch as part of the next version of our API. Prior to the change, the functions raised a "Permission Denied" exception when used to define roles. The change had the unintended side effect of causing the Identity() function to raise a “BadRequest” exception with the description "Authentication does not contain an identity.” This issue was not caught locally because our unit tests lacked breadth of coverage across functions that can be called as part of Attribute Based Access Control (ABAC). The problematic code change passed code review and was cherry picked into a release branch to be deployed along with another time-sensitive hotfix; because of the urgency of getting that change out, the combined set of changes did not make it through the full integration test suite prior to deployment. Eight hours after rolling out to production, the first user reported the issue in our community Slack. All production servers were successfully rolled back twenty hours after the initial roll out that caused the event.

We’re taking the following steps to improve:

1. Adding additional unit tests that cover the combination of Identity(), ABAC roles, UDFs, and other functionality.
2. Instituting a new policy that requires VP approval to deploy a build that hasn’t passed our integration test suite to production.
3. Standardizing our incident response process and tooling for issues that are reported via ad-hoc (eg. Slack) channels.

We prioritize the availability, security, and performance of our service above everything else and apologize for any inconvenience that this event caused you. If you have further questions/comments about the event or suggestions on additional steps that we could have taken to provide a better customer experience during the event, please reach out to support@fauna.com.