October 13, 2021
8:49am UTC
Some users are reporting delayed engagements and graph updates.
9:31am UTC
Analysis reveals that the bottleneck is being caused because post engagements and graph updates are running on the same process, resulting in a race condition.
Work is being done to separate these two processes.
10:52am UTC
We estimate that it may be another hour before the fix is in place..
11:02am UTC
The update is “code-complete”. We are awaiting approval from the business that the new code matches business rules specifications before proceeding to testing and then deployment to production.
11:20am UTC
Business rules checking is complete and testing is underway.
4:50pm UTC
A workaround has been put into place but engagements are still running 5 mins behind. We are continuing to work on this issue.
October 14, 2021
5:57am UTC
Engineers are continuing to work around the clock to resolve this issue.
8:55am UTC
Incident resolved. Whilst the code fix worked, engagements were still delayed by a few minutes because there was another issue caused by human error on the configuration of the most recently deployed server farm (that was deployed to support the new code that was written overnight).
