October 13, 2021
Some users are reporting delayed engagements and graph updates.
Analysis reveals that the bottleneck is being caused because post engagements and graph updates are running on the same process, resulting in a race condition.
Work is being done to separate these two processes.
We estimate that it may be another hour before the fix is in place..
The update is “code-complete”. We are awaiting approval from the business that the new code matches business rules specifications before proceeding to testing and then deployment to production.
Business rules checking is complete and testing is underway.
A workaround has been put into place but engagements are still running 5 mins behind. We are continuing to work on this issue.
October 14, 2021
Engineers are continuing to work around the clock to resolve this issue.
Incident resolved. Whilst the code fix worked, engagements were still delayed by a few minutes because there was another issue caused by human error on the configuration of the most recently deployed server farm (that was deployed to support the new code that was written overnight).