✔ Incident Resolved – Engagements and Graph Updates Delayed – October 13-14, 2021

cars on road
Incident resolved. The cause was human error on the configuration of the most recently deployed server farm (that was deployed to support the new code that was written overnight).

Share This Post

Share on facebook
Share on linkedin
Share on twitter
Share on email

October 13, 2021
8:49am UTC

Some users are reporting delayed engagements and graph updates.

9:31am UTC

Analysis reveals that the bottleneck is being caused because post engagements and graph updates are running on the same process, resulting in a race condition.

Work is being done to separate these two processes.

10:52am UTC

We estimate that it may be another hour before the fix is in place..

11:02am UTC

The update is “code-complete”. We are awaiting approval from the business that the new code matches business rules specifications before proceeding to testing and then deployment to production.

11:20am UTC

Business rules checking is complete and testing is underway.

4:50pm UTC

A workaround has been put into place but engagements are still running 5 mins behind. We are continuing to work on this issue.

October 14, 2021
5:57am UTC

Engineers are continuing to work around the clock to resolve this issue.

8:55am UTC

Incident resolved. Whilst the code fix worked, engagements were still delayed by a few minutes because there was another issue caused by human error on the configuration of the most recently deployed server farm (that was deployed to support the new code that was written overnight).

More To Explore

Malcare WordPress Security