October 6, 2021
10:57pm UTC
Graph updates are being delayed. Post engagements are continuing without issues.
11:59pm UTC
A review has determined that post statistics updates are being delayed. Additional servers have been launched to cover the delay.
5:15pm UTC
Graph updates have returned to normal.
This incident has not affected post engagements. Post engagements are occurring normally
12:15pm UTC
The post-incident review has been completed. Surprisingly, the root cause is something else entirely: A recent large uptick in the number of new users in Lempod has led to more user statistics updates than we had capacity for. The capacity limit is due to a performance bottleneck in user security and identification matching. The improvement for this is scheduled for approximately late October / early November.
Until then, the workaround solution is to update user statistics once every 36-48 hours instead of every 12-24 hours.
12:49pm UTC
We are in the process of deploying the fix to production. Along the way there has been a need for additional code fixes. This is an emergency fix and all hands are working hard. We apologize for the delay.
