Incident Analysis:
At around 11:15am UTC on 12th January, performance of the Memory Cache layer on one of our zones began to degrade, and eventually caused the above services to drop. Dashboard and softphone applications would also have been impacted, as parts of these also pull information such as user availability.

As we began to reroute traffic for affected customers into healthy zones, we saw the same issue present itself on a second zone, leaving one healthy zone without any interruption to service.

To mitigate the impact, we implemented a forced-flush of the system at around 13:00pm UTC, and managed to restore service to one zone shortly afterward.

The second affected zone did not recover with the same method. Our infrastructure team then rolled out a planned upgrade to our SIP proxies to that affected zone, at around 13:30pm, which dramatically reduced the load and allowed all services to resume on the final zones within minutes.

We took the decision to wait to implement this same upgrade across the other two healthy zones out of hours, however we unfortunately saw the same issue occur later in the day which caused us to roll this out early to a second zone, at around 15:55pm UTC.

The fix was implemented and service restored within 10 minutes of the first report in this instance.
Later that evening, we successfully rolled out the upgrade to the remaining zone.

Unfortunately, as a result of these upgrades, this caused the BLF indicators on some handsets to become stuck. The following day, on the 13th January at 12:47pm, our team implemented a flush of all active BLF connections, which caused an unforeseen, temporary interruption to those accounts. Service resumed automatically for all handsets at around 12:57pm.

Conclusion and Next Steps
After reviewing the data surrounding this incident, we are confident that the catalyst for the initial degradation on the affected zones is related to the ever-increasing load on the platform. While we have made significant progress in many areas since beginning this overhaul in July 2025, we recognise that there is still some way to go before it can be considered fully complete.
At the time of writing, all zones now have the relevant upgrade applied, which gives us a permanent solution to this issue: we have now fully retired the legacy module that caused yesterday’s issues.

In addition to this upgrade to our Memory Cache layer, we are in the final stages of QA for an additional major project that involves the complete re-design of our presence infrastructure, again dramatically reducing the load and enabling far greater scalability in future.