Real Time collaboration Issues detected
Incident Report for MURAL

The degradation on the real time collaboration service that occurred on Oct 30th was caused by a self-inflicted DDOS based on our websocket authentication protocol that would retry forever on a timeout. This was triggered by a high server side response time (caused by high load on our servers triggered by a slow deployment process) that made timeout of the authentication handshake to occur more often. We fixed the issue by improving the authentication protocol to allow for more flexibility in server side response time and by adding an "exponential backoff" strategy that would prevent a self-inflicted DDOS in the future.

Posted Nov 05, 2018 - 11:05 GMT-03:00

This incident has been resolved.
Posted Oct 30, 2018 - 14:15 GMT-03:00
Issue was identified and fixed. We continue to monitor the behavior of the web application.
Posted Oct 30, 2018 - 14:14 GMT-03:00
Sockets authentication service is down. Users cannot edit murals. We are working on a fix.
Posted Oct 30, 2018 - 12:30 GMT-03:00
We're investigating problems with real time collaboration in murals.
Posted Oct 30, 2018 - 12:22 GMT-03:00
This incident affected: Canvas.