Resolved -
All screen generation workers and APIs have been restored. Internal infrastructure analysis and adjustments will continue with no impact on customers or operations.
Mar 26, 15:58 EDT
Update -
All workers are caught up, but one worker is currently still offline. While this is technically degraded, the existing workers can handle the ongoing load.
Mar 26, 14:22 EDT
Update -
Screen generation workers continue to work through the backlog. This can result in:
- Existing plugins not showing up-to-date information (old screeens)
- New devices showing "You're all caught up!" due to no screens yet generated
Mar 26, 13:45 EDT
Monitoring -
We've identified the issue and restored site functionality. We are still monitoring the system as we identify the underlying cause.
Screen generation will still see lag due to the backlog and one worker being taken offline.
Mar 26, 11:33 EDT
Investigating -
We are currently investigating the issue as the main database has 32GB of replica lag.
Mar 26, 10:22 EDT