Incident resolved. On July 9 at 18:13 EDT we deployed major changes to the Framework design system. This included an asset deduplication strategy that created a bottleneck in our image generation pipeline for calendar plugins and private plugins.
Within 30 minutes the issue was identified, and within 3 hours patched. Additional edge cases were discovered on July 10 between 06:00-10:10 EDT and also patched.
Posted Jul 10, 2025 - 10:30 EDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 10, 2025 - 00:45 EDT
Investigating
Google Calendar, Private Plugins, and Recipes leverage our Firefox browser pool, which is timing out during screen generation. Investigating.
Posted Jul 09, 2025 - 22:36 EDT
This incident affected: Screen Generation Workers.