Failed database migration

Incident Report for TRMNL Eink Display

Resolved

During non-peak hours we deployed a migration that drops a legacy table, "Playlist Groups." This was responsible for scheduling different plugins through the day, but has since been replaced by the Scheduler, which was deployed to all accounts on September 8, 2025.

Database CPU quickly climbed to 100%, preventing us from connecting + debugging. Engineers upsized the database, connected, and discovered the table being dropped (+ foreign key being migrated) had legacy records with a composite index. This composite index was used in the service invoked by GET requests to /api/display, which caused some devices to render a "Technical Difficulties" image.

After fixing + adding back the composite index, and modifying the Playlist query that is triggered by GET requests to /api/display, service returned to normal.
Posted Oct 23, 2025 - 01:30 EDT