Skip to main content
Ubiquiti publishes a critical firmware advisory for EdgeRouter X devices. The ISP operates 14,000 of them across residential subscribers. The rollout has to stage firmware centrally, go out regionally with per-region maintenance windows, notify subscribers before each window, watch reboot success rates, and roll back the handful that don’t come back cleanly.

​
Systems involved

SystemRole
Vendor advisorySource announcement and firmware image.
TR-069 ACS (GenieACS)Firmware distribution to CPE fleet.
Studio inventoryCPE records tagged by region.
Twilio SMSSubscriber pre-maintenance notice.
GmailBusiness-tier subscriber email comms.
Atlassian StatuspageMaintenance windows published.
SplynxSubscriber region and contact lookup.
LibreNMSReboot and reachability verification.
Slack #cpe-fleetOperational channel.

​
Walkthrough

1

Verify and stage the image

Copilot downloads the firmware from the vendor advisory, validates the SHA against the published hash, and uploads it to the ACS repository. The image appears in the ACS catalogue with the advisory ID.
2

Plan the rollout

Split the fleet by region — 14 regions, roughly 1,000 CPEs each. Each region gets a 2-hour window spread over ten nights. Business subscribers are scheduled last so any issues surface on residential first.
3

Pre-window subscriber comms

48 hours before each regional window, Twilio sends an SMS to residential subscribers: brief outage, window, self-service URL for status. Business subscribers get a personalised email through Gmail.
4

Publish maintenance

Statuspage publishes all 14 maintenance windows with the affected regions and the advisory reference. The IVR hold message picks up an automated region-aware notice 30 minutes before each window.
5

Execute the first window

The CPE firmware rollout procedure targets Region 1, 10 percent of the CPEs at a time. For each batch, the ACS queues the upgrade. Copilot watches the CPEs come back online against LibreNMS reachability within the expected reboot interval.
6

Watch the success rate

Target threshold is 99.5 percent reboot-and-reauth within the window. Region 1 hits 99.7 percent. Seven CPEs didn’t come back — Copilot flags each one with the last-known state and queues them for individual attention.
7

Handle the stragglers

For each failed CPE, Copilot pulls the RADIUS last-accounting record, the ACS session history, and the LibreNMS last-seen. Five come back on the next day’s reboot. Two are dispatched for field swap.
8

Subsequent regions

Each following night, the rollout procedure runs for the next region. Statuspage and #cpe-fleet maintain a running status board. Residential complaints are near-zero because the comms went out ahead.
9

Close

After all 14 regions, generate the rollout report: fleet coverage, success rate, rollback count, field-swap count, advisory closed. The report is filed to the ISP’s security advisory register and the post-mortem is auto-scheduled.

​
Where Studio earns its keep

  • The rollout is gated — each region only starts when the previous region hits the success threshold, so the first problem is caught on 1,000 subscribers, not 14,000.
  • The SMS, the email, and the status page all point at the same regional schedule — there is no gap between “when we said” and “when it happened.”
  • Failed CPEs are handled individually from the same workspace with the full history available, not marked as errors in a report for someone else to chase next Tuesday.
  • The runbook is parameterized by region, so the next advisory from any CPE vendor reuses the same structure.

Procedures

CPE firmware rollout with advisory ID and region as arguments.

Connectors and MCP

GenieACS, Twilio, and Splynx wired as connectors.