Skip to main content
The quarterly DR drill arrives. The voice carrier has to prove it can fail SBCs, RADIUS, and routing to the secondary site without dropping carrier-grade SLAs. The drill needs to be announced, executed, validated, reverted, and signed off — with the regulator’s evidence pack assembled the same day.

​
Systems involved

SystemRole
Studio ProceduresVoice DR drill runbook.
Primary SBC + Secondary SBCActive and standby.
FreeRADIUS primary + secondaryActive and standby.
Anycast / DNS routingWhere carrier-side traffic is steered.
TestCallin / SIPpSynthetic call generation for validation.
Bandwidth / TelnyxVerify carrier sees the new SBC IP and accepts.
Microsoft Teams #voice-opsDrill war room.
Gmail / OutlookPre-drill customer notice (large enterprise tenants).
Studio FilesCompliance evidence pack.

​
Walkthrough

1

Pre-drill announcement

72 hours before, Copilot drafts the customer notice through Gmail: drill window, expected impact (none), what to do if real symptoms appear, a single point of contact for the window. Sent to the enterprise tenant contacts.
2

Open the drill war room

At T-15 minutes, open the #voice-ops Teams thread with the drill checklist, the rollback path, the on-call names, and the validation criteria. The drill procedure starts.
3

Capture pre-drill state

Copilot snapshots: active calls per SBC, RADIUS auth rate, carrier-side reachability checks, the current Anycast announcement state. The snapshot is the baseline for validation.
4

Fail SBC to secondary

Through SSH and the routing connector, withdraw the primary SBC’s Anycast announcement. The secondary becomes preferred. Existing calls on the primary continue; new calls land on the secondary within 30 seconds.
5

Fail RADIUS to secondary

Stop the primary RADIUS. The secondary takes over. Auth rate stays inside SLA. The procedure captures the failover transition time.
6

Validate inbound and outbound

Run a TestCallin sweep: inbound on five test DIDs, outbound to five test endpoints, both with and without media. Bidirectional audio confirmed in all cases.
7

Validate 911 routing

The single most important test. Place a 911 test call from a test endpoint with a known address. Confirm it routes to the test PSAP for the address, not to a stale primary-site path.
8

Hold for the soak window

Stay on the secondary for the contracted soak window (often 2 hours). Watch every metric. Anything outside SLA terminates the drill into rollback and is documented.
9

Fail back

Reverse the procedure: re-announce the primary, restart RADIUS primary, validate. Capture the restoration time and the call-continuity status.
10

Compliance report

Generate the drill evidence: timeline, snapshots, validation results, soak-window metrics, 911 evidence, sign-offs. PDF goes into the compliance file and is emailed to the regulator’s contact and the customer’s compliance officer for any tenant who requested evidence.

​
Where Studio earns its keep

  • The drill runs as a procedure, so the next quarter’s drill is the same drill — not a rewrite from memory.
  • 911 validation is a hard step, not an afterthought — the procedure does not pass without it.
  • The evidence pack is the procedure’s output, not a separate document someone has to write the next week.
  • The customer notice and the regulator evidence reference the same drill ID, so there’s a clean audit trail without manual cross-referencing.

Procedures

Build the Voice DR drill once and run it every quarter.

Files and artifacts

The compliance evidence pack lives in the workspace.