Pilot topology#
A controlled pilot may use a coordinator, transactional state store, object storage, work queue and isolated executors. The database and queue products are implementation decisions. Selection must satisfy atomic state changes, tenant isolation, backup recovery and audit requirements.
OPS-01 Authoritative state lives in durable records. Queue messages are notifications, not the sole source of task truth. Use a transactional outbox or equivalent method to recover when a state commit succeeds but notification delivery fails. Assume at-least-once delivery and deduplicate consumers.
Swipe or scroll to view all columns →
| Incident | Required response |
|---|---|
| Coordinator restart | Recover durable task state; reconcile leases and pending outbox events. |
| Executor disconnect | Stop new allocation; evaluate lease expiry and safe retry conditions. |
| Object store outage | Keep acceptance pending; retry authorized reads within the deadline. |
| Checker failure | Preserve delivery; do not let the executor self-approve. |
| Payment notification loss | Query the original external payment identifier before any new action. |
| Registry or key incident | Quarantine affected authority, preserve evidence and apply revocation policy. |
Change management#
OPS-02 Pin every released executable to a version, digest and dependency inventory. Publish migration requirements, compatibility impact and rollback limits with the artifact. In-flight tasks retain their frozen workload and acceptance profiles during an upgrade.
Drain nodes before planned maintenance. Recovery exercises must include state, authorization, evidence objects and accounting relationships; restoring a database alone does not establish complete service recovery.
Production operating decisions#
Before offering a paid service, define on-call ownership, incident severity, customer notification, backup retention, recovery point and recovery time objectives. Set values from the actual deployment and exercises. Static website availability or a local test pass does not establish these service objectives.