The level changes when you leave
This week, Ars Technica covered Quake's new 30th-anniversary mission pack, which introduces a looping world where revisiting the present changes what happens next. It is a great game mechanic because the world remembers what you did.
Production systems work the same way, except we keep pretending they do not.
A deployment rollback restores an earlier code or configuration version. It does not rewind the database, unsend the email, cancel the webhook, erase the model output, or reconstruct the queue exactly as it was five minutes ago. The release may go backward. The system's history keeps moving forward.
That distinction matters more as we deploy AI features, distributed workers, and agent-driven workflows that can change state outside the application that launched them.
A release is not the whole system
Teams often describe a deployment as if it were a single artifact: commit abc123 went live, so deploying def456 should put production back.
That model is only accurate for stateless services with no meaningful side effects. Most systems are not like that.
A single release can:
- Apply a database migration that changes the shape or meaning of stored data.
- Publish messages that consumers process asynchronously.
- Send emails, issue refunds, create tickets, or update a customer's CRM record.
- Generate embeddings or model classifications that become inputs to later decisions.
- Change feature flags, caches, search indexes, or external service state.
- Trigger an agent to call an API that has no undo operation.
If version 42 writes an invalid status to 10,000 records, deploying version 41 does not restore those records. If a worker sends duplicate invoices, the old binary cannot make the duplicates disappear. If an AI workflow approves a bad recommendation and another service acts on it, the original prompt and model version are now part of the incident.
Rollback controls executable behavior. Recovery controls the consequences of behavior.
The audit trail must include state, not just code
We have already argued in Why Agent-Authored Deploys Break Your Audit Trail that a commit alone is not enough to explain why a change shipped. Stateful recovery raises the bar again: you need to know not only who deployed the code, but what the code changed while it was running.
For every meaningful state transition, record enough information to reconstruct the decision:
- Release ID and immutable image or build digest.
- Database schema version and relevant configuration.
- Model, prompt, tool, and policy versions for AI-assisted work.
- Event ID, correlation ID, actor, and timestamp.
- Input references and the external request ID when another service is involved.
- Whether the operation is reversible, compensatable, or forward-only.
This is why a deploy log that says success is operationally weak. It tells you that a command completed. It does not tell you which state was live, which events were emitted, or which downstream systems accepted them.
The practical test is simple: can you answer, in under a minute, what version made a specific change and which other systems saw it? If not, a rollback button is mostly theater.
Recovery moves forward
The safer mental model is not rewind. It is a new, explicit state transition that repairs the current world.
There are three common recovery paths:
- Roll back code when the current behavior is wrong and existing data remains compatible with the older version.
- Apply compensating actions when the system has created an external side effect, such as issuing a reversal for a payment or sending a correction for a bad message.
- Ship a forward fix when the bad release changed data, emitted events, or altered a contract that the old code cannot safely understand.
These paths are not interchangeable. Calling all three of them rollback creates dangerous ambiguity during an incident.
A reliable service makes the distinction visible in its design. Event handlers should be idempotent. Migrations should have a documented compatibility window. Outbound work should use an outbox or durable queue so it can be inspected and replayed. Consumers should retain enough event history to identify what was processed, not merely the latest aggregate value.
The goal is not to preserve an imaginary past. The goal is to make the current state explainable and repairable.
Build a recovery contract before you need one
For each state-changing workflow, write a short recovery contract. Not a generic rollback checklist, but a description of what recovery means for that specific operation.
For example, an AI triage worker might define:
- The invariant: every customer request has exactly one active disposition.
- The durable evidence: request ID, event ID, model version, prompt version, and disposition history.
- The containment action: pause new triage decisions without deleting queued work.
- The compensation: mark affected dispositions for human review and notify downstream systems.
- The forward fix: deploy a corrected classifier, then replay only events from the affected release range.
- The recovery proof: compare active dispositions, replay counts, and downstream acknowledgements against expected totals.
That contract gives responders something better than a command to copy and paste. It defines the evidence required, the actions that are safe, and the condition that ends the incident.
It also exposes bad architecture early. If you cannot explain how to identify affected events, you do not have recovery. If you cannot replay or compensate them, you do not have a safe forward path. If nobody can declare the invariants restored, you have not defined recovered.
AI makes the past harder to reconstruct
AI systems add another layer of state because behavior can change without a conventional code diff. A model update, prompt edit, retrieval index refresh, tool permission change, or altered temperature can produce different results from the same application build.
That means the rollback unit cannot always be the container image. You may need to restore a model and prompt combination, freeze a retrieval snapshot, or route new work to a deterministic fallback while old decisions are reviewed.
This extends the production lesson from Autonomous Mining Is Not Production Until It Can Fail: observability is not just a dashboard showing whether the service is up. It is the ability to attribute an outcome, limit its impact, and prove that recovery worked.
The same principle applies to autonomous agents. If an agent can write data or call external tools, its action log is part of your operational state. Deleting the run record after a failed deploy does not restore safety. It destroys the evidence needed to repair the system.
Design for the world that exists after failure
Loop Desk is built around durable queue state, activity history, cycle history, and resumable work sessions because continuous systems need memory of what happened, not just the latest deployment status. That same principle belongs in every production workflow that can create consequences outside its process.
Before the next release, identify the state it can change, record the versions that explain those changes, and define the forward action that repairs each failure mode.
A rollback can restore code. Only a designed recovery path can restore trust.