Rollbacks and zero-downtime deploys

Every deploy is a small bet that the new version works. A rollback is how you cancel the bet in seconds, and a zero-downtime deploy is how you place it without users noticing. They solve different problems, but they rely on the same habit: never overwrite what is running, always keep the previous version ready.

Three release strategies cover almost every case: rolling, blue-green and canary. Two failures can break any of them: abrupt process shutdowns and database migrations.

What makes a rollback possible?#

A rollback is only fast when the previous version still exists as a complete, runnable artifact. That means three rules:

• Immutable, versioned builds. Tag Docker images with a version or commit SHA, never only latest. A tag that moves cannot be rolled back to.
• Config separate from code. Environment variables live outside the build, so reverting code does not silently revert (or keep) the wrong settings. See environment variables.
• Old release kept around. Keep at least the last few images or release folders. Kubernetes, for example, keeps old ReplicaSets for rollback, controlled by revisionHistoryLimit (set it to 0 and you lose the ability to roll back).

With that in place, a rollback is a redeploy of an older version, not a rebuild.

How do you roll back, step by step?#

Docker Compose. Pin the image by tag through a variable:

services:
  app:
    image: registry.example.com/myapp:${APP_VERSION}
    restart: unless-stopped

Deploy and roll back by changing one value:

APP_VERSION=1.4.2 docker compose up -d   # deploy
APP_VERSION=1.4.1 docker compose up -d   # rollback

Kubernetes. Deployments record revisions for you:

kubectl rollout history deployment/myapp
kubectl rollout undo deployment/myapp                  # previous revision
kubectl rollout undo deployment/myapp --to-revision=2  # a specific one

A new revision is created only when the pod template changes (image, labels, env), so scaling does not pollute the history.

Plain servers. Deploy each release into its own folder and point a current symlink at it. Rolling back is moving the link and reloading the process:

ln -sfn /srv/myapp/releases/1.4.1 /srv/myapp/current

After any rollback, check that it worked: hit the health endpoint, then read the app logs to confirm the old version is the one answering.

Which zero-downtime strategy should you use?#

Rolling is the default in Kubernetes (maxSurge and maxUnavailable both default to 25%). It needs no extra environment, but for a while two versions serve traffic at once, so they must be compatible with each other.

Blue-green keeps the old environment (blue) untouched while the new one (green) is started and tested. When you flip the router, every user moves at once, and flipping back is your rollback. The price is running two copies.

Canary limits the blast radius: 1% of traffic, then 10%, then everyone, watching error rate and latency between steps. It pays off when you have enough traffic for the numbers to mean something.

All three need the same two ingredients: at least two instances behind a router (see load balancers), and a health check that tells the router when the new instance is ready.

How do you reload without dropping requests?#

You do not need an orchestrator for a basic version of this.

Nginx. nginx -s reload checks the new configuration first. If it is invalid, Nginx keeps running with the old one. If it is valid, new workers start and the old workers finish their current requests before exiting. Always test before reloading:

nginx -t && nginx -s reload

PM2 (Node.js). In cluster mode, pm2 reload reloads your processes with no downtime, unlike pm2 restart, which kills and restarts the process:

pm2 start app.js -i 2     # two instances in cluster mode
pm2 reload app            # zero-downtime reload

This only helps with two or more instances, and your app must be stateless: each process has its own memory, so keep sessions in Redis or a database.

Health checks. A router should only send traffic to an instance that proves it is ready. In Compose:

services:
  app:
    image: registry.example.com/myapp:${APP_VERSION}
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:3000/healthz"]
      interval: 5s
      timeout: 3s
      retries: 3
      start_period: 10s

Make /healthz check what the app really needs (for example, a database ping), not just return 200 as soon as the process starts. The example assumes your image includes wget; swap in curl or a small script if it does not. Compose itself only records the result (the status shows up in docker compose ps); your router still has to act on it, either by reading that status or by running its own check against /healthz.

Why do deploys still drop requests?#

When an orchestrator or process manager stops an old instance, it sends a termination signal first (SIGTERM, or SIGINT with PM2). If your app ignores it, in-flight requests are cut. Handle it by stopping new connections and finishing the ones in progress:

// server.js (Node.js, ES modules)
import http from "node:http";

const server = http.createServer((req, res) => {
  if (req.url === "/healthz") return res.end("ok");
  res.end("hello");
});
server.listen(3000);

let shuttingDown = false;
function shutdown() {
  if (shuttingDown) return;
  shuttingDown = true;
  // Stop accepting new connections; the callback runs when open ones finish.
  server.close(() => process.exit(0));
  // Safety net: do not hang forever on a stuck connection.
  setTimeout(() => process.exit(1), 10_000).unref();
}
process.on("SIGTERM", shutdown);
process.on("SIGINT", shutdown);

Other common causes:

• The router keeps sending traffic to an instance that is stopping. Fix it with health checks and a short drain delay.
• The new instance receives traffic before it has warmed up. Fix it with a readiness check, plus start_period so a slow start is not marked unhealthy.
• Long-lived connections such as WebSockets. A restart always interrupts them, so clients need to reconnect on their own.

How do database migrations break rollbacks?#

Code rolls back easily. Data does not. If version 2 drops a column that version 1 reads, rolling back to version 1 crashes it. During a rolling or canary release the problem is worse, because both versions run at once against the same database.

The usual fix is the expand and contract pattern, split across separate releases:

1. Expand. Add the new column or table. Keep the old one. Both versions work.
2. Migrate. Deploy code that writes to both and reads the new one. Backfill old rows.
3. Contract. Once no running version uses the old column, and you no longer need to roll back past that point, drop it.

Rules of thumb: make migrations backward compatible with the previous release, never rename a column in one step (add the new name, copy, remove the old one later), and take a database dump before any destructive migration. See how to back up a database.

What should you check before and after every release?#

Before: tests pass, the previous version is tagged and still available, and the migration is backward compatible. Deploy through a pipeline so the steps are repeatable, as in automatic deploys with GitHub Actions.

After: the health endpoint answers from the new version, the error rate and latency match the old baseline, and logs show no new exceptions. Decide the rollback trigger in advance (for example, error rate above a fixed threshold for five minutes) so nobody debates it during an incident.