Kelvin’s Engineering Wiki

Incident operations

5 min read

Three practices recur across every runbook in this wiki: troubleshoot one layer at a time, change production through a small reversible loop, and close an incident only when stated evidence holds. Each is described once here and linked from the runbooks.

Layered troubleshooting

Layered troubleshooting treats a failure as one broken link in a chain the request or workload depends on. You collect read-only evidence, walk the chain in order, and act only on the first broken link. All four operational sources in this wiki follow it.

SourceThe chain it walks
Kubernetes IP or ENI exhaustionNode health, Pod subnet presence, subnet free IPs, node ENI/IP allocatable, IPAM and admission components1
Express BFF incidentsGateway, PM2 and BFF process, route and app logic, downstream dependency, host resources2
Nginx and OpenResty deploymentsConfiguration test, port ownership, upstream, file permissions, request mapping, ACME34

Rules the sources share

  • Evidence before action. Start with read-only commands and preserve evidence before changing processes, releases, routes, credentials, or downstream targets.
  • One symptom is not a cause. A Pending Pod does not establish IP exhaustion, and one event string is not enough; correlate several signals.1 A downstream 4xx/5xx is not proof the BFF is broken.
  • Liveness is not health. ZooKeeper’s ruok answering imok only proves the process is bound to its port, not that it is in quorum, and freeing disk space does not repair a truncated transaction log.5 Likewise an open port is not an application acceptance test.
  • A restart is not a diagnosis. A restart can restore service temporarily without proving the root cause.2
  • Do not widen the blast radius. Raising timeouts before finding the failing layer, chmod -R 777, or restarting everything are listed as things not to do.3

Safe change procedure

The sources in this wiki apply production changes with the same loop: record intent, back up, make one small change, validate before applying, apply gracefully, verify, and keep a way back.

The loop

flowchart LR
    accTitle: Safe change loop
    accDescr: Back up, change one concern, validate before applying, apply gracefully, then verify each layer. If verification fails, roll back to the backup or the known-good release.
    B[Back up] --> C[Change one concern]
    C --> V[Validate before applying]
    V --> A[Apply gracefully]
    A --> K{Every layer verified?}
    K -->|Yes| D[Done]
    K -->|No| R[Roll back cleanly]
  1. Record intent and back up. Back up the file, configuration, or release being changed.
  2. Change one concern. One small edit at a time.4
  3. Validate before applying. For packages, apt-get -s simulates the change so the plan can be read before anything is installed or removed; the APT guide calls this simulation the step that protects you from held versions, phased rollouts, and locks.6 For servers and releases: nginx -t or openresty -t before any reload; for a release, npm ci, tests, and node --check in the new release directory before switching.
  4. Apply gracefully. A reload validates the new configuration and then gracefully replaces workers, which is safer than a routine restart; PM2 reload does the same for cluster workers.
  5. Verify each layer. Health, an approved real request, and logs, not just an open port.
  6. Roll back cleanly. Restore the backup and test it before reloading, or repoint current to the known-good release.37

One member at a time in a quorum system

For a replicated system the loop gains a stop rule. The ZooKeeper guide changes one member, waits for it to rejoin, checks all three members, and continues only if there is still one leader and two followers; otherwise it stops and rolls back that one member. Two members are never changed at once, because three members tolerate only one failure.8

Restarts are a decision, not a reflex

The Kubernetes runbook says adding capacity does not justify restarting every workload: restart only the affected one, after checking replicas, update strategy, and PodDisruptionBudgets, and with the service owner’s approval.1

Incident closure criteria

An incident is closed against explicit acceptance evidence, not because the alert stopped. Both incident runbooks in this wiki end with such a list.

What the lists have in common

  • Every affected layer is shown healthy. For Kubernetes: nodes Ready, a usable same-zone subnet, ENI/IP capacity no longer exhausted, a new Pod that schedules and gets an IP, DaemonSets at DESIRED = CURRENT = READY, and no new InsufficientIPOrENI events during the observation period. For the BFF: stable PM2 workers, private health, the gateway route, and a representative authorized flow.
  • Recovery of a dependency is not proof it was the only cause. After rebuilding a ZooKeeper member, the runbook repeats the original dependent-service request rather than assuming the application is fixed.5 After a quorum-loss restore, closure also needs matching last_zxid values on every member and application owners’ approval of their ACL and dependency checks.9
  • An agreed observation period. Health must hold over time, not at one moment.
  • The record separates facts from guesses. Observed evidence, completed actions, and pending validation are kept apart; the first failing layer and the mitigation are recorded separately from hypotheses.
  • Sanitized records. Environment identifiers, raw logs, and screenshots stay in the authorized incident system; follow-ups get an owner.12

Footnotes

  1. Kubernetes IP or ENI Exhaustion Scheduling Failure Runbook, original ↩ ↩2 ↩3 ↩4

  2. Express BFF: Ten Common Incidents Runbook, original ↩ ↩2 ↩3

  3. Nginx Production Deployment and Operations for Beginners, original ↩ ↩2 ↩3

  4. OpenResty Production Deployment and Operations for Beginners, original ↩ ↩2

  5. ZooKeeper Disk-Full Recovery Runbook, original ↩ ↩2

  6. Common Ubuntu APT Operations, original ↩

  7. Express BFF Production Deployment for Beginners, original ↩

  8. ZooKeeper Production Deployment Guide for DevOps Beginners, original ↩

  9. ZooKeeper Quorum-Loss Snapshot Restore Runbook, original ↩

Backlinks