Field Notes
Short pieces about keeping websites up: migrations, caches, certificates, and the small mistakes that cost a weekend.
Migration sentinel KAPSULE-NX-B7B2F602EDC4CAA5FEE299DC
Planted by the Next.js migration drive moments before it ran.
Two checks that disagree
When one gate says pass and another says fail, the interesting question is what each of them can see.
What a copy does not carry
Dependencies, build output and secrets are the three things you must deliberately not copy, and deliberately replace.
The process nobody was supervising
It was started by hand, it survived for months, and then the machine rebooted.
Backups you have never restored
An untested backup is a strongly held belief.
Ports allocated from the wrong source
Deriving an answer from a record of what you think exists, rather than from the machine.
The migration that reported success
The worst outcome is not a failed migration. It is a failed migration that told the customer it worked.
Status 200 is not a render
A database error page, a fatal error, a blank body and a directory listing are all HTTP 200.
A lazy chunk is a chunk nothing tests
The code that only loads when somebody clicks is the code your entire test suite is blind to.
The build that baked in the old address
Some environment variables are read when the software starts. Others are compiled into it. Confusing the two moves a broken site.
rsync can succeed at copying nothing
Exit status zero from a pipeline is the status of the last command in it, and the last command is usually harmless.
Caches that lie in only one direction
A cache that serves stale content is a bug you will find. A cache that stores the wrong content is one you will not.
The certificate that renewed into nothing
A renewal succeeded, the file on disk changed, and the server kept serving the old one for eleven days.