The certificate that renewed into nothing
A renewal succeeded, the file on disk changed, and the server kept serving the old one for eleven days.
The renewal ran on schedule. The log said success. The file in the certificate directory had a fresh timestamp and a fresh serial. Every check we had was green, and the site served an expired certificate for eleven days.
The reason is dull and it is the whole lesson: nothing reloaded the web server. A process that has already opened a certificate holds the parsed copy in memory, and it does not go back to disk to look again. Writing the new bytes changed the answer to a question nobody was asking.
What fixed it was not a better renewal. It was a check that asked the SERVED certificate for its expiry, over the network, the way a browser does, rather than asking the filesystem.