On July 16th, a routine image update quietly killed the Cloudflare tunnel on Randy. I found out on July 19th. Not from an alert. From the family shopping list, which did not load.
Three days. Four public services were dark: the shopping list, media requests, analytics, and remote Plex. The house said nothing.
A network that no longer existed
Here is the failure, because it is a good one. The tunnel container was created when my media stack had a different name. Its network attachment — 1_media-net — was pinned at creation time. Later, I renamed the stack, and the old network went away. The running container did not care. It was already attached.
It ran happily for weeks on a network that technically no longer existed. It was like a guy who still uses a gym membership from a closed location.
Then the image updated. The update recreated the container. The new container tried to join 1_media-net by name. No such network existed anymore. So the container just… did not start.
Created, never started
And this was the kicker: it failed with zero logs. The container sat in Docker’s “Created” state. Not running. Not restarting. Not crashing. Just created, the way a form is created before anyone fills it out.
Nothing wrote a line anywhere. The clue, when I finally looked, was a timestamp. The container’s StartedAt was zero. The container never started. No alert exists for a thing that never began.
My monitoring made it worse, in an instructive way. Four services rode that tunnel, and I monitored one. My check on that one could not catch this failure. So one blind spot hid three more behind it. Silent failures travel in groups.
The small fix
The fix was so small that it was almost an insult. I deleted the container. I recreated it on Docker’s default bridge network, with the same mounts and restart policy. That network always exists. It will also survive my next fit of stack renames. The total repair time was a few minutes. The total diagnosis time was most of a morning. That is the usual ratio.
Then came the actual work: a proper uptime check on the tunnel itself. The check polls every five minutes. It sends notifications that reach me. The monitor expects the specific response of a healthy but locked-down endpoint. So it catches “tunnel dead,” and I expose nothing extra to make the tunnel checkable.
Presence of success
I relearn this lesson in a new costume each time. The failures that hurt are not the loud ones. A crashed container restarts itself, or at least it screams into a log. A container in the “Created” state is a tree that falls in a forest with no microphones.
Every system I run has a version of this: a thing that can stop without a technical failure. The only defense is monitoring that checks for the presence of success, not the absence of errors.
Anyway. The shopping list works again. Nobody in the house noticed the gap, except me. Either the system is not that important, or the family learned to route around my infrastructure. I decided not to investigate which.