Bubbles is the Mac Mini that runs our family AI assistant. You text him, and he texts back. One morning, he stopped. The monitoring said “gateway down.” The AI’s API host looked unreachable. Every signal pointed in the same direction: an upstream outage. Not my problem. Wait it out.
Except the API was fine from every other machine in the house. Only Bubbles could not reach it.
The crime scene
Here is the crime scene. The gateway process was alive, and it was alive for more than a day. Pings worked. Inbound SSH worked. But every outbound connection failed instantly. That included connections from Bubbles to itself. A machine that you can reach, but that cannot reach its own loopback, is a special kind of broken.
The kernel error told the story: it could not assign a local port. Each TCP connection borrows one of about 16,384 ephemeral ports. When a connection closes, the port waits for a few seconds. Then the port returns to the pool.
I counted the ports in that wait state: 18,357. Out of 16,384. The pool was not just empty. The janitor that recycles the ports was also stuck. The count sat frozen, byte-for-byte identical, minute after minute.
Sixteen days of retries
So what opened connections fast enough to drain the pool? A process list, sorted by CPU, put the answer in the first row. It was a one-shot debug query that I ran against Bubbles’ knowledge base. A command of that type returns in two seconds, and you never think about it again. The runtime of a command is directly correlated with your certainty that it finished. This one hung at 100% CPU for sixteen days. It failed, retried, and opened and closed connections the entire time. Orphaned, parentless, invisible, and very busy.
The one open door
The insult on top: macOS gives userspace no way to flush that stuck port table. The only fix is a reboot. I was not home, and the hung state made the normal remote tools half-useless. The escape was a graceful AppleScript restart over SSH — the one door that was still open. Thirty seconds later, Bubbles came back up, with 45 ports in wait instead of 18,357. He answered texts again as if nothing happened.
A reaper, not a bigger buffet
The fix for next time is the fun part. The old healthcheck script asked one question: “does the gateway process exist?” The answer was yes for the entire outage. Useless.
The new script runs every sixty seconds. It kills each one-shot query that is older than five minutes. It also acts when the waiting-port count crosses 8,000. It kills the disease and spares the daemon that it protects. It also logs the top CPU offender, so an unknown runaway still gets caught. I tested it with a decoy hang, and the reaper killed it in one minute.
I deliberately skipped the “real” sysadmin fix: a wider port range. A wider range only gives a runaway process a bigger buffet. The reaper kills the thing that eats.
The total cause of the outage, in the end: one forgotten debug command. Sixteen days. The machine did exactly what I told it to do. That is the whole problem with computers.