Barb is the smallest machine in my fleet. She is a small cloud droplet, named after the woman who owns the trailer park. Her job is to own the place and keep tabs on everything. She runs the monitoring, the backups, and the automation jobs. Barb watches the watchers.
Then I made an awkward discovery. No person watched Barb’s watchers.
Six containers, three suspects
I wired Barb into a new monitoring setup. During that work, I asked a question that I did not ask for a long time: what runs on this box? The answer was six containers. Three of them were a Grafana + InfluxDB + Telegraf stack. The stack had one job: collect Barb’s own system metrics and draw them on dashboards.
Then I checked the date of the last look at those dashboards.
Ten months without a look
Grafana’s own database does not lie. Three dashboards; last edit, October 8 of last year. The last admin login, October 9. Ten months.
In that time, Telegraf collected metrics every few seconds, as instructed. InfluxDB stored all of them, as instructed. The database grew to 756 MB. It sat on the most disk-constrained machine that I own, and that disk was 82% full. The stack had one measurable effect on the system. It became a thing that monitoring must warn you about.
Two of the three dashboards were empty drafts. The third had seven panels. I could not describe one of them under oath.
An afternoon of deletion
The teardown took one afternoon. First, I archived the dashboard definitions and the compose file. I can restore the full stack if I miss it. I will not miss it. Then I removed the three containers and pruned the leftovers.
The disk went from 82% full to 78% full. The container count went from six to three. Each of the three earns its keep.
The real disk usage
The good part came next, when I checked the real content of the disk. The monitoring stack was not the problem. Of 59 GB in use, 48 GB were backups shipped over from Randy. The backups are legitimate and necessary. They were the real growth driver, and no person had noticed.
The bloated containers were a rounding error on top of the real answer. An audit has a classic shape. The thing that you enter to fix is rarely the biggest thing that you find.
A shorter job description
The up/down checks still run. The disk usage now has real alerts with real thresholds. Barb’s job description is shorter and more honest.
Deletion is maintenance. It does not feel like progress, because nothing new exists after it. But “nothing new exists” was the goal. The droplet breathes, the backups have room, and no process on that machine performs work for an audience of zero.
Ten months of metrics, collected with care, never read. Somewhere in that InfluxDB, a chart showed its own disk usage as it climbed. It was a good chart. No person saw it.