Outside of machine vision I enjoy building infrastructure.
Projects
- NAS and storage systems
- Linux servers
- Docker environments
- Documentation platforms
- Local AI tooling
- Automation utilities
Rules I have settled on
- 3-2-1 backups, actually tested - three copies, two media, one offsite. The part everyone skips is the restore test; I schedule one quarterly, and it has found a silently broken backup chain more than once.
- One service, one container, one compose file - everything a service needs lives in its own directory: compose file, config, data volume. Migrating a service to a new machine means copying a directory. Resist the mega-compose-file that runs everything; it turns every change into a risk to everything.
- Boring base, adventurous edges - the storage, networking, and backup layers run the most boring, conservative software available. Experiments happen in containers on top, where their blast radius is one directory.
- Write the runbook while it works - the restore procedure, written and tested on a calm afternoon, is what 2 AM me follows mechanically. 2 AM me makes terrible decisions without a script.
- Monitor the few things that matter - disk health, backup completion, certificate expiry. Ten alerts that are always real beat a hundred that get ignored.
A failure that shaped the rules
An over-clever storage setup - cascading caches, deduplication, tiered volumes - delivered impressive benchmarks for a year. Then one layer failed in a way that corrupted the layer underneath, and the recovery required understanding the exact interaction of four technologies at once. The replacement is a plain mirrored pool with nightly snapshots. The benchmarks are worse. Nothing else is.
The recurring lesson
Simple systems survive. Complex systems become support cases.
Every layer of cleverness you add is a layer someone has to understand at 2 AM when it breaks. I now optimize for boring, predictable, and recoverable over impressive.
The overlap with my day job is no accident: a production inspection system and a home server fail for the same reasons - unwatched drift, untested recovery paths, and complexity nobody fully understood. Practicing on one makes you better at the other.