Nothing on this box fails loudly

Most of what gets called building with AI turns out to be plumbing, and I say that as someone who spent six months finding it out. The model is the part you rent by the token and it arrives already knowing how to write. Everything genuinely difficult sits around the outside of it, the memory that has to survive the end of a conversation, the search that isn't somebody's advertising index, the file with your real data in it, the credential that lets a script act on your behalf without you pasting it into a chat window. All of that has to run somewhere, and a laptop that goes to sleep when you shut the lid is not somewhere.

So the answer, rather predictably, was to go and buy a box to put all of it on. Obelisk is a small HP mini PC running Unraid, four cores and four threads of an i3 with 32 GB of RAM behind it, and it is not a datacenter machine nor is it pretending to be one. It exists to hold the tools I build for myself, which mostly run overnight and mostly run without me, and the arithmetic there was never close, since a cloud instance with this much memory and this much disk hanging off it would bill me every month for what the machine charged me once.

What actually runs on it is a fleet of MCP servers, which is a dull way of describing something rather useful, a way for a chat session to reach a real system and do real work inside it. There is one for administering the box itself, and one holding long-term narrative memory so that a session can recall a decision I made three weeks ago. Beyond those sits a private search stack with its own meta-search engine and page fetcher, one server exposing my finances out of a self-hosted budgeting app, one for the media stack and one for the photo library. Around them sits the unglamorous half, a secrets sync at quarter past one in the morning, the job sourcing sweep at two, the appdata backup at four, and the container auto-updater half an hour behind that.

The first thing the box taught me was that my intuition about it was wrong. It felt slow, and slow feels like a memory problem, so twice I went looking for the RAM ceiling and twice the audit came back saying the same thing, that CPU is the constraint and nothing else is. At rest it sits around twenty per cent aggregate CPU with a load average near two across four threads, memory is comfortable, and the read cache is already serving upwards of ninety-nine per cent of reads out of RAM. The correct lever was never buying more of anything, it was cutting the idle footprint, and stopping a seasonal Minecraft server that reserved close to seven gigabytes with nobody playing on it, along with the database backend of an analytics stack whose front end had already exited, freed about seven gigabytes in a single pass.

The second thing it taught me was rather more expensive than the first. Chasing a segfault in the storage layer, I ran a pool upgrade on the theory that unsupported feature flags might be behind it, which enabled somewhere around forty-five new ZFS features at once, one of which tripped a kernel null pointer dereference on any write to that pool at all. ZFS upgrades only go one way so there was nothing to undo, the pool dropped to read only, and it had to be rebuilt from scratch. It was, in hindsight, a guess I had promoted to a diagnosis, and the rule that came out of it is now written down in three separate places, never run a pool upgrade on production, and on Unraid never run one at all.

The third is the one I find most instructive, because nothing failed. One container started failing its backup verification every night for the dullest imaginable reason, a log file it was still writing to changed in between being archived and being checked. The backup plugin only prunes old sets after a fully successful run, and says as much in its own log. That single cosmetic failure therefore froze retention quietly across the entire system, good sets stopped expiring, failed ones stacked up behind them, and the offsite Google Drive mirror grew in lockstep to about a hundred and forty gigabytes before I noticed it.

The fourth arrived several days late, which is rather the point of it. I spent an evening hardening the box, rebinding service ports so that they stopped listening on every interface, and within two days four of those containers were back on wide open bindings. Unraid's nightly updater recreates a container from its template rather than from whatever the running container currently happens to look like, so a change made at the run layer only survives until the next image update, and the database passwords I set that same evening survived precisely because those had gone into the template.

What all four have in common is that none of them made a sound. Docker will report a port as free when it is bound to a private interface, so you assign it and then the bind fails underneath you. A restart looks exactly like a recreate from the outside, so a configuration change can appear applied and be entirely inert. Cloudflare will authenticate you at the edge and then hand you an error from an origin that has quietly dropped off the container network, which reads as an authentication problem and is nothing of the sort. Containers wear an unhealthy badge in the interface while serving traffic perfectly well, because the health check is the thing that is broken.

So what I have ended up optimising for stopped being uptime, which was never seriously in question, and became legibility, by which I mean the number of places I have to look before I can trust an answer about the state of the machine. A box that falls over tells you it fell over. A box that carries on serving while three of its assumptions have quietly stopped being true will happily let you keep building on top of it for a fortnight.

In practice that comes down to a handful of rules, all of them cheap to follow and expensive to skip. The template rather than the running container is the source of truth, so a durable change means editing both, the live container to test it and the template so that it survives the next update. The vault is the only place a secret exists, and containers never take one as an inline value, they read a file that has been rendered onto disk for them. Two service accounts handle that rendering and reading, and they are split by how much damage each of them could do. The one that writes those files can read the managed collections but never hands a value back to me, while the one exposed to a chat session cannot see the important collections at all and can write to exactly one throwaway collection. A compromised conversation is therefore structurally incapable of reaching anything that matters, which is a rather better guarantee than my remembering to be careful.

Reachability is the other half of it and it runs in three layers, with the cheapest one as the default. Nearly everything is reachable only across a private overlay network, where membership of the network is the entire credential and there is no public route to attack in the first place. Inside the box a container network handles internal routing, which is how the public tunnel finds its origins by name rather than by an address that changes every time something is recreated. Only the few surfaces that a hosted chat has to reach are exposed publicly at all, and those sit behind a single wildcard access app instead of one app per hostname. One app is one thing to audit, whereas per-host apps invite the mistake of cloning one that happened to carry an exemption, which is how a service of mine was public for an afternoon.

The alerting, finally, has been narrowed down until it only ever carries bad news. Every script on the box reports failures to a single Discord webhook and reports success to nobody at all, which sounds rather lazy and is the only reason the channel still works, because a nightly run of green ticks would have trained me to ignore it months before that backup mirror started quietly growing.

It isn't finished, and the honest gap is not the one people tend to assume. Compute is fine and the audits settled that, while the real exposure is redundancy. Five and a bit terabytes of media sit on an eight terabyte drive that is roughly six years old, carries reallocated sectors, hangs off a USB port because the chassis has no bay left to put it in, and has no parity behind it whatsoever. The data I would actually grieve over, photos and configuration and the vault, lives on the mirrored pool with a daily backup and an offsite copy, so the gamble is deliberate and it is scoped, but it remains a gamble and I would rather say so than describe the setup as sturdier than it is. There is a smaller version of the same problem sitting in a named volume that holds working configuration and that the backup cannot see, queued for the next maintenance window. The hardware itself is frozen until I move country in a couple of years, at which point the whole storage question gets asked again properly.

None of this is the interesting part of working with AI, which is more or less why I bothered writing it down. The interesting part, the tools that actually change how a week goes, only works if the layer underneath is legible enough to trust, and six months of maintaining that layer has taught me more about building things that survive contact with reality than any amount of prompting was ever going to. I am looking for work in Singapore, building AI automation and low-code tooling, and this is the half of the portfolio nobody thinks to ask about.

Nathan Wong is in Singapore, building AI automation and low-code tooling, most of it self-hosted on an Unraid server called Obelisk. Portfolio