A clean build of titan: Debian 13, one k3s cluster with namespaces and policy doing the isolating, designed around short-lived agents that stop stepping on each other.
In order of standing-up, not of importance. Everything runs in a container and anything can be brought down; there is no hierarchy of what wins when resources are tight. Where a choice is justified by something not on this list, the choice is wrong.
| # | Use case | Whose | What it needs |
|---|---|---|---|
| 1 | Game servers | The kids | Public ports, network isolation, saves restored |
| 2 | Media server | Household | GPU transcoding, bulk sequential storage |
| 3 | Backup server | Everyone | The 40 TB drive, per-machine shares, tested restores |
| 4 | Agent environments | Jack | Isolation that holds, instant spin-up, ~24h resume, a walkable filesystem when one dies |
| 5 | Foundry VTT | Jack | Public, persistent, and a public-read S3 asset store so players load maps and audio directly. Already answers on a second domain |
| 6 | Code projects | Jack | Git in the house, some projects published |
| 7 | Local models | Jack | The GPU, on demand, with nothing changing on the server to enable it |
Governing constraints. Open source, and not driven by an American corporation where there is a choice. Nothing phones home. Low maintenance: the machine is administered in short sessions months apart, so anything that needs regular attention is a design fault.
Read off the machine on 18 September 2026. The rebuild is from scratch, so none of this constrains the design — but three of these turn assumptions into measurements.
| Finding | Detail |
|---|---|
| No redundancy anywhere | Both volumes are linear LVM concatenations. /media holds 4.3 TB across two drives with no parity — either failing loses all of it. |
| One drive does all the work | During a backup: 2.5 GbE link, 60 MiB/s actual, sdb at 84% utilisation and 216 ms latency while sda sat at 0.00. A linear layout puts each file wholly on one disk. This is the evidence for AM. |
| Minecraft will not start | 698 restarts. Fabric rejects the mod set: create_dragons_plus resolved to 1.11.8 where 1.11.4-p2 is required, plus two mods for other game versions. A floating dependency moved and the server never came back — the case CQ answers. |
| Already present | Cockpit serving on :9090. A group model begun (admin, samba). Helm in use, eight releases. GPU on driver 595.84 with the open kernel module, which is the Blackwell requirement. |
| Memory is not the constraint | 4.4 GB of 31 GB in use, PSI flat at zero, one OOM event in thirty days. AV is headroom for the front-end migration, not relief from present pressure. |
| Loose ends | Apache serving the stock page on :80. 26 orphaned Jellyfin pods since March. 146 pending updates. No backup automation of any kind. Satisfactory on :latest. |
Ordered as it will be built. Decided is settled. Recommended is what happens if nothing is said. Deferred was considered and postponed. Phase — governs throughout. References are stable: a later row takes the next free reference rather than renumbering.
| Ref | Phase | Title | Status | Who | Decision, what it gives, what it costs |
|---|---|---|---|---|---|
| AA | — | Design principle | Decided | Jack | Convention over configuration — defaults with options, nothing welded shut. Gives: a working system without choosing everything first. Costs: defaults must be documented or they become invisible. |
| AB | — | Build order | Decided | Jack | Games, media, backup, agents, Foundry, projects, models. Order of standing-up only. Gives: a checklist. Costs: nothing — it is not a resource hierarchy; anything can yield to anything. |
| AC | — | Provisioned, not running | Decided | Jack | Every capability plumbed whether or not it is used. Gives: a new workload is a choice, not a rebuild. Costs: more built up front than is immediately needed. |
| AD | — | Runbook form | Decided | Jack | An agent-executable specification: precondition, command, expected output, failure branch. Gives: the build can be handed off. Costs: far slower to write than a guide. |
| CC | — | Measure, don’t presume | Decided | Jack | Where a cause or figure is unknown, build the measurement rather than design around a guess. Gives: answers arrive from data. Costs: instrumentation to build and read. |
| AE | 00 | Backup verified | Decided | Jack | A real restore, opened and checked, before any disk is touched. Hard stop. Gives: the only proof the backup exists. Costs: delays the build by however long it takes. |
| AF | 00 | Backup drive | Decided | Jack | A target only. Never mounted into a cluster, never a hostPath, never a PVC. Gives: no workload can reach it by any path. Costs: backup jobs run on the host, not in the cluster. |
| AG | 00 | Family video offsite | Deferred | — | 17 GB, the only irreplaceable data, currently with no copy outside the house. Deferred knowingly. Gives: nothing to maintain and no external dependency. Costs: fire or theft takes the server and the backup drive together. |
| CL | 00 | Backup drive air-gapped | Decided | Jack | Physically disconnected for the whole rebuild, held on another machine. Gives: an unplugged disk cannot be reformatted by a mistyped device name — stronger than any software control. Costs: AE must happen before it is unplugged, because afterwards it cannot be tested. |
| AH | 01 | Clean build | Decided | Jack | From scratch, not a migration. Gives: no inherited state. Costs: everything must be re-established deliberately. |
| AI | 01 | Disk reformat | Decided | Jack | Every disk except the backup drive. Gives: a known starting layout. Costs: irreversible; gated by AE. |
| AJ | 01 | Host OS | Decided | Jack | Debian 13 “trixie”, currently 13.7 (12 Sep 2026). Gives: the only major distribution with no owner — a constitution, an elected leader, nobody able to sell it or change its terms; the widest package archive; AppArmor rather than SELinux, so no policy friction on Samba, hostPath volumes or non-standard document roots; a transparent-hugepage default of madvise, which is the right one for a long-lived JVM; an official NVIDIA repository; and roughly half the major upgrades before 2035 of the alternative considered. Costs: no rollback tooling out of the box, which AK exists to fix, and no vendor testing k3s against it — though its real-world deployment base is far larger than any vendor QA. |
| AK | 01 | Rollback | Decided | Jack | btrfs root with snapper — timeline snapshots plus apt pre/post hooks, wired up in Phase 01. Not optional. Gives: an upgrade that breaks the machine becomes a reboot and pick the previous snapshot, rather than a rescue disk and a lost evening. It is the one genuinely superior thing the alternative distribution offered by default, and it is worth an hour to reproduce. Costs: an hour in Phase 01, and the root filesystem must be btrfs — which AL already specifies. |
| AL | 01 | NVMe layout | Decided | Jack | btrfs RAID1 across both NVMe — OS, containers, databases, game worlds, appdata. Gives: survives one drive failing, plus snapshots, data checksums and self-healing from the good copy. This is where copy-on-write earns its keep. Costs: half the raw capacity. |
| AM | 01 | HDD layout | Decided | Jack | XFS on mdadm RAID0 across both HDDs, for the media library only. Gives: reads served by both spindles instead of one — the survey measured 60 MiB/s with sdb at 84% and sda idle, which is what striping fixes. XFS also suits the hardware: these are drive-managed SMR, where copy-on-write scatters writes into the worst pattern for shingled media, and snapshots are of no use on a write-once library. Costs: no data checksums, so bit rot is invisible until playback — acceptable because the library is backed up and restic checksums what it stores. No redundancy either: one drive failing loses the array, which is why AE gates the build. |
| AN | 01 | No new disks | Decided | Jack | Four drives is the budget; the chassis is too shallow. Costs: no room for RAID on the media tier. |
| AO | 01 | Privilege model | Decided | Jack | Privileges attach to groups, never to accounts. Gives: adding a person is one membership change. Costs: group sprawl if not kept short. |
| AP | 01 | Fixed GIDs | Decided | Jack | GIDs pinned explicitly. Gives: container fsGroup matches the host. Costs: the numbers must be honoured everywhere, forever. |
| AQ | 01 | Default shell | Decided | Jack | zsh with oh-my-zsh for human accounts. Gives: the shell Jack actually uses. Costs: one more package on the host. |
| AR | 01 | Root shell | Decided | Jack | bash. Gives: root can still log in when /usr is broken. Costs: none. |
| AS | 01 | Escalation | Decided | Jack | sudo with your own password; never su -. Gives: attributable actions. Costs: none. |
| AT | 01 | Login banner | Decided | Jack | One banner for everyone; the health and diary block only after authentication. Gives: state visible on arrival. Costs: none. |
| AU | 01 | Paging tripwire | Decided | Jack | PSI, reported at login. Gives: the honest memory signal instead of swap-allocated. Costs: none. |
| AV | 01 | RAM | Decided | Jack | 64 GB — 2×32 @ 3600 CL18, arriving 25 Sep. The build does not wait for it: it can start on the existing 4×8 GB and swap afterwards, because fitting memory is a reboot and a MemTest86 pass rather than a rebuild, and nothing in the design depends on the amount. Gives: headroom for the front-end migration once fitted, and two slots left free for 128 GB later. Costs while on 32 GB: games and agents will not comfortably run at the same time, and four DIMMs is 2 DPC, so the modules clock lower than two would. Do not sell the 4×8 GB modules until the new kit has passed MemTest86 — they are the fallback if it has not. On fitting: set A-XMP for 3600 and confirm the fabric holds 1800 in 1:1, stepping to 3466 then 3200 rather than fighting it. |
| CD | 01 | Restart classification | Decided | Jack | Exit code separates crashes from clean stops; a reason file separates intent. Four categories to the diary: crash, scheduled, update, unexplained clean stop. Gives: a real failure stops hiding inside a restart count. Costs: whoever stops a service must write the reason file. |
| CE | 01 | Human accounts | Decided | Jack | Four people, three tiers. Jack administers. Max: admin in the games and personal namespaces, read-only cluster-wide. Orlando: games, media, share. Kaire: media and share, no shell. Groups: titan-admin 3000, titan-k8s 3001, titan-games 3002, titan-media 3003, titan-backup 3004, titan-share 3005, titan-k8s-view 3006, titan-appdata 3007, titan-registry 3008. Gives: everyone has exactly what they need. Costs: four accounts to keep current. |
| CF | 01 | Dormant admin account | Decided | Jack | hobbsy is not recreated. Gives: the standing root access disappears at no cost. Costs: none — the account is unused since March. |
| CG | 01 | Credential recovery | Decided | Jack | A labelled USB stick taped to the outside of the case holding the root password and a second SSH key; unplugged, rewritten at each test. Gives: one mechanism, in the one place nobody has to remember. Costs: destroyed with the machine — which costs nothing, because credentials for a machine that no longer exists are worthless. Test at build and annually. |
| CN | 01 | Disk encryption | Decided | Jack | None on the internal drives. Gives: unattended reboot, and a console recovery path that cannot be locked out. Costs: physical theft of the machine exposes its contents. The security boundary is the front door. |
| CP | 01 | Name resolution retained | Decided | Jack | 192.168.50.69 is a DHCP reservation keyed to the NIC’s MAC; titan.local comes from avahi. Gives: the address survives reinstall untouched. Costs: the name does not — it needs the hostname, the avahi package, mDNS opened in firewalld and nss-mdns, and it fails silently. Verify by resolving from another machine. |
| CU | 01 | Pre-install verification | Decided | Jack | Before installing, and before every later upgrade, check what the release actually broke. Point releases are not automatically safer than major ones — a regression is newest where the release is newest, and 13.7 was six days old when this was decided. Check, in this order: the errata page for the release; the release announcement and debian-devel-announce for post-release problems; the bug tracker for release-critical bugs against debian-installer, since installer regressions are the ones a fresh build cannot route around; kernel regressions affecting btrfs and XFS specifically, because a filesystem bug is the one class that loses data rather than time; and that the NVIDIA driver supports the shipped kernel. Verify the image checksum and its signature, not just the checksum. Gives: the difference between a known risk and a discovered one. Costs: twenty minutes, and sometimes the answer is to install the previous point release and update into the current one — same packages, older installer. This is a standing gate, not a one-off: it applies to every install and every upgrade, including the ones AK exists to roll back. |
| AW | 02 | What runs on the host | Decided | Jack | Base OS, admin tooling and platform only. No heavyweight, internet-facing or experimental workloads. Gives: the host stays comprehensible and rarely changes. Costs: a few admin services (docs, backup) are deliberate exceptions. |
| AX | 02 | GPU sharing | Decided | Jack | Time-slicing, proven with a test pod before anything depends on it. Gives: every namespace can reach the one card. Costs: no VRAM isolation and no fault isolation — an oversized model can disturb transcoding. MIG does not exist on consumer Blackwell, and passthrough is unavailable. |
| AY | 02 | Ingress | Decided | Jack | Traefik v3, in the cluster, holding 80 and 443 and routing every domain by name with a certificate each. Everything internet-facing runs in a container, which rules out a reverse proxy as a host service whatever its merits. Gives: every HTTP requirement passes with no configuration — WebSockets for Foundry, Jellyfin and Cockpit; no default request body limit, where nginx would reject Foundry’s 300 MB uploads at 1 MB; and no response buffering, so Jellyfin streams. Native TCP and UDP routing if the game servers are ever put behind it. k3s ships it, so it costs nothing to adopt. Costs: nothing is served while the cluster is down. Traefik Labs is a single VC-backed company (Lyon, MIT licence) rather than a foundation project — Envoy Gateway is the CNCF, multi-vendor alternative and fits equally well, at the price of not being bundled. Replacing an ingress controller is cheap, which is what makes the governance risk tolerable. Limit worth knowing: hostname routing needs SNI, and the Minecraft protocol is not TLS — so a game server can have a memorable name pointing at the address, but still needs its own port. Two Minecraft servers cannot share 25565 split by name. Rejected: Istio (a service mesh for eight services on one node); Cilium (fits, but replaces the CNI — a larger change than the problem). Note for anyone reading older guidance: ingress-nginx was archived in March 2026 and its intended successor InGate was archived in June, so most documentation found online now recommends something that no longer exists. |
| AZ | 02 | Cluster identities | Decided | Jack | system:masters for Jack via titan-k8s; titan-admins for day-to-day; titan-viewers read-only; per-namespace roles for everyone else. RoleBindings only — never ClusterRoleBindings for people. Gives: the same group-not-person model as the host. Costs: discipline, since a single wrong binding is the main way namespace isolation fails. |
| BA | 02 | CPU policy | Decided | Jack | Requests set, no limits. Gives: bursting into idle capacity. Costs: none at this scale. |
| BB | 02 | Memory policy | Decided | Jack | Requests at the floor, generous limits; everything Burstable. Gives: density. Costs: eviction order depends on usage over request, so requests must be honest. |
| BC | 02 | Packaging | Decided | Jack | Flux deploys from git; third-party components as pinned HelmRelease; our own as plain manifests. Never helm install by hand. Gives: the cluster matches the repository, and every change reverts. Costs: nothing reaches the cluster without a commit. |
| BW | 02 | IPv6 exposure | Decided | Jack | Explicit default-deny inbound v6 at the router. Gives: parity with v4. Costs: none — but required, because with static v6 there is no NAT accidentally hiding anything. |
| CH | 02 | SSH hardening | Recommended | — | Add fail2ban and set PermitRootLogin no. Gives: rate limiting and the removal of the most-targeted account. Costs: one package. Key-only authentication is already in force and is the control that matters. |
| CO | 02 | Documentation | Decided | Jack | The titan-support set with an index, markdown rendered, in a lightweight container behind Traefik — LAN and VPN only, never public. It describes the addressing, the account and group model and what is exposed, which is a map worth nobody else having. Gives: it obeys the container rule, needs no host service, and adds nothing to the host to maintain. Routing lives in one place with everything else. Costs: it is unreadable exactly when the cluster is down, which is when it is most wanted. That is accepted, because the copies that matter are offline anyway: a plain file copy on the backup drive, readable with no server and no network, which is the one that works during a rebuild; and the published artifact, readable from any device and the only copy that survives the house. Git remains the source of truth, so an update is a commit. |
| CS | 02 | Nothing phones home | Decided | Jack | Default-deny egress at the cluster level; outbound access is allow-listed per workload. Gives: a component that tries to call out fails because there is nowhere to go, rather than because a setting was right. Costs: every genuine external dependency — Modrinth, registries, ACME — must be named. Verify by watching egress, per CC. |
| CV | 02 | Egress control | Decided | Jack | Squid as an allow-listing forward proxy, with NetworkPolicy denying all other egress. Necessary because vanilla NetworkPolicy cannot express hostname rules — it understands CIDRs and selectors only, so “allow Modrinth” is not expressible against a CDN on rotating addresses. Run in plain CONNECT mode, never SSL-bump: the proxy sees the hostname, allow-lists on it, and intercepts nothing — no certificate authority to distribute, no man-in-the-middling your own traffic. Gives: the hostname control CS requires, and a log of every outbound request, which is what turns CS from an assertion into something auditable. GPL, community-maintained since 1996, no vendor. Costs: workloads reach it via HTTP_PROXY/HTTPS_PROXY, and software that ignores those variables simply fails to reach the internet — correct default-deny behaviour, but it reads as a bug the first time, so it belongs in the docs. |
| CX | 02 | Local name resolution | Decided | Jack | blocky in a pod on titan. Germany, Apache-2.0, stateless — one YAML file, no database and no volume. The router does port forwarding and nothing else: no DNS records, no firmware change. Gives: LAN clients resolve our own domains straight to 192.168.50.69 instead of turning round at the router, so the server sees real client addresses rather than the router’s, and LAN traffic stops crossing NAT twice. Configuration lives in git like everything else. Costs — and this one lands on the household, not on you. titan becomes the primary resolver, and it has to be primary: if the router is asked first it answers with the public address and the override never applies, so ordering cannot be used as a safety net. When titan is down, Windows clients lose about a second per lookup and then fail over. When titan is up and answering wrongly — a crashlooping pod, an empty config — NXDOMAIN is accepted as authoritative and the secondary is never consulted at all. There is no failover from a wrong answer, only from silence. Required, not optional: a liveness probe that resolves a known internal name, so an up-but-wrong pod restarts instead of lying; hostNetwork, keeping it out of the cluster’s own dependency chain rather than behind ServiceLB, which is documented as unreliable on UDP 53; and DNSStubListener=no in /etc/systemd/resolved.conf, because Debian’s resolver holds 127.0.0.53. Context: ASUS NAT loopback is enabled by default and cannot be switched off, so this is an improvement on a working path rather than a fix for a broken one. |
| BD | 03 | Admission control | Decided | Jack | Kubewarden in enforce mode, with Pod Security Admission underneath it. Both, not either — PSA lives in the API server and costs nothing, so it remains the floor if the policy server is unavailable. Between them they deny privileged pods, hostPath, host networking, host PID and IPC. Gives: the mechanism that makes a namespace a real boundary; CNCF-governed and SUSE-maintained, speaking Rego, CEL and Kyverno syntax so policies stay portable. Beyond PSA it can enforce two things already decided elsewhere — images only from allow-listed registries (supporting CS) and resource requests required (supporting BB). Costs: a webhook in the pod-creation path, roughly 250 MB, and one more component to upgrade. It validates and mutates but does not generate, so nothing auto-creates policy for a new namespace — BO handles that by creating it atomically instead, which is better than generation because there is no window. Install requirement: exclude kube-system and Kubewarden’s own namespace from the webhook, or a downed policy server blocks the pod creation needed to restart it. |
| BE | 03 | Isolation mechanism | Decided | Jack | Namespaces, not virtual clusters. Separation comes from RBAC, NetworkPolicy, ResourceQuota and admission policy. Gives: instant creation and teardown — which is what the agent workload needs — one coherent GPU view for every workload, no extra vendor, no control-plane overhead, and nothing to un-pick if a layer is added later. Costs: isolation depends on policy being correct. A ClusterRoleBinding where a RoleBinding was meant leaks everywhere, and CRDs are cluster-wide. Prove the boundary by attempting to escape it, per CC. |
| BF | 03 | Namespace per concern | Decided | Jack | Games, media, AI, projects and agents each get their own namespace with its own quota and default-deny network policy. Gives: segregation of concerns, and any one can be emptied without touching the others. Costs: they share one API server, so cluster-scoped objects are common ground. |
| CI | 03 | Max’s namespace | Decided | Jack | Max is admin within his own namespace and the games namespace, read-only cluster-wide, and deliberately not in titan-k8s. Gives: real autonomy over what he runs, plus the visibility to diagnose anything. Costs: he cannot install CRDs or cluster-scoped resources. Cluster-admin on the host cluster is root on the host by another route, so it is not on offer. |
| BG | 04 | Game servers | Decided | Jack | Minecraft and Satisfactory in the games namespace, day one, spun down when not wanted. Gives: present from the start. Costs: the largest memory consumer while up. |
| BX | 04 | Minecraft version | Decided | Jack | 1.21.11, Fabric. Gives: matches what the players run. Costs: mods must exist for it. |
| BY | 04 | Mod management | Decided | Max | Mods resolved from Modrinth at container start. Gives: updates without rebuilding an image. Costs: Modrinth becomes a runtime dependency of the server starting; the ? suffix makes individual mods non-fatal. |
| BZ | 04 | BlueMap exposure | Decided | Jack | Public, behind Traefik with TLS rather than its own port; tiles on the NVMe. Gives: a shareable map, served like everything else public. Costs: 12 GB of tiles, regenerable. |
| CA | 04 | Minecraft front end | Decided | Jack | Velocity moves onto titan and becomes the public front door; the lobby moves with it; deimos (192.168.50.9) is retired from the role. 25565 forwards to Velocity and nothing else. Gives: the modded server stops being directly reachable. Costs: ~2 GB while games are up, and the parallel-running fallback only exists while the Pi still serves. |
| CB | 04 | Minecraft resources | Decided | Jack | 7.5 GB heap inside an 8 GB limit; CPU request of one core with no limit; world data on the NVMe mirror. Gives: headroom outside the heap, bursting for chunk generation, and random writes on media that can take them. Costs: the single largest reservation on the machine. |
| CQ | 04 | Minecraft versions | Decided | Jack | Max owns the mod set, the versions, and pinned-or-floating. The build starts fully pinned and he unpins at will. Gives: a rebuild that boots, and an undo — configuration in git means returning to the last working commit rather than archaeology. Costs: the pinned baseline must be captured from a server that actually starts. |
| CR | 04 | Front-end requirements | Recommended | — | Velocity 512 Mi heap / 768 Mi limit; lobby 1 Gi / 1.5 Gi, tuned as a lobby — view and simulation distance 4–6, mob spawning off, generation disabled, adventure mode. Backends become ClusterIP with a NetworkPolicy admitting only Velocity; the forwarding secret becomes a long random value in a Kubernetes Secret. Gives: ~2.3 GB instead of ~6, and the proxy bypass closes because no route exists. Costs: sizing is unmeasured — derived from what the workloads do, and Max’s to overrule. |
| BH | 05 | Media | Decided | Jack | Jellyfin in the media namespace, transcoding on the GPU, library on the HDD array. Gives: hardware transcoding and bulk capacity. Costs: shares one GPU with everything else. |
| CJ | 05 | Jellyfin sign-in on the TV | Decided | Jack | Try the client setting first (Android TV: uncheck “Always ask for credentials”; Roku: Remember Me). Failing that, a non-admin living-room account with the password cleared and no delete or download rights. Gives: a TV that never asks. Costs: only acceptable while Jellyfin is LAN or VPN only — it must never be publicly exposed. Quick Connect does not help; it still needs a code each time. |
| BI | 06 | Backup split | Decided | Jack | restic to the 40 TB drive; separate repositories per source. Gives: deduplication and encryption at rest. Costs: a repository password that matters. |
| BJ | 06 | Samba placement | Decided | Jack | Samba on the host, not in a container. Gives: direct access to the backup drive without mounting it into the cluster. Costs: a host service, and a deliberate AW exception. |
| CK | 06 | Shared storage on the backup drive | Decided | Jack | A general SMB share alongside the backups, as two separate trees, two shares, two groups — titan-share and titan-backup. The backup tree is not writable by ordinary users. Gives: one drive serving both purposes. Costs: a writable share is reachable by ransomware, so the separation is what stops both copies going at once. |
| BK | 07 | Git server | Decided | Jack | Forgejo. Gives: GPL-3.0, a German non-profit, an LTS track. Costs: one more service to keep updated. |
| BL | 07 | Git modes | Decided | Jack | Forgejo is the origin; GitHub is a mirror where a project needs to be public. Gives: the source of truth stays in the house. Costs: mirroring to maintain. |
| BM | 07 | GitHub credentials | Decided | Follows | A scoped token per agent, never a personal account. Gives: revocable, attributable pushes. Costs: token lifecycle to manage. |
| BN | 07 | GitOps | Decided | Jack | Flux reconciles everything from git. Gives: the cluster is a function of the repository; every change reverts. Costs: drift is corrected automatically, so manual fixes do not survive. |
| BO | 07 | Agent isolation | Decided | Jack | A namespace per agent, created by the controller together with its NetworkPolicy, ResourceQuota and ServiceAccount in one operation. Gives: instant spin-up, a walkable filesystem when one dies, and no window where a namespace exists unprotected. Costs: the controller is responsible for policy at creation, since Kubewarden cannot generate it (BD). |
| BP | 07 | Agent egress | Decided | Jack | Deny 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16. Gives: an agent cannot reach the LAN, the host, Samba or the cloud metadata endpoint. Costs: anything an agent legitimately needs must be named. |
| BQ | 07 | Agent credentials | Decided | Jack | No host cluster token; scoped rights inside its own namespace only. Gives: the blast radius is one disposable namespace. Costs: none. |
| BR | 07 | zsh in agent image | Decided | Jack | No — oh-my-zsh is for humans on the host. Gives: a smaller image. Costs: none. |
| BS | 07 | Coordinator location | Decided | Jack | Runs from the laptop, not the server. Gives: it still works when titan is the problem. Costs: the laptop must be on to spawn agents. |
| BT | 08 | Agent coordination | Decided | Jack | A Postgres task board plus an MCP endpoint. Gives: agents coordinate without a shared filesystem, and options stay open. Costs: a database to run and back up. |
| BU | 09 | Certificates | Decided | Jack | TLS-ALPN-01 over 443. Gives: no port 80, no DNS API, no webhook. Costs: 443 must be reachable. |
| BV | 09 | Web control panel | Decided | Jack | Start and stop workloads from a browser. Cockpit is already installed and serving on :9090 and should be assessed before anything is built. Gives: spinning things up without a shell. Costs: later work; not in the first build. |
| CM | 09 | Max and the local model | Decided | Jack | The model is offered as a service endpoint with an API key, not as GPU device access. Gives: unlimited use with no ability to crash the card. Costs: direct GPU access for CUDA or training is a separate, deliberate grant. |
| CW | 09 | Foundry asset storage | Decided | Jack | Garage — AGPL-3.0, from the Deuxfleurs collective, a French non-profit rather than a company. A public-read asset bucket on the NVMe mirror, writable only by Foundry, with anything private in a separate bucket. MinIO is not an option: its community edition went to maintenance mode in December 2025 and was archived in April 2026. SeaweedFS is the alternative if a larger ecosystem matters more than governance. Gives: players fetch maps and audio straight from the store instead of through the Foundry process, which is the only reason to use S3 here at all. Costs: a third public service alongside Foundry and BlueMap. The part that is hard to undo: Foundry writes absolute URLs into world data — scene backgrounds, tiles, journals, playlists — as assets are placed. The endpoint hostname is therefore a commitment for as long as those worlds exist. Name it for the job rather than the implementation, so that replacing Garage later is a configuration change and not a database rewrite. Hostname not yet chosen, but it must be a hostname from the first upload. Foundry itself can be reached by IP quite happily — nothing records it. The asset endpoint is the exception: an IP used “for now” is written into every world created while it is in place, and static means stable rather than immutable, so changing ISP or moving the store off titan would break it. A provisional DNS record costs two minutes and can be repointed at anything; rewriting world databases cannot. Foundry already answers on a second domain, so this sits inside the wider question of how titan serves several domains (AY). |
| CY | 09 | Mail server | Open | — | Requirement (Jack): a small mail server with spam filtering and antivirus. A new use case — the first thing making the machine responsible for something people outside the house depend on receiving, so it is listed here rather than slipped into an existing row. The build is the easy half; deliverability is not, and is largely outside your control. Receiving on your own domains is very achievable. Sending to Gmail and Microsoft from an address with no reputation means spam folders or outright rejection for months, and no configuration fixes that — only time, or relaying through a reputable provider, which reintroduces a third party. AAISP is unusually good ground for this: static address, no block on port 25, and reverse DNS you can set, which most ISPs will not do and which is a hard requirement. (a) Mailcow — German, GPL, complete: Postfix, Dovecot, rspamd, ClamAV, webmail. Costs: docker-compose by design, so it sits awkwardly beside everything else here. (b) docker-mailserver — MIT, community, the same components in one container, fits a pod cleanly. Costs: assembled rather than integrated; no webmail. (c) Stalwart — AGPL, Rust, a single binary with JMAP and filtering built in. Costs: considerably younger than the others, for a service where losing mail is unacceptable. Budget 4–6 GB — ClamAV alone holds 1–2 GB resident, which is comfortable at 64 GB and tight at 32 (AV). Recommend (b) on fit with the rest of the design, (a) if you would rather have one supported thing than four assembled ones. |
Isolation is not one mechanism. These are independent: breaking one gains nothing on its own.
| Mechanism | Stops | Does not stop |
|---|---|---|
PSA restricted | Privileged pods, hostPath, host network, host PID/IPC — in the API server, so it cannot fail on its own | Anything needing a custom rule |
| Kubewarden | Image registries, required resource requests, anything beyond PSA’s three profiles | Nothing, while its webhook is down — which is why PSA sits underneath |
| NetworkPolicy, default deny | Reaching the LAN, the host, Samba, other namespaces, the metadata endpoint | Anything explicitly allow-listed |
| RBAC, RoleBindings only | Reading or changing another namespace’s objects | Cluster-scoped objects such as CRDs and nodes |
| ResourceQuota | One workload starving the others | Bad behaviour within quota |
| Not mounted | Any container reaching the backup drive — there is no path to request | Nothing. This is the strongest one |
Stated plainly: containers share the host kernel, so a kernel exploit escapes all of it. VMs would not, and they cost the GPU. What this design reliably stops is the realistic failure — an agent writing to the wrong place, a runaway process, an accidental port — which is the threat actually present.
The weak point is RBAC discipline. With one API server, a ClusterRoleBinding where a RoleBinding was meant grants across every namespace at once. That is the main way this design fails, and it fails silently. The runbook tests it by attempting to escape a namespace rather than by inspecting the YAML.
| Tier | Layout | Holds | If a drive dies |
|---|---|---|---|
| 2×4 TB NVMe | btrfs RAID1 | OS, containers, databases, game worlds, BlueMap tiles — everything random-write | Survives. Replace and rebalance |
| 2×5 TB SMR | XFS on mdadm RAID0 | Media library. Sequential reads only | Array lost. Restore from backup — which is why AE and AG matter |
| 40 TB USB3 | Single, ext4 | restic repositories, plus a separate shared SMB tree | Restore from source. Not redundant, and not intended to be |
These are drive-managed SMR: writing anywhere in a zone rewrites the whole band, so sustained random writes collapse. Nothing that writes randomly goes near them — and the filesystem choice follows the same logic, since copy-on-write scatters writes into exactly the pattern shingled media handles worst. Hence btrfs where snapshots and self-healing pay for themselves, XFS where they would only cost.
| Phase | What | Gate |
|---|---|---|
| 00 | Prove the backup by restoring from it | Hard stop. Nothing proceeds without a verified restore |
| 01 | Check the release errata and installer bugs first (CU), then install Debian 13, lay out disks, users and groups, snapper rollback, name resolution, recovery stick | Errata read and judged; then resolve titan.local from another machine |
| 02 | GPU on the host, k3s, GPU inside a container, Traefik, egress policy, docs | A test pod sees the GPU |
| 03 | Kubewarden, namespaces, quotas, network policy, RBAC | Attempt to escape a namespace and fail |
| 04 | Velocity, lobby, Minecraft, Satisfactory | A player connects through the proxy; the backend is unreachable directly |
| 05 | Jellyfin with NVENC | A transcode uses the GPU; the TV does not ask for a password |
| 06 | Backup server for Windows and Mac; shared tree | A restore from each source |
| 07 | Forgejo, Flux, the agent farm | An agent spawns, works, dies, and its filesystem is still walkable |
| 08 | Postgres task board, MCP endpoint | Two agents coordinate without shared storage |
| 09 | Foundry, projects, model endpoint, control panel | A model loads on request with nothing else changing |
| Thing | Why not |
|---|---|
| Offsite backup (AG) | Deferred knowingly. The consequence is that fire or theft takes both copies |
| Virtual clusters | Namespaces meet every stated requirement. vcluster or Capsule sit above namespaces, so either can be added later for one tenancy without unpicking anything |
| High availability | One machine. Redundancy here would be theatre |
| Monitoring stack | PSI at login and the diary answer the questions actually asked. Prometheus if that stops being true |
| Web control panel | Cockpit already exists and should be assessed first (BV) |