I Lost the Cluster: Six Weeks Dark, and the Rebuild That Followed

Between March and April 2026 I published four retrospectives about my infrastructure. Eight AI agents orchestrated on one Oracle box. Forty-seven apps moved off Vercel onto a self-hosted Coolify cluster. That cluster torn down nine days later and rebuilt on k3s with ArgoCD, SOPS, and CloudNativePG. Then a trenches post about the bugs that showed up three days in.

I was proud of all four. The last one even opened with a joke about how every infra retrospective ends with "and now everything works."

On June 15, 2026, at 04:54 UTC, the last write-ahead log segment shipped to my off-site backup bucket. Nothing shipped after it.

I did not find out until July 21.

By the time I looked, both Contabo nodes had been suspended for non-payment, then terminated. The disks were wiped. The IP address of my primary node had been reassigned to a different Contabo customer — who was, at that moment, serving their own Nextcloud on it, on a domain my DNS records still pointed at.

Almost every app I run was dark from June 15 to late July. This is the post-mortem, and the rebuild.

Part 1: What actually happened

The setup that failed

The k3s cluster was three nodes:

node	provider	role
vps50	Contabo	k3s server, CloudNativePG primary, GitLab CE, MinIO, MongoDB, Ghost, Uptime Kuma
vps10	Contabo	async Postgres replica, Loki
ada	Oracle Cloud (Always Free, ARM)	agent node

Two of those three nodes were on the same paid account with the same provider. That is the whole story in one sentence, and everything below is a footnote to it.

The timeline

• ~June 15 — Contabo billing lapses. Services suspended. WAL archiving to Backblaze B2 stops at 04:54 UTC. Last base backup was 04:04 the same morning.
• ~June 16 — mbp23, the MacBook that held my SOPS age key, my node SSH keys, and my kubeconfig, dies. It never powers on again. Multiple repair shops fail to revive it.
• June 15 – July 21 — I am heads-down on client and agrobusiness work. Apps are down. I assume it's a transient thing I'll get to.
• July 21 — I sit down to fix "the cluster" — on a new MacBook, which has none of it: no SOPS age key, no node SSH keys, no kubeconfig, not one real .env. Everything I'd need to touch the fleet died with the old machine. I pay the outstanding Contabo bill through payment links buried in the suspension emails, because the web panel won't let me log in. I probe 173.212.219.250, see port 443 answering, and conclude vps50 is alive but wedged.
• July 22 — I discover the server answering on that IP is not mine. rDNS says vmd200597.contaboserver.net. It serves nextcloud.zumbahnhof40.de. Both my VPSes are terminated, both disks wiped, and my primary node's IP now belongs to a stranger. vps10 had already passed Contabo's 30-day recovery window and was unrecoverable by policy.

That July 21 reading — "the box is up, the cluster is just wedged" — is the part that still bothers me most. I was reading liveness off a machine I did not own and had no way to distinguish from my own. A port answering is not proof of identity.

What died, ranked

Gone, no copy anywhere:

• GitLab CE — issues, merge requests, wikis, CI history, container registry. It ran as Omnibus with its own bundled Postgres inside a 60 GiB PVC, not on CloudNativePG. My backup job only ever touched the CNPG cluster. There was no gitlab-rake backup job. I had believed for months that GitLab was covered.
• Loki — all log history, on the node that got terminated first.
• MinIO object storage, MongoDB, Uptime Kuma history — no off-node backups at all, ever.
• Ghost — the blog engine and its content. (The posts themselves survived in a separate export; more on that below.)
• All Postgres writes between June 15 and June 30.

Survived:

• Every repository. GitLab CE one-way mirrored to GitHub, so the code was safe even though the surrounding project management was not.
• The CNPG app database, through June 15, in B2.
• ada — the Oracle node. It's on the Always Free tier, so it was never billed, so it never lapsed. The one node I hadn't been paying for is the one that lived.

And one thing I had already written off. For weeks I carried sahelprosperity-wv in my head as the unrecoverable loss — GitLab-only, never mirrored, gone. It wasn't. The work was sitting on the main branch of the sahelprosperity GitHub repo the entire time, invisible to me because production deployed from master and master was the only branch I ever looked at.

That is its own small lesson, and not a flattering one: my inventory of what I owned was as stale as my backups. I spent part of an outage mourning something I already had. If you keep a list of what you'd lose, check it against git branch -a before you believe it.

The honest cause

It is tempting to write this as "self-hosting bit me" or "k3s was too complex." Neither is true, and both would be a cheap ending.

The actual failure chain:

1. Billing was an unmonitored production dependency. I watched pods. I did not watch invoices. An unpaid invoice took down more infrastructure than any outage in the previous two years.
2. My backups covered one database, not my platform. I had a mental model that said "Postgres is backed up, therefore my data is backed up." Six stateful services said otherwise. I never wrote down the list of things that hold state — if I had, the gaps would have been obvious in ten minutes.
3. I never tested a restore. My infra docs referenced two runbooks: restore-postgres-from-b2.md and replace-k3s-node.md. Neither was ever written. An untested backup is a hypothesis.
4. Key custody was a single point of failure that HA cannot fix. Three nodes, two providers, replicated Postgres — and one laptop holding the only copy of the SOPS age key and every SSH key. When that laptop died, my cluster was cryptographically unreachable regardless of how healthy it was. High availability of compute means nothing if access has a single copy.
5. Two of three nodes shared one failure domain. Same provider, same account, same invoice. I built what looked like a distributed system and it had the blast radius of one machine.

There is a smaller, more embarrassing finding too. When I later got into ada and looked at disk usage, a 146 GB ClickHouse volume turned out to be 99.9% ClickHouse's own internal logs — trace_log at 127 GB, text_log at 18 GB. The actual analytics data I cared about was 324 KB. A disk that full almost certainly caused kubelet disk-pressure evictions and contributed to the instability I'd been shrugging at for weeks.

Part 2: The rebuild

Ten days, roughly, from "I understand what happened" to "everything is green." Here's the order it went in, because the order mattered.

Step 0 — Stop pointing my domains at a stranger

Before restoring anything, I had to stop actively endangering myself and my users. My DNS still pointed dozens of hostnames at IPs that now belonged to other people.

I audited both zones and deleted:

• 85 A records aimed at the two reassigned Contabo IPs, across afrotomation.com and tioye.dev
• 96 stale TXT records — dead _acme-challenge entries and external-dns heritage junk

Zone sizes went from 95 → 15 records on afrotomation.com and 143 → 42 on tioye.dev.

Two of those records deserve calling out. cloud.afrotomation.com still pointed at the reassigned IP — meaning any device still running a Nextcloud sync client would have been offering credentials to a stranger's server. And Cloudflare's security scanner had flagged the second reassigned IP as an exposed RDP host: the new owner was running RDP on an address my DNS was still advertising as mine.

If you take one operational thing from this post: when a server dies, deleting its DNS records is the first task, not the last. A dead A record is not inert. It's a redirect to whoever gets that IP next.

Step 1 — Emergency landing zone on Netlify

I needed things visible before I needed them right. Netlify was the fastest path to that.

This immediately hit a licensing wall: neither Netlify's nor Vercel's free tier will deploy private organization-owned repos. So I transferred the deployable repos from the afrotomation org back to my personal GitHub account, and pushed the priority apps out:

• tioye.dev — the portfolio. The domain had actually lapsed into expired-grace during the outage; I renewed it and transferred the registrar to Cloudflare the same day.
• auth.afrotomation.com — the SSO server, which everything else depends on. This had to come up first.
• afrotomation.com, plus ClickRise, SahelFoods, Bugginator, Sahel Prosperity.

Reviving SSO took three stacked fixes, each hiding the next:

1. My password hash predated the June 15 backup, so the credentials in the restored database were older than the ones in my password manager. I generated a fresh Better Auth scrypt hash locally and updated the row directly.
2. Sign-in then returned 429 for everyone. The rate limiter was failing closed because its Upstash Redis credentials had died with the cluster. I changed it: production without Redis now degrades to in-memory limiting, with RATE_LIMIT_FAIL_CLOSED=true available to restore strict behavior deliberately.
3. get-session then threw "Failed to decrypt private key." The JWKS row in the database was encrypted with the old BETTER_AUTH_SECRET — which was gone with the cluster. Deleting the row let Better Auth regenerate the keypair.

Every one of those three is a "your secrets were cluster-side" problem in a different costume.

Netlify's own limits then bit: I burned 363 of 300 monthly credits, and the failure mode was strange — external uptime checkers reported 200s while the sites were genuinely unreachable from some edge locations. Don't trust third-party "it's up globally" checks for an over-limit account. I upgraded, and reachability was restored immediately.

Step 2 — Get back into the Oracle box

ada had survived. I could not log into it.

The authorized SSH key was on the dead laptop. I tried the documented escape hatches:

• OCI Run Command — unavailable. The Ubuntu Minimal image ships 11 plugins, and Compute Instance Run Command isn't one of them.
• OCI Bastion — created the bastion and a managed SSH session, watched it hang in Creating for eight minutes, and gave up. Not a firewall problem, since public SSH answered fine. The plugin reported Enabled and Running and simply did not work.

What worked was a boot-volume swap: detach ada's boot volume, attach it as a data volume to a second throwaway instance, mount it, write my new public key into authorized_keys, detach, reattach, boot. Crude, reliable, no vendor tooling involved.

One thing to know if you ever do this: ada has two users, opc at UID 1000 and ubuntu at UID 1001. Assuming 1000 is the login account will waste you an hour.

Step 3 — Demolish the zombie

ada was still running a k3s agent pointed at a control plane that no longer existed. ingress-nginx was 404ing on 80/443 with an afrotomation.com certificate that had expired on July 23. The API server was down, etcd quorum permanently lost. Nothing there was recoverable.

Uninstalling reclaimed 164 GB — the box went from 190/193 GB used to 35 GB.

A gotcha worth writing down: if both k3s-uninstall.sh and k3s-agent-uninstall.sh exist on a node, the server uninstaller bails with "Additional k3s services installed, skipping" and removes only the systemd unit. The agent uninstaller is the one that actually deletes /var/lib/rancher.

Step 4 — Coolify, and the four things that blocked it

Coolify 4.1.2 went on the box. The dashboard lives on port 8000 and I deliberately never opened that port publicly — it's reachable over Tailscale only.

Then four fleet-wide blockers, in the order they surfaced:

1. Ingress: the OCI security list only allows port 22. Not the instance firewall — the VCN itself. I proved it with tcpdump: 0 packets captured while curling from outside. The confusing part is that Oracle's virtual network completes the TCP handshake before dropping you, so nc cheerfully reports success on a port that will never deliver a byte. Use curl or tcpdump, never nc, to test this.

The fix wasn't allowlisting Cloudflare's CIDRs. ada was already running a healthy cloudflared tunnel left over from an earlier era. I repointed its ingress at the new Traefik and the origin stayed sealed — curl http://<public-ip>/ returns nothing at all, while everything serves fine through Cloudflare.

2. The tunnel has to speak HTTPS to Traefik. Routing *.afrotomation.com to http://127.0.0.1:80 produces an infinite redirect loop, because Coolify's Traefik force-redirects HTTP→HTTPS and the response bounces back out through Cloudflare. The working shape:

  - hostname: "*.afrotomation.com"
    service: https://127.0.0.1:443
    originRequest:
      noTLSVerify: true

noTLSVerify is fine here — the hop is loopback and the tunnel is already encrypted.

Also: cloudflared tunnel route dns only works for the zone the tunnel's cert covers. Point it at a tioye.dev hostname and it will silently create tioye.dev.afrotomation.com instead of erroring. Second-zone hostnames need the CNAME created by hand.

3. bun does not install on ARM + Alpine — and it broke 47 of my 55 Dockerfiles. Every one of them had RUN npm install -g bun@X, which fails on aarch64 musl with Failed to find package "@oven/bun-linux-aarch64". Bun's npm distribution has no musl+ARM build. The one-line fix, which also respects my no-Docker-Hub policy since the GCR mirror carries it:

COPY --from=mirror.gcr.io/oven/bun:1.2.17-alpine /usr/local/bin/bun /usr/local/bin/bun

4. Deploy keys were disabled at the GitHub org level, so Coolify could not clone a single private repo. One org setting, forty broken deploys.

(There was a fifth, chronic one: stale bun.lock files from Renovate PRs that were never rebuilt. bun install --frozen-lockfile dies with "lockfile had changes, but lockfile is frozen." bun install --lockfile-only, commit, move on. This same problem had already bitten me on Netlify.)

With those cleared, deploys became routine: dedicated read-only deploy key per repo, POST /applications/private-deploy-key, DNS via the tunnel, deploy. About three minutes per app. As of August 1, all 41 resources in the Coolify project report running.

Step 5 — The data

One Coolify-managed PostgreSQL instance, one database per app.

The image matters: postgis/postgis is amd64-only — the exact wall that had kept Postgres off the ARM node in the old cluster. The multi-arch community build imresamu/postgis:17-3.5 works, reporting PostgreSQL 17.6 on aarch64.

Then I restored the B2 backup into a scratch instance and merged it into the live databases. Additive only — INSERT ... WHERE NOT EXISTS (pk), never update, never delete, converges to zero rows on a re-run. Run it repeatedly to a fixpoint, because child rows need their parents from an earlier pass.

Roughly 14,000 rows came back:

database	rows recovered
clickrise	9,260
bookshelf	4,462
sahelfoods	190
codeniserver	24
everything else	~22

Four staging traps each produced a "successful" merge that silently skipped most tables. The only reason I caught them was asserting that the staged table count equalled the source table count:

1. \restrict in plain-SQL dumps. Modern pg_dump wraps plain output in \restrict/\unrestrict; psql then rejects the \. COPY terminators and most tables fail to load. Use --format=custom and pg_restore.
2. Missing uuid-ossp. Columns default to uuid_generate_v4(); without the extension, every such table fails to create.
3. ALTER SCHEMA public RENAME TO bk moves the extensions with it, so those defaults become bk.uuid_generate_v4() — which won't exist in the live database. Strip defaults and FK/CHECK constraints from the staging schema after renaming (keep the primary keys; the merge needs them).
4. Per-account ID drift. The backup and the live database issued different IDs for the same email. You have to build an old→new map by joining on lower(email) and rewrite every foreign key onto users. Hard-coding one admin ID recovers almost nothing — that mistake is what initially hid two OAuth clients owned by a different test identity.

Two apps turned out not to have lost data at all — they'd drifted behind their own migrations. bookshelf was nine Prisma migrations behind, blocked by a failed migration inherited from its previous host that ended in a GRANT to a role that didn't exist on the new box. sahelfoods had no Drizzle migrations table whatsoever and only 61 of its 86 tables. Both recovered once the schema caught up.

Every application database is now on the box. Zero Neon references remain in any deployed app.

Step 6 — Split the fleet on purpose

The old architecture put everything behind one control plane. The new one deliberately does not:

tier	platform	what lives there
Dynamic apps	Coolify on Oracle ada (4 OCPU / 24 GB ARM)	41 resources — Next.js apps, Postgres, supporting services
Static sites	Cloudflare Pages	6 CRA/static builds, direct-upload via wrangler
Overflow / legacy	Netlify	11 remaining non-cornerstone sites
Blog	dev.to	43 posts

Cloudflare Pages gotchas, since they cost me time: it rejects any file over 25 MiB (a CRA source map killed one deploy — GENERATE_SOURCEMAP=false), it sets CI=true which promotes CRA ESLint warnings to build errors, and its default npm ci fails across this entire fleet because every repo uses bun.lock and none has a package-lock.json.

The sizing constraint on the Coolify box is CPU, not memory: 24 GB fits about 40 containers, but 4 OCPUs realistically supports 25–30 apps with concurrent builds set to 1.

Step 7 — The blog moved to dev.to

Ghost died with the cluster. I did not want to run a blog engine again.

I considered Hashnode — I'd used it before — but its API went Pro-only in May 2026, and I'm not paying a subscription for write access to my own words. dev.to it is. All 43 posts are live there, and tioye.dev/blog renders from the dev.to API through a lib/devto.ts module that replaced the dead Ghost client with the same interface.

Two dev.to API lessons, both of which cost me real time:

• RSS-imported articles embed front matter inside body_markdown, and a published: false in that embedded block overrides the API's published field. You have to rewrite the body's front matter to actually publish them.
• The API rejects backdated published_at — "only future or current." So 23 older posts are all stamped with their import date, and the site parses each post's true date out of its opening "Originally published on…" line and sorts by that instead.

And one gotcha that had nothing to do with dev.to: after the swap, browsers reported "Failed to fetch" while server-side renders worked perfectly. My next.config.js CSP connect-src still allowed only the dead Ghost origin. When you change a client-fetched data source, update your CSP and your image remotePatterns in the same commit.

Step 8 — The things that quietly depended on the cluster

Weeks later I was still finding these.

My portfolio's project screenshots came from a k3s CronJob. With that gone, the grid fell back to a live third-party capture API on every card render, blew through its free daily quota instantly, and rendered "coming soon" placeholders across the whole page. The replacement is a headless Chromium container on the box, cron'd for Sunday mornings, uploading to Cloudinary — 41 real covers, zero placeholders.

(Docker note, since this one wasted an afternoon: headless Chromium needs --shm-size=1g alongside --disable-dev-shm-usage. With Docker's default 64 MB /dev/shm, the GPU process crashes mid-capture and writes no file at all — no error, no image.)

While auditing costs I also found the actual money leak, and it wasn't hosting. 99 repositories were running Renovate on bare config:recommended — no schedule, no grouping. Every dependency PR fired a CI build. That, plus 37 dead workflow files still pointing at a runner set that no longer existed, was the burn. Throttled, deleted, and Dependabot removed from the 30 repos where it overlapped with Renovate.

What I'd tell April-me

Write the list of stateful things. Not "is Postgres backed up" — an actual enumerated list of every service that holds state, with the backup mechanism next to each one, and a blank next to the ones without. Mine would have had six blanks and I'd have seen it in ten minutes.

A backup you haven't restored is a hypothesis. Both of my restore runbooks were referenced in documentation and never written. Restore into a scratch environment on a schedule, or you don't know what you have.

Get your keys off your laptop. Not "back them up eventually" — today. HA compute plus single-copy credentials is not a resilient system, it's a fragile system with expensive decoration. My cluster was fine. I just couldn't get in.

Billing is production. Put invoices and card expiries in the same alerting channel as your uptime checks. A lapsed payment did more damage to me than any technical failure in two years.

Spread the blast radius across providers, not just machines. Three nodes across two providers still meant two-thirds of my cluster on one invoice. The node that survived is the one on a tier that cannot lapse for non-payment.

Delete DNS for dead hosts immediately. A dangling A record isn't a broken link. It's a live pointer at whoever the cloud provider hands that IP to next — and in my case that was someone running an exposed RDP service on an address my domain was still vouching for.

Don't trust liveness you didn't design. I spent a day working from the belief that my server was up, because something answered on its IP. Check identity — a TLS certificate, a health endpoint you wrote, an rDNS lookup — not just a port.

Was self-hosting the mistake?

No, and I want to be precise about why, because "just use a PaaS" is the wrong lesson to draw from this.

Nothing here was caused by Kubernetes being hard, or by running my own metal. The cluster didn't crash. It was deleted, by a billing event, and I couldn't recover it because I'd centralized the recovery path into one laptop and one incomplete backup job. Those are operational failures I'd have committed on any platform. Vercel taught me the same lesson from the opposite direction last year — that's why I self-host.

What did change is the shape. The old fleet was one control plane, one key, one backup target: a distributed system with the blast radius of a single machine. The new one is three independent tiers on three independent providers, with credentials that live somewhere other than my desk. If the Oracle box vanishes tomorrow, the static sites stay up and the blog stays up, and the recovery path is a rebuild script and a password manager rather than a boot-volume swap and a prayer.

Six weeks dark is a real cost. But I'd been running on an untested backup for months and simply hadn't found out yet. I'd rather have found out this way than the next way.

Still open

Being honest to the end, this isn't finished:

• SSO's auth database is off Netlify but still on a managed provider — it hasn't been migrated to the box yet.
• Eleven legacy sites are still on Netlify pending review.
• Distributed rate limiting still needs a Redis to come back.
• Every credential that ever sat on those Contabo disks — OAuth client secrets included — still needs rotating. Someone else owned those disks. It is the longest item on the list and the easiest to keep postponing.

---

This is part of the Afrotomation Startup series — building an automation agency in public, including the parts that go badly. Previous entries: 47 Apps, One Week · From Click-Ops to GitOps · Three Days In.