Amanda O'BrienWriting about distributed systems, AI-driven applications, and the craft of building things.https://cyberrhizome.ca/en-caClosing the Loop: What We Actually Shipped from the Roadmaphttps://cyberrhizome.ca/blog/10-closing-the-loop-what-we-shipped/https://cyberrhizome.ca/blog/10-closing-the-loop-what-we-shipped/Post #05 ended with a list of half-built features and a promise to ship them. This is the delivery receipt: compliance export, idempotency hardening, and the systems work that made them production-ready.Thu, 04 Jun 2026 00:00:00 GMTTriple-Defense Idempotency in a Crash-Prone Event Streamhttps://cyberrhizome.ca/blog/09-triple-defense-idempotency/https://cyberrhizome.ca/blog/09-triple-defense-idempotency/At-least-once delivery guarantees every message is processed — not exactly once. This post traces the three idempotency layers in the Sentinel-L7 Axiom pipeline and the specific failure modes each one defends against.Wed, 03 Jun 2026 00:00:00 GMTFour Signals for Quality Scoring in LLM Pipelineshttps://cyberrhizome.ca/blog/08-quality-scoring-and-domain-rag/https://cyberrhizome.ca/blog/08-quality-scoring-and-domain-rag/An LLM pipeline that produces no exceptions is not necessarily a pipeline that works. This post covers four signals for scoring the quality of compliance audit narratives produced by Sentinel-L7.Tue, 02 Jun 2026 00:00:00 GMTClosing the Gap: Persistent Tokens, Containers, and the Road to k3shttps://cyberrhizome.ca/blog/event-horizon-2026-06-03-closing-the-production-gap/https://cyberrhizome.ca/blog/event-horizon-2026-06-03-closing-the-production-gap/The previous post listed what EventHorizon needed before it could run in production. This one closes the list — persistent resume tokens, a multi-stage Dockerfile, a dependency-aware health check, and k3s manifests. Along the way, a silent data-loss bug that the test suite missed entirely.Fri, 29 May 2026 00:00:00 GMTGraduated Backpressure: Moving Beyond Binary Pauseshttps://cyberrhizome.ca/blog/07-graduated-backpressure/https://cyberrhizome.ca/blog/07-graduated-backpressure/The first version of Sentinel-L7's producer guard was binary: above the threshold it stopped, below it ran full speed. This post covers why that fails under real load and what graduated backpressure looks like in practice.Tue, 26 May 2026 00:00:00 GMTSelf-Healing Worker Pools: Embedding XAUTOCLAIM in Every Workerhttps://cyberrhizome.ca/blog/06-self-healing-worker-pools-xautoclaim/https://cyberrhizome.ca/blog/06-self-healing-worker-pools-xautoclaim/Sentinel-L7 runs two long-lived worker processes on Redis Streams. When a worker crashes mid-message, that message stays stuck in the PEL indefinitely. This post covers why the dedicated reclaimer daemon was replaced with embedded XAUTOCLAIM recovery in every worker.Tue, 19 May 2026 00:00:00 GMTThe Catalyst of the Flawed First Move; or, The Freedom of the Sacrificial Drafthttps://cyberrhizome.ca/blog/catalyst-or-sacrificial-draft/https://cyberrhizome.ca/blog/catalyst-or-sacrificial-draft/Why providing a deliberately flawed starting point is the fastest way to kill ambiguity and surface requirements.Mon, 11 May 2026 00:00:00 GMTWhat Comes Next: Multi-Server Deployments, Persistent Resume Tokens, and the Patterns That Transferhttps://cyberrhizome.ca/blog/event-horizon-2026-05-05-what-comes-next/https://cyberrhizome.ca/blog/event-horizon-2026-05-05-what-comes-next/EventHorizon is feature-complete as a learning vehicle. The interesting question is which of its patterns survive the move to a real production environment, and which need to be reworked entirely.Tue, 05 May 2026 00:00:00 GMTOne AI, One Context Filehttps://cyberrhizome.ca/blog/event-horizon-2026-05-02-one-ai-one-context-file/https://cyberrhizome.ca/blog/event-horizon-2026-05-02-one-ai-one-context-file/I started this project with two AI assistants and a sync problem. I'm finishing it with one assistant and three documents that act as the project's durable memory. Here's what changed.Sat, 02 May 2026 00:00:00 GMTWhat Comes Next: Multi-Tenancy, Compliance Exports, and Closing the Loop to EventHorizonhttps://cyberrhizome.ca/blog/05-sentinel-l7-what-comes-next-multi-tenancy-exports-and-closing-the-loop/https://cyberrhizome.ca/blog/05-sentinel-l7-what-comes-next-multi-tenancy-exports-and-closing-the-loop/A speculative look at what Sentinel L7 is pointing toward — the features that are half-built, the decisions still open, and what building this thing has actually taught me.Thu, 30 Apr 2026 00:00:00 GMTExtracting the App Factory to Make Tests Stop Lyinghttps://cyberrhizome.ca/blog/event-horizon-2026-04-29-app-factory-tests-stop-lying/https://cyberrhizome.ca/blog/event-horizon-2026-04-29-app-factory-tests-stop-lying/Importing a server module should not bind a port. It should not connect to MongoDB. It should not start a metrics interval. Once your tests can't import your server without ten unrelated things happening, you've already broken your own contract.Wed, 29 Apr 2026 00:00:00 GMTZod Won. Here's What It Cost.https://cyberrhizome.ca/blog/event-horizon-2026-04-28-zod-won-heres-what-it-cost/https://cyberrhizome.ca/blog/event-horizon-2026-04-28-zod-won-heres-what-it-cost/I evaluated Valibot. I picked Zod. I lost half a day to a UUID validator that was technically more correct than the previous version. Here is what I'd tell my past self about validation libraries.Tue, 28 Apr 2026 00:00:00 GMTTests, Rough Edges, and What's Still Ahead for RhizoBookhttps://cyberrhizome.ca/blog/06-rhizobook-testing-polish-next-steps/https://cyberrhizome.ca/blog/06-rhizobook-testing-polish-next-steps/The backend Jest strategy, frontend Vitest setup, two bugs found while writing tests, and an honest account of what the app still can't do.Sun, 19 Apr 2026 00:00:00 GMTResume Tokens and the Lie of 'Just Reconnect'https://cyberrhizome.ca/blog/event-horizon-2026-04-10-resume-tokens-and-just-reconnect/https://cyberrhizome.ca/blog/event-horizon-2026-04-10-resume-tokens-and-just-reconnect/The dashboard said everything was fine. Stats updated. Connection dot was green. Nineteen thousand events had moved through the pipeline and the live feed was a frozen screenshot.Fri, 10 Apr 2026 00:00:00 GMTTesting What You Can, Naming What You Can'thttps://cyberrhizome.ca/blog/event-horizon-2026-04-05-testing-what-you-can/https://cyberrhizome.ca/blog/event-horizon-2026-04-05-testing-what-you-can/There are parts of this system I have no automated tests for, and that is a deliberate design decision. The trick is being honest about which parts those are and why.Sun, 05 Apr 2026 00:00:00 GMTBackpressure Is Just a Number You Have To Pickhttps://cyberrhizome.ca/blog/event-horizon-2026-04-03-backpressure-is-just-a-number/https://cyberrhizome.ca/blog/event-horizon-2026-04-03-backpressure-is-just-a-number/AMQP prefetch is one integer in one config file. It is also the difference between competing consumers and one worker eating the entire queue while the others sit idle.Fri, 03 Apr 2026 00:00:00 GMTDraining 657 Messages and Running Out of AI on the Way Downhttps://cyberrhizome.ca/blog/04-sentinel-l7-draining-657-messages-and-running-out-of-ai-on-the-way-down/https://cyberrhizome.ca/blog/04-sentinel-l7-draining-657-messages-and-running-out-of-ai-on-the-way-down/Integrating a Python anomaly sidecar, hitting Gemini quota limits within minutes, and why a pluggable driver abstraction is not over-engineering.Thu, 02 Apr 2026 00:00:00 GMTAt-Least-Once Is a Lie Without an Idempotent Receiverhttps://cyberrhizome.ca/blog/event-horizon-2026-04-02-at-least-once-idempotent-receiver/https://cyberrhizome.ca/blog/event-horizon-2026-04-02-at-least-once-idempotent-receiver/How I almost shipped duplicate documents into MongoDB, and why I now think of unique indexes as load-bearing infrastructure rather than schema decoration.Thu, 02 Apr 2026 00:00:00 GMTRAG Is Easy. Getting It to Actually Retrieve Anything Is Hard.https://cyberrhizome.ca/blog/03-sentinel-l7-rag-is-easy-getting-it-to-actually-retrieve-anything-is-hard/https://cyberrhizome.ca/blog/03-sentinel-l7-rag-is-easy-getting-it-to-actually-retrieve-anything-is-hard/On building a policy knowledge base for compliance analysis — and the silent failure that made it look like everything was working when nothing was.Wed, 01 Apr 2026 00:00:00 GMTThree Strikes and the Dead Letterhttps://cyberrhizome.ca/blog/event-horizon-2026-04-01-three-strikes-dead-letter/https://cyberrhizome.ca/blog/event-horizon-2026-04-01-three-strikes-dead-letter/Why the worker republishes failed messages instead of nacking them with requeue=true, and how a single boolean flag is the difference between a healthy pipeline and a frozen one.Wed, 01 Apr 2026 00:00:00 GMTThe Seven-Step Shutdownhttps://cyberrhizome.ca/blog/event-horizon-2026-03-31-seven-step-shutdown/https://cyberrhizome.ca/blog/event-horizon-2026-03-31-seven-step-shutdown/There are exactly seven steps in EventHorizon's graceful shutdown, in exactly one order, and every reordering loses data in a specific way.Tue, 31 Mar 2026 00:00:00 GMTFour Planes, One Direction: The Shape of EventHorizonhttps://cyberrhizome.ca/blog/event-horizon-2026-03-30-four-planes-one-direction/https://cyberrhizome.ca/blog/event-horizon-2026-03-30-four-planes-one-direction/Why I named the four stages of the pipeline before I wrote any code, what naming bought me, and what backflow between them would have cost.Mon, 30 Mar 2026 00:00:00 GMTThe Hard Part: Appointment Scheduling, Conflict Detection, and Role-Aware Querieshttps://cyberrhizome.ca/blog/05-rhizobook-appointments-business-logic/https://cyberrhizome.ca/blog/05-rhizobook-appointments-business-logic/How conflict detection, a status state machine, and a single role-aware endpoint handle the core scheduling logic in RhizoBook.Sun, 29 Mar 2026 00:00:00 GMTWhat Strangers Shouldn't See: Securing the Public Provider APIhttps://cyberrhizome.ca/blog/04-rhizobook-securing-public-apis/https://cyberrhizome.ca/blog/04-rhizobook-securing-public-apis/How Prisma's include silently exposed sensitive fields on a public endpoint, and how explicit select fixed it at the query level.Sun, 08 Mar 2026 00:00:00 GMTSemantic Caching, or: Why I Spent Three Days Making a Cache Hit Workhttps://cyberrhizome.ca/blog/02-sentinel-l7-semantic-caching-or-why-i-spent-three-days-making-a-cache-hit-work/https://cyberrhizome.ca/blog/02-sentinel-l7-semantic-caching-or-why-i-spent-three-days-making-a-cache-hit-work/Fingerprints, embeddings, and the exact-timestamp bug that gave me a 0% cache hit rate until I stopped being clever and started being dumb.Fri, 27 Feb 2026 00:00:00 GMTWiring Up the Frontend: Next.js App Router, NextAuth, and a Rewrite Proxy That Saved Mehttps://cyberrhizome.ca/blog/03-rhizobook-nextjs-auth-proxy/https://cyberrhizome.ca/blog/03-rhizobook-nextjs-auth-proxy/How I structured the Next.js frontend with two route groups, wired up NextAuth JWT sessions, and replaced a fragile build-time env var with a server-side rewrite proxy.Sun, 22 Feb 2026 00:00:00 GMTBuilding a Compliance Engine That Doesn't Block the Web Serverhttps://cyberrhizome.ca/blog/01-sentinel-l7-building-a-compliance-engine-that-doesnt-block-the-web-server/https://cyberrhizome.ca/blog/01-sentinel-l7-building-a-compliance-engine-that-doesnt-block-the-web-server/Why I designed Sentinel L7 around three separate processes, Redis Streams, and a rule-based fallback that fires more than I expected.Thu, 19 Feb 2026 00:00:00 GMTBuilding RhizoBook: Structuring a NestJS Backend from the Ground Uphttps://cyberrhizome.ca/blog/02-rhizobook-backend-foundation/https://cyberrhizome.ca/blog/02-rhizobook-backend-foundation/Why I chose NestJS over plain Express, how I structured domains as modules, and the schema decisions that shaped the whole app.Sun, 01 Feb 2026 00:00:00 GMTWhy I'm Building an Appointment Schedulerhttps://cyberrhizome.ca/blog/01-rhizobook-intro/https://cyberrhizome.ca/blog/01-rhizobook-intro/From identifying a recurring problem in healthcare SaaS to designing a solution I actually want to use.Fri, 30 Jan 2026 00:00:00 GMTThe 72-Hour Christmas Miracle: My First Amigurumihttps://cyberrhizome.ca/blog/personal/mr-robot/https://cyberrhizome.ca/blog/personal/mr-robot/Crunch time in Santa's workshopSun, 28 Dec 2025 00:00:00 GMTon building in publichttps://cyberrhizome.ca/blog/on-building-in-public/https://cyberrhizome.ca/blog/on-building-in-public/Why I'm starting to write more and consume less — and what this site is for.Mon, 15 Dec 2025 00:00:00 GMT