The tech stack
A Python core for the walk and the search, served by FastAPI, run as jobs by Celery workers over RabbitMQ, with Postgres for state, Redis for progress, and a Next.js UI, all in Docker Compose behind Caddy.
You
React 19
over HTTPS, to Caddy only
caddy
Caddy 2.11 · Let's Encrypt
HTTPS on 80 and 443; the only port open to the internet
web
Next.js 16 · Node 24
the UI and its BFF
api
FastAPI · Uvicorn · Python 3.14
REST and Server-Sent Events; stateless
postgres
PostgreSQL 18
models, concepts, searches, jobs
rabbitmq
RabbitMQ 4.4
the cpu and search queues
redis
Redis 8.8
one progress stream per job
worker-cpu
Celery 5.6
concept walks, from the cpu queue
worker-search
Celery 5.6
corpus scans, from the search queue
data/
NumPy · SciPy arrays
models, corpora, concepts, searches
builder
msr CLI · spaCy 3.8
builds models and corpora, then exits
You pick a model and a term. The browser posts to web's BFF, which validates the request and forwards it to FastAPI.
The API hashes the request's parameters into a cache key. A concept already built is returned at once. Otherwise one INSERT … ON CONFLICT against the active jobs either creates a job or joins the one already running, so two identical requests never walk twice.
A new job's task goes to RabbitMQ's cpu queue, and the page moves to the job's progress view.
A cpu worker takes the task, counts the attempt in Postgres, loads the model from disk, and runs the walk, appending progress to the job's Redis stream as it goes.
The API streams those events as Server-Sent Events, and web relays them to the browser, validated, with their ids, so a dropped connection resumes where it stopped.
The worker saves the concept to disk and Postgres and marks the job succeeded in one transaction. The last event names the concept, and the page opens it.
A search runs the same way on the search queue: its worker reads each concept’s terms from Postgres, scans the raw corpus chunk by chunk, and stores the ranked hits.
Caddy 2.11, Let's Encrypt
Terminates HTTPS in front of web, the only container reachable from the internet. It gets and renews its certificate on its own, redirects HTTP to HTTPS, and passes the Server-Sent Events streams through unbuffered. It runs only in production (compose.prod.yaml).
Next.js 16, React 19, Tailwind 4, Zod
Serves the UI and a backend-for-frontend: the browser only ever calls web's own /api/* route handlers, which call FastAPI inside the stack and validate every response against Zod schemas generated from the API's OpenAPI schema. It also proxies the API itself under /fastapi/, docs included.
FastAPI, Uvicorn, psycopg 3, Pydantic 2
Answers REST requests and streams job progress as Server-Sent Events. It holds no files and no state of its own: it reads Postgres, publishes tasks to RabbitMQ by name, and reads progress from Redis. A correlation id follows each request from the BFF through the API and into the task's message headers.
PostgreSQL 18, raw SQL
The source of truth: the model registry and vocabularies, concept tables, corpora, search hits, and every job's status and attempts. Plain-SQL migrations run once by a one-shot migrate container before anything else starts, and each service logs in as its own role with only the privileges it needs.
RabbitMQ 4.4, durable direct exchanges
The Celery broker. Each queue (cpu for concept walks, search for corpus scans) is durable and bound to a direct exchange of its own name. The consumer timeout is raised to about a year, since a walk or scan can run for hours while its message stays unacknowledged.
Redis 8.8 Streams
Holds only progress: one stream per job, which the worker appends to and the API reads. A page that reconnects sends its Last-Event-ID and replays from there. Streams are capped and expire an hour after their job ends.
Celery 5.6, NumPy, SciPy
One task at a time per container; scale by running more containers. Concept workers mount the models read-only and walk on threads; search workers mount the corpora read-only and never load a model. Both write their results to disk and Postgres, then mark the job done in the same transaction.
the msr CLI, spaCy 3.8
A run-and-exit container for the heavy, offline steps: downloading and parsing a dataset, counting co-occurrences, building the transition matrix and the comparison walks, and registering the model. Builds are staged and checksummed, so an interrupted one resumes where it stopped.
The whole Compose stack runs on a single EC2 instance (8 vCPUs, 32 GB). Models, corpora, concepts, and searches live on a separate 100 GB EBS volume; images and the database's volumes stay on the root disk.
Production is compose.yaml plus compose.prod.yaml, picked up from COMPOSE_FILE in .env, so the same task commands run it as run a laptop. The overlay only adds Caddy, restart policies, and log caps.
Caddy holds a Let's Encrypt certificate in a Docker volume and renews it itself. Plain HTTP redirects to HTTPS, browsers are told to stay on HTTPS (HSTS), and HTTP/3 is on.
Every long-running service restarts unless it was stopped on purpose, and Docker starts with the machine. A worker being stopped gets 30 minutes to finish its job first.
Each container keeps at most five compressed 20 MB log files, so a chatty service can't fill the disk.
Datasets are downloaded and models built on the box itself, in the builder container, while the site stays up. Each model and corpus registers in Postgres as it finishes and appears in the UI.
A task's message stays on the broker until the task returns, so a worker killed mid-walk has its job redelivered to another. Postgres counts the attempts, and a job fails for good after three.
Walks and scans run on separate workers, so a long walk never holds up a search, and a search worker never needs a model's worth of memory.
Concepts are cached under the model, root, and a hash of every parameter that changes the result. Changing only a scoring setting reuses the saved walk and skips the minutes it took.
Postgres, RabbitMQ, Redis, the API, and the workers sit on an internal Docker network with no route out. Only Caddy is published to the internet, and web on localhost; only the builder can reach the internet, for its downloads.
Pydantic models define the API; its OpenAPI schema generates the BFF's TypeScript types and Zod validators, and CI fails if they're stale. Python is checked with strict mypy.
A uv workspace with one package per deployable part (core, nlp, db, api, worker, build) and one lockfile, so each container installs only what it runs.