Operations

Running it day to day: what to look at, what the common symptoms mean, and what is intentionally absent.

Observability

Logs. Structured JSON via structlog to stdout, collected by Compose or journald. Every line in a unit of work carries correlation_id, bound once at the API and carried through the Celery task or the desktop's in-process driver and into the engine. Beside it sits the identity it names: run_id for a research run, session_id for a session. The two are never interchanged — a research_runs.id is not a session and reading one as the other is how a trace stops joining. There is no print in application code.

The trace. The agent_logs table is the run trace. It is durable, ordered, and replayable after the fact — the same rows the live feed replays are the ones you read when debugging a run that finished yesterday.

Health.

Endpoint	Meaning
`GET /health`	Liveness. The process is up
`GET /health/ready`	Readiness. Database and Redis both answered; 503 until they do

Tracing. LangSmith is available via LANGCHAIN_TRACING_V2, off by default.

Metrics. GET /metrics serves Prometheus text on the server. It is unauthenticated, like /health, and deliberately sits outside /api/v1 so a reverse proxy can withhold it with one rule. Seven families, all counters and one build-info gauge:

Metric	What it answers
`research_run_outcomes_total{outcome}`	How many runs finished, and how. `completed`, `failed`, `cancelled` only — a run waiting at a gate has not finished and is not counted
`research_run_cost_usd_total`	Estimated spend of terminated runs
`research_run_cost_confidence_total{confidence}`	How far that figure can be trusted: `priced`, `unpriced`, or `unstamped`
`research_run_tokens_total{direction}`	Tokens attributed to terminated runs
`corpus_ingest_documents_total{outcome}`	Uploads by what happened to them
`corpus_ingest_bytes_total`	Bytes actually indexed
`research_build_info{version,git_sha}`	Which build answered this scrape

Read the cost two metrics at a time. Cost estimation returns 0 for endpoint-defined providers (ollama, custom, openrouter), so research_run_cost_usd_total under-reports whenever research_run_cost_confidence_total{confidence="unpriced"} is moving, and says nothing either way when {confidence="unstamped"} is. Cap spend at the provider.

No research content, no user, project or run identifier, and no free-form text reaches a label: every value comes from a closed list, so the series count is fixed and cannot grow with usage.

Not exposed on the desktop, by design. The desktop app records the same counters through the same code; it serves no /metrics, because nothing scrapes a laptop.

Deferred, and named rather than faked: per-role token attribution, retrieval latency, and run duration. The last is worth stating plainly — the pipeline records elapsed_seconds only on a path the run surface never takes, so the column is NULL on every run and every bundle. There is no measured compute duration to expose, and deriving one from timestamps would report how long the reviewer took to approve.

Runbook

Symptom	First checks	Likely cause and action
Session stuck `PENDING`	Worker logs; `celery inspect ping`; is Redis up?	The worker is down or the broker is unreachable. Restart the worker; the task is picked up
Session stuck `RUNNING`	Tail `agent_logs` for that session; lock TTL	The worker crashed mid-run. The lock expires and the session is resumable from its checkpoint. If a budget or time limit was crossed it self-fails with the reason
Live feed empty, research completes	Is the SSE endpoint reachable? Is a proxy buffering?	An intermediary is compressing the stream. The API sets `no-transform` and `X-Accel-Buffering: no`; make sure a CDN or extra proxy honours them
All searches failing	Retriever chain logs — which retrievers errored	Tavily/Brave keys missing or exhausted and DuckDuckGo rate-limiting. Add a key; check the cache hit rate
Provider 429s or errors	Provider status; per-role routing	Switch a role to a different provider. Budgets prevent runaway retries; nothing auto-retries an expensive task
Cost spike	`sessions.total_cost_usd` by day; diff the evaluation results	A model or prompt change without an evaluation run. Revert; check the price catalog matches the provider's published prices
Reported cost is `$0.00` but the bill is not	Which provider is routed	`openrouter` and `custom` are unpriced by the catalog, so the cap cannot fire and cost reads zero. Cap at the provider
Project chat returns 503	`GET /projects/{id}/memory/status`	No embeddings provider is reachable. It fails closed rather than answering ungrounded
Memory looks incomplete	Same endpoint: `pending_reports`, `stale_models`	An ingestion failed, or chunks were written by an embedding model no longer configured

Useful commands

# Service state and logs
docker compose -f docker-compose.full.yml ps
docker compose -f docker-compose.full.yml logs -f worker

# Is the worker actually consuming?
docker compose -f docker-compose.full.yml exec worker \
  celery -A app.workers.celery_app.celery_app inspect ping
docker compose -f docker-compose.full.yml exec worker \
  celery -A app.workers.celery_app.celery_app inspect active

# Readiness
curl -fsS http://localhost:3031/api/v1/../../health/ready || \
  docker compose -f docker-compose.full.yml exec api curl -fsS http://localhost:8000/health/ready

# Schema state
docker compose -f docker-compose.full.yml exec api alembic current

A worker that answers a ping may still be importing LangGraph and the provider clients. That import cost lands entirely on the first task — measurably tens of seconds — and is why CI waits for an explicit preload signal rather than for a ping.

Cost control

Token usage is read from each model response and accumulated on the session, so sessions.total_cost_usd and the token columns are a real record rather than an estimate of an estimate — with the caveat that the price is a catalog lookup, and two providers have none.

Practical order of controls:

Cap at the provider. This is the only control that works on every route.
DEFAULT_MONTHLY_TOKEN_LIMIT — counts tokens actually sent, so it works everywhere.
MAX_INPUT_TOKENS per session — likewise.
MAX_COST_PER_SESSION_USD — only binds where the catalog has prices.

Depth is the user-facing cost dial; MAX_PARALLEL_TASKS trades wall-clock against how much can already be in flight when a ceiling is crossed.

Data and retention

Deleting a user cascades to their sessions, logs, chat messages, audit rows, and memory chunks, at the database level.
Deleting a session additionally drops its LangGraph checkpoints explicitly — cascades cannot reach them, and they hold fetched page content.
Archiving is reversible and loses nothing.
Corpus documents live in per-project SQLite files under CORPUS_DIR, outside Postgres.

Housekeeping

Runtime artifacts must not be committed: data/corpus/*.sqlite, __pycache__, .venv, .next, and build output are ignored. Note that an ignore rule does not untrack a file that is already tracked.

CORPUS_DIR defaults to a path resolved against the backend package root rather than the process working directory, so running from the repository root and from backend/ cannot produce two different corpus roots.

Committed evaluation results are write-once. Never modify a file under backend/evals/results/; add a new one named eval-<date>-<routing>-run<N>.json, because a date alone is not a run identity. CI enforces this — a frontend commit once silently overwrote a real measurement, and nothing failed.