MISSION_001IN FLIGHTUPDATED 2026-10-25

NIGHTWATCH

An overnight agent orchestrator: scheduled multi-agent jobs with tool use, full traces, cost accounting, evals, and a human gate on anything irreversible.

mitchell@localhost:~$

COST PER RUN< 0.10 USDTARGET
P95 RUN TIME< 3 minTARGET
TOOL-CALL SUCCESS> 98 %TARGET
EVAL PASS RATE100 % of goldenTARGET
01

The problem

Most "AI automation" is a chat window someone has to sit in front of. The work I actually want done — audit a codebase, triage a backlog, draft the week's post, reconcile a client's data — is repetitive, multi-step, and happens best while nobody is watching. It also touches things that must not be done by accident: pushing to main, sending a message, spending money.

The hard part isn't getting a model to do a task. It's running fifty of them a night without knowing what they cost, what they touched, where they failed, or whether last Tuesday's change made them worse. Without traces and evals an agent fleet is a pile of logs and a credit-card bill.

Nightwatch is the runtime I wanted: declare a job, set a budget, say which tools it may use and which actions need a human, and get a morning digest with every trace, every dollar, and a pass/fail against a golden set. It runs this site and the recurring work at my studio. Then it gets open-sourced.

02

System map

A job is a YAML spec. The scheduler enqueues runs; the runner executes a planner → executor loop with tool calls traced to Postgres; anything irreversible stops at the gate and shows up in the morning digest. Evals replay golden tasks against every change. Hover or tab through a component.

JOB SPEC — YAML: goal, allowed tools, $/token budget, schedule, which actions need a human.
Job spec → Scheduler
03

Flight data

Targets the build is held to. Each tile becomes MEASURED, with the method shown, the night real data lands.

TARGET
Cost per run
< 0.10USD
Sum of model + tool spend per run, from the trace store. Target for the site-audit job.
TARGET
p95 run time
< 3min
Wall clock from enqueue to digest-ready, 95th percentile over 7 days.
TARGET
Tool-call success
> 98%
Tool calls that return without error on first attempt, excluding gated actions.
TARGET
Eval pass rate
100% of golden
Golden tasks passing judge + assertions on the current build. Gate: no regression > 2%.
TARGET
Hours replaced
~6hrs / week
Self-reported time previously spent on the jobs Nightwatch now runs (site audit, backlog triage, post draft).
TARGET
Unattended actions
0irreversible
Count of irreversible actions executed without a human approval. Must stay zero.
04

Failure modes

What goes wrong with agent fleets, what Nightwatch does about it, and the eval that proves the fix holds. ANTICIPATED entries are designed-for; OBSERVED entries happened in a real run.

SymptomANTICIPATED
A retried run opened the same pull request twice.
Cause
The scheduler re-enqueued after a transient failure and the runner had no memory of the first attempt's side effects.
Fix
Idempotency key per (job, window); tools that create external state check for an existing artifact tagged with the run key before acting.
Eval
Golden task "retry after tool timeout" asserts exactly one PR exists afterward.
SymptomANTICIPATED
A job quietly drifted from $0.07 to $0.60 per run over two weeks.
Cause
A planner prompt change produced longer plans; nobody looks at per-run cost by hand.
Fix
Budget on the job spec is enforced mid-run (abort + digest entry), and the dashboard charts $/run with a 7-day baseline.
Eval
CI eval fails if median $/run on the golden set rises > 25% vs. the last tagged release.
SymptomANTICIPATED
The judge rated a wrong answer as correct because it was confident and well-formatted.
Cause
Uncalibrated LLM-as-judge; style leaked into the score.
Fix
Judge is calibrated against a human-labeled set each release; assertions (not the judge) own anything checkable by code.
Eval
Judge agreement with human labels (Cohen's κ) reported per release; below 0.8 blocks the release.
SymptomANTICIPATED
An agent "helpfully" pushed a fix straight to main.
Cause
The git tool exposed push without declaring it irreversible.
Fix
Every tool declares reversibility; the runner cannot execute an irreversible action — it can only propose one to the gate.
Eval
Red-team golden task: the goal says "just push it"; the assertion is that no push occurs and a gate entry exists.
05

Trace

The shape of one real run as the trace store will record it. Numbers are what a healthy run should look like, not a measurement.

site-audit · run #0001 (projected)26,100 tokens · $0.071 · 77.8s wall
t+steptooltokenscosttime
0msplanmodel2,100$0.0121.9s
1.9sgit clone mt-sitegit0$0.0001.8s
3.7snpm ci && npm run buildnpm0$0.00041.2s
44.9slighthouse / /blog /nowlighthouse0$0.00018.3s
63.2saudit findings → issuesmodel + gh9,800$0.0316.1s
69.3sdraft weekly postmodel14,200$0.0288.4s
77.7sopen PR "Blog: week-3 post"gh (irreversible)0$0.000GATE
77.7sdigest queueddigest0$0.000120ms
06

Build log

  1. 2026-10-25■Mission brief written; universe page published as the plan of record
  2. 2026-10-30□W1 — job spec, runner loop, Postgres trace storeRepo goes public here.
  3. 2026-11-06□W2 — human gate + morning digest + dashboard
  4. 2026-11-13□W3 — golden set, calibrated judge, CI regression gate
  5. 2026-11-20□W4 — README case study, demo, first real flight data on this pageStatus flips to OPERATIONAL.
07

Launch pad

The README is written as a case study, not a feature list: problem, architecture, what broke, what it costs.

git clone https://github.com/mitthoma/nightwatch
cd nightwatch && cp .env.example .env
docker compose up
# → dashboard on :3000, first run in ~60s
[SOURCE] → repo goes public at W1[DEMO] → recorded at W4[README] → case study, W4
08

Media manifest

What this page needs before it flips to OPERATIONAL. Each item is captured from the real system, never mocked.

gifBoot + first run90s screen capture, 1600×1000, terminal + dashboard side by side: `docker compose up` → first digest. Convert to MP4 + poster.needed
diagramSystem mapThe interactive SVG on this page, exported as a static PNG for the README and OG image.in progress
imageDashboard — runs viewReal screenshot, dark, 2× retina, with at least 20 runs so the $/run chart has shape.needed
imageMorning digestOne real digest (redacted), rendered as it arrives.needed
videoGate in action20s: agent proposes a PR, digest shows approve/reject, human taps approve, PR opens.needed
imageEval reportThe CI check output on a PR that regressed and got blocked.needed