Agent evaluations — ZigBase

ZigBase includes a provider-neutral harness for testing whether a coding agent can turn a product idea into a secure, tested, deployed application. The harness runs a user-supplied command in an isolated workspace and grades artifacts and executable behavior. It does not inspect the agent’s transcript or trust the agent’s self-report.

The Genesis flagship passed its release gate on 2026-08-15: commit 3f3224d5 produced three consecutive score-4 runs with zero interventions using codex-cli-0.147-default. The sanitized summaries are preserved under evals/agents/evidence/genesis/2026-08-15-3f3224d5/. The immutable tag agent-eval-genesis-2026-08-15 resolves to the recorded full commit 3f3224d50d2bc8e45b3888011ac118397103cd75. The PocketBase migration flagship passed its release gate on 2026-08-16: commit 9a72b656 produced three consecutive score-4 runs with zero interventions using the same agent. Its sanitized summaries are preserved under evals/agents/evidence/pocketbase/2026-08-16-9a72b656/. The immutable tag agent-eval-pocketbase-2026-08-16 resolves to the recorded full commit 9a72b656b7df8cc732aa249b7ba9f84ddd89dba6. The Rails API migration flagship passed its release gate on 2026-09-01: commit 8fec10ae produced three consecutive score-4 runs with zero interventions using codex-cli-0.150-default. Its sanitized summaries are preserved under evals/agents/evidence/rails-api/2026-09-01-8fec10ae/. The immutable tag agent-eval-rails-api-2026-09-01 resolves to the recorded full commit 8fec10ae45aca023bf1f46da3f73d4a320735cde. The Rails full-stack coordination flagship passed an earlier evaluation checkpoint on 2026-09-03: commit da8b9d4e produced three consecutive score-4 runs with zero interventions using claude-code-2.1.259-opus. Those runs predate later review fixes and do not qualify the current branch tip; they remain as historical evidence under evals/agents/evidence/rails-fullstack/2026-09-03-da8b9d4e/. The immutable tag agent-eval-rails-fullstack-2026-09-03-14 resolves to the recorded full commit da8b9d4edbc096d8aaaeaaa6a41a89b981e0c704. Deterministic harness tests block CI; real-agent runs remain opt-in so model tokens and credentials are never required by ordinary builds.

Install the scenario skill

The runner installs the scenario’s declared skill into the isolated workspace:

• genesis uses skills/zigbase-app-genesis/;
• pocketbase uses skills/zigbase-migrate-pocketbase/;
• rails-api uses skills/zigbase-migrate-rails-api/.
• rails-fullstack uses skills/zigbase-migrate-rails-fullstack/.

Scenarios that exercise repository tooling list its canonical paths in repository_files; the runner materializes those files into the workspace instead of maintaining fixture copies.

When running an agent outside this harness, use that product’s normal local-skill installation mechanism and preserve the directory name. The harness deliberately does not know how any provider stores credentials.

The skill embeds canonical ZigBase references. CI fails if those copies differ from the matching files under docs/, if required metadata disappears, or if SKILL.md grows beyond its context budget.

Select an agent command

Set ZIGBASE_AGENT_COMMAND_JSON to a JSON array of argv entries. The runner never invokes a shell. It substitutes an entry that is exactly {workspace} with the fresh writable workspace and an entry exactly {prompt} with the absolute prompt-file path. If an entry is exactly -, the prompt body is also sent on stdin.

For a Codex CLI configured to load the skill:

export ZIGBASE_AGENT_COMMAND_JSON='["codex","exec","--ephemeral","--approve-for-me","--skip-git-repo-check","-C","{workspace}","-"]'
python3 -m evals.agents.run genesis --agent codex

For a commit-pinned Genesis release run, resolve the Zig package content hash before entering the agent’s isolated workspace and pass that non-secret value explicitly:

commit="$(git rev-parse HEAD)"
export ZIGBASE_EVAL_HASH="$(mise exec zig@0.16.0 -- zig fetch "git+https://github.com/valthon/zigbase.git#$commit")"
python3 -m evals.agents.run genesis --agent codex --pass-env ZIGBASE_EVAL_HASH

The runner itself supplies ZIGBASE_EVAL_COMMIT; the explicit hash lets a network-restricted agent write a complete immutable build.zig.zon without guessing or omitting Zig’s required integrity field.

Use the same command with pocketbase to evaluate an unattended migration from the committed, synthetic PocketBase 0.39.11 snapshot:

mise exec zig@0.16.0 -- zig build
ZIGBASE_EVAL_BINARY="$PWD/zig-out/bin/zigbase" \
  python3 -m evals.agents.run pocketbase --agent codex \
  --pass-env ZIGBASE_EVAL_BINARY

That scenario provides only the pinned snapshot, the supported converter, the migration skill, its prompt, and the read-only ZigBase binary built from the scenario’s repository revision. The agent verifies and copies that binary into its local deployment image; it must not substitute an older public image or clone a different source revision. The scenario requires a deterministic bundle, exact durable decisions, autonomous public signup, owner/file authorization, bcrypt rehash-on-login, historical timestamps, parity evidence, and a restartable named-volume Compose deployment. It neither downloads nor runs PocketBase.

Use rails-api to evaluate an unattended migration from the committed, frozen Rails 8.1 API-only snapshot:

mise exec zig@0.16.0 -- zig build
ZIGBASE_EVAL_BINARY="$PWD/zig-out/bin/zigbase" \
  python3 -m evals.agents.run rails-api --agent codex \
  --pass-env ZIGBASE_EVAL_BINARY

That scenario provides the frozen snapshot, the supported converter, the migration skill, its prompt, and the read-only binary. Ruby is never installed and the application is never booted: the recorded inventory is the only observation available, which is the point — an agent that fabricates one fails completion.

It is deliberately harder than a data copy. The snapshot carries a default_scope that hides a row from Rails itself, an encrypted attribute, single-table inheritance, a polymorphic association, database triggers behind a counter cache, and a serialized column — each a decision the converter refuses to take silently. It also carries the scope gate that gives the scenario its name: the only ERB in the application is a mailer layout, so the correct answer is that no frontend is retained, and a report claiming Rails views were migrated fails however good the data is.

Use rails-fullstack to evaluate the coordinating step after the backend and presentation adapters have both run against one frozen representative application:

python3 -m evals.agents.run rails-fullstack --agent codex

The live grader needs the already-built application binary and a Python environment containing Playwright plus its Chrome channel:

ZIGBASE_EVAL_BINARY=./zig-out/bin/zigbase \
PLAYWRIGHT_PYTHON=/path/to/playwright-python \
python3 -m evals.agents.run rails-fullstack --agent codex

The scenario supplies observed Rails routes, ZigBase OpenAPI, all-passing backend replay findings with their request capture, plus a generated Zigapagos v0.5.0 target, presentation manifest, and handoff. The agent must produce exact route decisions, run the shipped coordinator twice, preserve an unsupported Vue root as an explicit blocker, and reconcile every reviewed public rule. The grader independently reruns reconciliation, applies the schema, runs production doctor, serves the generated site with the pinned ZigBase binary, executes HTTP and Playwright parity, restarts the target, and boots a restored copy. It does not trust the agent’s operational claims.

Metacharacters are literal argv data. Do not wrap the JSON command in sh -c. The scenario caps command size, runtime, and captured output.

The child receives a small environment allowlist and isolated HOME/TMPDIR. If a command truly needs an environment credential, opt into that one name explicitly:

python3 -m evals.agents.run genesis --agent example --pass-env EXAMPLE_API_KEY

--pass-env never accepts runner-control variables, missing names, duplicates, or unbounded values. Prefer an agent’s existing authenticated local configuration when it can operate without copying a secret into the process environment.

Results and artifacts

The runner prints exactly one compact JSON result on stdout. Field order is stable:

{"zigbaseAgentEval":1,"scenario":"genesis","commit":"…","agent":"codex","started_at":"…","duration_ms":123,"agent_exit":0,"timed_out":false,"interventions":0,"completion":true,"rules_locked":true,"tests_green":true,"deployed":true,"score":4,"failures":[]}

The four booleans are independently graded. For genesis they mean:

• completion: expected app shape, trusted server logic, cursor/expand usage, and a successful build;
• rules_locked: production doctor output exactly matches the reviewed public-rule inventory;
• tests_green: in-process Zig tests and the declared client/browser boundary test pass; and
• deployed: pinned Compose config has durable /data, health and metadata respond, production doctor is clean, and exact teardown succeeds.

For pocketbase they mean:

• completion: the source and converter match their pinned hashes, the bundle verifies every output hash and expected row/file count, and the strict migration report is complete;
• rules_locked: syntax and full-depth rule lint plus production doctor reconcile exactly with durable decisions and security/public-rules.json, including members.createRule as a reviewed warning for open signup;
• tests_green: fixed schema/auth/data/file commands, the declared integration boundary, timestamps, relations, anonymous signup, owner denial/allowance, protected files, and bcrypt-to-argon2 login upgrade all pass; and
• deployed: production-shaped Compose serves health, metadata, migrated data, auth, and files before and after restart, uses named /data, passes doctor, and tears down exactly.

For rails-api they mean:

• completion: the snapshot is untouched and still observed, the bundle is bound to that snapshot’s inventory digest and attests every output, every blocker carries a decision with a usable rationale, row counts match including the row default_scope hides, and the report answers the scope gate without claiming the frontend moved;
• rules_locked: the emitted schema, security/public-rules.json, the report and production doctor all agree that the public surface is exactly users.create — anonymous signup — as a reconciled warning rather than a suppressed one;
• tests_green: the agent’s declared boundary passes, and the running target enforces the semantics the guide demands: the unauthenticated posts list, whose rule hides every row, answers 200 with an empty array rather than 401 or 403 — an access rule is not an authentication failure — a post the caller may not see is concealed as 404, the notifications list returns the actor’s own rows and only those, an invalid signup is a validation failure, and the migrated bcrypt credential logs in and rehashes to argon2id; the migrated author’s protected attachment serves with their bearer token and is concealed without it; and
• deployed: the rehearsed target is copied whole — the documented SQLite backup — a server boots on the copy, direct database inspection verifies the complete migrated row set and timestamps, and the migrated credential logs in there. A backup that cannot serve is not a backup.

The rehearsal steps themselves — schema dry-run then apply, auth imported separately from ordinary data with --preserve-timestamps, files installed, doctor, and the live probes — belong to tests_green, not to deployed.

For PocketBase, Rails API, and Genesis, the grader runs agent-authored build and integration tests after the deterministic rehearsal and deployment checks. This keeps command ordering stable and makes failures easier to attribute. Genesis keeps build:-backed Compose services supported by running that boundary immediately before docker compose up --build; image-backed services retain the normal tests-last order.

For rails-fullstack they mean:

• completion: the immutable source set is unchanged, the released presentation and handoff schemas validate, every route has a reconciled disposition, evidence retains producer semantics, and the manifest is exact canonical coordinator output;
• rules_locked: the reconciled auth boundaries and production doctor agree on the complete reviewed public surface, with no errors, skipped checks, or unreviewed public rules;
• tests_green: the schema applies, HTTP parity runs against a live same-origin target, the generated Playwright journey passes, and the stopped target restarts with parity intact; and
• deployed: a copy of the stopped complete data directory boots independently and serves both the generated frontend and API as a synthetic cutover; after it stops, the original unit boots again and accepts the pre-switch credential as an executable rollback.

Exit 0 means all four grades passed. Exit 1 means the harness or agent command failed or timed out. Exit 2 means the agent completed but deterministic grading found a product condition that needs attention.

Raw bounded stdout/stderr and grader command logs default to /tmp/zigbase-agent-evals/<scenario>-<timestamp>-<id>/. Choose another location with --artifacts-dir. Write a sanitized result copy below that directory with:

python3 -m evals.agents.run genesis \
  --artifacts-dir /tmp/zigbase-genesis-runs \
  --out results/latest.json

--out cannot escape the selected artifact directory. Raw logs may contain source, prompts, or credentials printed by the external command. Run directories are created with mode 0700 and process logs with mode 0600. Commands run in their own process group, which is terminated on a timeout or interruption. Keep logs local or in a short-lived restricted CI artifact; never commit transcripts. Only sanitized result summaries belong in release evidence.

Deterministic development checks

Real-agent execution is not part of ordinary CI. Run the blocking converter, contracts, fake-agent cases, negative grader fixtures, and Genesis live Docker boundary with:

ZIGBASE_TEST_BINARY=./zig-out/bin/zigbase \
  python3 -m pytest tests/pocketbase tests/rails tests/rails_fullstack tests/tools tests/agent_evals -q
python3 -m pytest --noconftest tests/admin/test_skill_sync.py -q
python3 tools/sync_skill_references.py --check
python3 -m ruff check \
  --extend-exclude evals/agents/scenarios/rails-fullstack/fixture/source/presentation-target/test/journey_playwright.py \
  tools/pocketbase tools/rails tools/replay evals/agents tests/pocketbase tests/rails \
  tests/rails_fullstack tests/tools tests/agent_evals
ZIGBASE_DOCKER_EVAL_TEST=1 \
  python3 -m pytest tests/agent_evals/test_genesis_docker.py -q

The live Docker test uses the pinned ghcr.io/valthon/zigbase:0.13.0 image, a unique Compose project, a random loopback port, and down -v --remove-orphans in every path. It requires access to a local Docker daemon. PocketBase grader tests use deterministic positive and one-failure-per-grade fixtures; a real PocketBase agent run supplies the migration-specific deployment for live grading.

Ordinary CI downloads the already-built ZigBase artifact, never a PocketBase release, and runs no model command. The committed PocketBase database, local files, bcrypt credential, and parity data are synthetic fixtures. Do not replace them with a production snapshot. Keep generated agent workspaces and raw logs local; commit only source fixtures, deterministic grader fixtures, and sanitized result summaries.

Three-run release evidence

Run any scenario from a fresh agent context with only its installed skill, scenario prompt, fixture, and ordinary repository-visible tools. Do not provide the grader source, an implementation plan, or coaching from an earlier failure. A failure resets that scenario’s consecutive count. Fix the owning tool, docs/skill, or grader and add a deterministic regression test before restarting.

The flagship criterion is three consecutive results from the same commit where all four booleans are true, score is 4, interventions is 0, and failures is empty. Preserve those sanitized summaries and record the agent/model identifier supplied to --agent.

The qualifying sets are the tagged 2026-08-15 Genesis 3f3224d5, 2026-08-16 PocketBase 9a72b656, 2026-09-01 Rails API 8fec10ae, and 2026-09-03 Rails full-stack da8b9d4e evidence linked above. Raw transcripts and scenario workspaces stay out of the repository.