real-test-pi-crew

real-test-pi-crew

End-to-end verification discipline for pi-crew changes. Distilled from the broker Phase-4 rollout (commits 1cb2dca → d599578 → 612e18b → 4186284, July 2026). The pain this skill prevents: shipping code that compiles + unit-tests-green but breaks in the user's live Pi session, or hangs the verifier worker.

When to use: after any change to src/runtime/crew-broker*.ts, src/ui/, src/config/, src/extension/registration/lifecycle-handlers.ts, src/runtime/child-pi-spawn.ts, src/runtime/plan-templates.ts, src/schema/team-tool-schema.ts (or any Type.Unsafe({...}) schema definition), src/extension/registration/team-tool.ts, workflows/*.workflow.md, or before any commit touching these paths. Schema changes additionally require Tier 9 (feature battery) because the team tool's TypeBox schema is validated by pi-ai BEFORE the handler runs — a too-strict or malformed schema breaks every action silently.

Core principle: disk ≠ live Pi

Two locations hold pi-crew state:

1. Source (src/, test/, package.json, workflows/, src/runtime/plan-templates.ts) — git-tracked, git diff shows it.
2. Bundle (dist/index.mjs) — pre-built, loaded by Pi at extension cold-start only.

The 3-way resolution order for dist/index.mjs (per index.ts:5-22):

1. dist/index.mjs (pre-built bundle) if present  ← DEFAULT since the v0.9.17 bundle-as-default rollout
2. Inline strip-types loading — fallback when bundle missing
   OR PI_CREW_USE_BUNDLE=0

Note on version pins: this skill mentions specific versions (v0.9.17, v0.9.46, v0.9.47) as anchors for when a behavior was introduced, not as a constraint on which version the skill applies to. The verification discipline (Tiers 1–8) applies to every pi-crew release. Verify the version pin is still accurate via git log --oneline -- index.ts and git log --oneline -- src/ui/run-dashboard.ts.

Workflow files are runtime data — workflows/*.workflow.md and task prompt strings inside src/runtime/plan-templates.ts are loaded per-call, NOT bundled. Edits take effect immediately, no rebuild needed.

The most common silent-failure mode: edit src/, run npm test (pass!), rebuild bundle (good md5!), but the session still has the old code because Pi wasn't /quit-ed + reopened.

Prerequisites

Before running any tier, verify these are available:

Working directory should be the pi-crew repo root:

cd ${PWD}
ls package.json  # must exist

CI integration

The skill maps to existing CI gates as follows:

To add Tier 1 to a pre-commit hook:

# .git/hooks/pre-commit (or via husky / pre-commit framework)
npm run test:critical || {
  echo "✋ test:critical failed — fix before commit"
  exit 1
}

To add Tier 1 to CI as a fast-feedback gate (under 30s):

# .github/workflows/fast.yml
- name: Critical unit tests
  run: npm run test:critical
- name: Disabled-path proof
  run: PI_CREW_BROKER=0 npm run test:critical
- name: Explicit-on proof
  run: PI_CREW_BROKER=1 npm run test:critical

Tier 1 — Critical unit tests (~25s, 101 tests, the only suite you need for broker/UI changes)

What: run the curated 14-file fast subset.

Why this exists: full npm run test:unit runs 642 files, >4 minutes. Verifier worker timeout is 300s → worker killed mid-run, run = "hang". The fix (introduced in commit 1cb2dca) splits out a test:critical subset covering exactly what changed in the broker/UI work.

How:

time npm run test:critical

Expected output: # tests 101 # pass 101 # fail 0 # duration_ms ~26000. (Count was 97 at v0.9.46; 101 since the model-routing merge — verify with the actual run; the skill's hard-coded numbers drift between releases.)

References:

Run after: any edit to src/runtime/crew-broker*.ts, src/ui/, src/config/, src/extension/registration/lifecycle-handlers.ts, or src/runtime/child-pi-spawn.ts.

Tier 2 — Three-path kill-switch proof

What: prove all three precedence paths in effectiveEnabled() still resolve correctly.

Why: any change to DEFAULT_BROKER (in src/config/defaults.ts:169) or effectiveEnabled() (in src/extension/registration/lifecycle-handlers.ts:819-833) can silently break the precedence chain. The chain:

PI_CREW_BROKER=0     → disabled (env always wins)
broker.enabled=false → disabled (config)
PI_CREW_BROKER unset → enabled (DEFAULT_BROKER=Phase 4 default-on)
PI_CREW_BROKER=1     → enabled (explicit; redundant under default-on)

How:

# 1. default path (whatever DEFAULT_BROKER.enabled is right now)
npm run test:critical
# 2. env kill switch
PI_CREW_BROKER=0 npm run test:critical
# 3. env explicit-on (must still work under default-on)
PI_CREW_BROKER=1 npm run test:critical

All three must show # pass 101 # fail 0. Measured times in this session (2026-08-11): ~26s for default, ~26s for PI_CREW_BROKER=0, ~26s for PI_CREW_BROKER=1 (varies ±1-2s run-to-run).

References:

Tier 3 — Typecheck + bundle rebuild + md5 sync

What: prove the bundle actually contains the source you just edited.

How:

npm run typecheck    # ~20s, exits 0 with "strip-types import ok"
npm run build:bundle # <1s, prints "[build-bundle] dist/index.mjs NNNN KB in NNN ms"
md5sum dist/index.mjs

Compare the printed md5 against what the user's Pi session loaded. If they differ → the session is running stale bundle.

References:

Tier 4 — Bundle sync into a live Pi session

What: ensure the user's running Pi sees your changes.

The immediate-vs-rebuild rule (which edits take effect without a rebuild):

• workflows/*.workflow.md edits → immediate, no rebuild, no restart
• src/runtime/plan-templates.ts taskTemplate strings → immediate, runtime data
• Everything else (src/ edits, package.json) → must npm run build:bundle THEN user /quit + reopen Pi

How to verify in this session:

md5sum dist/index.mjs
# then in Pi session, the user runs `md5sum` in a shell tool
# if they differ, user needs to /quit + reopen

How to verify in a fresh pty/tmux session without disturbing the user's main Pi:

tmux -S /tmp/sock new-session -d -x 160 -y 50 -s pi \
  "cd ${PWD} && exec pi 2>&1"

References:

Tier 5 — Live TUI probe via tmux send-keys

What: drive a real Pi session's keystrokes from the shell, capture screen state.

Why tmux and not raw pty: tmux gives you a clean separation — session persists across your bash commands, capture-pane gives ASCII screenshot, send-keys with hex escapes covers \x1b[A (legacy CSI), \x1bOA (app-cursor-mode), and Kitty-protocol variants.

How:

# Spawn (160x50 fits ~standard TUI)
tmux -S /tmp/sock new-session -d -x 160 -y 50 -s pi \
  "cd ${PWD} && exec pi 2>&1"

# Wait for pi to start
sleep 2

# Send slash command
tmux send-keys -t pi '/team-help' Enter
sleep 1
tmux capture-pane -t pi -p | tail -40

# Send raw escape sequence (app-cursor-mode up arrow)
tmux send-keys -t pi $'\x1bOA'
sleep 0.5
tmux capture-pane -t pi -p > /tmp/screen-after-up.txt

Key gotcha: terminals send arrow keys as one of 3 byte sequences. pi-crew's matchesKey() helper (src/ui/key-utils.ts:37-42, the keyOf() function) normalizes all of them — but verify it does in your probe:

References:

Tier 6 — Live TUI probe via Python pty (bulk keys + diag)

What: send many keys in sequence + capture per-keystroke diag output.

When to use: when you need to probe dispatch across multiple keypresses, or want to verify each key reached the component's handleInput.

How (simplified inline example — for the full hardened script with zombie reaping, non-blocking read, and escape-sequence decoding, use scripts/pty_probe.py directly):

#!/usr/bin/env python3
"""pty_probe.py — bulk-key + diag probe for pi-crew TUI components."""
import os, sys, time

CMD = ['pi']
ENV = dict(os.environ)  # keystroke diag env var REMOVED (see note below)

pid, fd = pty.fork()
if pid == 0:
    os.execvpe(CMD[0], CMD, ENV)
else:
    time.sleep(2)  # initial pi startup
    keys = [
        'j', 'j', 'k',                      # vim nav (run dashboard)
        '\x1b[A',                            # legacy CSI up
        '\x1b[B',                            # legacy CSI down
        '\x1bOA',                            # app-cursor-mode up
        '\x1bOB',                            # app-cursor-mode down
        'q', 'q',                            # quit (double-tap)
    ]
    for k in keys:
        os.write(fd, k.encode())
        time.sleep(0.3)
    time.sleep(1)
    sys.stdout.write(os.read(fd, 65536).decode(errors='replace'))

⚠️ The inline code above is a teaching example. For real use, run the bundled script (scripts/pty_probe.py, 161 lines) which adds zombie reaping (_reap_child), non-blocking read (select.select with 5s timeout), exec error handling, and --keys escape-sequence decoding:

python3 scripts/pty_probe.py [--keys '\x1bOA,q,q'] [--cwd /path] [--startup-sleep 3]

The inline code works for a quick one-off but leaks a zombie pi process on exit.

Keystroke diag env var REMOVED (2026-08-10): PI_CREW_BROKER_DIAG_UI=1 made run-dashboard's handleInput write a [PI-CREW-DIAG] line to stderr per keystroke. It was removed in e3ee6fe2 (PR-B5: remove TEMP DIAGNOSTIC from run-dashboard, UI-8) — there is no replacement in src/. To prove keystroke arrival now, rely on screen-change evidence (Tier 5 tmux capture-pane before/after each key, or the pty output diff): a key that changes screen state reached the TUI; a key that does not was consumed or never arrived. Capture the probe output to a file with 2>&1 | tee /tmp/pty-probe.log and diff the rendered frames.

References:

Tier 7 — Smoke team run (verifier prompt doesn't hang)

What: prove the verifier worker completes within RESPONSE_TIMEOUT_MS (300s).

Why this is its own tier: test:critical covers unit-level invariants, but the verifier LLM is a separate failure mode — it reads the verifier prompt from src/runtime/plan-templates.ts:143, 190 (taskTemplate strings) or from workflows/*.workflow.md:24, 30, 31 (workflow verifier sections), then decides which bash command to run. If the prompt says "Run tests" without specifying which, the LLM runs npm test and the worker hangs at 300s with exit 143.

How (from parent Pi session — team is a tool, not a shell command):

# illustrative — the actual tool takes positional + named params:
#   team action='run' team='fast-fix' workflow='fast-fix' goal='...' async=false
team:
  action: run              # run | status | events | cancel | retry | ...
  team: fast-fix           # team (a role-set): default / fast-fix / implementation / parallel-research / research / review
  workflow: fast-fix       # workflow (a phase DAG): default / fast-fix / plan-execute / implementation / review / research / parallel-research / pipeline / chain
  goal: "Smoke-verify <X>. Run `npm run test:critical && npx tsc --noEmit` once, cache output, report exact pass/fail counts + total time. Confirm verifier completes without hang (must be <300s)."
  async: false             # synchronous: wait for completion before returning

The team tool is described in the agent's system prompt. Use team action='status' <runId> to inspect mid-run, team action='events' <runId> <limit> for the event log, team action='cancel' <runId> to abort.

Real measured outcomes from this session:

References:

Two known failure modes for verifier:

1. Verifier LLM runs npm test (full unit + integration suite, >4 min) instead of npm run test:critical. Symptom: worker killed with exit 143 after exactly 300s. Fix: rewrite the verifier prompt to specify the exact fast command AND include "Do NOT run npm test or npm run test:unit".
2. Verifier LLM improvises with a clean-cache npm test run anyway. The cache directive ("cache to .crew/cache/", "do NOT re-run") catches this — the second worker that observes a cached log should not re-run.

Tier 8 — Bundle-vs-session md5 sync (operational check)

What: the concrete md5 comparison step that proves Tier 4's claim. Tier 4 explains when you need a rebuild; Tier 8 is the command you run to confirm the session picked it up. Run Tier 8 as the final integrity check after Tier 3-4.

How:

# Disk
md5sum dist/index.mjs

# Session (ask user to run in their pi shell tool)
# The symlink is in the CONSUMING project, not inside pi-crew:
readlink ../node_modules/pi-crew/dist/index.mjs 2>/dev/null \
  || readlink "$(npm root -g)"/pi-crew/dist/index.mjs \
  || md5sum "$(npm root -g)"/pi-crew/dist/index.mjs
# (the consuming project loads pi-crew via this symlink — see index.ts:5-22)

If the two md5s match → session is on the latest code. If not → user must /quit + reopen Pi.

Agent-inside-session caveat: when the agent doing the testing runs inside the very Pi session under test, the agent cannot restart its own session — only the user can. Pattern that works: (1) edit source + rebuild bundle, (2) ask the user to /quit + reopen, (3) on resume, re-check md5sum dist/index.mjs then issue a probe tool call (e.g. team action='list'). If the probe returns the old error (e.g. Unknown type, Validation failed for tool team), the session did NOT reload — there may be multiple pi PIDs and the user reopened a different one. Verify with ps -eo pid,lstart,tty,args | grep pi which PID is yours (the one whose session log is being appended to right now).

References:

Tier 9 — Feature battery (live action coverage)

What: drive the team tool + subagent tools through a spread of actions from the parent Pi session to prove the full surface works end-to-end, not just one smoke run.

Why this exists: Tier 7 proves one team run completes. But pi-crew has ~50 team actions plus 4 subagent tools (Agent, crew_agent, get_subagent_result, crew_agent_steer), dispatched through several code paths (sync run, async run, chain, parallel, direct subagent). A schema or registration regression can break some paths while others still pass. The battery catches path-specific breakage.

When required: any change to src/schema/team-tool-schema.ts, src/extension/registration/team-tool.ts, src/extension/team-tool/*.ts (handler dispatch), or the subagent-tool registration. Optional but cheap for any change — the read-only actions are free.

How (run from the parent Pi session — these are tool calls, not shell):

1. 9a. Read-only actions (free, no subagent spawn — run these first as a fast battery):
   • team action='list' — teams/workflows/agents
   • team action='recommend' goal='...' — planner routing
   • team action='health' — run-state scan
   • team action='doctor' focus='zombies' — orphan subagent scan (read-only)
   • team action='status' runId='<recent>' details=false — compact
   • team action='events' runId='<recent>' — full event lifecycle
   • team action='summary' runId='<recent>' — cost/by-role report
   • team action='get' resource='workflow' team='implementation' — resource inspect
   • team action='explain' runId='<recent>' — markdown render
   • team action='worktrees' runId='<recent>' — workspace listing
2. 9b. Spawn paths (cost tokens — one probe each is enough):
   • team action='run' sync (fast-fix, trivial goal) — proves sync run + child-pi spawn + provider-extension loading
   • team action='run' async=true — proves background dispatch
   • team action='run' chain='"A" -> "B"' — proves sequential handoff (chain runner). Omit workflow — passing workflow:'chain' forwards it to each step and fails fast (~58ms silent; issue #44).
   • Agent direct subagent — proves the direct-subagent tool
   • crew_agent run_in_background=true then get_subagent_result — proves background subagent lifecycle
3. Acceptance: every action returns without Unknown type / Validation failed for tool team / empty error text; every spawn path completes with consistency=1 and the expected probe token in the agent output.

Real measured outcome (this session, after the v0.9.57 schema fix): 9a (15 team actions) + 9b (4 subagent tools / 3 run paths) exercised; all green; the two silent-failure modes that motivated this tier (Unknown type from Type.Unsafe without Kind, and Validation failed for tool team from empty-string-strict schema) were caught ONLY by this battery — Tier 1-8 all passed while the team tool was broken live. The session also surfaced the unauthorized-agent-edit anti-pattern (a chain-run agent edited chain-runner.ts mid-smoke) — see Anti-patterns.

Not covered by the cheap battery above — the actions below need extra setup, cost, or user confirmation. Run them only when the change touches their code path, and prefer a throwaway cwd / config so you don't mutate the user's real state. Organised by cost/safety. As of 2026-08-11 (extended battery, run report real-test-2026-08-11-scratchpad-I-batch.md), 9c/9e/9f have been exercised live once each — they are no longer unproven, but still require explicit scope+confirmation to re-run.

9c. Lifecycle / recovery (needs a running run — start an async run, then exercise these against its runId):

• team action='wait' runId='...' — block until completion
• team action='steer' runId='...' message='...' — inject a steering note mid-run
• team action='status' runId='...' details=true — full dump mid-run
• team action='cache' subAction='...' runId='...' — snapshot cache ops
• team action='checkpoint' runId='...' — state checkpoint
• team action='cancel' runId='...' — ⚠️ destructive (kills the run); use a throwaway run
• team action='invalidate' runId='...' — cache invalidation
• team action='resume' runId='...' / retry — resume a completed/failed run
• team action='respond' taskId='...' message='...' — mailbox reply (needs a waiting task)
• subagent steering: crew_agent run_in_background=true a long task (e.g. sleep 60), then crew_agent_steer while it runs, then get_subagent_result — proves the steer arrived (timing-sensitive; assert the agent's output reflects the steer)

9d. Destructive (⚠️ requires explicit user confirmation per the delegation policy — never run unprompted):

• team action='prune' keep=<N> — delete old finished runs
• team action='cleanup' — sweep stale workspaces/state
• team action='forget' runId='...' — delete one run's state
• team action='doctor' focus='zombies' is READ-ONLY (safe) but the follow-up kill <PID> it suggests is destructive — confirm with the user before killing

9e. Admin / mutation (mutates config or workflow files — use a scratch project cwd or back up first):

• team action='create' resource='team' ... / update / delete — manage teams/agents/workflows
• team action='init' / config / validate / autonomy / settings — project setup
• team action='workflow-create' / workflow-save / workflow-delete / workflow-get / workflow-list — workflow CRUD
• team action='import' / imports / export — run data portability
• team action='parallel' tasks=[...] — parallel dispatch (spawn path, costs tokens per task)

9f. Background / scheduled (expensive or niche):

• team action='run' runKind='goal-loop' — the goal loop runs many turns judging an objective; smoke with a trivial objective + low maxTurns (e.g. 2) to prove dispatch without burning budget
• team action='schedule' cron='...' ... / scheduled / subAction='remove' — cron; assert the job registers then remove it (cleanup)
• team action='auto-summarize' / anchor / auto_boomerang — background features; assert no-throw on a completed run
• team action='api' — programmatic surface

Acceptance for 9c–9f: the action returns a structured result (not Unknown type / not an empty error), and for spawn/lifecycle paths the run reaches the expected terminal status. For 9d/9e, the mutation is reversible or confined to scratch state.

Anti-patterns (the cost is real, observed in this session)

Failure symptoms + recovery

When a tier fails, the recovery is usually quick. Match the symptom to the cause:

Performance budget (per-tier soft limits)

If a tier runs over the hard limit, stop and investigate — don't bump the budget silently. The budget exists precisely so regressions in test runtime (which usually means a regression in test setup/teardown) are caught early.

Edge cases

macOS specifics

Non-standard paths

The skill assumes pi-crew is at ${PWD} (the directory you cd'd into). If you have it elsewhere:

export PI_CREW_ROOT=/path/to/pi-crew
cd $PI_CREW_ROOT
# Now ${PWD} resolves correctly inside the skill

The cd ${PWD} calls appear in the Prerequisites section, Tier 4, Tier 5, and the Quick reference section — all use the same path. Once you cd into the repo once, all commands that reference ${PWD} resolve correctly. Tier 6 uses scripts/pty_probe.py --cwd instead, and Tier 8 uses readlink (no cd needed).

No-tmux fallback

If tmux is not installed, use Tier 6 (Python pty) instead. Tier 6 doesn't depend on tmux; it spawns pi directly under a pty. The trade-off: Tier 5 gives you capture-pane for ASCII screenshots; Tier 6 gives you per-keystroke diag output.

Stale /tmp/sock (tmux session already exists)

If a previous Tier 5 run left a /tmp/sock server running, tmux new-session -S /tmp/sock will reuse it instead of creating a fresh session. The new pi instance attaches to the existing session, which may have leftover state. To force a fresh session:

tmux -S /tmp/sock kill-server 2>/dev/null  # clean up
tmux -S /tmp/sock new-session -d -x 160 -y 50 -s pi "cd ${PWD} && exec pi 2>&1"

Multiple concurrent Pi sessions

When the user has multiple Pi sessions open (e.g., main + scratch), each loads the same dist/index.mjs. The md5sum check is global — if any session loaded the old bundle, you need to restart ALL of them, not just the one you're testing in. Tier 8 covers this only for the user's "main" Pi; warn them about siblings.

Broker on Windows

broker.enabled=true is silently no-op on native Windows (no unix-domain socket). Users on WSL1/2 get full broker behavior. Don't waste time running Tier 7 smoke tests on native Windows — the verifier will run fine but the broker won't actually do anything. Use PI_CREW_BROKER=0 to skip the broker entirely.

Cross-skill notes

This skill overlaps with these built-in/project skills. Pick the right one:

The "skill stack" for a typical pi-crew change:

1. Edit src/
2. tier 1 (test:critical)        ← this skill
3. tier 2 (3-path proof)         ← this skill, if broker change
4. tier 3 (typecheck + bundle)   ← this skill
5. tier 5/6 (live TUI)           ← this skill, if ui change
6. tier 7 (smoke team)           ← this skill, if plan/workflow change
7. commit + push
8. verify-before-complete        ← make the "done" claim with evidence

Maintenance

The skill mentions specific commits, line numbers, and version pins. As the code evolves, these will drift. Maintenance playbook:

The skill does NOT need to be updated for every commit — only when the cited lines/files move. Consider it a "living reference" not a "live spec".

Quick reference — exact commands

# Tier 1 (critical unit, ~25s, 101 tests)
npm run test:critical
# Tier 2 (3-path proof, broker changes only)
PI_CREW_BROKER=0 npm run test:critical
PI_CREW_BROKER=1 npm run test:critical
# Tier 3 (compile + bundle)
npm run typecheck
npm run build:bundle
md5sum dist/index.mjs
# Tier 4 (sync check — symlink is in the CONSUMING project)
readlink ../node_modules/pi-crew  # dev: → ../pi-crew
readlink "$(npm root -g)"/pi-crew  # global install
# Tier 5 (tmux probe)
tmux -S /tmp/sock new-session -d -x 160 -y 50 -s pi \
  "cd ${PWD} && exec pi 2>&1"
tmux send-keys -t pi '<key>' ; sleep 0.5
tmux capture-pane -t pi -p
# Tier 6 (pty probe)
python3 scripts/pty_probe.py 2>&1 | tee /tmp/diag.log
# Tier 7 (smoke team)
# from parent Pi session only — uses the `team` tool, not shell
# Tier 8 (final md5 sync — compare disk vs loaded bundle)
md5sum dist/index.mjs
md5sum "$(npm root -g)"/pi-crew/dist/index.mjs 2>/dev/null \
  || md5sum ../node_modules/pi-crew/dist/index.mjs
# Tier 9 (feature battery — from parent Pi session, tool calls not shell)
#   read-only: team action=list / recommend / health / doctor / status / events / summary / get / explain / worktrees
#   spawn:     team action=run (sync) ; team action=run async=true ; team action=run chain='"A" -> "B"'
#              Agent (direct) ; crew_agent run_in_background=true + get_subagent_result
#   reproduce the two silent schema failures:
#   node --input-type=module -e "import {Value} from '@sinclair/typebox/value'; import {TeamToolParams} from './src/schema/team-tool-schema.ts'; Value.Check(TeamToolParams, {action:'list', skill:'', config:{}})"  # throws 'Unknown type' = Type.Unsafe-without-Kind bug

Done-criteria checklist

Before claiming "tested":

• Tier 1: test:critical fresh-run, all pass (<25s). Count varies by release — was 97 at v0.9.46, 101 since the model-routing merge (v0.9.66); record the actual count in the report.
• Tier 2: 3-path proof all pass — required if you touched src/config/defaults.ts or src/extension/registration/lifecycle-handlers.ts
• Tier 3: npm run typecheck exit 0, npm run build:bundle exit 0
• Tier 4: bundle md5 matches what the session loaded (or user has /quit-ed + reopened)
• Tier 5/6: live TUI smoke for any src/ui/ change — keystroke reached handleInput
• Tier 7: smoke team run for any src/runtime/plan-templates.ts or workflows/*.workflow.md change — completed, no hang, verifier output under 60s
• Tier 8: final md5 sync check passed
• Tier 9: feature battery — required if you touched src/schema/team-tool-schema.ts, src/extension/registration/team-tool.ts, any Type.Unsafe({...}) schema, or any armed-role tool list (agents/*.md / src/config/role-tools.ts). 9a read-only batch all return clean; one probe per 9b spawn path (sync / async / chain / Agent / crew_agent+get_subagent_result) completes with consistency=1. Run 9c–9f only when the change touches their code path; at least one full 9c/9e/9f sweep per release is recommended so the battery stays proven (see real-test-2026-08-11-scratchpad-I-batch.md); 9d (destructive) requires explicit user confirmation. After every run: git status to catch unauthorized agent edits.
• Output report: save docs/real-test/reports/real-test-<YYYY-MM-DD>-<slug>.md from skills/real-test-pi-crew/REPORT-TEMPLATE.md, filled DURING the run with per-tier evidence (counts/md5/runId) — not reconstructed from memory afterward. This is what makes past runs verifiable instead of trust-the-summary.

"All 9 tiers pass" is a claim that needs per-row evidence. Tier 9 means 9a and 9b and whichever of 9c–9f applies to the change — not "9a passed, therefore 9 passed". If any required item above is unchecked or lacks concrete evidence (a number, an md5, a runId), the answer to "is it tested?" is no — say so explicitly instead of rounding up to "pass".

File-anchored references (full index)

Decision docs:

• docs/decisions/2026-07-21-broker-phase4-default-on.md — interim default-off (SUPERSEDED)
• docs/decisions/2026-07-22-broker-phase4-gated-on.md — default-on flip + risk + monitoring + rollback
• docs/decisions/2026-07-21-broker-windows-perms.md — Windows named-pipe perms + Phase-4 update note

Source files (critical paths):

• src/config/defaults.ts:155-187 — DEFAULT_BROKER + resolveBrokerEnvOverride
• src/extension/registration/lifecycle-handlers.ts:819-833 — effectiveEnabled() (precedence)
• src/runtime/child-pi-constants.ts:23 — RESPONSE_TIMEOUT_MS = 300_000
• src/runtime/plan-templates.ts:143, 146, 190, 193 — verifier taskTemplate + verificationCommand
• src/runtime/crew-broker.ts — broker server (per-connection gate, NDJSON framing)
• src/runtime/crew-broker-client.ts — client (isEventFrame() distinguishes event vs response frames)
• src/runtime/crew-broker-tokens.ts — BrokerTokenRegistry with timingSafeEqual
• src/runtime/broker-issuer.ts — per-run broker issuer (env injection at spawn)
• src/runtime/crew-broker-child.ts — child-side broker client wiring
• src/ui/key-utils.ts:37-42 — keyOf() using pi-tui matchesKey()
• src/ui/keybinding-map.ts — dispatch using matchesKey() (commit f05a10d)

Test files (the 14 in test:critical):

• test/unit/crew-broker-{handshake,stale-socket,feature-flag,server-gate,client-fallback,mailbox-observer,close-during-reconnect,steer-dedup,symlink-steering}.test.ts
• test/unit/keybinding-map.parity.test.ts
• test/unit/pi-tui-dispatch-probe.test.ts
• test/unit/session-utils-extract.test.ts
• test/unit/config-schema-sync.test.ts
• test/unit/child-pi-env-spread.test.ts

Integration tests (Tier 1 covers none — these are for full E2E):

• test/integration/crew-broker-msg.test.ts — 5 tests (Phases 1)
• test/integration/crew-broker-phase2-3.test.ts — events.subscribe + task.waitStatus + steer.push + escalate

Workflow files:

• workflows/fast-fix.workflow.md:24 — verifier prompt (commit d599578)
• workflows/default.workflow.md:31 — verifier prompt
• workflows/plan-execute.workflow.md:30 — verifier prompt
• workflows/review.workflow.md:31 — verifier prompt

Commits (chronological, the patterns they introduced):

• 1cb2dca — test:critical script + plan-templates verifier fix
• d599578 — 4 workflow verifier prompt fixes
• 612e18b — Phase 4 default-on flip (code + decision doc)
• 4186284 — mark default-off doc SUPERSEDED + index update

Real team runs (Tier 7 outcomes):

• team_20260722083504_cae04a2804a24d79 — full-implementation, 3/4 phases done, 04_verify hung (root cause investigation)
• team_20260722095143_2e58fce2ce91af19 — first fast-fix smoke, 3/3 PASS (after test:critical introduced)
• team_20260722100811_9bf95bebff2b052a — final fast-fix smoke, 3/3 PASS, verifier used cached output (449s wall-clock)