sk-create-manual-testing-playbook

Manual Testing Playbook Creation

create-manual-testing-playbook is the manual-validation package workflow for the sk-doc family. It authors manual-testing-playbook/ packages for skills and systems that need reproducible operator-facing scenarios, evidence capture, release-readiness review, and realistic orchestration or multi-agent execution planning.

Core principle: keep shared rules in the root playbook, keep execution truth in per-feature files, and make every scenario deterministic enough that another operator can reproduce the verdict.

1. WHEN TO USE

Activation Triggers

Use this workflow when the request involves:

• /create:manual-testing-playbook.
• Creating manual-testing-playbook/manual-testing-playbook.md.
• Building one-file-per-feature manual test scenarios.
• Designing deterministic prompts, command sequences, expected signals, evidence, pass/fail criteria, and failure triage.
• Planning manual validation across multiple operators, agents, CLIs, MCP tools, or runtime surfaces.
• Converting an ad hoc release checklist into a reusable evidence-driven playbook package.
• Aligning a manual testing playbook with a feature catalog.

Keyword triggers: manual testing playbook, /create:manual-testing-playbook, testing playbook, playbook system, deterministic scenario, evidence collection, operator validation, multi-agent execution, release readiness.

Strong signals that a playbook is warranted:

• 5+ distinct features need manual validation.
• Release decisions depend on structured evidence.
• Realistic orchestration behavior matters.
• Multiple operators or agents will execute the same scenarios.
• Automated tests cover internals but not operator-visible behavior.

When NOT to Use

Use another sk-doc packet when:

• Test steps fit cleanly in a spec folder checklist.
• The feature is one-off or experimental.
• The system has only a few manually testable behaviors.
• Automated tests already cover the only meaningful acceptance criteria.
• The user asks for a feature catalog rather than executable validation scenarios. Use create-feature-catalog.
• The user asks for a README, install guide, skill, agent, command, benchmark package, flowchart, or changelog. Use create-readme, create-skill, create-agent, create-command, create-benchmark, create-flowchart, or create-changelog.
• The task audits, validates, scores, or optimizes an existing playbook without creating or extending the package. Use create-quality-control.

2. SMART ROUTING

Decision Rule

Decision rule:

Need reusable manual validation with captured evidence?
YES -> Create a playbook package
NO  -> Keep test steps in spec/checklist docs

Family Boundary

This packet owns manual testing playbook packages only. It consumes shared sk-doc standards from ../shared, but the advisor identity lives at the sk-doc hub root. Do not add a packet-local graph-metadata.json.

Smart Router Pseudocode

For this flat-reference packet, the canonical resilient router discovers resources at call time, guards and loads only what exists, scores the two authoring scopes, and returns a disambiguation checklist rather than silently loading nothing:

from pathlib import Path

SKILL_ROOT = Path(__file__).resolve().parent
RESOURCE_BASES = (SKILL_ROOT / "references", SKILL_ROOT / "assets")
DEFAULT_RESOURCE = "references/README.md"

# Two authoring scopes; keywords come from this packet's activation triggers.
INTENT_MODEL = {
    "root_playbook": {"weight": 4, "keywords": ["manual testing playbook", "/create:manual-testing-playbook", "testing playbook", "release readiness"]},
    "per_feature_scenario": {"weight": 4, "keywords": ["deterministic scenario", "evidence collection", "operator validation", "multi-agent execution"]},
}
UNKNOWN_FALLBACK_CHECKLIST = [
    "Confirm the target skill or system and the feature set under test",
    "Confirm root playbook index scope vs per-feature scenario file scope",
    "Confirm the evidence and pass/fail validation expectations",
]

def discover_markdown_resources() -> set[str]:
    docs = []
    for base in RESOURCE_BASES:
        if base.exists():
            docs.extend(path for path in base.rglob("*.md") if path.is_file())
    return {doc.relative_to(SKILL_ROOT).as_posix() for doc in docs}

def _guard_in_skill(relative_path: str) -> str:
    resolved = (SKILL_ROOT / relative_path).resolve()
    resolved.relative_to(SKILL_ROOT)
    if resolved.suffix.lower() != ".md":
        raise ValueError(f"Only markdown resources are routable: {relative_path}")
    return resolved.relative_to(SKILL_ROOT).as_posix()

def load_if_available(relative_path, inventory, loaded, seen) -> None:
    guarded = _guard_in_skill(relative_path)
    if guarded in inventory and guarded not in seen:
        load(guarded)
        loaded.append(guarded)
        seen.add(guarded)

def score_intents(request) -> dict:
    text = request.text.lower()
    scores = {intent: 0 for intent in INTENT_MODEL}
    for intent, cfg in INTENT_MODEL.items():
        for kw in cfg["keywords"]:
            if kw in text:
                scores[intent] += cfg["weight"]
    return scores

def route_manual_testing_playbook_request(request):
    inventory = discover_markdown_resources()
    loaded, seen = [], set()
    scores = score_intents(request)

    if max(scores.values() or [0]) < 4:                      # Tier 1: unclear scope
        load_if_available(DEFAULT_RESOURCE, inventory, loaded, seen)
        return {
            "load_level": "UNKNOWN_FALLBACK",
            "needs_disambiguation": True,
            "disambiguation_checklist": UNKNOWN_FALLBACK_CHECKLIST,
            "resources": loaded,
        }

    scope = max(scores, key=scores.get)                       # Tier 2: happy path
    # Flat resource topology: no references/<key>/ subdirectories. The scope selects the
    # authoring target documented above, not a keyed subtree; load the flat refs that exist.
    for path in sorted(inventory):
        load_if_available(path, inventory, loaded, seen)
    return {"scope": scope, "resources": loaded}

3. OUTPUT PACKAGE CONTRACT

Canonical Package Shape

Author this layout:

manual-testing-playbook/
|-- manual-testing-playbook.md
|-- category-name/
|   |-- feature-name.md
|   `-- another-feature-name.md
`-- another-category/
    `-- feature-name.md

Package invariants:

• The root file is always manual-testing-playbook.md.
• Per-feature files live in root-level category folders.
• Category directories use descriptive kebab-case names such as category-name (no numeric prefix).
• Per-feature files use stable kebab-case slugs such as feature-name.md; no numeric file prefix.
• Per-feature snippet order is defined by the root playbook listing order.
• Display order is owned by the root playbook index (manual-testing-playbook.md), not the folder name.
• Benchmark tier is owned by the per-feature file's optional stage: frontmatter field (routing default, or holdout/negative), not by a filename token.
• Every feature ID maps to exactly one per-feature file.

Contract boundary:

• The operator-scenario contract is the five-section, execution-evidence contract defined here and checked by scripts/validate-playbook-package.cjs.
• The routing-gold contract is the compact typed-routing schema checked by sk-create-skill/scripts/validate-playbook-topology.cjs.
• playbook-corpus-manifest.json is an explicit whole-tree override map consumed only by the operator validator. A listed routing-gold tree is excluded from operator checks; for every other file, a non-empty expected_workflow_mode plus at least one expected_leaf_resources pair in frontmatter classifies that file as routing gold. Files without that signature are operator-scenario files. The topology gate and the Lane-C loader do not read this manifest and keep their existing boundary.

Do not create:

• A snippets/ subtree for canonical per-feature files.
• Separate canonical review_protocol.md.
• Separate canonical subagent_utilization_ledger.md.
• Duplicated scenario truth across multiple sidecar files.

Root Playbook Responsibilities

manual-testing-playbook/manual-testing-playbook.md is the package directory and review surface. It owns:

• Frontmatter and H1 intro.
• Global overview and coverage note.
• Global preconditions.
• Global evidence requirements.
• Deterministic command notation rules.
• Integrated review protocol and release-readiness rules.
• Integrated orchestration and wave-planning guidance.
• Category sections with short per-feature summaries.
• Automated test cross-reference section.
• Feature catalog cross-reference index.

Root summaries should be concise but useful:

• Provide enough context to understand what each scenario covers.
• Preview operator intent and prompt shape.
• Do not duplicate the full execution matrix from the per-feature file.

Root-to-feature rule: the root document explains package-level policy; per-feature files carry scenario-specific execution truth.

Per-Feature File Responsibilities

Each per-feature file is the canonical scenario contract for one feature ID.

Required per-feature section order:

1. ## 1. OVERVIEW
2. ## 2. SCENARIO CONTRACT
3. ## 3. TEST EXECUTION
4. ## 4. REFERENCES or ## 4. SOURCE FILES
5. ## 5. SOURCE METADATA

Each per-feature file must include:

• Frontmatter with title, description, and a 4-part version.
• Realistic user request when it clarifies user intent.
• Operator prompt or orchestrator prompt.
• Exact prompt in the scenario table when a table is used.
• Exact command sequence.
• Expected signals.
• Evidence requirements.
• Pass/fail criteria.
• Failure triage.
• Root playbook link.
• Feature catalog link when applicable.

4. RESULTS STORAGE CONTRACT

A playbook is a corpus. Running it produces evidence, and that evidence has a home: the same benchmark/ tree that holds every other measurement of the skill. Without this rule a run leaves nothing behind, and the next person has no way to know the playbook was ever executed.

Where A Run Lands

<skill>/
|-- manual-testing-playbook/          # the corpus, an input, never rewritten by a run
`-- benchmark/
    `-- reports/
        |-- README.md                 # the run index, one row per folder
        `-- 2026-07-29--manual-testing-playbook--goal-hook/
            |-- README.md
            |-- skill-benchmark-report.json
            |-- skill-benchmark-report.md
            |-- results.csv
            |-- failed-runs.md
            |-- findings-and-recommendations.md
            `-- source.md

Run folders are named <YYYY-MM-DD>--<subject>--<variant>, dated by execution. When a run is feature-scoped — a hand-derived validation of one feature or scenario group rather than a full-corpus harness sweep — name the <variant> for the feature (e.g. goal-hook) and record the model/executor inside the report, so the folder stays legible across models. A full-corpus harness run auto-names the <variant> from the executor identity instead (see below). create-benchmark owns the grammar in full; see its storage sections for the field vocabulary and the one carve-out.

What Writes It

The Lane C harness reads a skill's playbook as its default corpus and writes the whole folder, including the index row:

node .opencode/skills/system-deep-loop/deep-improvement/scripts/skill-benchmark/run-skill-benchmark.cjs \
  --skill <skill-id>

Given no --outputs-dir, it derives the path above from the skill root, the execution date and the executor identity in the environment. Pass --outputs-dir only to send a run somewhere else on purpose; a run outside a reports/ directory is deliberately left out of the index.

Manual scenario completion

A manual scenario is incomplete until its PASS, FAIL, or SKIP outcome and reason are persisted by the canonical wrapper into the skill's benchmark/reports/<dated-run-label>/ folder. The renderer owns skill-benchmark-report.md and any results.md or report.md output; never hand-author those files.

node .opencode/skills/system-deep-loop/deep-improvement/scripts/skill-benchmark/run-manual-playbook-scenario.cjs \
  --skill <root-or-id> \
  --scenario <ID> \
  --variant <feature-slug> \
  --verdict PASS|FAIL|SKIP \
  --reason "<text>" \
  --stage <slug> \
  [--evidence <comma-paths>]

Lane C scoring remains owned by scoring-contract.md; this completion rule does not restate it.

Every PASS uses --outcome-json, sets executionContext.requireDurableEvidence to true, and selects one controlled evidence class: unit, adapter-driven, registered-path, or native-host-delivered. Evidence paths must resolve beneath executionContext.evidenceRoot through non-symlink regular files; reports record repo-relative paths, byte counts, and SHA-256 values. A PASS also records the exact command, runtime plus observed version, sanitized payload fixture or an explicit not-applicable reason, observed executor or reason, and observed model or reason. Corrected runs list prior immutable report folders in executionContext.supersedes; the wrapper updates the external supersession manifest. Requested --executor and --model labels remain requested labels unless the outcome marks them observed.

Rules

• The corpus is an input. A run never edits manual-testing-playbook/, and gold that needs to change gets a corpus revision rather than a rewritten scenario.
• skill-benchmark-report.md is renderer-owned and regenerated from the JSON. Never hand-edit it.
• Curated summaries and result tables live in the run folder. Raw transcripts and copied artifacts stay in the spec packet that produced them, named in source.md.
• A run whose result changes gets a new folder. A prior run is never overwritten.
• Every file in the folder derives from the run record. A field the run did not capture reads as not recorded; it is never filled in from inference.

5. HOW IT WORKS - AUTHORING WORKFLOW

Follow this sequence:

1. Confirm the target skill or system, package owner, feature set, and whether a feature catalog already exists.
2. Decide whether a manual testing playbook is appropriate using the decision rule in this skill.
3. Define category directories using a descriptive kebab-case slug such as category-name.
4. Define stable feature IDs using a consistent {PREFIX}-{NNN} pattern.
5. Create the root manual-testing-playbook/ directory.
6. Create manual-testing-playbook/manual-testing-playbook.md from assets/manual-testing-playbook-template.md.
7. Create one per-feature file for each feature ID from assets/manual-testing-playbook-snippet-template.md.
8. Write root package policy before writing scenario-specific exceptions.
9. Write each per-feature prompt and execution truth before polishing root summary prose.
10. Fill each scenario contract with the nine display fields: Feature ID, Feature Name, Scenario Objective, Exact Prompt, Exact Command Sequence, Expected Signals, Evidence, Pass/Fail Criteria, Failure Triage. The operator validator's eight unconditional checks cover Feature ID plus the seven execution/link fields; realistic user request, table prompt, and feature-catalog link are conditional checks.
11. Write root category summaries and link every per-feature file.
12. Add automated-test anchors when they exist.
13. Add feature-catalog cross-references when a catalog exists.
14. Explicitly note when a scenario has no dedicated catalog entry.
15. Review destructive scenarios and isolate them in root preconditions and recovery guidance.
16. Validate the root document with shared validators.
17. Manually spot-check per-feature structure, prompt synchronization, feature ID counts, and local links.
18. Report any remaining manual validation scope honestly.

Authoring sequence matters:

• Decide categories and IDs before writing summaries.
• Write root package policy before scenario exceptions.
• Write execution truth before root prose.
• Keep scenario-specific execution truth in per-feature files.

6. SCENARIO DESIGN RULES

Determinism

Each scenario must be reproducible by another operator. Include exact prompts, exact command sequences, observable expected signals, captured evidence, and binary pass/fail criteria.

Execution status is limited to:

• PASS
• FAIL
• SKIP with a specific sandbox blocker

Do not classify scenarios outside the PASS / FAIL / SKIP enum. A SKIP must name a specific sandbox or runtime blocker.

Prompt Quality

Prompts should be:

• Realistic, not bare command paraphrases.
• Deterministic enough to produce stable evidence.
• Explicit about what to capture.
• Explicit about the user-facing verdict or outcome.
• Compact enough for a 9-column table cell.

Weak prompt:

Test search

Acceptable prompt:

Use memory_context in auto mode for the flaky index scan retry issue, capture the returned bounded context, and return a concise pass/fail verdict with the main reason.

Natural-Human vs RCAF Voice

The canonical Prompt: field defaults to natural-human voice. Match how a real user would phrase the request to an AI in conversation.

Use the RCAF wrapper only when the actor is an AI orchestrator:

As a {ROLE}, {ACTION} against {TARGET}. Verify {EXPECTED_OUTCOME}. Return {OUTPUT_FORMAT}.

Use natural-human voice when:

• A human asks the AI directly in conversation.
• The scenario is a code review, commit/git workflow, bug fix, code question, or preference question.
• The scenario could plausibly originate from a Slack DM or terminal prompt.

Use RCAF when:

• The actor is an AI orchestrator delegating to another tool, AI, agent, CLI, or runtime.
• Role context determines safety refusal behavior.
• The validation contract treats the orchestrator as the operator.

When in doubt, prefer natural-human voice. The Real user request: field is always natural-human and serves as the voice reference baseline.

Prompt Synchronization Gate

These fields must agree:

• Structured prompt field in SCENARIO CONTRACT.
• Exact Prompt column in the execution table.
• Any root summary prompt text.

Do not ship unsynchronized prompt fields.

7. VALIDATION AND RELEASE GATES

Automated Checks

Run shared validation on the root playbook before delivery from the repo root (replace <SKILL_PATH> with the target skill directory, e.g. .opencode/skills/system-spec-kit):

# New-content naming guard. The staging root must contain only the newly authored
# canonical manual-testing-playbook package, never an ancestor with shipped legacy roots.
python3 .opencode/skills/sk-doc/shared/scripts/check_no_hyphenated_catalog_content.py <new-content-staging-root>

python3 .opencode/skills/sk-doc/shared/scripts/validate_document.py <SKILL_PATH>/manual-testing-playbook/manual-testing-playbook.md --type reference
python3 .opencode/skills/sk-doc/shared/scripts/extract_structure.py <SKILL_PATH>/manual-testing-playbook/manual-testing-playbook.md

The staging scope is mandatory until shipped underscore roots are migrated. Do not run this guard against .opencode/skills or another ancestor containing legacy feature_catalog/ or manual_testing_playbook/ trees.

Also check:

• Broken local links, by grep/script or the repository markdown-link guard.
• Feature ID counts between the root index and per-feature files.
• Automated-test links, when referenced.
• Feature-catalog links, when referenced.

Automated Checks

Run the operator-contract validator from the repository root:

node .opencode/skills/sk-doc/sk-create-manual-testing-playbook/scripts/validate-playbook-package.cjs \
  --package .opencode/skills/<skill-id>/manual-testing-playbook

The command validates the operator-scenario contract, not the routing-gold contract. It walks every non-excluded scenario tree and checks, per feature: five-section ordering; title, description, and four-part version frontmatter; Feature ID; operator/orchestrator prompt; exact command sequence; expected signals; evidence; pass/fail criteria; failure triage; root-playbook link; allowed verdicts; filename/category shape; unique IDs; root-index bijection; local links; evergreen truth; and placeholder exclusion. Conditional checks cover a realistic user request when user intent is explicitly being clarified, an exact prompt when a scenario table is present, and a feature-catalog link when catalog applicability is declared.

The validator derives scenario and category counts at run time. A root's hand-typed census is reported as a warning, including a mismatch, so documentation repair remains separate from enforcement. Existing measured packages are listed in the validator's staged warning set for the first fleet run; clean packages and new playbooks fail closed. Promotion removes a package from that warning set only after a clean run.

The validator also checks the root playbook for the wrapper completion marker and the complete PASS / FAIL / SKIP vocabulary as advisory warnings. Missing either item never creates a new fail-closed violation for an existing package.

Exit codes are direct: 0 means conforming or staged warning, 1 means a fail-closed contract violation, and 2 means a usage or boundary error. Strict mode is on by default; --no-strict is local triage only and must not be used by CI.

The existing validate-playbook-topology.cjs remains the routing-gold consumer and is intentionally unchanged. The Lane-C loader also remains unchanged. Both consumers continue reading their current playbook paths without consulting the additive corpus manifest.

8. RULES

✅ ALWAYS

1. Use manual-testing-playbook.md as the root file name.
2. Put per-feature files in root-level category folders named with the bare descriptive slug.
3. Keep one canonical per-feature file for each feature ID.
4. Use exact prompts, exact command sequences, observable expected signals, captured evidence, and binary pass/fail criteria.
5. Keep shared review, orchestration, and release-readiness rules in the root playbook.
6. Link scenarios back to the matching feature catalog entry when a catalog exists.
7. Explicitly document exceptions when no matching catalog entry exists.
8. Mark destructive scenarios and include safe recovery expectations.
9. Validate the root playbook and manually review per-feature files before delivery.

⛔ NEVER

1. Create a snippets/ subtree for canonical per-feature files.
2. Create separate canonical review_protocol.md or subagent_utilization_ledger.md files.
3. Duplicate full execution truth in the root summary.
4. Renumber published feature IDs just to remove gaps.
5. Use packet-history or spec-phase references as scenario identity.
6. Ship unsynchronized prompt fields.
7. Add packet-local graph-metadata.json.
8. Add numeric prefixes to category folder names or per-feature filenames; the root playbook index owns display order and the per-feature stage: field owns benchmark tier.
9. Never use underscores in category folder or per-feature filename path segments.

⚠️ ESCALATE IF

1. The target feature set, category boundary, or package owner is unclear.
2. A scenario would require destructive actions without a safe recovery path.
3. The user expects external credentials, live production access, or privileged tool execution.
4. Feature catalog links are required but no stable catalog exists.
5. Validation fails after structural fixes.
6. The requested package shape conflicts with existing published playbooks.

9. SUCCESS CRITERIA

A create-manual-testing-playbook run is done when:

• The package uses the canonical shape: root manual-testing-playbook.md plus kebab-case category folders of per-feature files, with no snippets/ subtree and no separate review_protocol.md or subagent_utilization_ledger.md.
• Every feature ID maps to exactly one per-feature file, and no packet-local graph-metadata.json was added.
• Every scenario is deterministic: exact prompt, exact command sequence, observable expected signals, captured evidence, and binary PASS/FAIL/SKIP verdicts.
• Prompt fields are synchronized across the scenario contract, the execution table, and any root summary.
• Destructive scenarios are marked and isolated with safe recovery expectations.
• The root playbook passes shared validate_document.py, per-feature files are manually spot-checked, and any remaining manual scope is documented honestly.

10. RESOURCES FOR DEEP DETAIL & REFERENCES

The core executable workflow lives in this SKILL.md. Use these only for overflow detail, exhaustive examples, or template text:

• references/README.md - reference map routing to the overflow detail below.
• references/prompt-voice.md - natural-human vs RCAF decision table and voice guidelines.
• references/common-pitfalls.md - recurring package defects and correct fixes.
• references/examples.md - shipped reference playbooks and scaffold templates.
• assets/manual-testing-playbook-template.md - root playbook scaffold.
• assets/manual-testing-playbook-snippet-template.md - per-feature file scaffold.
• ../shared/references/core-standards.md - shared markdown structure rules.
• ../shared/references/validation.md - shared validation and DQI workflow.
• ../shared/references/frontmatter-versioning.md - 4-part version expectations.
• ../shared/references/evergreen-packet-id-rule.md - evergreen current-state wording.

The source assets keep the filenames manual-testing-playbook-template.md and manual-testing-playbook-snippet-template.md until their separate source-file migration. Those filenames are authoring inputs, not emitted package names.