cli-e2e-testcase-writer

CLI E2E Testcase Writer

Work on one domain per run. Produce exactly two artifacts for that domain:

• workflow testcase files under tests/cli_e2e/{domain}/
• tests/cli_e2e/{domain}/coverage.md

Focus on domain testcase files. Do not change shared E2E support code such as tests/cli_e2e/core.go unless the user explicitly asks. Treat tests/cli_e2e/demo/ as reference only.

Core standard

• Make the testcase scenario-based and self-contained.
• Prove one workflow end to end: create plus follow-up read, or mutate plus teardown.
• Prefer one file per workflow or one closely related feature.
• For mutable flows, prove persisted state with read-after-write assertions, not just exit code.
• Leave prerequisite-heavy paths uncovered when they cannot be proven, and explain why in coverage.md.

Workflow

1. Explore the live CLI before writing code

lark-cli --help
lark-cli <domain> --help
lark-cli <domain> +<shortcut> -h
lark-cli <domain> <group> --help
lark-cli <domain> <group> <method> -h
lark-cli schema <domain>.<group>.<method>

2. Count leaf commands for the denominator

• A leaf command is one that executes an action — it has no further subcommands.
• If lark-cli <domain> <group> --help lists no subcommands, <group> itself is the leaf.
• Count task +create as one leaf and task tasks get as one leaf.
• Do not count parameter combinations.
• Reuse coverage already present under tests/cli_e2e/{domain}/. Do not count tests/cli_e2e/demo/.

3. Choose the proof surface before editing

Identify the provable risks for the touched workflow: invalid input, missing prerequisite, identity or permission, state transition, output shape, cleanup safety. If only the happy path is testable, document the blocked risk areas in coverage.md.

4. Add or update the workflow testcase

• Use clie2e.RunCmd(ctx, clie2e.Request{...}).
• Put command path and plain flags in Args; put JSON in Params (URL/path parameters) and Data (request body).
• Prefer one top-level test per workflow with t.Run substeps.
• Register teardown on parentT.Cleanup so it survives subtest failures.
• When touching an existing command, verify the JSON response shape is stable: assert status type, field paths, and identifiers consumed by later steps before changing assertions.

5. Run and iterate

Run go test ./tests/cli_e2e/{domain} -count=1 while iterating and before finishing. If command shape or behavior is unclear, re-check help or schema (step 1) before changing assertions.

6. Refresh the domain outputs

• Update the workflow testcase files.
• Update coverage.md: recompute the denominator from live help output, mark each command as shortcut or api, and keep one command table for the whole domain.

Testcase rules

• Override BinaryPath, DefaultAs, or Format on clie2e.Request only when the testcase truly needs it.
• Use require.NoError, result.AssertExitCode, result.AssertStdoutStatus, assert, and gjson.
• Shortcut responses ({ok: bool}) assert true; API responses ({code: int}) assert 0.
• Use t.Helper() only for setup or assertion helpers that are called from multiple tests.
• Use table-driven tests only when the scenario shape repeats across inputs.
• For expected failures, assert stderr content and exit code when the environment makes them deterministic.
• If identity or external fixtures cannot be proven, leave the command uncovered and document the prerequisite rather than faking confidence.

coverage.md

Keep coverage.md brief and mechanical. Include:

• a domain-specific H1 title
• a metrics section with denominator, covered count, and coverage rate
• a summary section restating each Test... workflow, key t.Run(...) proof points, and main blockers
• one command table for all commands

Recommended structure:

# <Domain> CLI E2E Coverage

## Metrics
- Denominator: N leaf commands
- Covered: N
- Coverage: N%

## Summary
- TestXxx: ... key `t.Run(...)` proof points ...
- Blocked area: ...

## Command Table
| Status | Cmd | Type | Testcase | Key parameter shapes | Notes / uncovered reason |
| --- | --- | --- | --- | --- | --- |
| ✓ | task +create | shortcut | task_status_workflow_test.go::TestTask_StatusWorkflow | basic create; create with due | |
| ✕ | task +assign | shortcut |  | none | requires real user open_id |

• Mark each command shortcut or api.
• Write testcase entries in go test -run friendly form.
• Commands only exercised in parentT.Cleanup teardown are not counted as covered.
• Do not split covered and uncovered commands into separate sections.

Guardrails

• Run as bot identity only; do not assume --as user works.
• Do not place new real coverage under tests/cli_e2e/demo/.
• Do not depend on preexisting remote data.
• Do not fabricate open_ids, chats, docs, or other remote fixtures.
• Prefer deterministic negative cases over tenant-dependent assertions.
• Do not guess Params or Data fields when help or schema can tell you the exact shape.
• Do not hardcode obvious defaults unless the command truly requires explicit flags.
• Do not put agent, model, or vendor brand names in visible remote test data; use neutral prefixes such as lark-cli-e2e- or <domain>-e2e-.
• A command is covered only when the testcase asserts returned fields or persisted state, not just exit code.
• Cleanup-only execution is not primary coverage, except delete in the same workflow that created the resource.