meta-planning-ai-planning

AI Planning Frameworks

Quick Guide: Default to the simplest tier that satisfies the requirement — most features are one well-built model call. Pin the model id, define the output contract with a repair-vs-reject policy, budget in tokens and money rather than adjectives, enumerate the failure modes, and make quality measurable with an eval plan before implementation starts. Apply a framework only when the spec touches its artifact class — a feature with no retrieval needs no retrieval section.

<critical_requirements>

CRITICAL: Before Specifying AI Features

All specifications must be grounded in the codebase's real model clients, prompt modules, schemas, and eval fixtures — reference specific files with line numbers

(You MUST justify the approach against the simpler tier — a fixed code-orchestrated chain beats an agentic loop whenever the step sequence is known)

(You MUST pin an explicit model id in configuration with a named fallback — never a floating alias, never inline in code)

(You MUST define the output contract completely: mechanism, schema, validation boundary, and a repair-vs-reject policy)

(You MUST state budgets as numbers — tokens per call, calls per request, cost per request, p95 latency — never as adjectives)

(You MUST identify where untrusted input enters every prompt, and require adversarial eval cases wherever it does)

(You MUST apply each framework only when the spec touches its artifact class — an unused section is omitted, never filled)

</critical_requirements>

Auto-detection: AI spec, LLM feature spec, prompt design spec, model selection, RAG spec, retrieval design, tool calling spec, agent loop spec, eval plan, token budget, structured output

When to use:

• Specifying a feature that calls a model (classification, extraction, generation, summarization)
• Choosing between a single call, a fixed chain, retrieval, and an agentic loop
• Specifying prompt architecture, versioning, and untrusted-input boundaries
• Specifying structured output contracts and validation policy
• Specifying retrieval parameters, tool contracts, or loop guards
• Specifying eval datasets, metrics, thresholds, and budgets

When NOT to use:

• When implementing AI code (use the relevant AI implementation skill)
• For the API surface that exposes the capability (use the api planning skill)
• For the planning PROCESS itself — research, scope fencing, success criteria — which the PM agent carries

Key patterns covered:

• Approach selection (single call → chain → retrieval → agentic loop)
• Model and provider selection with pinned ids and fallback chains
• Prompt architecture: role split, versioning, untrusted-input boundaries
• Structured output strategy and repair-vs-reject policy
• Retrieval design parameters
• Agentic loop guards and tool side-effect classification
• Eval design and adversarial cases
• Failure-mode matrix
• Token, cost, and latency budgets

Detailed Resources:

• examples/core.md - Per-artifact spec section templates and a worked example specification

Philosophy

Non-determinism is the material; contracts are what make it buildable. A model's output cannot be trusted by construction, so every boundary — schema, budget, failure behavior, eval threshold — must be decided in the spec, or it gets decided implicitly in production.

When specifying AI work:

• Trace one existing model call end to end before specifying a new one — client, prompt module, validation, consumer
• Name the requirement that forces each tier of complexity; if nothing does, use the simpler tier
• Establish the baseline before proposing a change: current cost, latency, and quality numbers, or an explicit "unmeasured"
• Decide product questions in the spec — may the model answer from parametric knowledge when retrieval is empty? — rather than leaving them to the implementer

When NOT to specify:

• Don't specify an agentic loop for a step sequence known in advance
• Don't add retrieval infrastructure when the knowledge fits in the context window
• Don't fill a tool-contract or retrieval section for a feature that has neither
• Don't include implementation code — contracts, budgets, and behaviors only

Core principles:

• Simplest sufficient tier: most features are a single well-constructed call
• Validate on receipt, every branch: provider-side enforcement reduces violations; it does not eliminate them
• Budgets are numbers: "fast and cheap" is not implementable or reviewable
• Refusals are surfaced, never retried: retrying with softened wording is a bypass attempt
• Eval thresholds gate the merge: thresholds set after seeing results ratify instead of gating

Core Patterns

Pattern 1: Choosing the Approach

Does the task need knowledge that is not in the model and not in the request?
├─ NO  → Single model call with a well-built prompt. Stop here. Most features end here.
└─ YES → Where does that knowledge live?
    ├─ A bounded set that fits the context window (< ~30% of it) → Pass it directly. No retrieval infrastructure.
    ├─ A large or growing corpus → Retrieval (RAG)
    └─ A live system of record (database, third-party API) → Tool calling, not retrieval

Does the task need multiple dependent actions the model must sequence itself?
├─ NO  → Single call, or a fixed chain of calls you orchestrate in code
│         (a fixed chain is cheaper, more debuggable, and easier to eval than a loop)
└─ YES → Agentic loop with an explicit step budget and termination conditions

Default to the simplest tier that satisfies the requirement. Specify the more complex tier only when you can name the requirement that forces it, and record the rejected alternatives with the reason each was rejected.

Pattern 2: Model and Provider Selection

Score the candidates against the actual requirement and record the table in the spec.

Rules:

• Always pin an explicit model id in configuration, never a floating alias, and never inline in code.
• Always name a fallback, even if the fallback is "fail closed with error X" — an unnamed fallback becomes an unhandled exception.
• Re-run the eval set when changing model id, prompt version, or retrieval parameters. Any of the three can move quality.

Pattern 3: Prompt Architecture

Every prompt the spec introduces is named, located, versioned, and trust-annotated.

• Location and version. A prompt is a module with a version suffix in its filename (prompts/summarize-ticket.v1.ts), following the shape of an existing prompt module. The version is logged with every call so eval results attribute to a specific revision. Editing a shipped prompt in place is out of scope; ship the next version instead.
• Role split. Instructions live exclusively in the system message. Task input is interpolated only into the user-role message. Few-shot examples, when used, are assistant-role and trusted.
• Untrusted-input boundary. Name every variable, its source, and whether it is untrusted. Untrusted text is delimited with the codebase's existing fence helper, and a fenced instruction ("ignore your previous instructions") must not change behavior — the eval set includes this case. Retrieved document text is untrusted too.
• Variables. List each variable with its type and source. A variable the template reads but the spec never mentions is a contract gap.

Pattern 4: Structured Output Strategy

Does the provider support tool calling / function calling?
├─ YES → Is the output a single well-defined record?
│   ├─ YES → Tool calling with a single submit-style tool. Strongest schema adherence.
│   └─ NO  → Tool calling with one tool per action, plus a terminal final-answer tool
└─ NO  → Does it support a JSON/structured output mode?
    ├─ YES → JSON mode + schema validation on receipt
    └─ NO  → Delimited free text + strict parser. Requires the widest eval coverage.

Every branch validates on receipt. Provider-side schema enforcement reduces violations; it does not eliminate them, and it never covers semantic correctness (a valid-shaped record with a hallucinated value passes schema validation).

Repair-vs-reject policy — state it explicitly:

Pattern 5: Retrieval Design

Always specify: what happens on empty retrieval, what happens on low-similarity retrieval, whether answers must cite sources, and whether the model may answer from parametric knowledge when retrieval returns nothing. That last one is a product decision, not an implementation detail — decide it in the spec.

Pattern 6: Agentic Loop Guards

Every loop specification names all six:

Tool side effects must be classified. Every tool is read-only or mutating. Mutating tools need: idempotency strategy, whether they require confirmation, and whether they are permitted on a retry after an ambiguous failure.

State carried between steps is named exactly — nothing else persists. On step-budget exhaustion, return the partial answer flagged as truncated; never silently return an unfinished result as complete.

Pattern 7: Eval Design

Include adversarial cases whenever untrusted text reaches the prompt. At minimum: instruction override, delimiter escape, and exfiltration attempts against any tool the loop can call.

Pattern 8: Failure-Mode Matrix

Enumerate every applicable row for the feature. Absent rows become production incidents.

Pattern 9: Budgets and Observability

Budget in tokens and money, not adjectives. Per call: input and output token ceilings. Per user request: maximum model calls, blended cost target at current pricing, p95 end-to-end latency, and time-to-first-token for streamed surfaces. State the overflow behavior: what is shed first when the input budget is exceeded — never the system prompt.

Observability names four things: what is logged per call (model id, prompt version, token counts, latency, cost, outcome, correlation id), the trace span boundaries (retrieval, each model call, each tool call), what is redacted (prompt bodies with user data, retrieved content, keys, PII — named field by field), and what alerts (threshold breaches worth notifying).

<decision_framework>

Decision Framework

Which Spec Sections Does This Feature Need?

Apply a framework only when the spec touches its artifact class. The per-artifact section templates live in examples/core.md.

Does the feature call a model at all?
├─ YES → Approach rationale (Pattern 1) + Model Selection (Pattern 2) + Prompt Architecture (Pattern 3)
│        + Output Contract (Pattern 4) + Budgets (Pattern 9) + Failure Modes (Pattern 8) + Eval Plan (Pattern 7)
├─ Does it retrieve from a corpus? → Retrieval Design section (Pattern 5)
├─ Does the model call tools?      → Tool Contracts section (see examples/core.md)
├─ Does the model sequence steps?  → Agentic Loop section (Pattern 6)
└─ None of the above touched       → the section is omitted, never filled

Common Specification Mistakes

</decision_framework>

<red_flags>

RED FLAGS

High Priority Issues (a spec with one of these is incomplete):

• A model call with no output schema or validation boundary
• A floating model alias, or a model id inline in code
• An agentic loop missing any of the six guards
• Untrusted input reaching the system message, or fenced input with no adversarial eval case
• Budgets as adjectives, or no budgets at all
• No behavior defined for a failure row the feature can hit

Medium Priority Issues:

• A loop specified where the step sequence is known in advance
• Retrieval infrastructure for knowledge that fits the context window
• An unversioned prompt, or an in-place edit to a shipped prompt version
• Empty-retrieval behavior left undecided
• No regression eval set named

Common Mistakes:

• Treating provider-side schema enforcement as validation
• Repair attempts on semantic violations (laundering the bad value)
• Acting on streamed partial JSON before completion
• Specifying "handle API errors gracefully" instead of a per-row failure matrix

Gotchas & Edge Cases:

• A valid-shaped record with a hallucinated value passes schema validation — semantic checks are separate
• Retried refusals read as bypass attempts; surface them unchanged
• Changing any of model id, prompt version, or retrieval parameters moves quality — all three trigger re-eval
• A mutating tool retried after an ambiguous failure can double-apply; idempotency is part of the tool contract

</red_flags>

<critical_reminders>

CRITICAL REMINDERS

All specifications must be grounded in the codebase's real model clients, prompt modules, schemas, and eval fixtures

(You MUST justify the approach against the simpler tier — a fixed code-orchestrated chain beats an agentic loop whenever the step sequence is known)

(You MUST pin an explicit model id in configuration with a named fallback — never a floating alias, never inline in code)

(You MUST define the output contract completely: mechanism, schema, validation boundary, and a repair-vs-reject policy)

(You MUST state budgets as numbers — tokens per call, calls per request, cost per request, p95 latency)

(You MUST identify where untrusted input enters every prompt, and require adversarial eval cases wherever it does)

(You MUST apply each framework only when the spec touches its artifact class — an unused section is omitted, never filled)

Failure to specify these contracts produces AI features whose outputs go unvalidated, whose costs are unbounded, whose injections ship undetected, and whose regressions cannot be traced to a prompt revision.

</critical_reminders>