cli-testing

CLI Testing — Exercise the Assistant End-to-End

Drive a real assistant from the terminal only. The vellum CLI (cli/, package @vellumai/cli) manages instance lifecycle; vellum message / vellum events exercise a running instance. See cli/AGENTS.md and the root README.md § CLI for command reference.

0. Prerequisites

export PATH="$HOME/.bun/bin:$PATH"   # bun + the linked `vellum` binary
vellum ps                            # sanity check the CLI resolves

If vellum is missing, run ./setup.sh from the repo root once (installs deps, links the vellum command). Docker must be running for the default flow below.

1. Provide an LLM provider key (from the environment)

Local-mode and Docker-mode instances need one LLM provider key. The CLI reads it straight from the host environment — just export it before hatching/setup:

export ANTHROPIC_API_KEY=sk-ant-...   # or OPENAI_API_KEY / GEMINI_API_KEY /
                                      # FIREWORKS_API_KEY / OPENROUTER_API_KEY /
                                      # MINIMAX_API_KEY

In Devin sessions ANTHROPIC_API_KEY is typically already present in the environment — check with echo "${ANTHROPIC_API_KEY:0:7}" before asking for one. The CLI maps providers to env vars in cli/src/shared/provider-env-vars.ts.

2. Hatch — default to a Docker hatch built from source

Always default to --remote docker. It runs the assistant, gateway, and credential-executor in isolated containers that mirror production and keep the test off your host process table. Reserve --remote local (§5) for the rare case where Docker is unavailable.

Build from source — that's the point of testing. A bare vellum hatch --remote docker pulls the published platform images even when the CLI itself runs from your checkout, so it would test released code, not your changes. Source-build is opt-in via a flag (resolveDockerHatchMode in cli/src/lib/docker.ts):

• --source <path> — build images once from the source tree at <path>, no watcher. Default for testing: picks up your current changes and is robust for a scripted one-shot run.
• --watch — build from source and start a file-watcher that rebuilds the affected image on change (watches each service's src/, package.json, and Dockerfile). Use while iterating. The watcher is a long-lived foreground process, so prefer --source for unattended/scripted runs.

vellum hatch --remote docker --source . --name clitest   # build from cwd
# → "Mode: build-from-source" then "Images (local build): vellum-assistant:local-clitest …"

If --source/--watch is passed but no full source tree is found (e.g. the CLI is running from a packaged app bundle), the CLI falls back to pulling the published images and says so — watch for that line if you expect a build. Building all three images takes ~1–2 min the first time.

Hatch attached — do not pass -d. An attached hatch leases the guardian token and configures the provider credential from your environment inline, then returns once the containers are healthy — no follow-up vellum setup needed. Detached mode (-d) defers the guardian-token lease, so a later vellum setup cannot authenticate against the gateway and fails with an invalid_signature 401. Confirm readiness with vellum ps (🟢 healthy) before messaging.

3. Verify functionality

vellum message is async (returns a message id, not the reply — --json only adds {accepted, messageId}). vellum events streams the reply but is long-running, so background it, send, wait, then read.

Assert on a token the assistant must generate, never one you put in the prompt. vellum events echoes your prompt as **You:** <text> (cli/src/commands/events.ts), so grepping for a word that appears in the prompt passes even when the assistant never replied. Ask a question whose answer is absent from the prompt:

( vellum events > /tmp/vel_events.log 2>&1 & )   # stream in background
sleep 2
vellum message "What is 6 multiplied by 7? Reply with only the number."
sleep 25                                          # let the assistant respond
pkill -f "vellum events"
grep -w 42 /tmp/vel_events.log                    # "42" is NOT in the prompt,
                                                  # so a match proves a real reply

The assistant's streamed reply is written as plain text (no **You:** prefix), so a match on a generated answer confirms the round-trip worked. If you must use a fixed sentinel string, strip the echoed prompt first (grep -v '^\*\*You:\*\*' /tmp/vel_events.log | grep <sentinel>).

Common verification commands

4. Tear down

vellum retire clitest --yes          # stops containers and removes the instance

retire is destructive (removes per-instance Docker volumes); always clean up test instances when done.

5. Fallback: local mode (no Docker)

Only when Docker is unavailable. Runs the daemon + gateway as plain host processes; configures the provider key automatically from the env at hatch time:

vellum hatch --name clitest          # defaults to --remote local
# verify via the `vellum events` + generated-answer pattern in §3, then:
vellum retire clitest --yes