inference

Inference Reference — ModelProvider + VirtualModel + Switchyard

Prerequisites

This skill assumes the platform is already running and any required plugins (nemo-switchyard, nemo-guardrails) are loaded. For local-platform startup, Switchyard install, and DB-reset choices, follow the setup playbook (SETUP.md at the repo root) first and then return here.

API key environment variables

Applies to both branches. Before creating secrets or running notebook/test harness flows, confirm API key environment variables:

1. Check whether INFERENCE_NVIDIA_API_KEY and/or NVIDIA_API_KEY are already set in the user's shell.
2. Treat INFERENCE_NVIDIA_API_KEY as the source of truth for the https://inference-api.nvidia.com/v1 provider secret and notebook/test harness flows.
3. If INFERENCE_NVIDIA_API_KEY is not set, ask whether the user has the same token in another environment variable. Common alias: NVIDIA_INFERENCE_API_KEY.
4. If the user identifies an alias variable, export it into INFERENCE_NVIDIA_API_KEY for the current shell/session before continuing.
5. If NVIDIA_API_KEY is required for a Build/NIM-style provider or seeding path such as https://integrate.api.nvidia.com, and is not set, ask whether another env var already contains that token and export it into NVIDIA_API_KEY if the user confirms.

Example alias normalization:

export INFERENCE_NVIDIA_API_KEY="$NVIDIA_INFERENCE_API_KEY"

Use INFERENCE_NVIDIA_API_KEY for inference-api secret creation:

printf '%s' "$INFERENCE_NVIDIA_API_KEY" | nemo secrets create nvidia-inference-key \
  --from-file - --workspace my-workspace

Environment

• Port: 8080 (CLI default — do NOT pass a custom --base-url)
• export NMP_BASE_URL=http://localhost:8080 — required when targeting a local platform. If your ~/.config/nmp/config.yaml already points at a remote cluster, the CLI uses that base URL and ignores the local platform entirely. Setting this env var overrides the config file for the current shell session.
• Workspace: my-workspace (substitute as needed; default also works)
• Backend: https://inference-api.nvidia.com/v1

For platform startup (nemo services run), Switchyard install, and state reset, see the setup playbook (SETUP.md at the repo root).

CLI gotchas (real failures observed)

• nemo secrets create uses --from-file (pipe key in). No --value flag.
• nemo inference providers create takes <name> as a positional arg. Same for update-status, get, delete.
• nemo virtual-models create is a top-level command (not under nemo inference) and takes <name> as positional.
• There is no nemo inference chat completions create command. Use nemo inference gateway model post <path> <vm-name> --workspace <ws> --body '<json>'.
• example is not a valid --services arg. Valid services: audit, auth, customization, data-designer, entities, evaluation, files, guardrails, hello-world, inference-gateway, intake, jobs, models, safe-synthesizer, secrets, studio.
• The reconciler re-syncs served_models from backend discovery every few seconds and drops manually-registered entries. Always point VM models at auto-discovered entity IDs (e.g. my-workspace/aws-anthropic-claude-opus-4-5) — these survive reconciler cycles. If a VM 404s shortly after working, the reconciler overwrote your entry. Re-run update-status with the full list (it replaces, not appends) or switch to auto-discovered IDs.
• update-status replaces the entire served_models list — include all entries, not just the new one.

Step 1 — Workspace + Secret + Provider

my-workspace is not created automatically. Create it first, then the secret and provider.

nemo workspaces create my-workspace

printf '%s' "$INFERENCE_NVIDIA_API_KEY" | nemo secrets create nvidia-inference-key \
  --from-file - --workspace my-workspace

nemo inference providers create nvidia-inference \
  --workspace my-workspace \
  --host-url "https://inference-api.nvidia.com/v1" \
  --api-key-secret-name "nvidia-inference-key"

nemo wait inference provider nvidia-inference --workspace my-workspace

Auto-discovery runs within ~3s.

For NVIDIA Build / NIM via https://integrate.api.nvidia.com, use NVIDIA_API_KEY:

printf '%s' "$NVIDIA_API_KEY" | nemo secrets create nvidia-build-key \
  --from-file - --workspace my-workspace

nemo inference providers create nvidia-build \
  --workspace my-workspace \
  --host-url "https://integrate.api.nvidia.com" \
  --api-key-secret-name "nvidia-build-key"

nemo wait inference provider nvidia-build --workspace my-workspace

For Anthropic direct (needs auth header rewrite):

printf '%s' "$ANTHROPIC_API_KEY" | nemo secrets create anthropic-api-key \
  --from-file - --workspace my-workspace

nemo inference providers create anthropic \
  --workspace my-workspace \
  --host-url "https://api.anthropic.com" \
  --api-key-secret-name "anthropic-api-key" \
  --auth-header-format "X-Api-Key: {{ auth_secret }}" \
  --default-extra-headers '{"anthropic-version": "2023-06-01"}'

{{ auth_secret }} is Jinja2 — substituted at request time. Without it the gateway defaults to Authorization: Bearer which Anthropic rejects.

Step 2 — Discover available models

After the provider is ready, served_models contains auto-discovered entries. Always use these entity IDs in VM --models — they survive the reconciler.

# All entity IDs (use these in --models and inference body)
nemo inference providers get nvidia-inference --workspace my-workspace \
  --output-format json | jq -r '.served_models[].model_entity_id'

# Side-by-side: served_model_name → model_entity_id
nemo inference providers get nvidia-inference --workspace my-workspace \
  --output-format json \
  | jq -r '.served_models[] | "\(.served_model_name)  →  \(.model_entity_id)"'

# Filter by vendor prefix
nemo inference providers get nvidia-inference --workspace my-workspace \
  --output-format json \
  | jq -r '.served_models[] | select(.served_model_name | startswith("aws/anthropic")) | .model_entity_id'

# Count
nemo inference providers get nvidia-inference --workspace my-workspace \
  --output-format json | jq '.served_models | length'

Entity ID normalization: slashes/dots → dashes, workspace prefix added. aws/anthropic/claude-opus-4-5 → my-workspace/aws-anthropic-claude-opus-4-5

Step 3 — Body + model entity convention

body["model"] must be a real backend entity ID that IGW can resolve — not the VM name. The VM name is resolved via the URL path (gateway model post v1/chat/completions <vm-name>), not the body.

Exception: VMs that rewrite body["model"] (random_routing, ModelFormatLookupProcessor in translate) — here the initial body model can be the VM name since the rewrite resolves it to the real entity.

For VMs without a rewriting middleware: always send the real auto-discovered entity in the body.

Common failure: HTTP 422 from chat completion

If nemo agents invoke …, an openai-SDK client, or a langchain-openai client returns HTTP 422 in well under a second with no provider call made, the gateway rejected the request's model field as malformed. The most common cause is sending the upstream catalog name (e.g. meta/llama-3.3-70b-instruct — vendor slash, dotted version) instead of the hyphenated entity ID (meta-llama-3-3-70b-instruct).

The entity ID is what nemo models list returns and what every gateway input expects. The slash-with-dots form is served_model_name — a human-display alias from the upstream provider's catalog. It appears in provider metadata and error messages, but is never a valid request field. (A workspace/entity-id prefix is fine — that's a workspace qualifier, not the upstream alias. The rejected form is specifically vendor/upstream.dotted.name.)

Diagnose and recover:

# 1. What entity IDs actually exist?
nemo inference providers get nvidia-build --workspace default \
  --output-format json \
  | jq -r '.served_models[] | "\(.served_model_name)  →  \(.model_entity_id)"'

# 2. If you set NEMO_DEFAULT_MODEL, confirm it's the hyphenated entity ID.
echo "$NEMO_DEFAULT_MODEL"

# 3. If a NAT-style deployed agent's resolved config carries the upstream
#    slash form in model_name, re-deploy with NEMO_DEFAULT_MODEL set to the
#    entity ID. The .config.llms.agent.model_name path is specific to NAT
#    workflows (e.g. the calculator-agent example); other workflow types store
#    the model elsewhere.
nemo agents deployments list --workspace default \
  | jq -r '.data[] | "\(.name)  model=\(.config.llms.agent.model_name // "n/a")"'

For an OpenAI-SDK or LangChain client, pass the entity ID (with or without the workspace/ prefix) as model=:

client.chat.completions.create(
    model="default/meta-llama-3-3-70b-instruct",   # OR "meta-llama-3-3-70b-instruct"
    messages=[...],
)
# NOT: model="meta/llama-3.3-70b-instruct"  → 422

This is intentional gateway behavior: accepting arbitrary slash-prefixed names would force IGW to guess provider attribution for every request instead of resolving it from a registered entity.

Step 4 — VirtualModel patterns

Random routing — same format (deterministic test: strong_probability=1.0)

nemo virtual-models create vm-random-strong --workspace my-workspace \
  --models '[
    {"model":"my-workspace/nvidia-mistralai-mixtral-8x22b-instruct-v01","backend_format":"OPENAI_CHAT"},
    {"model":"my-workspace/nvidia-qwen-qwen3-32b","backend_format":"OPENAI_CHAT"}
  ]' \
  --request-middleware '[{"name":"nemo-switchyard","config_type":"random_routing","config":{
    "strong":{"model":"my-workspace/nvidia-mistralai-mixtral-8x22b-instruct-v01"},
    "weak":{"model":"my-workspace/nvidia-qwen-qwen3-32b"},
    "strong_probability":1.0,
    "rng_seed":42,
    "enable_stats":false
  }}]'

Random routing — cross format (chain routing + translate)

random_routing picks the backend; translate rewrites the request format. Order matters: routing first, translate second. Use response_middleware too for full round-trip translation back to the client's format.

nemo virtual-models create vm-random-cross --workspace my-workspace \
  --models '[
    {"model":"my-workspace/aws-anthropic-claude-opus-4-5","backend_format":"ANTHROPIC_MESSAGES"},
    {"model":"my-workspace/nvidia-nvidia-nemotron-nano-31b-v3","backend_format":"OPENAI_CHAT"}
  ]' \
  --request-middleware '[
    {"name":"nemo-switchyard","config_type":"random_routing","config":{
      "strong":{"model":"my-workspace/aws-anthropic-claude-opus-4-5"},
      "weak":{"model":"my-workspace/nvidia-nvidia-nemotron-nano-31b-v3"},
      "strong_probability":0.5,
      "enable_stats":false
    }},
    {"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"auto","enable_stats":false}}
  ]' \
  --response-middleware '[{"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"auto","enable_stats":false}}]'

Translate — cross format, full round-trip ✓

Client sends OpenAI shape, backend is Anthropic, response comes back as OpenAI. Must list translate in BOTH request_middleware and response_middleware.

nemo virtual-models create vm-translate-cross --workspace my-workspace \
  --models '[{"model":"my-workspace/aws-anthropic-claude-opus-4-5","backend_format":"ANTHROPIC_MESSAGES"}]' \
  --request-middleware '[{"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"anthropic","enable_stats":false}}]' \
  --response-middleware '[{"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"anthropic","enable_stats":false}}]'

Guardrails — adding content safety to a VirtualModel

Guardrail rails are attached as a nemo-guardrails MiddlewareCall. Configs are created separately with nemo guardrail configs create — see the nemo-guardrails skill for rails config JSON, prompt templates, task LLMs, and streaming output rails. This section only covers the VirtualModel wiring.

The same call must appear in --request-middleware (input rails), --response-middleware (output rails), or both (full coverage). A call only on the request side fires input rails only; only on the response side fires output rails only.

Output rails only — block bad bot responses (most common):

nemo virtual-models create vm-guarded --workspace my-workspace \
  --models '[{"model":"my-workspace/<backend-entity-id>","backend_format":"OPENAI_CHAT"}]' \
  --response-middleware '[{
    "name":"nemo-guardrails",
    "config_type":"guardrail_config",
    "config_id":"my-workspace/content-safety"
  }]'

Input + output rails — full coverage. Include the call in both lists:

nemo virtual-models create vm-guarded-full --workspace my-workspace \
  --models '[{"model":"my-workspace/<backend-entity-id>","backend_format":"OPENAI_CHAT"}]' \
  --request-middleware '[{"name":"nemo-guardrails","config_type":"guardrail_config","config_id":"my-workspace/content-safety"}]' \
  --response-middleware '[{"name":"nemo-guardrails","config_type":"guardrail_config","config_id":"my-workspace/content-safety"}]'

The guardrails plugin doesn't route — the VirtualModel's --models array decides the upstream backend. Inline configs (no stored entity) are also supported via "config":{...} instead of "config_id".

Guardrails + Switchyard translate — ordering matters

The nemo-guardrails plugin only understands OpenAI chat completions shape. When the backend is in a different format (e.g. ANTHROPIC_MESSAGES) and a translate middleware is also in the chain, guardrails must see the OpenAI form on both sides:

• Request stack: guardrails BEFORE translate. Guardrails inspects the OpenAI request from the client, then translate rewrites it to the backend format.
• Response stack: guardrails AFTER translate. Translate converts the backend response back to OpenAI first, then guardrails runs output rails on the OpenAI shape.
• Inference calls: keep the request body in OpenAI shape regardless of the backend's backend_format. The translate middleware handles the conversion to/from backend format; sending Anthropic-shaped bodies bypasses guardrails' input rails because the plugin can't parse them.

nemo virtual-models create vm-guarded-translate --workspace my-workspace \
  --models '[{"model":"my-workspace/aws-anthropic-claude-opus-4-5","backend_format":"ANTHROPIC_MESSAGES"}]' \
  --request-middleware '[
    {"name":"nemo-guardrails","config_type":"guardrail_config","config_id":"my-workspace/content-safety"},
    {"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"anthropic","enable_stats":false}}
  ]' \
  --response-middleware '[
    {"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"anthropic","enable_stats":false}},
    {"name":"nemo-guardrails","config_type":"guardrail_config","config_id":"my-workspace/content-safety"}
  ]'

Same principle applies when random_routing mixes OpenAI and Anthropic backends behind one VirtualModel: put guardrails first in the request chain (before any routing/translate decisions), and put guardrails last in the response chain (after translate has normalized everything back to OpenAI).

If a rails config declares a task LLM via models[] (e.g. content_safety, topic_control), the task LLM must itself be addressable as OpenAI chat completions — point models[].model at an OpenAI-format entity ID, not at an Anthropic-format backend. The guardrails plugin doesn't translate task-LLM calls.

Step 5 — Making inference calls

Preferred (nemo CLI):

nemo inference gateway model post v1/chat/completions <vm-name> \
  --workspace my-workspace \
  --body '{"model":"my-workspace/<entity-id>","messages":[{"role":"user","content":"hi"}],"max_tokens":15}'

Verify routing by running multiple times (model field in response flips between backends at the configured probability). Parse with Python — see the parsing-pitfalls subsection below for why jq is the wrong tool here:

for i in $(seq 1 10); do
  nemo inference gateway model post v1/chat/completions vm-random-cross \
    --workspace my-workspace \
    --body '{"model":"my-workspace/vm-random-cross","messages":[{"role":"user","content":"hi"}],"max_tokens":400}' \
    | python3 -c 'import json,sys; d=json.loads(sys.stdin.read(), strict=False); print(d.get("model"))'
done

For reasoning models (e.g. nemotron-nano-*), use max_tokens ≥ ~200; the model spends most of its budget on a hidden reasoning trace and produces an empty content if cut off mid-trace (finish_reason: length). That is not an inference error, just a budget shortfall.

curl (streaming or when you need raw output):

curl -s -X POST \
  "http://localhost:8080/apis/inference-gateway/v2/workspaces/my-workspace/openai/-/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{"model":"my-workspace/<entity-id>","messages":[{"role":"user","content":"Hello"}],"max_tokens":64,"stream":true}'

Parsing pitfalls — jq and reasoning-model responses

Reasoning models (nemotron-nano-* and similar) sometimes emit responses that contain raw control characters (literal U+0000–U+001F bytes — typically unescaped newlines or tabs) inside the reasoning_content field. This is technically invalid JSON. Python's json.loads rejects it by default but can be configured with strict=False to accept it. jq does not — it bails out with:

jq: parse error: Invalid string: control characters from U+0000 through
U+001F must be escaped at line N, column M

In a routing-verification loop driven by jq -r '.model', every nemotron response will look like a failure (empty model field) while every opus response parses fine — making the split look catastrophically broken when inference is actually working. Always parse these responses with Python using strict=False (or a similarly lenient parser) instead of jq:

echo "$resp" | python3 -c '
import json, sys
d = json.loads(sys.stdin.read(), strict=False)
print(d.get("model"), d["choices"][0].get("finish_reason"))
'

If you must stay in shell, jq --slurp -R 'fromjson?' will tolerate most of these payloads but is brittle and not worth the effort. Python is the right tool.

Verifying routing decisions (DEBUG logs)

Start with LOG_LEVEL=DEBUG. Filter service output:

grep -E "RandomRoutingRequestProcessor|picked tier|FormatTranslate|StampOriginalFormat" \
  <service-output-file>

Expected output:

debug RandomRoutingRequestProcessor: picked tier=strong model=my-workspace/aws-anthropic-claude-opus-4-5
debug RandomRoutingRequestProcessor: picked tier=weak   model=my-workspace/nvidia-nvidia-nemotron-nano-31b-v3

tier= and model= in DEBUG logs match response.model returned to the client.

Failure cases

❌ Translate in request_middleware only — response shape mismatch

The most common mistake. Request translates fine, backend call succeeds, but the client receives the backend's native response shape instead of OpenAI:

Fix: always add response_middleware with the same translate config.

❌ Path mismatch without translate → 502

Sending /v1/messages traffic to a VM backed by an OpenAI-format model with no translate fails because IGW forwards the inbound path unchanged. Switchyard's PathUpdateProcessor only rewrites request.path when translate is configured.

❌ Reconciler-induced 404

VM works briefly then returns "Model entity not found". The reconciler overwrote served_models from upstream auto-discovery. Fix: use auto-discovered entity IDs in VM --models (they survive reconciler cycles) instead of manually-registered aliases.

❌ NVIDIA hub silently accepts format mismatches

OpenAI body at /v1/chat/completions routed to an Anthropic backend without translate does not fail at the upstream — NVIDIA hub accepts OpenAI-shaped bodies for Anthropic models and returns the Anthropic-native response. The visible failure is the response shape mismatch above (case 1), not a 4xx/5xx.

❌ Guardrails after translate (request) or before translate (response)

nemo-guardrails parses OpenAI chat shape only. With an Anthropic backend and request_middleware=[translate, guardrails], guardrails sees the already-translated Anthropic body and either skips its rails or errors on the unexpected shape. Symmetrically, response_middleware=[guardrails, translate] runs guardrails against the raw Anthropic response. Fix: keep guardrails on the OpenAI side of translate in both chains — request=[guardrails, translate], response=[translate, guardrails]. Same applies to clients sending Anthropic-shaped request bodies: input rails won't fire. Send OpenAI shape and let translate do the conversion.

Troubleshooting

DB disk I/O error on startup — orphaned WAL journal files. Delete all three:

rm -rf ~/.local/share/nemo

nemo-switchyard fails to load at startup — switchyard.lib not importable. Run uv sync from the repo root with default groups enabled to install the plugin and its vendored switchyard library (plugins/nemo-switchyard/vendor/switchyard/), then restart services.

401 from Anthropic — missing or wrong --auth-header-format. Verify:

nemo inference providers get anthropic --workspace my-workspace --output-format json \
  | jq '.auth_header_format'

No served models after provider creation — wait ~10s for the first reconciler cycle, then check:

nemo inference providers get nvidia-inference --workspace my-workspace \
  --output-format json | jq '.served_models | length'

Cleanup

# Delete all switchyard test VMs
for vm in $(nemo virtual-models list --workspace my-workspace --output-format json \
  | jq -r '.data[].name' | grep vm-); do
  nemo virtual-models delete "$vm" --workspace my-workspace
done

nemo inference providers delete nvidia-inference --workspace my-workspace
nemo secrets delete nvidia-inference-key --workspace my-workspace
nemo workspaces delete my-workspace