troubleshoot-ssi

Troubleshoot APM SSI on Kubernetes

Triggers

Invoke this skill when the user expresses intent to:

• Debug why a pod is not being instrumented
• Investigate why traces are not appearing in Datadog
• Diagnose admission webhook or init container injection failures
• Follow up on failed checks from verify-ssi
• Report that a specific service or pod has no traces

Do NOT invoke this skill if:

• SSI has not been enabled yet — run enable-ssi first

Prerequisites

• kubectl configured to target cluster — kubectl config current-context

pup-cli: check, install, and authenticate

Claude runs

pup --version

If not found, install it (OS-aware):

Claude runs

if [[ "$(uname)" == "Darwin" ]]; then
  brew tap datadog-labs/pack && brew install pup
else
  PUP_VERSION=$(curl -s https://api.github.com/repos/datadog-labs/pup/releases/latest | grep '"tag_name"' | cut -d'"' -f4)
  curl -L "https://github.com/datadog-labs/pup/releases/download/${PUP_VERSION}/pup_linux_amd64.tar.gz" | tar xz -C /usr/local/bin pup
  chmod +x /usr/local/bin/pup
fi
pup --version

Check auth:

pup auth status

If not authenticated:

Claude runs

pup auth login

This opens a browser tab for OAuth. Complete the login there — Claude will continue once the command exits.

If no browser available: export DD_APP_KEY=<your-app-key>.

Context to resolve before acting

How SSI Works — Domain Knowledge

Read this before investigating. It gives you the mental model to reason about novel failures, not just known ones.

Injection chain:

1. Admission webhook (registered by Cluster Agent) intercepts pod creation
2. Webhook mutates the pod spec — adds a datadog-lib-<language>-init init container
3. Init container downloads the tracer library onto a shared volume
4. LD_PRELOAD env var is set pointing to the library .so file
5. Application process loads the library automatically on startup via LD_PRELOAD

What each diagnostic layer can see:

• pup — sees what Datadog's backend received. Blind to cluster-side injection failures. If pup shows no tracer telemetry for the service, the tracer was either never injected (cluster-side — confirm with the kubectl init-container check) or injected but unable to report yet (connectivity, DD_SITE, API key, or telemetry lag). Don't assume the cluster; cross-check kubectl.
• kubectl — sees cluster state. Blind to whether data reached Datadog. If kubectl shows the init container but pup shows no traces, the problem is post-injection.

What healthy looks like:

• pup fleet tracers list shows the service as active, with the expected language
• kubectl get pod -o jsonpath='{.spec.initContainers[*].name}' includes datadog-lib-<language>-init

Known silent failures — SSI produces no error when these occur:

• Existing ddtrace or OTel instrumentation — SSI detects it and silently disables itself
• Unsupported runtime version — silently skipped
• admission.datadoghq.com/enabled: "false" annotation — webhook skips the pod entirely
• Pod not restarted after SSI enabled — injection happens at startup; existing pods keep running uninstrumented
• Pod in Agent namespace — SSI never instruments its own namespace

Reasoning shortcuts:

• No init container → webhook didn't fire → check: namespace targeting, pod-selector, opt-out annotation, webhook registration, pod not restarted
• Init container present + no traces → check whether the service is in pup fleet tracers list: absent → tracer not reporting (Agent connectivity, DD_SITE mismatch, API key, blocked egress; or the tracer never loaded — existing ddtrace/OTel, unsupported runtime); present → reporting telemetry but spans not arriving (no traffic, sampling, ingestion/retention filter)

Step 1: Triage

Run all seven simultaneously and surface them back to the user as the diagnostics you're running. Everything after this is driven by what you find here. Resolve <NODE_HOSTNAME> from kubectl get pod <POD_NAME> -n <APP_NAMESPACE> -o jsonpath='{.spec.nodeName}' once you have a pod name; if no pod context yet, run the pup commands without --hostname first.

Claude runs

pup traces search --query "service:<SERVICE_NAME>" --from 1h --limit 5
pup fleet tracers list --filter "service:<SERVICE_NAME>"
pup apm troubleshooting list --hostname <NODE_HOSTNAME> --timeframe 1h
pup apm service-library-config get --service-name <SERVICE_NAME> --env <ENV>
kubectl get pod <POD_NAME> -n <APP_NAMESPACE> \
  -o jsonpath='{.spec.initContainers[*].name}'
kubectl describe pod <POD_NAME> -n <APP_NAMESPACE> | grep -A 10 "Events:"
kubectl get mutatingwebhookconfigurations | grep datadog

The last command confirms the Admission Controller webhook is registered cluster-wide — this is the precondition for SSI injection working at all and must be checked even when most other services are being instrumented (any deviation in one webhook config can silently skip a subset of pods).

pup apm troubleshooting list surfaces injection errors that Datadog's backend received from the cluster — these point to cluster-side mutation failures that may not be visible from kubectl describe alone. pup apm service-library-config get shows the runtime SDK config the tracer is operating under; an empty result with ddTraceConfigs configured, or unexpected values, points to UST/config-propagation issues.

Presenting your findings (required)

Your final response is the deliverable — not your investigation transcript. It must include every diagnostic from this skill that you ran or that applies, each with its purpose and what you found. Three failure modes to avoid:

• Running a check but not reporting it. If you ran kubectl get mutatingwebhookconfigurations, the namespace admission.datadoghq.com/mutate-pods label check, or any other command during investigation, state the command and its result in your response. A check you ran but didn't surface gives the reader nothing — and the namespace-label and webhook checks in particular must appear explicitly.

• Omitting the two required pup diagnostics. Every diagnosis must explicitly include these two commands, by name, for each affected service — they are mandatory triage output, not optional:

  • pup apm troubleshooting list --hostname <NODE_HOSTNAME> — surfaces injection errors Datadog received from the node
  • pup apm service-library-config get --service-name <SERVICE_NAME> --env <ENV> — shows the tracer's runtime SDK config

  Run them if pup is available; recommend them for the user to run if it isn't. Do not substitute pup traces search for these — it is a different check and does not satisfy the runbook. If you don't know <ENV>, state your assumed value and run the command anyway.

• Stopping at the first root cause. When multiple services are affected, investigate and report each one independently — they may have different causes — and give per-service remediation.

Step 2: State Your Hypotheses

Before investigating, explicitly state your ranked hypotheses based on triage output. Do not skip this step.

When the user reports multiple affected services in the same namespace, diagnose each independently. Two pods can fail injection for entirely different reasons (one opt-out annotation, one missing namespace label, one with pre-existing ddtrace). Do not assume a shared root cause — investigate each service's pod spec, annotations, and runtime separately and surface findings per-service.

State your top 1-3 hypotheses explicitly: "Based on triage, I think the most likely cause is X because Y."

Step 3: Investigate

Use only the tools relevant to your hypotheses. Each observation informs your next action.

Cluster-side investigation tools

Is the pod in the Agent namespace? SSI never instruments pods in the same namespace as the Datadog Agent.

kubectl get pods -n <AGENT_NAMESPACE>

Were pods restarted after SSI was enabled?

Confirm with the user before restarting. Tell the user: "Pods must be restarted for SSI to inject into them. I'll restart <DEPLOYMENT_NAME> in <APP_NAMESPACE>. Ready to proceed?" Wait for confirmation.

Claude runs

kubectl rollout restart deployment/<DEPLOYMENT_NAME> -n <APP_NAMESPACE>
kubectl wait --for=condition=Ready pod -l app=<APP_LABEL> -n <APP_NAMESPACE> --timeout=120s

Claude runs

# Primary — authoritative and immediate: confirm the freshly-restarted pods carry the init container.
# Label-scoped, not by <POD_NAME>: a rolling restart replaces the old pod with new-suffixed ones.
kubectl get pod -l app=<APP_LABEL> -n <APP_NAMESPACE> \
  -o jsonpath='{.items[0].spec.initContainers[*].name}'
# Secondary — eventual: the service reappears here once the restarted pod reports telemetry
# (subject to a propagation delay of a minute or more, and only if the pod serves traffic)
pup fleet tracers list --filter "service:<SERVICE_NAME>"

The kubectl init-container check is the authoritative post-restart signal — the pod is injected the moment that init container appears. tracers list is telemetry-derived and lags, so don't read an empty result immediately after a restart as "injection didn't happen."

Does the namespace carry the Admission Controller opt-in label? When the Admission Controller runs with mutateUnlabelled: false, injection happens only in namespaces explicitly labeled admission.datadoghq.com/mutate-pods=true. A namespace missing this label silently has SSI skipped for every pod in it — a common cause when most cluster services are instrumented but one namespace's services aren't.

kubectl get namespace <APP_NAMESPACE> -o jsonpath='{.metadata.labels}'
kubectl get namespace <APP_NAMESPACE> --show-labels

Fix: label the namespace, then restart the affected deployments so the AC mutates them on pod recreate.

kubectl label namespace <APP_NAMESPACE> admission.datadoghq.com/mutate-pods=true

Is namespace targeting filtering the pod out?

kubectl get datadogagent datadog -n <AGENT_NAMESPACE> -o yaml | grep -A 15 instrumentation

Fix: update enabledNamespaces in datadog-agent.yaml.

Claude runs

kubectl apply -f datadog-agent.yaml

Is a podSelector target filtering the pod out? If targets with podSelector is configured, only pods whose labels match the selector are instrumented. Check whether the app pod's labels match any target:

kubectl get datadogagent datadog -n <AGENT_NAMESPACE> -o yaml | grep -A 20 targets
kubectl get pod <POD_NAME> -n <APP_NAMESPACE> --show-labels

Fix: add a matching label to the pod template, or broaden the podSelector, then apply and restart.

Is a pod annotation opting it out — or missing the AC's injection-success annotation? Two annotations to look for:

• admission.datadoghq.com/enabled: "false" — explicit opt-out, AC skips the pod.
• admission.datadoghq.com/status: injected — set by the AC after successful mutation; its absence on a running pod is positive evidence the AC never mutated it.

kubectl get pod <POD_NAME> -n <APP_NAMESPACE> -o jsonpath='{.metadata.annotations}'
kubectl get pod <POD_NAME> -n <APP_NAMESPACE> -o yaml | grep -A 10 annotations

Fix: remove an opt-out annotation from the Deployment pod template, then apply and restart.

Are the expected DD_* environment variables present in the running pod? SSI injects DD_SERVICE, DD_ENV, DD_VERSION, DD_TRACE_*, and LD_PRELOAD into the container env when it mutates a pod. Their absence confirms the mutation did not run; their presence with unexpected values points to UST label mismatches or ddTraceConfigs issues.

kubectl exec -n <APP_NAMESPACE> <POD_NAME> -- env | grep -E '^(DD_|LD_PRELOAD)'
kubectl describe pod <POD_NAME> -n <APP_NAMESPACE> | grep -E 'DD_|LD_PRELOAD'

Claude runs

kubectl apply -f <your-app-deployment.yaml>

Confirm with the user before restarting. Tell the user: "I need to restart <DEPLOYMENT_NAME> in <APP_NAMESPACE> for this change to take effect. Ready to proceed?" Wait for confirmation.

Claude runs

kubectl rollout restart deployment/<DEPLOYMENT_NAME> -n <APP_NAMESPACE>

Does the app have existing custom instrumentation? SSI silently disables itself when it detects existing tracer code. Scan source files for:

• Python: import ddtrace, ddtrace.patch_all()
• Node.js: require('dd-trace'), DD.init()
• Java: GlobalTracer.register(, dd-java-agent
• .NET: Tracer.Instance, DD.Trace
• Ruby: require 'ddtrace', Datadog.configure
• PHP: DDTrace\

Also check dependency manifests: requirements.txt, package.json, Gemfile, pom.xml.

Fix: remove the import/package, rebuild image, reload into cluster, restart pod.

Is the base image Alpine (musl libc)? K8s SSI injects LD_PRELOAD as an environment variable into the pod — it does not rely on /etc/ld.so.preload, so musl/Alpine images are supported. This is not a blocker for Kubernetes SSI.

Is the runtime version supported?

kubectl exec -n <APP_NAMESPACE> <POD_NAME> -- python --version
kubectl exec -n <APP_NAMESPACE> <POD_NAME> -- node --version
kubectl exec -n <APP_NAMESPACE> <POD_NAME> -- java -version

Verify against SSI compatibility matrix.

Is the admission webhook registered?

kubectl get mutatingwebhookconfigurations | grep datadog
kubectl get pods -n <AGENT_NAMESPACE> -l app=datadog-cluster-agent
kubectl logs -n <AGENT_NAMESPACE> -l app=datadog-cluster-agent --tail=100

Did injection produce errors? Get the node hostname first, then query Datadog for injection errors:

kubectl get pod <POD_NAME> -n <APP_NAMESPACE> -o jsonpath='{.spec.nodeName}'
pup apm troubleshooting list --hostname <NODE_HOSTNAME> --timeframe 1h

Is the Agent sending data to Datadog?

kubectl exec -n <AGENT_NAMESPACE> \
  $(kubectl get pod -n <AGENT_NAMESPACE> -l app=datadog-agent -o name | head -1) \
  -- agent status | grep -A 5 "APM Agent"

Datadog-side investigation tools

Is the tracer reporting?

pup fleet tracers list --filter "service:<SERVICE_NAME>"

Does APM recognise the service?

pup apm services list --env <ENV>

What SDK configuration is the service running with? Shows env vars the tracer is configured with (e.g. DD_TRACE_ENABLED, DD_SERVICE, DD_ENV, sampling rules). Empty output is expected if ddTraceConfigs was not set in enable-ssi; a populated output mismatching what was configured indicates the change didn't propagate.

pup apm service-library-config get --service-name <SERVICE_NAME> --env <ENV>

Are traces arriving?

pup traces search --query "service:<SERVICE_NAME>" --from 1h --limit 10

Which agent is the tracer connected to? Use if connectivity between tracer and Agent is suspected.

pup fleet agents list --filter "hostname:<NODE_HOSTNAME>"
pup fleet agents tracers <AGENT_KEY> --filter "service:<SERVICE_NAME>"

Step 4: Reflect Before Concluding

Before applying any fix, answer:

1. What evidence confirms my hypothesis?
2. What evidence would contradict it — and have I checked?
3. Is there a simpler explanation I haven't considered?

If the conclusion doesn't hold up, return to Step 2 with new hypotheses. Keep iterating until you can defend the conclusion against all three questions.

Step 5: Fix

Apply the fix for the confirmed root cause. If the fix requires a code or Dockerfile change, rebuild and reload:

Claude runs

docker build -f <DOCKERFILE_PATH> -t <IMAGE_NAME> <BUILD_CONTEXT>

[DECISION: cluster type]

• kind (local): load the image into the cluster

Claude runs

kind load docker-image <IMAGE_NAME> --name <CLUSTER_NAME>

• Registry-based: skip — image will be pulled on next deployment

Confirm with the user before restarting. Tell the user: "I need to restart <DEPLOYMENT_NAME> in <APP_NAMESPACE> to apply the fix. Ready to proceed?" Wait for confirmation.

Claude runs

kubectl rollout restart deployment/<DEPLOYMENT_NAME> -n <APP_NAMESPACE>
kubectl wait --for=condition=Ready pod -l app=<APP_LABEL> -n <APP_NAMESPACE> --timeout=120s

Step 6: Verify

Re-run triage to confirm the fix worked:

Claude runs

pup traces search --query "service:<SERVICE_NAME>" --from 1h --limit 5
pup fleet tracers list --filter "service:<SERVICE_NAME>"

If traces are arriving — resolved (the service may take a minute to reappear in pup fleet tracers list; an empty tracers list immediately after the restart is expected telemetry lag, not a failure). Automatically proceed to onboarding-summary now — do not ask the user for permission.

ERROR: No traces arriving — return to Step 2 with the new triage data and form updated hypotheses.

Security constraints

• Never write a raw API key into any file or chat message
• Never run kubectl delete without user confirmation
• Never modify admissionController settings directly
• docker push to a registry always requires user confirmation