cxc-dev-debugging

dev-debugging — Systematic Root Cause Analysis

This skill is the thinking process for fixing bugs. As a routing role it activates by change-surface (an error/bug to diagnose), not by any external dispatcher. It enforces a structured phases 0-4 methodology for every technical issue — test failures, runtime errors, build failures, performance regressions, integration bugs.

Boundary: This skill covers how to reason about bugs. For test harness, reproduction frameworks, and verification tooling, see dev-testing. For domain-specific context (API errors, hydration issues, query performance), consult dev-backend or dev-frontend.

C0/C1 work (small local patches): See dev §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references.

dev is canonical: dev §0.2 Rule Classes, §3 Verification Gate, and §5 Safety Rules apply to all work governed by this skill.

dev-debugging = root cause methodology (the thinking)
dev-testing   = test harness for reproducing/verifying (the tooling)
dev §2        = summary pointer to this skill (the overview)

Core Principle

Check if the problem is structural before debugging code. Complete root cause investigation before proposing any fix. If Phase 1 is not done, keep investigating.

When to Activate

• Test failures, runtime errors, build failures, performance regressions
• Integration issues (API, database, third-party), CI pipeline failures
• Especially: when under time pressure or when "just one quick fix" seems obvious — that's when methodology matters most

The Phases

Phase 0: Is This a Bug or a Design Problem?

Before debugging code, ask: "Could this be a structural/design issue rather than a code bug?" Patching symptoms of architectural debt creates an endless stream of "bugs" that are really design consequences.

Decision Tree — escalate to architecture review if any apply:

If structural: escalate to architecture review. Do not patch the symptom — the patch creates the next bug.

Symptom vs Root Cause Fix:

If none of the above apply — proceed to Phase 1 (it's a code bug, not a design problem).

Phase 1: Root Cause Investigation

Feedback loop gate: For UI, browser, TUI, visual, streaming, or agent-output bugs, first create a red-capable loop that can fail before the fix: screenshot/assertion, recorded terminal bytes, Playwright visual check, log fixture, or a manual repro script with explicit pass/fail evidence. Do not patch from screenshots alone when a repeatable probe can be built in reasonable time.

Complete these before attempting any fix:

1. Read the full error — stack trace, line numbers, error code, surrounding context. Do not skim. The answer is often in the error message itself.

2. Reproduce consistently — exact steps to trigger the bug. If intermittent, document frequency, conditions, and environment state. A bug you cannot reproduce is a bug you cannot verify as fixed.

3. Check recent changes — run git log --oneline -10 and git diff. Check new dependencies, config changes, environment variables. Bugs correlate with recent changes most of the time.

4. Trace data flow — where does the bad value originate? Trace backward from the failure point through the call stack until you find the source. Follow the full causal chain from trigger → boundary → bad state → failure. Removing the visible symptom is not a fix unless the defect that creates the bad state is gone.

5. Instrument component boundaries — for multi-layer systems (API → service → database, CI → build → deploy), log input/output at each boundary BEFORE proposing fixes.

6. Trace-first for distributed/async/agent failures (DEFAULT) — capture the evidence trail before hypothesizing: request IDs, OpenTelemetry spans/logs, Playwright traces/videos, exact agent tool transcripts. For order-dependent or intermittent failures logs cannot explain, use time-travel/replay debugging (Microsoft TTD on Windows, rr on Linux).

For EACH component boundary:
  - Log what data enters the component
  - Log what data exits the component
  - Verify environment/config propagation
Run once → analyze evidence → identify failing layer → investigate THAT layer

Work through these steps; skip only if clearly irrelevant to the problem at hand.

Phase 2: Pattern Analysis

1. Find working examples — similar working code in the same codebase. If it worked before, use git bisect to find the breaking commit (see references/tool-guides.md).

2. Compare systematically — list every difference between working and broken code. No matter how small. Resist assuming "that can't matter."

3. Read reference docs completely — official documentation for the library, API, or framework involved. Don't skim — read the full relevant section.

4. Check known issues — GitHub Issues, changelogs, migration guides. Someone may have hit the same bug. Search with the exact error message.

When the bug depends on third-party library/API/framework behavior, current error workarounds, upstream issues, changelogs, or migration guides, read the active search skill and follow its source-fetch and evidence-status rules before treating external material as proof.

Phase 3: Hypothesis and Testing

STRICT (DEBUG-RCA-EVIDENCE-01): Before investigating any single root-cause hypothesis or making a root-cause claim, write at least three orthogonal hypotheses (H1/H2/H3) and one falsifier for each. Collapse duplicates, test against disconfirming evidence, and do not claim root cause until competing hypotheses have been ruled out by evidence. If fewer than three are plausible, state why.

1. State the leading hypothesis explicitly — "X is the root cause because evidence Y shows Z." If you can't articulate it clearly, you don't understand it yet.

2. Design a test to disprove — falsification is stronger than confirmation. What would you expect to see if your hypothesis is wrong?

3. Test one variable — smallest possible change, one variable at a time. Never fix multiple things at once.

4. If it fails → move to another listed hypothesis. Revert the failed change and start from clean state. Stacking fixes obscures the root cause.

5. Keep the rejection record — preserve rejected hypotheses and the evidence that rejected them. The final report must include them, not just the winning cause.

6. Admit ignorance — "I don't understand X" is a valid finding. Research further rather than guessing. Record the open question explicitly.

Phase 4: Implementation

STRICT (DEBUG-TOGGLE-PROOF-01): Enter implementation only after the captured value matches the hypothesis prediction, the repro repeats, and toggling the suspected cause off/on removes then restores the bug. Write one paragraph explaining the causal mechanism before patching.

1. Write a failing test first — the test reproduces the bug. It should fail before the fix. Use dev-testing for TDD patterns and test harness setup.

2. Make the minimal fix — address the root cause, not symptoms. One logical change only.

3. Verify: the test passes, no regressions (run the full test suite: npm test / pytest / equivalent).

4. Check for similar patterns — does the same bug class exist elsewhere in the codebase? Search for it. Fix all instances, not just the one you found.

5. Document — final report and commit message explain root cause AND fix, including rejected hypotheses and rejection evidence. Not "fixed bug" but "fix: race condition in session middleware caused by missing await on Redis write."

Red Flags — Return to Phase 1

If you catch yourself doing any of these, pause — root cause investigation was likely skipped.

Repeated Failure Rule: After repeated failed fix attempts, pause entirely. Each fix revealing a new problem in a different place is a sign of architectural issues, not simple bugs. Discuss with the user before attempting more fixes.

Slop Debugging Patterns

Slop debugging is spray-and-pray: guess, patch, pray, repeat.

Concrete Debugging Scenarios

Scenario A: API Returns 500

Root cause pattern: Missing input validation lets undefined values propagate into business logic. Instrument controller/service/repository boundaries to find where the bad value enters. Compare with a working endpoint that validates input with a schema. Fix: add schema validation at the entry point, write a test that sends invalid input and expects 400.

Worked example:

curl -i -X POST http://localhost:3000/api/orders \
  -H 'content-type: application/json' \
  -d '{"sku":"book-1"}'

Observed failure:

HTTP/1.1 500 Internal Server Error
TypeError: Cannot read properties of undefined (reading 'toFixed')
    at calculateTotal (src/orders/service.ts:42:21)
    at createOrder (src/orders/controller.ts:27:18)

Competing hypotheses before narrowing:

1. Request validation allows missing quantity.
2. Controller mapping drops quantity before service call.
3. Repository returns an order row with quantity = null.

Boundary instrumentation:

DEBUG=orders:* npm run dev
curl -s -X POST http://localhost:3000/api/orders \
  -H 'content-type: application/json' \
  -d '{"sku":"book-1"}' | jq .

Sample log output:

orders:controller input {"sku":"book-1"}
orders:controller mapped {"sku":"book-1"}
orders:service input {"sku":"book-1"}
orders:repository skipped insert due service error

Rejections: repository-null is rejected because the repository is never reached. Controller-drop is rejected because controller input already lacks quantity. Root cause: entry validation accepts a payload missing a required domain field. Fix at the entry boundary: schema rejects missing quantity; regression test posts the same payload and expects HTTP 400 with a stable error.code.

Scenario B: React Hydration Mismatch

Root cause pattern: Server renders a value (e.g., date, locale string) that differs from client-side rendering due to environment differences (UTC vs. local timezone). Compare with components that defer environment-dependent rendering to useEffect. Fix: move environment-dependent formatting into a client component.

Scenario C: N+1 Query Performance

Root cause pattern: List endpoint lazy-loads related records per item (1 query + N queries). Enable query logging to count queries, then compare with an endpoint that uses eager loading. Fix: add include/joinedload, write a test asserting bounded query count.

Scenario D: Flaky Test (Intermittent Failure)

Root cause pattern: Test passes in isolation but fails in suite due to shared mutable state (database rows, global variables, uncleared mocks). Compare with stable tests that use transaction rollback in beforeEach/afterEach. Fix: add proper test isolation, then search for other tests missing cleanup.

When to Escalate vs When to Keep Digging

Keep Digging When:

• You have untested hypotheses from Phase 2
• You haven't read the full error message or stack trace
• You haven't checked recent changes (git log, git diff)
• You haven't found working comparison code yet
• The bug is in YOUR code (not a third-party library)
• You still have untested approaches to try

Escalate When:

• Repeated fix attempts failed — likely architectural; needs human judgment
• Undocumented library behavior — file an issue upstream, work around it
• Environment-specific — requires access you don't have (prod DB, cloud IAM)
• Security-sensitive — don't debug auth/crypto/payment alone; flag for human review
• Multi-team dependency — bug is in another team's service or API contract
• Stalled: if investigation stalls, reassess approach

How to Escalate Well

Don't just say "I'm stuck." Provide: symptom (exact error), reproduction steps, evidence gathered (logs, traces, bisect results), hypotheses tested (including rejected hypotheses and rejection evidence), remaining hypotheses (untested), and a recommendation for next steps.

Post-Mortem Discipline

After resolving any bug that:

• Was user/customer-impacting
• Took >1 hour to diagnose
• Involved 3+ failed fix attempts (per postmortem-template.md)
• Revealed a systemic issue (same bug class exists elsewhere)

Fill out references/postmortem-template.md and include it in the PR or commit. The goal is learning, not blame. Every postmortem must produce at least one action item that prevents the same class of bug from recurring.

Modular References

Integration with Other Skills

Security-Sensitive Bugs

For security-sensitive bugs (auth bypass, data leak, injection), follow the incident response in dev-security/SKILL.md before applying a fix.

Compact Summary

When context is limited, preserve: (1) Phase 0 — is it a bug or a design problem?, (2) Core principle — no fixes without root cause, (3) phases 0-4 — architecture check → investigate → analyze → hypothesize → implement, (4) Repeated Failure Rule — after repeated failures, reassess, (5) one variable at a time, (6) evidence over intuition, (7) failing test first.