find-untested-sources

Find Untested Sources

Purpose

Coverage tools answer "which lines were executed?" — they require a green build and a passing test run, which is minutes-to-tens-of-minutes on a real repo. The question this skill answers is different and much cheaper:

Which source files have no test file referencing any of their declared types/symbols?

That's the question an agent asks before writing a new test — and it can be answered statically in a few seconds by parsing source files, with no build, no dependency resolution, and no compilation. The output is a deterministic test-pairing map that lets the agent pick the next file to test without reading the entire codebase first.

Two engines — pick one

This skill ships two interchangeable analyzers with a compatible JSON contract:

For a .NET-only repository, prefer the Roslyn engine — its namespace-aware pairing beats the polyglot engine's identifier overlap.

Required workflow

1. Use the narrowest repository or package root named by the caller. Do not scan a parent workspace when the request identifies a subdirectory.
2. Execute the appropriate analyzer once. Do not replace analyzer execution with manual globbing, filename matching, or visual inspection. For polyglot analysis, pass --include-tested when the answer must distinguish paired sources from unpaired sources.
3. Base the result on the analyzer's JSON. Preserve its paired/unpaired classification and suggested relative path; do not guess a different path.
4. When the caller named a subdirectory, prefix analyzer-relative paths with that subdirectory so reported paths are workspace-relative.
5. Report the requested result plus the static-pairing coverage caveat. Do not append build, package-install, test-run, or coverage commands. When paired sources exist, name their covering test files so the unpaired classification is auditable.

When to Use

• User asks "where should I add tests?", "which files have no tests?", "find untested code", "give me a test gap list", "what's the next file to test".
• Before invoking a test-generation agent, to produce a prioritized worklist.
• After generating tests, to verify each new test file pairs to a source file.
• To enumerate "weakly paired" source files (only one referring test) for follow-up depth checks.

When Not to Use

• Line/branch coverage — use coverage-analysis.
• CRAP-score / risk hotspots — use coverage-analysis.
• Are existing tests strong? — use test-gap-analysis (mutation reasoning) or assertion-quality.

Roslyn engine (C#)

Prerequisites

• .NET SDK that supports file-based apps (dotnet run script.cs). Pinned in the repo's global.json (SDK 11 preview or later).
• No internet access required beyond the initial NuGet restore of Microsoft.CodeAnalysis.CSharp on first run.

Usage

# From the skill folder
dotnet run scripts/Find-UntestedSources.cs -- <repo-root> [--top N]

# Save the report
dotnet run scripts/Find-UntestedSources.cs -- <repo-root> > pairing.json

# Iterate the untested list, highest-API-surface first
$report = Get-Content pairing.json | ConvertFrom-Json
$report.untested | Select-Object -First 10 source, decl_count, suggested_test_path

Diagnostics go to stderr; JSON goes to stdout.

Output schema

{
  "repo": "<absolute path>",
  "elapsed_ms": 8883,
  "counts": {
    "source_files": 3036,
    "test_files": 867,
    "untested_files": 1852,
    "paired_files": 1184
  },
  "untested": [
    {
      "source": "src/Foo/Bar.cs",
      "decl_count": 8,            // # of type declarations in the file
      "suggested_test_path":      // mirror of source under a discovered test project
        "tests/Foo.Tests/Bar/BarTests.cs"
    }
  ],
  "source_to_tests": {
    "src/Foo/Baz.cs": [
      "tests/Foo.Tests/BazTests.cs",
      "tests/Foo.IntegrationTests/Scenarios/BazScenarios.cs"
    ]
  }
}

How it works

1. File discovery — recursive walk pruning bin/, obj/, node_modules/, .git/, .vs/, packages/, and any dotted subdir. Skips generated files (.g.cs, .Designer.cs, .AssemblyInfo.cs).
2. Test vs source classification — walks up to the nearest .csproj and marks it a test project if the project name ends in .Tests, .Test, .UnitTests, .IntegrationTests, .E2E, .EndToEnd, .Spec, .Specs, or the content references Microsoft.NET.Test.Sdk, MSTest.Sdk, Microsoft.Testing.Platform, xunit, NUnit, TUnit, or <IsTestProject>true</IsTestProject>.
3. Source index (parallel) — parse each source file with CSharpSyntaxTree.ParseText (syntax only, no compilation); record every BaseTypeDeclarationSyntax / DelegateDeclarationSyntax as (ShortName, EnclosingNamespace, FilePath).
4. Test scan (parallel) — parse each test file, collect using directives + enclosing namespace, walk every IdentifierToken, look it up in the short-name index, and disambiguate strictly: an identifier is attributed only if the declaration's namespace matches one of the test file's using directives, the enclosing namespace, or a prefix of them. This avoids noise where common names like Settings or Context match every project.
5. Pairing & suggestion — invert into source → [tests]. Build a production-to-test project map from <ProjectReference> entries; for each untested source, mirror its in-project relative path under the referencing test project to suggest a path.
6. JSON emit — ordered by declaration count desc, then alphabetical.

Polyglot engine (tree-sitter)

Prerequisites

• Python 3.10+.
• pip install tree-sitter-language-pack (single self-contained wheel that bundles parsers for 300+ languages and the high-level process() API). No native build, no per-language grammar install.

Usage

# From the skill folder
python scripts/find_untested_sources.py <repo-root>

# Restrict to a language (repeatable)
python scripts/find_untested_sources.py <repo-root> --lang python --lang typescript

# Truncate the report (top 20 by declared API surface)
python scripts/find_untested_sources.py <repo-root> --limit-untested 20 > pairing.json

# Iterate, highest-API-surface first
$report = Get-Content pairing.json | ConvertFrom-Json
$report.untested_sources | Select-Object -First 10 path, declaration_count, suggested_test_path

Pass --include-tested to additionally emit tested_sources (omitted by default to keep the payload small for LLM consumption). Diagnostics go to stderr; JSON goes to stdout.

Output schema

{
  "repo_root": "<absolute path>",
  "summary": {
    "source_files": 3138,
    "test_files": 761,
    "tested_source_files": 1419,
    "untested_source_files": 1719,
    "orphan_test_files": 15,
    "languages": ["csharp"]
  },
  "untested_sources": [
    {
      "path": "src/Foo/Bar.cs",
      "language": "csharp",
      "declaration_count": 8,
      "declarations": ["Bar", "BarOptions", "IBar", "..."],
      "suggested_test_path": "src/Foo/BarTests.cs"
    }
  ],
  "orphan_tests": [
    { "path": "tests/SomeIntegrationTest.cs", "language": "csharp" }
  ]
}

How it works

1. File discovery — recursive walk pruning common build/vendor dirs (bin, obj, node_modules, target, dist, build, vendor, __pycache__, .venv, .git, …) and generated files (.d.ts, .g.cs, .Designer.cs, _pb2.py, *.min.js, AssemblyInfo.cs, …).

2. Language detection — detect_language_from_path maps the extension to a supported language; unknown extensions are skipped.

3. Test-vs-source classification — per-language path heuristics:

   Language	Test rule
   Python	path contains tests//test/; or filename starts with test_ or ends _test.py; or conftest.py.
   JS/TS/TSX	path contains __tests__, tests, test, spec, e2e; or filename contains .test./.spec..
   Go	filename ends _test.go.
   Java	path contains test/tests; or filename ends Test.java/Tests.java.
   Rust	path contains tests//benches/.
   C#	path contains tests/; or project segment ends .Tests/.Test/.UnitTests/.IntegrationTests; or filename ends Tests/Test.
   Ruby	path contains spec//test/; or filename ends _spec.rb/_test.rb.

4. Per-file extraction — process(text, ProcessConfig(structure, imports, symbols)) returns declared items, raw import statements, and a flat declared -name list.

5. Pairing — for each test file, union import resolution (per language, e.g. Python from pkg.mod import x → pkg/mod.py; Java import a.b.C; → a/b/C.java; C# using is namespace-not-file, so a no-op) with identifier overlap (word-like tokens, length ≥ 4, matched against declared names).

6. JSON emit — untested_sources ordered by declaration count descending.

Limitations (be honest with the agent)

Both engines are static, parse-only heuristics that trade a little accuracy for orders-of-magnitude lower cost than coverage. Known gaps:

• Reflection / DI-resolved types referenced only via a string name or container resolution won't be detected — the type's short name never appears in the test source.
• Extension methods invoked as instance methods (C#): the declaring static class is not named, so its file is not credited.
• var, target-typed new(), pattern matching lose the type token; the file-level union usually still catches it through other references.
• Short identifier names (polyglot, < 4 chars) are dropped to avoid noisy pairings on names like id, db, Tag.
• Monorepo path aliases (TS path mapping, Java module-info) are not resolved; a suffix-match fallback may pick the wrong source if two files share a trailing path segment.

For these cases, run actual coverage (coverage-analysis) on the unpaired candidates the agent has already triaged.

Always label the final result as a static pairing heuristic, not evidence of line or branch coverage. Include that caveat even when every requested source file has an obvious matching or missing test.

Outputs the agent should consume

• untested[*].source / untested_sources[*].path — pick the next source file to test (highest declaration count first).
• *.suggested_test_path — drop-in target for the new test file; the Roslyn engine honors the test project that already <ProjectReference>s the source's project, so dotnet sln add is not needed. The polyglot engine may suggest a co-located test when no test root is discoverable; that is a valid fallback, but prefer an established repository test directory when one exists.
• source_to_tests (Roslyn) / --include-tested tested_sources (polyglot) — verify a newly written test file lands in the list for the intended source.
• orphan_tests (polyglot) — tests that don't reference any same-language source file; useful for triaging stale or integration-only tests.