scoring

NLPM Quality Scoring Rubric

100-point quality scale for all NL programming artifacts. Apply penalties deterministically. Use calibration examples to anchor judgment on borderline cases.

Scoring Formula

base_score = 100
adjustments = sum of all applicable penalties (all penalties are negative)
final_score = max(0, min(100, base_score + adjustments))

Penalties stack. The floor is 0; the ceiling is 100. No bonuses — the default assumption is that an artifact is well-formed, and quality is measured by what is missing or wrong.

Penalty Tables

Skills

Scope-note discipline: R07 means "scope note when related skills exist." Do NOT apply R07 to missing example blocks — that is the new R06 row above (penalty -10, not -15). The 2026-05-13 lijigang/ljg-skills audit applied R07 + −15 fourteen times for missing example blocks; both labels were wrong (R07 is not example-related, and -15 is the agents penalty, not the skills penalty). The validator at auditor/scripts/validate-rule-ids.py catches this kind of drift in CI.

<example>-block counting discipline (added 2026-08-01, origin: xiaolai/cc-suite v1.3.1 remediation): an <example> block counts only when it sits outside fenced code blocks — in the body or in a frontmatter description block scalar. Tags inside a fenced template (```markdown … ```) are illustrative content, and prose that names the string `<example>` is a mention, not a block. A 2026-07-31 scoring pass credited a skill with example blocks that existed only inside a fenced template, hiding a real R06 violation across 13 files. Verify by reading the file, not by grepping for the tag.

name matches parent directory (added 2026-05-25, audit: google/skills): the open Agent Skills spec at agentskills.io makes this a MUST. Mismatch is deterministic, high-confidence, and reproducible by single-line diff (frontmatter name: vs basename($(dirname FILE))). Mark such findings confidence: high per the manifest-vs-disk-diff principle in agents/scorer.md step 6. Note: this penalty did not exist before 2026-05-25 — re-scoring past audits will yield slightly lower scores for any corpus containing this defect, but no contribute outcomes are retroactively affected since no PRs were ever opened against a name-mismatch finding under the prior rubric.

Agents

Commands

Shared Partials

Rules

Hooks — universal checks (apply to all tools)

Hooks (Claude Code — Tier 2-Claude only)

Authoritative event list: nlpm:conventions-claude §7. Per the multi-tool design (analysis/multi-tool-design-2026-05.md decision #4), Claude / Codex / Antigravity hook event vocabularies are NOT 1:1 mappable — three separate tables, no translation.

Hooks (Codex CLI — Tier 2-Codex only)

Authoritative event list: nlpm:conventions-codex §6.

Hooks (Antigravity / Gemini lineage — Tier 2-Antigravity only) — ADVISORY

Authoritative event list: nlpm:conventions-antigravity §5. All Antigravity-specific hook scoring is advisory-only (confidence:low) until the Antigravity 2.0 spec stabilizes — see analysis/multi-tool-design-2026-05.md decision #3.

plugin.json (Claude Code — .claude-plugin/plugin.json)

.codex-plugin/plugin.json (Codex CLI — Tier 2-Codex only)

Schema reference: nlpm:conventions-codex §3.

.agents/plugins/marketplace.json (Codex marketplace — Tier 2-Codex)

Schema reference: nlpm:conventions-codex §4. Schema is largely compatible with Claude's .claude-plugin/marketplace.json.

agents/openai.yaml (Codex skill sidecar — Tier 2-Codex)

Schema reference: nlpm:conventions-codex §2.

gemini-extension.json (Gemini/Antigravity — Tier 2-Antigravity) — ADVISORY

Schema reference: nlpm:conventions-antigravity §3. All Antigravity-specific manifest scoring is advisory-only until the post-2026-06-18 Antigravity spec stabilizes.

.gemini/commands/*.toml (Gemini slash commands — legacy/transitional, Tier 2-Antigravity)

Schema reference: nlpm:conventions-antigravity §4.

.mcp.json (Claude Code — .mcp.json at repo root)

.codex/config.toml (Codex configuration — Tier 2-Codex)

Schema reference: nlpm:conventions-codex §5.

.lsp.json (Claude Code LSP — Tier 2-Claude)

Schema details: nlpm:conventions-claude §12. Stable in 2026.

monitors/monitors.json (Claude Code monitors — Tier 2-Claude)

Schema details: nlpm:conventions-claude §13. Stable in 2026.

Settings Files (.claude/settings.json, .claude/settings.local.json)

CLAUDE.md

Memory Files

Applies to .md files located in ~/.claude/projects/*/memory/ directories.

Agent Workflow Programs (project-root program.md-style files)

A new artifact type recognized 2026-05-28 (see nlpm:conventions §2 and auditor/exemplars/karpathy-autoresearch.md): a project-root Markdown file driving an autonomous agent loop, hybrid between a memory file (AGENTS.md-shaped context) and a slash command (numbered workflow with output format + error paths).

No type-specific penalty rows. This artifact type is scored as the UNION of:

• Command rules R14–R17: numbered steps for multi-phase work, empty-input handling, output format, error paths.
• Memory file rules R33–R39: build/run commands, architecture overview, no stale refs, instructive-not-descriptive.
• Universal R01 (vague quantifiers) and R03 (positive framing).

Type-specific penalty rows are deferred until N ≥ 3 examples surface — the existing rules cover the artifact adequately as a hybrid, and inventing rows from N = 1 risks over-fitting (calibrated per the same discipline applied to multi-tool discovery deferrals).

Patterns this artifact type rewards (loaded on demand from nlpm:patterns):

• P10 (numeric anchoring of subjective principles)
• P11 (paired CAN/CANNOT contract)
• P12 (autonomy instruction + rationale + fallback ladder)
• P13 (vivid closing use-case)

All Artifact Types: Vague Quantifiers

Mention-versus-use exclusion (added 2026-08-01, origin: xiaolai/cc-suite audit-family false positives; design reviewed via Codex consultation): do not count a vague term when it is presented as a literal token AND the containing clause explicitly instructs the reader or a tool to detect, flag, reject, replace, avoid, or report that term — audit tooling must be able to name the words it hunts (e.g. Flag uses of `some`, `several`, `various` without concrete criteria is R01's own job description, not a violation). Backtick or quotation formatting alone does NOT qualify: a term that still modifies an action, criterion, or requirement is counted even when backticked — handle errors `properly` remains a violation.

All Artifact Types: Vocabulary Drift (R51 — opt-in, disabled by default)

Applied only when R51: { enabled: true, vocabulary_skill: <path> } appears in .claude/nlpm.local.md. Without the opt-in, R51 contributes zero penalty regardless of artifact content. The configured vocabulary_skill must contain a registry.yaml listing canonical and deprecated terms; without it, R51 emits an advisory and contributes zero penalty.

Why opt-in: vocabulary discipline is high-leverage for projects with accumulated drift but premature for projects still discovering their domain. Each project decides when it has enough literary warrant (P6) to lock terms in. See analysis/vocabulary-design-principles.md for the six principles R51 operationalizes.

Registry-declaration exclusion (added 2026-08-01, origin: xiaolai/cc-suite vocabulary-skill self-reference; design reviewed via Codex consultation): the file that declares a deprecation must name the deprecated term to do so. Within the configured vocabulary_skill path, do not count a deprecated term where it occurs in the declaration that registers it or maps it to its replacement — registry.yaml deprecated: lists and the SKILL.md deprecation tables. The exclusion covers ONLY those declaration term fields: deprecated terms in surrounding prose inside the vocabulary_skill path are counted, and the path is not categorically exempt. The same mention-versus-use principle as R01's exclusion above, applied to R51.

Cross-Artifact (--plugin flag)

Applied when linting an entire plugin rather than individual files.

Score Bands

Default pass threshold: 70. Configurable in .claude/nlpm.local.md.

Calibration Examples

Four worked examples — Excellent Agent (95), Rewrite Agent (41), Excellent Rule (92), Weak Rule (40) — live in references/calibration-examples.md. Load that file on demand when scoring a borderline case (around band boundaries: 88-92, 68-72, 58-62) and you need an anchored reference.

The examples are not needed for routine scoring — the penalty tables above are self-contained. They were extracted from this file 2026-05-28 to keep the rubric under R05's 500-line body budget while preserving the calibration material verbatim.

Scope Note

This skill covers the NLPM scoring formula, penalty tables, score bands, and calibration examples. It does NOT cover:

• Artifact schemas and valid field values → see nlpm:conventions (universal) and the tool-specific overlays nlpm:conventions-claude, nlpm:conventions-codex, nlpm:conventions-antigravity (the latter three created in PR-B; see analysis/multi-tool-design-2026-05.md)
• Patterns and anti-patterns catalog → see nlpm:patterns
• How to run the score command → see commands/score.md

Multi-tool scoring (PR-B landed 2026-05-25)

nlpm now scores artifacts across three tool ecosystems — Claude Code, Codex CLI, and Antigravity (which absorbs Gemini CLI on 2026-06-18). The tier classification in agents/scorer.md separates open-spec (Tier 1), Tier 1.5 open-spec corpora, and per-tool Tier 2 overlays (2-Claude / 2-Codex / 2-Antigravity).

• Hooks are scored per tool (Claude / Codex / Antigravity tables above). The three tools' event vocabularies are not 1:1 mappable; no universal translation layer. See analysis/multi-tool-design-2026-05.md decision #4.
• Codex-specific artifacts scored: .codex-plugin/plugin.json, .agents/plugins/marketplace.json, .codex/config.toml, agents/openai.yaml sidecars.
• Antigravity-specific artifacts scored (advisory-only until spec stabilizes): gemini-extension.json, .gemini/commands/*.toml, Antigravity hook events.
• Claude-specific additions in 2026: .lsp.json, monitors/monitors.json (validate JSON-parse only until detailed schemas land). New SKILL.md fields documented in nlpm:conventions-claude.

Known False Positive Patterns

The following findings have historically been reported by the scorer despite having no backing in this rubric. They MUST NOT be penalized:

When in doubt: if a finding cannot be cited to a specific row in the penalty tables above, drop it.