authoring-model-cards

Authoring Model Cards

A model card is the honest spec sheet for a model: what it's for, how well it works, where it breaks, and who it might fail. For clinical models this is governance-critical — an undocumented de-id model is one nobody can sign off on. This skill fills a model card directly from OpenMed eval outputs so the numbers are reproducible, not aspirational.

When to use this skill

• You're publishing or updating an OpenMed model and need its card.
• You have eval artifacts (GateReport, fairness_report, error_report) and need to turn them into intended-use, metrics, and limitations sections.
• A clinical AI governance / model-risk review needs a transparency document.

Run the evals first (see evaluating-with-leakage-gates, benchmarking-clinical-ner, auditing-subgroup-fairness); this skill documents their results — it does not generate the numbers.

Card sections (Mitchell et al., + clinical extensions)

See references/model-card-sections.md for the full section-to-source map. The load-bearing sections for an OpenMed model:

• Model details — repo id, family, tier, format, params, milestone, license (Apache-2.0). Pull from the GateReport identity fields.
• Intended use — the clinical task and the deployment envelope.
• Out-of-scope / misuse — explicitly: not a medical device; not for autonomous clinical decisions; de-id is verified, not assumed.
• Metrics — entity-level P/R/F1 and, for de-id, residual leakage + per-label recall floors and the gate decision.
• Quantitative analysis (subgroups) — per-group leakage/recall from fairness_report, including which groups lack data.
• Limitations — error patterns from error_report; calibration assumptions.
• Caveats & disclaimer — the medical-device disclaimer.

Quick start — fill the card from eval outputs

from openmed.eval import (
    run_suite, ReleaseGate, fairness_report, error_report,
)

report = run_suite("eval/gold/test.json", suite="golden",
                   model_name="OpenMed/Privacy-PII-Detection", device="cpu",
                   metadata={"family": "PII", "tier": "base",
                             "policy": "hipaa_safe_harbor"})

gate = ReleaseGate(milestone="v1.6", policy="hipaa_safe_harbor").evaluate(report)
fair = fairness_report("OpenMed/Privacy-PII-Detection", "golden")
errs = error_report("OpenMed/Privacy-PII-Detection", "eval/gold/test.json")

card = {
    "model_details": {
        "repo_id": gate.repo_id, "family": gate.family, "tier": gate.tier,
        "format": gate.format, "license": "Apache-2.0",
    },
    "metrics": {
        "exact_span_f1": report.metrics["exact_span_f1"]["f1"],
        "residual_leakage_rate": gate.residual_leakage_rate,
        "critical_leakage_count": gate.critical_leakage_count,
        "per_label_recall": dict(gate.per_label_recall),
        "release_decision": gate.decision,            # RELEASABLE / QUARANTINED
    },
    "subgroup_analysis": fair.to_dict(),              # per-group leakage/recall
    "limitations": errs.to_dict()["confusion_matrix"],
}
# Render `card` into Markdown front matter + body (or the HF card template).

error_report and fairness_report carry no plaintext PHI (offsets + hashes), so their output is safe to paste into a public card.

Workflow

1. Gather artifacts. Gate report, fairness report, error report — all from a pinned model + synthetic eval set.
2. Fill model details from the GateReport identity fields so the card, models.jsonl, and the README cannot drift (the gate's manifest_coherence and model_card checks enforce this).
3. Write intended use narrowly. Name the clinical task, language(s), and the deployment envelope. Over-broad intended-use is the most common card failure.
4. State out-of-scope and the disclaimer plainly (see template below).
5. Report metrics with their floors. For de-id, lead with leakage and the gate decision, not F1.
6. Report subgroups honestly, including the documentation gap: if race/ ethnicity isn't available, say so rather than implying parity.
7. List limitations from real errors, not boilerplate — cite the confusion matrix's worst cells.

Disclaimer block (paste & adapt)

This model assists clinical text processing and is not a medical device. It does not make autonomous clinical decisions. De-identification output must be independently verified before any data is shared; residual PHI risk is never zero. Validate on your own population before deployment.

Hand-off to / from OpenMed

• From evaluating-with-leakage-gates (GateReport), benchmarking-clinical-ner (error_report), and auditing-subgroup-fairness (fairness_report): these are the card's evidence.
• To building-with-openmed / models.jsonl: keep card front matter (license, task, languages) coherent with the manifest — the gate checks it.
• Pairs with gating-deid-leakage: cite the green gate as the card's release evidence.

Edge cases & gotchas

• Don't claim numbers you can't reproduce. Every metric in the card should trace to an eval artifact and a pinned eval-set hash.
• Intended use ≠ capability. Document the supported envelope; mark everything else out-of-scope.
• Subgroup silence is a finding. Omitting race because it wasn't collected is itself a limitation to state — don't let absence read as equity.
• Card/manifest drift fails the gate. License/task/language mismatches between the card and models.jsonl trip manifest_coherence.
• No raw PHI examples. Use the offset/hash examples from error_report; never paste real patient strings as "qualitative examples".
• Quantized variants need their own line. Report INT8/INT4 recall deltas (G4) per format; don't reuse the fp32 numbers.

Standards & references

• Mitchell et al., Model Cards for Model Reporting (FAT* 2019): https://arxiv.org/abs/1810.03993
• Hugging Face model card spec & template: https://huggingface.co/docs/hub/model-cards
• Sendak et al., Presenting machine learning model information to clinical end users (clinical "model facts" label): https://doi.org/10.1038/s41746-020-0253-3
• FDA, Clinical Decision Support Software guidance: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/clinical-decision-support-software
• Section-to-source map: references/model-card-sections.md.