ml-experiment-standards

ML Experiment Standards

Purpose: every ML job (quick prototypes included) is reproducible, leakage-free, and metric-justified. These are not optional polish; every skipped item typically returns as "the model collapsed in production" or "the result didn't replicate".

1. EDA comes first

Before any model, produce and show: distributions, missingness rates, outliers, target balance, salient correlations. Metric and loss choice depend on this information; a model recommendation without EDA is a guess.

2. Leakage audit

At every split decision, answer explicitly (and write the answer as a code comment): "Does the training set contain indirect information about any test sample?"

• Scalers/encoders/imputers are fit on train only; the clean path is sklearn.pipeline.Pipeline — CV then fits correctly by construction.
• Target-derived features (target encoding etc.) must be computed out-of-fold, and shown to be.

The spatial protocol in references/spatial-cv-protocol.md is the single canonical source for this repo — other skills link here; do not restate it.

3. Metric selection — justified

Never choose a metric by default; write a one-sentence rationale:

• Imbalanced classes → F1 / AUC-PR, not accuracy (accuracy rewards majority-class memorization).
• Segmentation → IoU/Dice (pixel accuracy is inflated by background).
• Regression → RMSE (sensitive to large errors) vs MAE (robust) vs R² (variance explained) — justify from the use case.
• Every point estimate gets uncertainty: bootstrap CI or mean ± std across CV folds. A single number hides whether a difference is signal or noise.

4. Reproducibility skeleton

Every training script follows this shape (script-first; no notebook magic):

"""Experiment: <name>. Goal and success criterion: <one sentence>."""
from dataclasses import dataclass, asdict
import json, random
import numpy as np

@dataclass
class Config:
    seed: int = 42
    lr: float = 1e-3
    batch_size: int = 32
    epochs: int = 100
    patience: int = 10  # early stopping

def set_seed(seed: int) -> None:
    random.seed(seed)
    np.random.seed(seed)
    # if torch: torch.manual_seed(seed); torch.cuda.manual_seed_all(seed)

def main(cfg: Config) -> None:
    set_seed(cfg.seed)
    ...  # data -> split -> pipeline -> train -> evaluate
    with open("runs/run_meta.json", "w", encoding="utf-8") as f:
        json.dump({"config": asdict(cfg), "metrics": metrics}, f, indent=2)

if __name__ == "__main__":
    main(Config())

• Config lives in a dataclass/YAML, never hardcoded — sweeps and run comparison depend on it.
• Pin library versions (pip freeze > requirements.txt).
• Use MLflow/W&B when available; the JSON log above is the minimum.

5. Deep learning extras

• Loss rationale: Dice/Dice+CE for imbalanced segmentation; write why. Focal only after comparison — not a free win.
• Augmentation rationale: state which transforms respect the physics of the problem (orientation-dependent tasks forbid some rotations; multispectral forbids naive color jitter).
• Overfitting control: early stopping with patience + a train/val curve in the report; no curve, no "the model is good".
• Capacity order: small model + simple baseline first (logistic regression, RF); a deep model that can't beat the baseline is a data problem, not an architecture problem.
• EO-specific chipping/inference details → geo-deep-learning.

6. System context (MLOps)

Position every model in its chain in one paragraph: data source → cleaning → features/versioning → training → evaluation → deployment (batch/real-time) → monitoring (data/model drift). Even for a prototype, note "what this step becomes in production".

7. Report format

## Experiment: <name>
- Data: n=<>, split: <strategy + rationale>
- Baseline: <model> → <metric ± CI>
- Model: <model> → <metric ± CI>
- Leakage audit: <what was checked>
- Next step: <single recommendation>

When reporting differences, respect statistical honesty: if the gap doesn't exceed the across-fold std, say "no clear difference" — no p-hacking, no selective reporting.

Execution contract

• Workflow: define prediction target and decision use; establish a baseline; audit leakage; create spatially valid splits; train reproducibly; quantify uncertainty; inspect errors and deployment fit.
• Decision rules: apply this skill only to predictive model experiments; use spatial statistics for inference, geostatistics for sampled-surface estimation, and descriptive analysis without forcing a model.
• Verification protocol: reproduce from a clean environment, compare against baseline across folds or seeds, inspect spatial residuals, verify split independence, and test the final decision threshold.
• Failure modes: invalidate uplift claims for leakage, post-split preprocessing, inappropriate metrics, non-independent test units, selective runs, or train-serving skew.
• Deliverables: experiment configuration, split and seed manifest, baseline and model metrics with uncertainty, leakage audit, error analysis, artifacts, and deployment caveats.
• Source freshness: consult the authoritative source registry before using version-sensitive split, metric, or reproducibility APIs.