Validation

The Validation dashboard is the trust layer: it checks the model against the real CFP Selection Committee on completed seasons, and against the current season's final rankings when they exist.

Three trust dimensions

The dashboard reports the same three dimensions the harness measures, using the labels you see in the app:

• Committee alignment: how closely the composite reproduces the committee's top-12 ordering (rank correlation plus overlap), per season.
• Era-correct field selection: each season judged against the playoff format that actually applied that year (4-team through 2023, 12-team from 2024), with the teams the model added or dropped and a first-team-out comparison.
• Predictive signal: how the composite's game-level signal scored completed games next to simpler baselines (Elo, SRS, home field). Retrospective scoring, never a live forecast or win probability.

Overlap measures how closely the model aligns with the committee's published selections, not whether those selections were objectively correct.

Model vs Committee

When final committee rankings exist for the active run's season, the page opens with a per-run Model vs Committee panel: field agreement, seed matches, and a rank-by-rank table. See Model vs committee for how to read it.

What validation does not prove

Validation does not prove that Selection Room replicates committee deliberation, insider criteria, or future outcomes. Field checks use actual participants under era-correct rules; committee top-12 overlap is a separate metric with its own limits. Sample and Live CFBD runs are not directly comparable.

Hosted vs local artifacts

On the hosted site, validation data comes from the seeded catalog; nothing to refresh. Running locally, validation is a single repo-level artifact (data/output/api/validation.json):

1. Run the validation harness: make validate (or ./bin/sroom validate --years 2014:2024).
2. Confirm data/output/api/validation.json regenerated for the target seasons.
3. Reload the Validation page. Offline development falls back to the seeded fixture (pnpm seed-fixtures copies it to web/lib/fixtures/validation.json).

If the artifact is missing, the dashboard shows an empty or partial state rather than inferred numbers.