Model Validation

The honesty layer: how well the model reproduces the committee's rankings, picks the right field under each era's rules, and scores completed games. Measured against history, not asserted.

This run vs the committee

The current run's projection against the committee's published final rankings for 2025.

Model vs Committee

2025

This run's projection next to the final 2025 cfp committee rankings. The model is transparent and rules-based, so where it lands on a different team, you can trace exactly why.

Field agreement

11 of 12

Teams in both the model's projected field and the committee's actual field.

First team out

BYU

Model's first team out. Committee: Notre Dame.

Seeds matched

7 of 12

Field teams the model seeded exactly where the committee did.

Reference

2025

Final 2025 CFP committee rankings

Where they disagree

The model takes

The committee selected

Click a team to open its resume and see the components behind the model's call.

Cmte	Model	Shift	Field
1	1	=	In both fields
2	2	=	In both fields
3	3	=	In both fields
4	4	=	In both fields
5	5	=	In both fields
6	9	3	In both fields
7	10	3	In both fields
8	7	1	In both fields
9	6	3	In both fields
10	12	2	Committee only
11	8	3	Model only
12	11	1	Out in both
13	15	2	Out in both
14	14	=	Out in both
15	17	2	Out in both
16	13	3	Out in both
17	34	17	Out in both
18	16	2	Out in both
19	27	8	Out in both
20	20	=	In both fields
21	47	26	Out in both
22	26	4	Out in both
23	29	6	Out in both
24	22	2	In both fields
25	19	6	Out in both
NR	18	—	Out in both
NR	21	—	Out in both
NR	23	—	Out in both
NR	24	—	Out in both
NR	25	—	Out in both

Committee ranks come from the final 2025 cfp committee rankings, checked into the repo from the official release. "NR" means a team was outside that side's published top 25. Shift is committee rank minus model rank; an up arrow means the model ranks the team higher than the committee did.

Historical validation

The same comparison run across every completed season with committee data.

How the model's output compares to the real CFP Selection Committee across 2014–2024 (11 seasons). This is a retrospective accuracy check on finished seasons. The model is transparent and rules-based, and it will disagree with the committee. Those disagreements are the point.

Validation scope

Retrospective
Completed seasons only
Era-correct rules
Outlier seasons labeled (2022)

Seasons in this artifact: 2014–2024

Each season is judged against the playoff format that actually applied that year, not today's rules. See Era-correct rules.

Committee Agreement

0.77

Moderate alignment with top-12 ordering

Mean top-12 overlap 78% across validated seasons (moderate).

Validated Field Accuracy

100%

Exact field size in every validated season

For 2014–2024 (11 seasons in this artifact). Mean field overlap 77%.

Predictive Signal

72%

Composite win-side accuracy on completed games

How the composite's game-level signal scored completed games, not a live forecast. Brier 0.178 (lower is better).

Committee alignment

How closely the model reproduces the committee's ranking, by season. Overlap counts how many of the committee's teams the model also placed in the same band.

2014Spearman (top-12) 0.83

In 2014, the model matched 9/12 of the committee's top 12 teams (rank correlation 0.83, close). TCU/Baylor co-champions excluded; committee controversy

Top-12 overlap9/12 (75%)

Bubble overlap0/3 (0%)

2015Spearman (top-12) 0.83

In 2015, the model matched 9/12 of the committee's top 12 teams (rank correlation 0.83, close).

Top-12 overlap9/12 (75%)

Bubble overlap1/3 (33%)

2016Spearman (top-12) 0.91

In 2016, the model matched 10/12 of the committee's top 12 teams (rank correlation 0.91, strong).

Top-12 overlap10/12 (83%)

Bubble overlap1/3 (33%)

2017Spearman (top-12) 0.57

In 2017, the model matched 11/12 of the committee's top 12 teams (rank correlation 0.57, loose). Alabama non-champion selected over Ohio State

Top-12 overlap11/12 (92%)

Bubble overlap2/3 (67%)

2018Spearman (top-12) 0.86

In 2018, the model matched 9/12 of the committee's top 12 teams (rank correlation 0.86, close).

Top-12 overlap9/12 (75%)

Bubble overlap0/3 (0%)

2019Spearman (top-12) 0.45

In 2019, the model matched 11/12 of the committee's top 12 teams (rank correlation 0.45, loose).

Top-12 overlap11/12 (92%)

Bubble overlap0/3 (0%)

2020Spearman (top-12) 0.57

In 2020, the model matched 6/12 of the committee's top 12 teams (rank correlation 0.57, loose).

Top-12 overlap6/12 (50%)

Bubble overlap0/3 (0%)

2021Spearman (top-12) 0.97

In 2021, the model matched 9/12 of the committee's top 12 teams (rank correlation 0.97, strong).

Top-12 overlap9/12 (75%)

Bubble overlap0/3 (0%)

2022Committee outlierSpearman (top-12) 0.78

In 2022, the model matched 11/12 of the committee's top 12 teams (rank correlation 0.78, moderate). Outlier year: low committee-formula agreement (TCU, Kansas State)

Top-12 overlap11/12 (92%)

Bubble overlap1/3 (33%)

2023Spearman (top-12) 0.86

In 2023, the model matched 10/12 of the committee's top 12 teams (rank correlation 0.86, close). Florida State undefeated ACC champion ranked #4

Top-12 overlap10/12 (83%)

Bubble overlap1/3 (33%)

2024Spearman (top-12) 0.83

In 2024, the model matched 10/12 of the committee's top 12 teams (rank correlation 0.83, close). First 12-team field; Clemson auto bid at rank #16; Alabama displaced

Top-12 overlap10/12 (83%)

Bubble overlap1/3 (33%)

Era-correct field selection

Each season is judged against the playoff format that actually applied that year (4-team, then 12-team), not today's rules.

2014Top 4Correct field size

In 2014, the model produced the correct field size under Top 4 rules (4/4 field overlap).

Field overlap4/4 (100%)

2015Top 4Correct field size

In 2015, the model produced the correct field size under Top 4 rules (3/4 field overlap); model added Stanford; model dropped Oklahoma.

Field overlap3/4 (75%)

Model added:

Stanford

Model dropped:

Oklahoma

2016Top 4Correct field size

In 2016, the model produced the correct field size under Top 4 rules (3/4 field overlap); model added Michigan; model dropped Clemson.

Field overlap3/4 (75%)

Model added:

Michigan

Model dropped:

Clemson

2017Top 4Correct field size

In 2017, the model produced the correct field size under Top 4 rules (2/4 field overlap); model added Auburn, UCF; model dropped Alabama, Oklahoma.

Field overlap2/4 (50%)

Model added:

Auburn

UCF

Model dropped:

Alabama

Oklahoma

2018Top 4Correct field size

In 2018, the model produced the correct field size under Top 4 rules (3/4 field overlap); model added Georgia; model dropped Oklahoma.

Field overlap3/4 (75%)

Model added:

Georgia

Model dropped:

Oklahoma

2019Top 4Correct field size

In 2019, the model produced the correct field size under Top 4 rules (3/4 field overlap); model added Penn State; model dropped Oklahoma.

Field overlap3/4 (75%)

Model added:

Penn State

Model dropped:

Oklahoma

2020Top 4Correct field size

In 2020, the model produced the correct field size under Top 4 rules (1/4 field overlap); model added BYU, Cincinnati, Coastal Carolina; model dropped Clemson, Notre Dame, Ohio State.

Field overlap1/4 (25%)

Model added:

BYU

Cincinnati

Coastal Carolina

Model dropped:

Clemson

Notre Dame

Ohio State

2021Top 4Correct field size

In 2021, the model produced the correct field size under Top 4 rules (4/4 field overlap).

Field overlap4/4 (100%)

2022Top 4Committee outlierCorrect field size

In 2022, the model produced the correct field size under Top 4 rules (4/4 field overlap).

Field overlap4/4 (100%)

2023Top 4Correct field size

In 2023, the model produced the correct field size under Top 4 rules (4/4 field overlap).

Field overlap4/4 (100%)

20245+7 champion-byesCorrect field size

In 2024, the model produced the correct field size under 5+7 champion-byes rules (11/12 field overlap); model added Alabama; model dropped Tennessee.

Field overlap11/12 (92%)

Model added:

Alabama

Model dropped:

Tennessee

First team out:

Alabama·

Miami

Seeds within one55%

Predictive Signal

How well the composite's game-level signal scored completed games, next to simpler baselines. Retrospective scoring on finished games, not a live forecast.

Higher Accuracy is better; lower Brier is better.

2014Composite win accuracy 72%

In 2014, the Composite beats listed baselines on both win-side accuracy and Brier score.

Composite

72% · Brier 0.181

ELO

56% · Brier 0.225

SRS

56% · Brier 0.296

Home field

56% · Brier 0.249

2015Composite win accuracy 76%

In 2015, the Composite beats listed baselines on both win-side accuracy and Brier score.

Composite

76% · Brier 0.169

ELO

57% · Brier 0.217

SRS

57% · Brier 0.300

Home field

56% · Brier 0.249

2016Composite win accuracy 71%

In 2016, the Composite beats listed baselines on both win-side accuracy and Brier score.

Composite

71% · Brier 0.183

ELO

58% · Brier 0.220

SRS

58% · Brier 0.293

Home field

58% · Brier 0.245

2017Composite win accuracy 73%

In 2017, the Composite beats listed baselines on both win-side accuracy and Brier score.

Composite

73% · Brier 0.176

ELO

58% · Brier 0.220

SRS

58% · Brier 0.296

Home field

58% · Brier 0.245

2018Composite win accuracy 72%

In 2018, the Composite beats listed baselines on both win-side accuracy and Brier score.

Composite

72% · Brier 0.177

ELO

59% · Brier 0.215

SRS

59% · Brier 0.288

Home field

59% · Brier 0.242

2019Composite win accuracy 73%

In 2019, the Composite beats listed baselines on both win-side accuracy and Brier score.

Composite

73% · Brier 0.176

ELO

60% · Brier 0.216

SRS

60% · Brier 0.289

Home field

60% · Brier 0.240

2020Composite win accuracy 72%

In 2020, the Composite beats listed baselines on both win-side accuracy and Brier score.

Composite

72% · Brier 0.173

ELO

57% · Brier 0.214

SRS

57% · Brier 0.300

Home field

56% · Brier 0.249

2021Composite win accuracy 72%

In 2021, the Composite beats listed baselines on both win-side accuracy and Brier score.

Composite

72% · Brier 0.178

ELO

58% · Brier 0.224

SRS

58% · Brier 0.297

Home field

59% · Brier 0.245

2022Composite win accuracy 69%

In 2022, the Composite beats listed baselines on both win-side accuracy and Brier score.

Composite

69% · Brier 0.190

ELO

58% · Brier 0.220

SRS

58% · Brier 0.277

Home field

58% · Brier 0.244

2023Composite win accuracy 74%

In 2023, the Composite beats listed baselines on both win-side accuracy and Brier score.

Composite

74% · Brier 0.176

ELO

58% · Brier 0.222

SRS

58% · Brier 0.292

Home field

58% · Brier 0.245

2024Composite win accuracy 70%

In 2024, the Composite beats listed baselines on both win-side accuracy and Brier score.

Composite

70% · Brier 0.184

ELO

59% · Brier 0.219

SRS

59% · Brier 0.272

Home field

59% · Brier 0.242

Validation artifact

Generated: Jul 3, 2026, 6:39 PM
Target: All
Seasons: 2014–2024
Outlier seasons: 2022

Refresh with sroom validate.