The Lab

Methodology — weekly win-probability forecast

cfb-model-lab produces a weekly college football win-probability forecast. This page is the public description of how the forecast is built, what it does and does not claim, and how it is graded. It is a forecast, not a betting recommendation.

What the model predicts

One number per game: the probability the home team wins, in [0, 1]. The projected winner is whichever side that probability favors. The confidence tier (high / medium / low) buckets distance from a coin flip and is informational only.

Model

StackedEnsemble: three base models — logistic regression, XGBoost, random forest — produce 5-fold out-of-fold predictions on the training data, and a logistic meta-model combines those into the final probability. Trained on historical FBS games from the Model Training Pack (model_pack/training_data.csv).

A simple baseline is kept for comparison: the market's closing spread. The dashboard reports the model's accuracy next to the market's every week, and specifically how the model does on games where the two disagree.

Features

Every feature comes in a home/away pair. Sources:

• Opponent-adjusted EPA — twelve families (overall, rushing, passing, success rate, standard/passing down success, explosiveness ×3, line yards, second-level yards, open-field yards), each on offense and defense. Fit by ridge regression over play-by-play data, corrected for schedule strength and home field.
• Talent composite — CFBD's recruiting-derived team talent score.
• Pregame Elo — CFBD's per-team rating published before kickoff.
• Conference and week — categorical, one-hot encoded against the exact category set seen in training.

Leakage rules (the important part)

A Week W forecast may use only information available before that week's kickoff. Concretely:

• Play-by-play for the opponent-adjusted ratings comes only from weeks before W (AsOfWeek enforces this; postseason games use the completed regular season, never the training pack's week=1 encoding).
• Elo is the pregame Elo from /games, not a rating that already reflects Week W's result.
• Season-level stats (/stats/season/advanced, talent) are used as published; when used as a raw stand-in for adjusted EPA the slate is explicitly announced as epa_basis="raw_ppa", because the approximation is not the same number the model was trained on.
• Demo validation is a random stratified 80/20 split of the supplied 2016-2024 training pack. It mixes seasons and is not time-aware. Its scores may be optimistic for forecasting later seasons.
• Separate research in analysis/walk_forward.py trains on seasons before each held-out season, with category encoding learned from training rows. analysis/model_comparison.py trains on 2016-2024 and predicts 2025. The ensemble's internal stacking folds remain shuffled in both cases.
• The 2024-2025 results informed feature development, so they are retrospective evidence, not untouched confirmation data. Historical feature availability must also be checked before treating a backtest as a prospective forecast.
• Prospective grading requires predictions frozen before kickoff and graded after results arrive. A retrospective rerun is not prospective evidence.

If a backtest in this repo ever looks excellent, the first hypothesis is that it used future information. That has happened here before; the sign-convention sanity check and the as-of-week cutoff exist because of it.

Missing-data rules

A game is only projected when every feature resolves. Games that cannot be resolved are reported by named reason (team_absent_from_stats, team_stats_null, elo_missing) — never silently imputed, never filled with zero. Export is refused outright if:

• fewer than 3 games resolve (the sign-convention sanity check cannot run), or
• coverage falls below 50% of the FBS slate (a failure that large is systemic, not a few FCS opponents).

When too little play-by-play exists to fit the adjusted ratings (early weeks), the slate falls back to raw season stats and says so on the export.

Live odds checks and rejected data

Live pick generation and settlement validate raw /lines responses against the captured /games identities before using any odds. Team IDs, provider, event ID, UTC kickoff, and neutral-site status must agree. Display names cannot substitute for missing team IDs. When /lines omits neutralSite, the guard uses the locked /games value; an explicit conflict is rejected.

Rejected responses are saved under data/external/odds_quarantine/ with their original bytes, SHA-256 hash, and a machine-readable rejection record. The guard does not change the picks ledger or odds state. Retrying the same rejection reuses its saved evidence, including after an interrupted write. tests/test_betting_controls.py exercises both sides of that write boundary and verifies that recovery preserves the exact rejected bytes.

Versioned predictions

Each week's forecast is frozen to exports/live_<season>_week<week>_predictions.json before kickoff, via uv run python -m cfb_model_lab.export live <season> <week>. One file per week, append-only, never edited after the fact. That file — not a rerun — is what gets graded.

Evaluation

After the week completes, uv run python -m cfb_model_lab.grade <season> <week> resolves the frozen predictions against final scores and closing lines and appends one row to exports/performance.csv plus a per-week exports/calibration_<season>_week<week>.json. Regrading is idempotent — it replaces the week's row rather than duplicating it.

Metrics reported:

• Accuracy — how often the projected winner won.
• Brier score — mean squared error of the probabilities. Rewards honest probabilities, not just right picks.
• Calibration — per-bucket predicted vs. actual win rate for the picked side. A calibrated model's 0.75 picks should win ~75% of the time.
• Market comparison — the closing spread's accuracy on the same games, the model's agreement rate with it, and how the model does specifically on games where the two disagree.

uv run python -m cfb_model_lab.dashboard builds exports/dashboard.html from those files: weekly accuracy vs. the market, aggregate calibration, and the per-week detail table. The dashboard is a view over the CSV/JSON — the files are the source of truth.

Model-comparison report

exports/model_comparison_2025.json separates straight-up winner accuracy from ATS cover rate. Its ATS strategy backs each model's predicted winner at the recorded spread; market_ats backs the market favorite. Pushes are excluded from cover-rate denominators, and the market baseline also excludes pick'em lines. These are not the dashboard's separate model-edge ATS picks.

Before the 2026-09-08 correction, the report mislabeled straight-up model and market-favorite accuracy as ats and market_ats. Those historical values were not spread performance. The corrected report and regression tests in tests/test_model_comparison.py distinguish a winner failing to cover from an ATS win and exclude pushes from the denominator.

What this is not

• Not a betting recommendation. No stakes, no picks-of-the-week, no locks.
• Not a claim of market-beating edge. The market comparison is published every week precisely so underperformance is visible.
• Not finished. Feature additions go through the same leakage and grading gates; the rules above do not change after seeing a week's results.

Interpreting early-season EPA rankings

The /rankings board describes season-to-date opponent-adjusted per-play performance. It is not a forecast that its first-ranked team would beat every other team. The live calculation uses only completed prior regular-season FBS-vs-FBS plays; it does not bring forward preseason strength, Elo, talent, or prior-season ratings. Its implementation is AsOfWeek.rate → oppadj.fit.

For each eligible play, ridge regression fits an intercept plus offense and defense team effects and a home-field term. Published offense is the intercept plus the offense effect; published EPA allowed is the intercept plus the defense effect. Higher offense and lower EPA allowed are better. Net is offense minus EPA allowed, so the common intercept cancels. Regularization shrinks the team effects toward zero. The selected penalty is not a measure of ranking certainty.

The September 14, 2026 source audit reproduces all published values from the immutable Weeks 1–2 capture. Texas ranks 54th at net 0.014208; Mississippi State ranks first at 0.159125. Their raw offensive PPA is 0.303773 and 0.544401, respectively, while raw defensive PPA allowed is 0.146295 and -0.070138. The fitted values, after adjustment and shrinkage, are 0.178818 offense / 0.164611 allowed for Texas and 0.260419 / 0.101295 for Mississippi State. These ranks follow the stated formula; they do not demonstrate a sign error.

The reliability limitation is substantial: the observed FBS opponent graph has 38 disconnected components, and these two teams belong to different components. With two games per team, cross-component ordering depends on the common baseline and regularization assumptions rather than a chain of shared opponents. This is why the page describes early-season efficiency rather than predictive power. Changing that policy requires a separately tested model decision, not an unannounced adjustment of surprising ranks.

The derived [audit receipt](../exports/rankings_audit_20260914.json) records sample sizes, the fitted penalty/intercept, opponents, and component membership. tests/test_rankings_snapshot_audit.py verifies the installed source capture through derived CSV, database, and public rankings; it skips only when the operational capture is not installed. See [ADR-0005](adr/0005-live-slates-run-the-ridge-instead-of-standing-in-raw-stats.md) for the source and temporal policy.