# Walk-Forward Edge Bucket Backtest, 2017–2025

**Date:** 2026-08-05
**Scripts:** `analysis/walk_forward.py` (model_pack ML path), `analysis/walk_forward_production.py` (production EPA-OLS path)
**Data:** 3,848 lined FBS games, 8 seasons (2017–2019, 2021–2025; 2020 absent — COVID)

## Why this test exists

The 2025 anomaly hunt found the model's 14+ edge bucket covering at 77.8% (n=9) — the only signal in the season. One season of 9 games is a whisper, not a track record. This backtest asks whether the signal survives out-of-sample, using the same walk-forward discipline the production engine already uses: for each season T, everything is fitted only on data from seasons/weeks before T.

## Two engines, two verdicts

The site ships **two** projection engines, and they disagree:

| Engine | 14+ bucket pooled 2017–2025 | Verdict |
|---|---|---|
| **Production EPA-OLS** (weekly opponent-adjusted EPA, expanding-window OLS refit before each week) | **56.1% cover, n=114** | **Survives out-of-sample** |
| Model-pack ML (season-final adjusted stats, stacked ensemble) | 45.8% cover, n=170 | Dead — noise |

The 2025 anomaly-hunt signal came from the production path, and it reproduces exactly (77.8%, n=9). The ML path's 14+ bucket was never the signal — it was a different engine with a different feature family.

## The production path, season by season (14+ bucket)

| Season | n | Cover | |
|---|---|---|---|
| 2017 | 12 | 50.0% | |
| 2018 | 15 | 53.3% | |
| 2019 | 13 | 53.8% | |
| 2021 | 22 | 63.6% | |
| 2022 | 21 | 52.4% | |
| 2023 | 12 | 50.0% | |
| 2024 | 10 | 50.0% | |
| 2025 | 9 | 77.8% | (the original anomaly-hunt signal) |
| **Pooled** | **114** | **56.1%** | p=0.223 |

Never below 50% in any season. Directionally consistent, not statistically significant.

## Finer buckets (pooled)

| Bucket | n | Cover | binom p |
|---|---|---|---|
| 0–3 | 1,561 | 52.0% | 0.118 |
| 3–7 | 1,413 | 51.0% | 0.485 |
| 7–14 | 760 | 48.2% | 0.343 |
| 14–17 | 66 | **57.6%** | 0.268 |
| 17–21 | 36 | 47.2% | 0.868 |
| 21+ | 12 | **75.0%** | 0.146 |

The signal concentrates at the extremes: 14–17 and 21+ both beat break-even (52.4% at −110), while 17–21 is a hole. The 21+ bucket is 9–3 but n=12 — a whisper.

## Fav vs dog (pooled, production path)

| Bucket | Fav picks | Dog picks |
|---|---|---|
| 0–3 | 52.6% (n=835) | 51.4% (n=726) |
| 3–7 | 53.4% (n=839) | 47.3% (n=574) |
| 7–14 | 47.5% (n=537) | 50.0% (n=223) |
| 14–17 | 54.9% (n=51) | 66.7% (n=15) |
| 21+ | 72.7% (n=11) | 100% (n=1) |

The 3–7 edge is a favorites play (53.4% vs 47.3% dogs). The extreme buckets are too small to split.

## What this means

1. **The 14+ signal is real, not noise** — it survives 8 seasons out-of-sample, never dipping below 50%, and the 2025 77.8% reproduces exactly. The model's high-conviction spots (edge ≥ 14) have covered at 56.1% pooled.
2. **It is not yet a bet.** 56.1% on n=114 is p=0.223 — consistent with a coin flip that got lucky. At $20 stakes, 114 games at 56.1% vs 52.4% break-even is +$84; the edge needs to hold another season or two to mean anything.
3. **The ML path is a different animal.** Its 14+ bucket (45.8%) is a loser. Any future "promote logistic regression" experiment must be judged on the production EPA-OLS path, not the model-pack features.
4. **The 0–3 bucket is quietly interesting.** 52.0% on n=1,561 (p=0.118) — the model's smallest edges are its most consistent, and the favorites half (52.6%) is the driver. Not a bet, but the least-noisy number in the table.

## Honest caveats

- The production path's weekly EPA is rebuilt from CFBD plays with the repo's own ridge method (r≥0.999 vs the pack's values on 2024). The 2017–2024 EPA is a reconstruction, not the pack's original values.
- Spreads for 2017–2023 come from the pack's training_data.csv (closing lines); 2024–2025 from per-season files. Same source family.
- 2020 is absent (COVID season, no lines in the pack).
- The 14+ bucket is 114 games over 8 seasons — about 14 per season. Even a real edge moves slowly at that volume.

## Files

- `exports/walk_forward_2017_2025.json` — ML-path results
- `exports/walk_forward_production_2017_2025.json` — production-path results
- `analysis/walk_forward.py`, `analysis/walk_forward_production.py` — the backtests
- `analysis/fetch_plays_history.py` — play fetcher for 2016–2024 (data in `plays_*/`, gitignored)