A reference implementation of procurement-oriented demand forecasting for many-SKU, intermittent retail demand. The forecast is not the deliverable: it is an input to an ordering decision made months ahead of demand. So the pipeline is built end to end around the question a buyer actually asks — how much do I order now to cover the lead window at a chosen service level? — and the accuracy metric that matters here is whether the resulting order over- or under-buys, not the point error on any single month. It is demonstrated on the public M5 (Walmart) dataset, California subset, aggregated to monthly.
The order for a lead window of L months is the quantile of the summed demand
over those months, q(Σ demandₘ) — computed by Monte-Carlo convolution of the
monthly distributions with an inter-month correlation ρ. It is not the sum
of the monthly quantiles, Σ q(demandₘ). Those two are equal only under perfect
positive month-to-month correlation (ρ → 1); otherwise the sum-of-quantiles is
systematically biased.
The bias is worst on intermittent SKUs, where it collapses the order to zero.
Take a SKU that sells in bursts: in a typical month it sells 0 units, and only
occasionally a handful. Its monthly distribution is roughly median = 0,
p80 = 0, p95 = 5. Order it for a L = 6 month lead at an 80% service level:
| policy | computation | order |
|---|---|---|
| sum-of-quantiles (legacy) | 6 × p80(month) = 6 × 0 |
0 |
| quantile-of-sum (this repo) | p80( Σ of 6 monthly draws ) |
> 0 |
The sum-of-quantiles buys nothing for a SKU that clearly sells — the 80th percentile of a single month is 0, and six zeros are still zero — so every burst over the lead window is a guaranteed stockout. The quantile-of-sum reconstructs each month's distribution, convolves the six months, and reads the 80th percentile of the total, which is strictly positive (over six months you expect to see bursts), producing a real buffer.
The same correction cuts over-ordering on smooth SKUs. Summing each month's upper
quantile assumes every month peaks together; with ρ < 1 the peaks diversify, so
q(Σ) sits below Σ q — between the sum of the medians and the sum of the
upper quantiles — freeing capital that the legacy policy would freeze in
inventory. See src/quantile_sum.py and
tests/test_quantile_sum.py.
Two more procurement-specific pieces sit on top of this:
- Explicit per-tier service levels (
procurement.service_level), a deliberate policy instead of quantiles that fall out of two-sided prediction intervals by accident. High-revenue A/B items are trimmed where they were over-buffered; the C tail keeps its buffer because a stockout there risks losing a client. - Order-level backtesting (
src/eval_recommendation.py): replay the order you would have placed at a past as-of date and compare it against the demand that actually materialised over the lead window, per tier (fill rate, stockout rate, over-order ratio). Point metrics such as RMSSE or WSPL on monthly values never touchorder_recommendation.xlsx, so they cannot tell you whether the ordering policy over- or under-buys — this is the metric that does.
Each stage maps to a module under src/ (orchestrated by pipeline.py):
prepare_m5.py M5 (California) download -> data/processed/cleaned.parquet
+ product_dim.parquet
|
v
data.load_cleaned load the cleaned transaction table
|
v
aggregate.aggregate_to_monthly daily -> monthly (item_id x channel) panel
. apply_filters drop dead SKUs; drop an incomplete trailing month
. build_static_features
. build_client_shares
|
v
tiers.compute_tiers ABC by 12-month revenue + NEW cold-start bucket
|
v
classify.classify_panel Syntetos-Boylan (ADI x CV^2):
smooth / intermittent / erratic / lumpy
|
v
per-class model layer
. forecast_stats.StatsRunner 10 classical models + conformal PI (CV + final)
. forecast_ml.MLRunner LightGBM cross-learning (lags/rolling/date)
+ direct quantile LightGBM
. forecast_neural.NeuralRunner NHITS + PatchTST (optional: --neural, GPU)
|
v
ensemble.compute_weights/apply ensemble winner per (channel x demand class)
by cross-validated WAPE
|
v
calibrate_pi.compute_scale_factors per-class/series PI calibration, asymmetric
/apply_scale lower and upper half-widths
|
v
reconcile.forecast_channels channel-level TSB anchor -> top-down rescale;
+ top_down_rescale lumpy-giant SKUs pass through un-rescaled
|
v
tier overrides
. tier_b_aggregate.tier_b_forecast B: brand x channel aggregate + historical share
. tier_c_rule.tier_c_forecast C: rule-based safety stock (no ML)
. cold_start.cold_start_forecast NEW: group-average forecast, sigma-capped PI
|
v
disaggregate.disaggregate_to_clients optional split of the forecast across clients
|
v
recommend.build_recommendation quantile-of-sum order over the lead window
-> outputs/forecasts/order_recommendation.xlsx
|
v
evaluate / eval_recommendation RMSSE, WSPL, PI coverage + order-level backtest
report.render_report -> outputs/reports/eval_report.html
pip install -r requirements.txt # core CPU stack; neural extras are optional (see the file)
python src/prepare_m5.py # download M5 (California) -> data/processed/*.parquet
python pipeline.py --horizon 6 # forecast + reconcile + tiering + order recommendationOutputs land under outputs/:
| path | contents |
|---|---|
outputs/forecasts/order_recommendation.xlsx |
the order plan — the primary output |
outputs/forecasts/forecast_monthly.parquet (+ .csv) |
per-(SKU x channel) monthly forecast with PIs |
outputs/forecasts/forecast_monthly_with_client.* |
forecast disaggregated to client |
outputs/forecasts/forecast_monthly_tiered.parquet |
tier-aware final forecast |
outputs/reports/eval_report.html |
HTML evaluation report |
outputs/reports/ensemble_summary.json, ensemble_cv.* |
metrics + backtest frame |
Leak-free as-of backtesting. python pipeline.py --asof 2015-06-01 --horizon 6
truncates everything — the panel, the static features and the client shares — to
the as-of date before any forecasting, so no stage can peek past the origin. Other
flags: --skip-ml, --skip-reconcile, --neural, --ensemble-mode.
Setup: M5 California (4 stores), aggregated to monthly. 3,047 series survive the
activity filters (of 3,049 items); history 2011-02 to 2016-05 — the trailing
2016-06 stub (19 of 30 days) is dropped automatically by the incomplete-month
guard. channel = department (7), client = store (4). Full run: single CPU,
n_jobs: 6, ~25 min.
Point accuracy — rolling 2-window CV, all models scored on identical rows (162,347 series-months):
| forecast | WAPE |
|---|---|
| per-(channel × class) ensemble (this pipeline) | 0.306 |
| SeasonalNaive | 0.362 |
| HistoricAverage | 0.390 |
Honest held-out (train clipped at 2015-11-01, scored on the following months, so model selection and PI calibration never see the eval window):
| metric | held-out | in-sample CV |
|---|---|---|
| RMSSE (median, <1 = beats naive) | 0.561 | 0.541 |
| WSPL | 0.139 | 0.148 |
| PI coverage @80 | 0.775 | 0.842 |
| PI coverage @95 | 0.916 | 0.936 |
The held-out/in-sample gap is the honest price of selection + calibration optimism. Coverage at the 80% level lands slightly under nominal on held-out data (0.775 vs 0.80) — the asymmetric per-class calibration is fit on CV residuals and inherits their optimism; the 95% level holds within 3.5pp.
Order-level backtest — the metric this repo exists for. The order is
replayed at two as-of origins (2015-05-31 and 2015-11-30, lead = 6 months) and
compared with the demand that actually materialised over each lead window
(pooled: 6,096 SKU-origin pairs, 6.35M units of realized demand). Reproduce
with scripts/order_eval_m5.py:
| order policy | unit fill rate | SKU stockout rate | over-order / demand | units ordered |
|---|---|---|---|---|
| median only (no safety stock) | 86.7% | 54.6% | 10.6% | 6.18M |
| sum of quantiles (legacy) | 94.4% | 27.6% | 52.2% | 9.32M |
| quantile of sum (ρ=0.5, tiered service levels) | 93.5% | 31.5% | 38.9% | 8.41M |
The quantile-of-sum policy keeps essentially the same service (−0.9pp unit fill) while releasing a quarter of the legacy policy's excess: over-ordering drops from 52% to 39% of demand, ~0.9M units (−10% of the total order). The effect concentrates exactly where the theory says it should — Tier A (smooth, high-volume): fill 98.6% → 98.0% while over-order falls 66% → 49%.
Honest caveats: ρ = 0.5 is the pipeline's default, not tuned on M5; and the
B-tier SKU stockout rate is high under every policy here (~54-57%) — the
brand × channel aggregate that Tier B rides on is coarse for M5, where "brand"
degenerates to the category level. Both are documented knobs, not hidden
constants.
All knobs live in config.yaml. The ones that shape the ordering
policy and runtime:
| key | default | meaning |
|---|---|---|
procurement.order_method |
quantile_of_sum |
ordering policy; sum_of_quantiles is the legacy biased baseline |
procurement.service_level |
A 0.85, B 0.85, C 0.80, NEW 0.80 |
explicit target service level per tier |
procurement.month_correlation |
0.5 |
inter-month correlation ρ for the quantile-of-sum convolution |
procurement.lead_time_months |
6 |
months from order placement to warehouse receipt |
compute.n_jobs |
6 |
parallelism cap for StatsForecast / MLForecast / LightGBM (throttles CPU on shared machines; -1 uses all cores) |
classify.adi_threshold / cv2_threshold |
1.32 / 0.49 |
Syntetos-Boylan cutoffs |
forecast.levels |
[80, 95] |
prediction-interval levels |
python -m pytest tests/ -qThe unit tests run on tiny synthetic frames — no network and no heavy forecasting
stack required (only numpy, pandas, scipy, pytest). The real M5 download is
exercised by a single opt-in test:
RUN_M5_DOWNLOAD=1 python -m pytest tests/test_prepare_m5.py -qContinuous integration (.github/workflows/ci.yml)
installs only that minimal set, runs a ruff syntax gate, and executes the suite
on Python 3.11.
The M5 dataset is © Walmart, released for the M5 forecasting competition
(Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2022). The M5 competition:
Background, organization, and implementation. International Journal of
Forecasting, 38(4), 1325–1336). This repository ships no data:
src/prepare_m5.py downloads it at runtime through the
datasetsforecast package (a public Nixtla mirror). Refer to the M5 competition
terms for redistribution and usage conditions.
Released under the MIT License — see LICENSE.