Core Demand Prediction — Unite Hackathon 2026

Team TED | 1st Place on L1 Leaderboard — €1,509,650

Challenge Overview

Predict Core Demand for 100 procurement buyers: identify which product categories (E-Class codes) each buyer will purchase in Q1 2026. This is an economic portfolio optimization problem — every prediction costs a €10 fee, but correct predictions capture 10% of the matched spend as savings.

Scoring formula: $$\text{Score} = \sum(\text{Savings}) - \sum(\text{Fees}) = \sum(10% \times \text{matched spend}) - \sum(€10 \times \text{predictions})$$

The dataset contains 8.37M anonymized procurement transactions across 36 months, with 100 test buyers split into:

• 48 Warm buyers — have transaction history
• 52 Cold buyers — no purchase history, only industry/size metadata

---

Final Result

Metric	Value
Score	€1,509,649.66
Predictions	47,868
Savings	€1,988,336
Fees	€478,680
Spend Captured	79.3%
Hits	23,490

---

Architecture

The pipeline combines 5 prediction sources through an ensemble approach:

                          ┌──────────────────┐
                          │  Raw Transactions │
                          │   8.37M rows TSV  │
                          └────────┬─────────┘
                                   │
                    ┌──────────────┼──────────────┐
                    ▼              ▼               ▼
            ┌──────────┐   ┌────────────┐   ┌──────────┐
            │ Feature   │   │  Temporal   │   │ Buyer    │
            │ Engineer. │   │  Split     │   │ Profiles │
            └────┬─────┘   └─────┬──────┘   └────┬─────┘
                 │               │                │
          ┌──────┴───┐     ┌────┴─────┐          │
          ▼          ▼     ▼          │          │
    ┌──────────┐ ┌──────┐ ┌────┐     │          │
    │Handformel│ │  ML  │ │ CF │     │          │
    │ Scoring  │ │Model │ │    │     │          │
    └────┬─────┘ └──┬───┘ └─┬──┘     │          │
         │          │       │        │          │
         └────┬─────┘       │        │          │
              ▼             │        │          │
      ┌───────────┐        │        │          │
      │  Ensemble │        │        │          │
      │  Merge    │        │        │          │
      └─────┬─────┘        │        │          │
            ▼              │        │          │
    ┌──────────────┐       │        │          │
    │   Portfolio  │       │        │          │
    │   Trimming   │◄──────┘        │          │
    └──────┬───────┘                │          │
           │                        │          │
    ┌──────┴─────────────┐         │          │
    │   SWAP Strategy    │◄────────┘          │
    │  Replace weak HF   │                    │
    │  with better CF    │                    │
    └──────┬─────────────┘                    │
           │                                   │
    ┌──────┴─────────────┐                    │
    │ Industry Expansion │◄───────────────────┘
    │ (Cold logic on     │
    │  warm buyers)      │
    └──────┬─────────────┘
           │
    ┌──────┴─────────────┐
    │    Cold Start       │
    │  (NACE matching)    │
    └──────┬─────────────┘
           │
           ▼
    ┌──────────────┐
    │  submission   │
    │    .csv       │
    └──────────────┘

Source Breakdown (Winning Run)

Source	Predictions	Description
Handformel + ML Ensemble	~18,000	Rule-based scoring + HistGradientBoosting, merged with consensus bonus
Collaborative Filtering	~4,400	Items similar buyers purchase (cosine similarity, sparse matrix)
Eclass Hierarchy	~77	Related product categories within the same E-Class prefix group
Association Rules	~1,075	Co-purchase patterns via frequent itemset mining
Industry Expansion	~2,700	Cold-start logic applied to warm buyers — items they should buy based on NACE industry peers
Cold Start	~27,000	3-tier NACE matching for 52 buyers with zero history

---

Pipeline Components

1. Data Loading & Preprocessing (data_loader.py, preprocess.py)

• Loads gzipped TSV files (tab-separated, UTF-8 BOM encoding)
• Column remapping to standardized names
• Removes rows without E-Class codes (~100k rows)
• Computes derived columns: spend = price × quantity

2. Feature Engineering (feature_engineering.py)

Builds a buyer×eclass feature matrix (2M+ combinations) with 11 features per pair:

Feature	Description
total_orders	Number of purchase events
total_quantity	Total units purchased
total_spend	Total € spent
mean_price	Average unit price
active_months	Months with at least one purchase
frequency_ratio	active_months / total_months (0-1)
recency_months	Months since last purchase
regularity	Std deviation of monthly purchase intervals (inverted)
spend_per_month	total_spend / active_months
span_months	First to last purchase span
q1_ratio	Fraction of purchases in Q1 (Jan-Mar) — seasonal indicator

Also builds buyer profiles aggregating across all items per buyer (total spend, industry, company size).

3. Warm-Start Scoring (warm_start.py)

Rule-based economic scoring for buyers with transaction history.

Core formula:

expected_future_spend = blend(conservative, aggressive, reliability)
  conservative = total_spend × (3/36)
  aggressive   = spend_per_month × 3
  reliability  = min(active_months / 16, 1.0)

savings_estimate = 10% × expected_future_spend × boosts
core_score = savings_estimate if > €3, else 0

Score boosts (multiplicative):

• Frequency: 1.0 + 0.9 × frequency_ratio (frequent buyers score higher)
• Regularity: 1.0 + 0.10 × regularity (regular patterns score higher)
• Recency: 1.08 if ≤3 months ago, 1.03 if ≤9 months, 1.0 otherwise
• Seasonality: 0.80 + 0.20 × q1_ratio (Q1 is the prediction period)
• Span correction: Up to 2× for buyers with short history (compensates for limited data)

Key insight — MIN_SAVINGS threshold: Items with savings_estimate > €3 get their actual savings as score. This was a critical breakthrough — the model underestimates real savings by ~2×, so items estimating €3 could have real savings of €6-10. Setting the threshold at €3 (not €10) captures these marginal-but-profitable items.

4. ML Model (ml_model.py)

• Algorithm: HistGradientBoostingClassifier (sklearn)
• Training: Temporal split — 33 months training, 3 months holdout
• Hyperparameters: max_iter=300, learning_rate=0.05, max_leaf_nodes=127, min_samples_leaf=50, random_state=42
• Scoring: P(purchase) × expected_savings, filtered by min_proba=0.25 and min_spend=€100
• Role: Confirms and boosts Handformel predictions. Items both methods agree on get a 20% consensus bonus.

5. Portfolio Trimming (portfolio.py)

• Max 400 items per buyer (proven optimal cap through extensive grid search)
• Sorted by core_score descending — only the highest-value items are kept
• No marginal benefit ratio filter (set to 0) — pure score-based cutoff

6. Collaborative Filtering (collab_filter.py)

• Builds sparse buyer×eclass interaction matrix (64,898 × 6,226) with log1p(spend) weights
• Cosine similarity between buyers
• For each warm test buyer: finds top-50 most similar buyers
• Recommends items those neighbors buy but the test buyer doesn't
• Score: confidence × expected_savings, where confidence = min(support/50, 1.0)
• Filters: min 2 neighbors must buy the item, avg_spend ≥ €100, score ≥ €5

7. Eclass Hierarchy Expansion (eclass_expand.py)

Two strategies:

• Hierarchy expansion: If a buyer purchases ≥2 items sharing a 6-digit E-Class prefix, recommend other items in that prefix group
• Association rules: Mine co-purchase patterns (min_support=0.005, min_confidence=0.3), recommend items frequently co-purchased with a buyer's existing portfolio

8. SWAP Strategy (in main.py)

Rather than simply adding CF predictions on top (which increases fee count), we replace the weakest Handformel items with better-scoring CF items. This keeps prediction count stable while improving quality.

For each warm buyer:

1. Sort HF items ascending by score (weakest first)
2. Sort CF items descending by score (best first)
3. Replace HF items where CF score > HF score (same count)
4. Also add CF items for buyers under the 400-item portfolio cap

9. Industry Expansion (in main.py)

The biggest breakthrough — apply cold-start NACE-matching logic to warm buyers.

Even warm buyers might be missing items that their industry peers commonly buy. We run predict_cold_buyers() on warm test buyers with max_items=200, then add any items not already in their portfolio. This found ~2,700 additional high-quality predictions worth ~€25k in score improvement.

10. Cold Start (cold_start.py)

3-tier NACE industry matching for 52 buyers with zero history:

1. Exact NACE-4: Find known buyers in the same 4-digit industry code
2. NACE-2 prefix: Fall back to 2-digit industry group
3. Global: Use all known buyers as neighbors

For each cold buyer: find top-50 neighbors by company size similarity, aggregate their portfolios, and recommend items with prevalence ≥ 5% among neighbors. Max 525 items per buyer.

---

Score Progression

Milestone	Score	Key Change
MVP baseline	€781k	Basic frequency-based predictions
ML ensemble + tuning	€1,267k	Added HistGradientBoosting, parameter optimization
Scoring quality fix	€1,484k	Used savings_estimate as score instead of flat 0.01 for marginal items
Cold cap optimization	€1,484k → €1,485k	Tuned cold-start cap from 400 → 525
Industry expansion + SWAP	€1,510k	Cold-start logic on warm buyers, CF swap strategy

What Worked

• Scoring quality (using actual savings_estimate as the ranking score) was the single biggest improvement (+€200k)
• Industry expansion for warm buyers was the breakthrough that secured #1 (+€25k)
• SWAP strategy improved quality without increasing prediction count
• Low MIN_SAVINGS threshold (€3 instead of €10) captured marginal-but-profitable items
• Aggressive CF params (min_support=2, max_items=300) cast a wide net

What Didn't Work

• Inflating CF scores with similarity weighting → too many bad swaps (-€34k)
• CF with 100 neighbors instead of 50 → diluted signal
• Eclass prefix-4 expansion → too broad, added noise
• MIN_SAVINGS=2 → too many low-value predictions, fees > savings

• 53k predictions → 81% spend captured but €534k in fees, net worse

---

Project Structure

core-demand-hackathon/
├── README.md                      # This file
├── core_demand/                   # Core pipeline
│   ├── main.py                    # Pipeline orchestration & entry point
│   ├── config.py                  # All hyperparameters & paths
│   ├── data_loader.py             # TSV/gzip data loading
│   ├── preprocess.py              # Data cleaning & column mapping
│   ├── feature_engineering.py     # Buyer×eclass feature matrix
│   ├── warm_start.py              # Rule-based economic scoring
│   ├── ml_model.py                # HistGradientBoosting classifier
│   ├── cold_start.py              # NACE-based industry matching
│   ├── collab_filter.py           # Sparse cosine CF
│   ├── eclass_expand.py           # Hierarchy + association rules
│   ├── portfolio.py               # Portfolio trimming
│   ├── backtesting.py             # Temporal split & calibration
│   ├── inference.py               # Submission formatting
│   └── utils.py                   # Logging & CSV utilities
├── data/                          # Training data (not in repo)
│   ├── plis_training.csv.gz       # 8.37M transactions
│   ├── customer_test.csv.gz       # 100 test buyers
│   ├── features_per_sku.csv.gz    # SKU features
│   └── nace_codes.csv.gz          # Industry codes
└── output/                        # Generated submissions (not in repo)
    └── submission_l1_ensemble3.csv

How to Run

# Install dependencies
pip install numpy pandas scikit-learn scipy

# Place data files in data/ directory
# Run the full pipeline (ensemble3 = all 5 sources)
cd core-demand-hackathon
python core_demand/main.py --skip-calibrate --mode ensemble3
# Output: output/submission_l1_ensemble3.csv

Pipeline Modes

Mode	Command	Description
Handformel only	python core_demand/main.py --mode handformel	Rule-based scoring only
ML only	python core_demand/main.py --mode ml	HistGradientBoosting only
Ensemble	python core_demand/main.py --mode ensemble	HF + ML with consensus bonus
Ensemble3	python core_demand/main.py --mode ensemble3	HF + ML + CF + Eclass + Industry (winning config)

---

Technical Details

• Python 3.9+ with numpy, pandas, scikit-learn 1.6, scipy
• Data format: Tab-separated, UTF-8 BOM (utf-8-sig), gzipped
• Scale: 8.37M transactions, 64,898 unique buyers, 6,226 E-Class codes
• Runtime: ~4 minutes on M2 MacBook Air

---

Key Configuration (Winning Run)

MAX_PORTFOLIO_SIZE = 400       # Warm buyer cap
MIN_SAVINGS = 3.0              # Threshold for including items
SAVINGS_RATE = 0.10            # 10% savings on matched spend
FEE_PER_ELEMENT = 10.0         # €10 per prediction
COLD_START_K_NEIGHBORS = 50    # Neighbors for cold/industry matching
COLD_START_MIN_PREVALENCE = 0.05
# CF: top_k=50, min_support=2, max_items=300
# Cold cap: 525 items per buyer
# Industry expansion: max_items=200 for warm buyers