empw-synth-data

A pipeline that turns raw smart-meter data from energy communities into a fully anonymised, statistically faithful synthetic dataset — synthetic metering points, synthetic households, one assembled synthetic energy community (EEG) with master data, participation factors and 15-minute energy time series, plus a built-in fidelity and privacy evaluation.

Developed in the CERTAIN project (Horizon Europe) for the EMPOWER pilot, at St. Pölten University of Applied Sciences. The methodology is described in the accompanying paper (link follows); this repository is the code. No real data is contained in or required by this repository — a demo input generator lets you run everything end-to-end.

What it does

1. Stage 1 — Preprocess: strip PII (real ids → UUID surrogates, restricted lookup kept apart), stream-scan the input corpus into per-meter features and a structural skeleton
2. Stage 2 — Cluster: rule-based, interpretable archetypes (generator technology, size, consumption patterns, behind-the-meter PV)
3. Stage 3 — Synthesise: per-archetype time-series synthesis. The released method is a conditional Gaussian copula with per-meter identity and optional weather conditioning (docs/methods.md); profile bootstrap, flat copula and marginal sampling serve as comparison baselines. VAE / GAN / diffusion synthesizers ship as placeholders — deliberately simple implementations that demonstrate the plug-in synthesizer interface, not tuned competitors
4. Stage 4 — Assemble: build the full EEG schema (community, participants, metering points, participation-factor history, settled energy data) around the synthetic meters
5. Stage 5 — Evaluate: fidelity (distributions, autocorrelation, load profiles) and privacy (nearest-neighbour distances, membership inference) per archetype

A Streamlit dashboard compares runs side by side.

Quickstart

Python 3.11/3.12. No real data needed:

pip install -r requirements.txt

# 1. fabricate a small, fully synthetic demo input corpus
python src/scripts/make_demo_input.py

# 2. run all five stages
python src/scripts/entrypoints/entrypoint_full_pipeline.py \
  --config_files \
    config/conf_base.yaml \
    config/conf_stage1_preprocessing.yaml \
    config/conf_stage2_clustering.yaml \
    config/conf_stage3_synthesis.yaml \
    config/conf_stage4_assembly.yaml \
    config/conf_stage5_evaluation.yaml \
    config/overrides/demo.yaml

# 3. inspect the run
streamlit run src/dashboard/app.py

Each run creates local_data/runs/<run_id>/ with a manifest.json (config snapshot, stage timings, lineage) and all artifacts. Runs can be resumed and forked (--resume_run_id, --fork_from_run_id, --from_stage) — see docs/setup.md.

To run on your own data: write one CSV per community in the input schema described in docs/data_format.md and point config/conf_stage1_preprocessing.yaml at it.

Repository layout

src/modules/        pipeline library (preprocessing, clustering, synthesis, assembly, evaluation, io)
src/scripts/        stage runners, entrypoint, analysis/evaluation scripts, demo input generator
src/dashboard/      Streamlit run-comparison dashboard
config/             one YAML per stage + example overrides
tests/              pytest suite (self-contained, no data needed)
docs/               setup & config reference, architecture, synthesis methods, data format

License & funding

Apache-2.0 — see LICENSE.

This work is part of the CERTAIN project, funded by the European Union's Horizon Europe research and innovation programme under Grant Agreement No 101189650, with support from the Swiss State Secretariat for Education, Research and Innovation (SERI). Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.