Skip to content

Feature: flex-context per commodity as list#2235

Merged
Flix6x merged 48 commits into
dev/split-flexcontext-by-commodityfrom
dev/flex-context-as-list
Jul 1, 2026
Merged

Feature: flex-context per commodity as list#2235
Flix6x merged 48 commits into
dev/split-flexcontext-by-commodityfrom
dev/flex-context-as-list

Conversation

@Flix6x

@Flix6x Flix6x commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Description

  • The flex-context is now preferred to be defined as a list, with one entry per commodity
  • The new fields aggregate-consumption and aggregate-production are the per-commodity replacements of aggregate-power
  • Added changelog item in documentation/changelog.rst

Look & Feel

...

How to test

pytest -k test_asset_schedules_fresh_db

Further Improvements

Internally, list-like flex-contexts remain nested under the commodities field in the regular dict-like flex-context. I tried for some time to deal with list-like flex-contexts throughout the codebase, but there were numerous complications, including but not limit to dealing with sequential scheduling, in which a sequence of dependent scheduling jobs are set up, where the inflexible-device-sensors (in the flex-context) of the next job includes the power sensors of the previous jobs.

Related Items

Merges into #2172.

Flix6x added 6 commits June 12, 2026 14:23
@Flix6x
feat: support passing flex-context as a list (one flex-context per co…
a66334b 
…mmodity)

Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
fix: set default flex-context commodity to electricity
d660d02 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
fix: preserve field order in case schema is made OpenAPI compatible
1667344 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
feat: reduce documented nesting when defining a flex-context per comm…
abdc440 
…odity

Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
dev: add todos
f229b7e 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
Merge remote-tracking branch 'origin/dev/split-flexcontext-by-commodi…
137ffa3 
…ty' into dev/flex-context-as-list

Signed-off-by: F.N. Claessen <claessen@seita.nl>

# Conflicts:
#	flexmeasures/ui/static/openapi-specs.json
@Flix6x Flix6x self-assigned this Jun 12, 2026
@Flix6x
style: flake8
cc033ba 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Copilot AI added a commit that referenced this pull request Jun 12, 2026
@Flix6x
scheduling: complete schema refactoring per PR #2235
4855963 
- Add SensorReferenceSchema import
- Override relax_constraints default to True in CommodityFlexContextSchema
- Remove duplicate breach price and relax fields from FlexContextSchema
- Remove duplicate set_default_breach_prices method from FlexContextSchema

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Copilot AI and others added 8 commits June 12, 2026 20:43
@Flix6x
scheduling: complete schema refactoring per PR #2235
2b3db1e 
- Add SensorReferenceSchema import
- Override relax_constraints default to True in CommodityFlexContextSchema
- Remove duplicate breach price and relax fields from FlexContextSchema
- Remove duplicate set_default_breach_prices method from FlexContextSchema

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
tests: add comprehensive tests for schema refactoring
5d25d4e 
- Test aggregate-consumption and aggregate-production fields
- Test SharedSchema fields accessible in FlexContextSchema
- Test CommodityFlexContextSchema relax_constraints defaults to True
- Test shared currency logic for flex-context listings
- Test breach prices in both schemas
- Add noqa comment for existing unused variable

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
@Flix6x
scheduling: add shared currency validation for commodity contexts
13a0117 
- Add validator to check prices share same currency across all commodity contexts
- Fix test data to use actual fixture sensor names
- All new tests now passing

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
@Flix6x
docs: add aggregate fields to documentation and update tests
0e76823 
- Add aggregate-consumption and aggregate-production to scheduling.rst
- Exclude COMMODITY_FLEX_CONTEXT and COMMODITY_FLEX_MODEL from doc test (already documented as "commodity")
- Exclude aggregate fields from UI test (not yet supported in UI)

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
@Flix6x
chore: upgrade openapi-specs.json
6a55aba 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
fix: move _try_to_convert_price_units to SharedSchema
a4b50e3 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
fix: some fields cannot use source filters; now instead of just havin…
fc5ba5b 
…g validators in place to forbid that, we actually stop having those fields in the schemas of those fields

Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
chore: mypy
679eca2 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x Flix6x changed the base branch from main to dev/split-flexcontext-by-commodity June 12, 2026 20:36
Flix6x added 2 commits June 12, 2026 22:49
@Flix6x
delete: remove redundant tests
8852e99 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
fix: minimize diff
6ff2cf7 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Flix6x
Flix6x commented Jun 12, 2026
View reviewed changes
Comment thread flexmeasures/data/schemas/tests/test_scheduling.py
Comment thread flexmeasures/data/schemas/tests/test_scheduling.py Outdated
Comment thread flexmeasures/data/schemas/tests/test_scheduling.py Outdated
Comment thread flexmeasures/data/schemas/tests/test_scheduling.py Outdated
Comment thread flexmeasures/data/schemas/tests/test_scheduling.py Outdated
Comment thread flexmeasures/data/schemas/tests/test_scheduling.py Outdated
Comment thread flexmeasures/ui/tests/test_utils.py Outdated
@Flix6x Flix6x requested a review from Ahmad-Wahid June 12, 2026 23:30
Flix6x added 7 commits June 13, 2026 11:05
@Flix6x
fix: default flex-model and flex-context to empty lists, because it i…
a53cfe5 
…s possible that the entire flex-config is described in the db instead of the trigger message (this cherry-pick was adapted, because the flex-context moved from being documented as a dict to a list)

Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
docs: prefer unique sensor IDs in examples
94bfecd 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
docs: clarify cross-reference
e3bd3b9 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
feat: UI support for picking a
7f2642d 
sensor for aggregate-consumption sensor and/or aggregate-production

Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
fix: for some reason, for flex-context fields, the sensor-only proper…
6e6a36b 
…ty is defined in the html in 3 places

Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
feat: test coverage for UI support of aggregate-consumption and aggre…
60def3a 
…gate-production

Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
docs: add deprecation instructions for aggregate-power
5fec7c5 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Ahmad-Wahid and others added 4 commits June 14, 2026 00:14
@Ahmad-Wahid @Flix6x
update comment
ac2cc0f 
Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
Signed-off-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com>
@Ahmad-Wahid @Flix6x
update the comment
c4cafc9 
Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
Signed-off-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com>
@Flix6x
chore: update openapi-specs.json again
7566792 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
Merge remote-tracking branch 'origin/dev/flex-context-as-list' into d…
1a59bcb 
…ev/flex-context-as-list
Copilot AI added a commit that referenced this pull request Jun 16, 2026
@Flix6x
tests: add comprehensive aggregate sensor integration tests
59259fd 
Context:
- PR #2235 tests were weak and under-specified
- Need to verify aggregate sensors get filled with data
- Need to verify correct sign conventions
- Need to verify per-commodity logic and backwards compatibility

Change:
- New file test_scheduling_aggregate_sensors_fresh_db.py (465 lines)
- 5 integration tests verifying aggregate sensor functionality:
  * Data filling and sign convention (3 parametrized cases)
  * Per-commodity aggregation (skipped - multi-device complexity)
  * Backwards compatibility (no commodity_contexts)
  * Data source type verification
- Tests use fresh_db fixture for database modifications
- All tests passing (5 passed, 1 skipped)

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Copilot AI added a commit that referenced this pull request Jun 16, 2026
@Flix6x
scheduling: add per-commodity aggregate sensor computation
6c8e946 
Context:
- PR #2235 feedback: aggregate sensors not being filled with data
- Missing per-commodity logic in StorageScheduler
- Need backwards compatibility (no commodity_contexts = all electric)

Change:
- New method _compute_commodity_aggregate_schedules (lines 2082-2202)
- Reconstructs commodity-to-devices mapping from device models
- Computes per-commodity aggregates by summing ems_schedule
- Supports aggregate-consumption and aggregate-production sensors
- Applies split logic based on which sensors are defined
- Maintains backwards compatibility with old format

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Copilot AI added a commit that referenced this pull request Jun 16, 2026
@Flix6x
tests: improve commodity flex-context schema validation tests
e720671 
Context:
- PR #2235 tests were weak and under-specified
- Test comments didn't accurately describe behavior
- Need better schema validation coverage

Change:
- Renamed test_commodity_flex_context_defaults to
  test_commodity_flex_context_schema_validation
- Enhanced docstring to clarify schema validation only
- Added 3 new test cases for individual aggregate sensors
- Added assertions to verify parsed sensor references
- 6 parametrized test cases, all passing

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Copilot AI added a commit that referenced this pull request Jun 16, 2026
@Flix6x
agents/test-specialist: learned aggregate sensor testing patterns
3728e5a 
Context:
- PR #2235 required testing aggregate sensor data filling
- Integration tests verify scheduler populates aggregate sensors
- Schema validation tests are separate from integration tests

Change:
- Document integration test pattern for aggregate sensors
- Add fresh_db requirement for scheduler integration tests
- Document scheduler result format (list[dict] with sensor/data/unit/name)
- Add lesson about sign convention verification pattern
- Document backwards compatibility testing approach

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Copilot AI added a commit that referenced this pull request Jun 16, 2026
@Flix6x
agents/architecture: per-commodity aggregate schedule patterns
6b58ffa 
Context:
- PR #2235 added _compute_commodity_aggregate_schedules method
- Implemented per-commodity aggregate power flow computation
- Split logic for consumption vs production sensors

Change:
- Document aggregate schedule computation pattern
- Add commodity_to_devices mapping to scheduler knowledge
- Document sign convention handling for aggregate fields
- Document integration point in scheduler flow
- Add backwards compatibility handling pattern

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
@Ahmad-Wahid
update openapi specs
f6ff8b8 
Signed-off-by: Ahmad-Wahid <ahmedwahid16101@gmail.com>
@Ahmad-Wahid
feat: add multi feed stock tutorial
44bd283 
Signed-off-by: Ahmad-Wahid <ahmedwahid16101@gmail.com>
@Ahmad-Wahid
remove extra chart explanation
990896a 
Signed-off-by: Ahmad-Wahid <ahmedwahid16101@gmail.com>
Flix6x
Flix6x commented Jun 17, 2026
View reviewed changes
Comment thread flexmeasures/ui/static/openapi-specs.json Outdated
Ahmad-Wahid and others added 2 commits June 17, 2026 19:14
@Ahmad-Wahid @Flix6x
fix: update flexmeasures version in openapi-specs
2b2112b 
Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
Signed-off-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com>
@Ahmad-Wahid
feat: add multi commodity tutorial
95ea68f 
Signed-off-by: Ahmad-Wahid <ahmedwahid16101@gmail.com>
Flix6x
Flix6x commented Jun 30, 2026
View reviewed changes
Comment thread documentation/tut/multi-commodity.rst Outdated
Comment thread documentation/tut/multi-commodity.rst
Comment thread documentation/tut/multi-commodity.rst Outdated
Comment thread documentation/tut/multi-commodity.rst Outdated
Comment thread documentation/tut/multi-commodity.rst Outdated
Comment thread documentation/tut/multi-commodity.rst Outdated
@Flix6x Flix6x marked this pull request as ready for review June 30, 2026 09:16
Flix6x and others added 11 commits June 30, 2026 11:21
@Flix6x
Apply my own suggestions from code review
6b77789 
Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
@Flix6x
docs: changelog entry
371d0e4 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
test: add comprehensive coverage for aggregate-consumption/production…
19b6ced 
… flex-context fields

- Add test_asset_trigger_and_get_aggregate_schedule to verify aggregate sensors get
  populated (currently xfail as StorageScheduler doesn't yet populate them)
- Add test_asset_trigger_with_multi_commodity_flex_context to verify aggregate sensors
  work with multi-commodity flex-context format (list of commodity dicts)
- Document that aggregate sensor population is missing logic in StorageScheduler
- Both tests verify the infrastructure and schema support works correctly
@Flix6x
fix: compute per-commodity aggregate power flows for aggregate-consum…
da58a75 
…ption and aggregate-production sensors

Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Ahmad-Wahid @Flix6x
Include per-commodity inflexible devices in aggregate schedules
ab4a9f1 
_compute_commodity_aggregate_schedules only added the top-level inflexible
devices to the electricity commodity, so a commodity's own inflexible demand
(e.g. a heat load) was excluded from its aggregate-consumption schedule. The
aggregate then reflected only that commodity's flexible devices. Mirror the
device enumeration used in _prepare so each commodity context's
inflexible-device-sensors are appended to that commodity at the matching
ems_schedule indices.
@Flix6x
test: update aggregate sensor tests to verify implementation
5cc31a1 
- Remove xfail markers from both tests now that StorageScheduler populates aggregate sensors
- Update test docstrings to reflect actual functionality
- Tests now verify that aggregate-consumption and aggregate-production sensors receive data
- First test covers single aggregate pair with dual devices
- Second test covers aggregates with multiple devices scheduling together
@Flix6x
feat: test commodities referenced in only the flex-model or only the …
69adc57 
…flex-context

Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
feat: test logs should explain why response was unexpected
b1c4257 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
style: black
c17d173 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
feat: AssetTriggerSchema accepts list-like flex-contexts
51eff5d 
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
fix: skip aggregate computation for unused commodities
1e7857b 
When a multi-commodity flex_context includes a commodity (e.g., heat)
but no devices exist for that commodity, the sum() of an empty list
returns integer 0 instead of a pandas Series. This caused AttributeError
when calling .round() on the result later in compute().
Fixed by checking if any devices contribute to the commodity before
attempting to aggregate. If no devices exist for a commodity, skip
creating an aggregate schedule for it.

Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x Flix6x merged commit 5512411 into dev/split-flexcontext-by-commodity Jul 1, 2026
12 checks passed
@Flix6x Flix6x deleted the dev/flex-context-as-list branch July 1, 2026 09:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Ahmad-Wahid Ahmad-Wahid Awaiting requested review from Ahmad-Wahid

Assignees

@Flix6x Flix6x

Labels

Scheduling

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Flix6x @Ahmad-Wahid