Speed up queries via materialized view by Muhammad-Moiz626 · Pull Request #1671 · FlexMeasures/flexmeasures

Muhammad-Moiz626 · 2025-08-19T15:22:32Z

Description

Our data model with multiple beliefs per event time (multiple horizons as well as sources) makes queries computationally expensive. Most-used are queries where we look only for the most recent beliefs about each event, when given any moment in time as max belief time.

Pre-computation with Postgres' materialized views can make queries much faster. This PR will enable hosts to use this performance enhancement.

Created migrations file to create materialized view.
Use the MV in queries.
Use SQL Alchemy for MV.
Automate the refreshing of the MV.
Make MV optional / configurable.
Document the process

...

Further Improvements

the full queries (used in playbacks, getting fulls set of beliefs for each event) might also enjoy a materialized view, but maybe it is just too much data, then we should just limit playback to shorter time ranges
give hosts a range of options (e.g. refresh time)
actually move to TimeScaleDB, which auto-refreshes their materialized views on inserts

Related Items

Closes 108

...

Sign-off

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

…and for refresh.

Flix6x

Some early review comments. Nice to see the refresh logic taking shape.

read-the-docs-community · 2025-08-25T15:19:04Z

Documentation build overview

📚 flexmeasures | 🛠️ Build #30701717 | 📁 Comparing aabd536 against latest (08d77fa)

🔍 Preview build

Show files changed (5 files in total): 📝 5 modified | ➕ 0 added | ➖ 0 deleted

File	Status
changelog.html	📝 modified
configuration.html	📝 modified
_autosummary/flexmeasures.data.models.generic_assets.html	📝 modified
_autosummary/flexmeasures.data.models.time_series.html	📝 modified
api/v3_0.html	📝 modified

Flix6x

Exciting work! Thanks for addressing my earlier comments. Just a short code review for now, so you don't have to wait for me testing. Looking forward to testing tomorrow.

nhoening

Also a quick review from me, before having tested myself, mostly about concepts.

I also suggest moving get_timed_belief_min_v() to timely-beliefs and add basic documentation there (not much more than the SQL command so anybody reading only that could in principle use it)

Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Signed-off-by: Muhammad-Moiz626 <muhammadmoiz4843@gmail.com>

…erialized-veiws # Conflicts: # documentation/changelog.rst # documentation/configuration.rst # flexmeasures/data/models/generic_assets.py # flexmeasures/ui/templates/base.html

Signed-off-by: F.N. Claessen <felix@seita.nl>

Resolved conflicts in changelog.rst, configuration.rst, time_series.py, assets.py, base.html, and views.py, keeping materialized view features alongside upstream changes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

base.html gained ~950 lines of stale template blocks (copies of JS that lives in includes/graphs.html) and openapi-specs.json had its version reverted. Restore both to main; the materialized-view UI toggle will be re-added on top of the real chart-loading code in includes/graphs.html. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qprz6PKnXkVE1TJ8d7PYe9

# Conflicts: # documentation/changelog.rst

…recorded refresh time - migration: frozen SQL from timely-beliefs' DDL generators (simplified view query, no CASCADE on downgrade), repointed onto main's migration head (drops the stale merge migration) - CLI: refresh-materialized-views records its run in latest_task_run via @task_with_status_report, supports --concurrent (autocommit connection), aborts with an error exit code on failure; monitorable with 'flexmeasures monitor latest-run' - config: replace FLEXMEASURES_MVIEW_REFRESH_INTERVAL with FLEXMEASURES_MVIEW_ALWAYS_INCLUDE_LIVE_TAIL (the refresh cadence now lives solely in how hosts schedule the CLI command; the recorded refresh time is the single source of truth) - queries: TimedBelief.search decides centrally whether to trust the view (it must exist at startup and have a recent enough recorded refresh) and passes a cutoff so events recorded since the last refresh are looked up in the beliefs table (live tail); new include_live_tail param threaded through Sensor/GenericAsset.search_beliefs and the chart data API endpoints - UI: 'Include latest data' toggle on asset graphs (shown only when the view is in use and the live tail is not already always included), wired to the real chart-loading code in includes/graphs.html; tooltip shows the actual last refresh time - docs: configuration.rst setting description, hosting docs section with cron example, changelog entry - tests: CLI refresh success/failure recording, mview/live search equivalence Note: requires the timely-beliefs release containing SeitaBV/timely-beliefs#204 (materialized_views module); the dependency pin should be bumped once that is released. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qprz6PKnXkVE1TJ8d7PYe9

Signed-off-by: F.N. Claessen <claessen@seita.nl>

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

socket-security · 2026-07-06T08:32:19Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	timely-beliefs@3.5.5 ⏵ 4.0.0	⁺¹

View full report

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Signed-off-by: F.N. Claessen <claessen@seita.nl>

…at/materialized-veiws # Conflicts: # uv.lock

WIP: added migration file.

3ca67f4

Muhammad-Moiz626 marked this pull request as draft August 19, 2025 15:23

nhoening added the Data label Aug 19, 2025

nhoening changed the title ~~WIP: added migration file.~~ WIP: Speed up queries via materialized view Aug 19, 2025

nhoening assigned Muhammad-Moiz626 Aug 19, 2025

Added materialized view boolean to backend logic and created CLI comm…

14226ba

…and for refresh.

Flix6x requested changes Aug 22, 2025

View reviewed changes

Added refresh interval, Fixed some bugs.

ce19cb0

Flix6x reviewed Aug 22, 2025

View reviewed changes

Comment thread flexmeasures/utils/config_defaults.py Outdated

Muhammad-Moiz626 added 3 commits August 25, 2025 17:59

Merged main

7dd77fe

Resolved comments.

fd41966

Tiny refactor.

e483d98

Muhammad-Moiz626 marked this pull request as ready for review August 27, 2025 12:00

Muhammad-Moiz626 added 2 commits August 27, 2025 21:17

Merge branch 'main' into feat/materialized-veiws

e3277eb

Merge branch 'main' into feat/materialized-veiws

3795934

Muhammad-Moiz626 requested a review from Flix6x September 1, 2025 14:42

Flix6x requested changes Sep 1, 2025

View reviewed changes

nhoening requested changes Sep 1, 2025

View reviewed changes

Muhammad-Moiz626 and others added 8 commits September 2, 2025 14:03

Update documentation/configuration.rst

74d6651

Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Signed-off-by: Muhammad-Moiz626 <muhammadmoiz4843@gmail.com>

Moved get_timed_belief_min_v and some other small changes.

c65ef8c

Merged main.

cc88389

Moved get_timed_belief_min_v to timely-beliefs repo.

942380c

Updated base.html according to materialized view refresh interval

b6ca3d4

Renamed timed_belief_min_v to most_recent_beliefs_mview

e2140a1

Merge branch 'main' into feat/materialized-veiws

475cf0d

Added changelog.

fa522fa

Muhammad-Moiz626 requested review from Flix6x and nhoening September 4, 2025 06:57

Merge branch 'main' into feat/materialized-veiws

e7b4599

Muhammad-Moiz626 changed the title ~~WIP: Speed up queries via materialized view~~ Speed up queries via materialized view Sep 8, 2025

Flix6x reviewed Sep 9, 2025

View reviewed changes

Comment thread documentation/configuration.rst Outdated

Flix6x added 2 commits December 15, 2025 16:46

Merge remote-tracking branch 'refs/remotes/origin/main' into feat/mat…

24be3f5

…erialized-veiws # Conflicts: # documentation/changelog.rst # documentation/configuration.rst # flexmeasures/data/models/generic_assets.py # flexmeasures/ui/templates/base.html

chore: db merge

aabd536

Signed-off-by: F.N. Claessen <felix@seita.nl>

Flix6x assigned Flix6x and unassigned Muhammad-Moiz626 Dec 15, 2025

nhoening mentioned this pull request Jan 22, 2026

Sensor page shows latest forecasts visually, but stats use all horizons #1290

Open

nhoening mentioned this pull request Mar 10, 2026

Support TimescaleDB #601

Open

Ahmad-Wahid and others added 6 commits June 4, 2026 21:46

Merge origin/main into feat/materialized-views

22cf553

Resolved conflicts in changelog.rst, configuration.rst, time_series.py, assets.py, base.html, and views.py, keeping materialized view features alongside upstream changes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into feat/materialized-veiws

f4726b9

# Conflicts: # documentation/changelog.rst

style: black

6fee373

Signed-off-by: F.N. Claessen <claessen@seita.nl>

chore(deps): bump timely-beliefs to >=4.0.0

5eb7e02

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Flix6x and others added 3 commits July 6, 2026 10:40

chore(deps): bump timely-beliefs to >=4.0.0

81a0c6f

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

refactor: flake8 complained about complexity

ccfabc8

Signed-off-by: F.N. Claessen <claessen@seita.nl>

Merge remote-tracking branch 'origin/feat/materialized-veiws' into fe…

b75b802

…at/materialized-veiws # Conflicts: # uv.lock

Uh oh!

Conversation

Muhammad-Moiz626 commented Aug 19, 2025 • edited by Flix6x Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Further Improvements

Related Items

Sign-off

Uh oh!

Flix6x left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

read-the-docs-community Bot commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

Flix6x left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nhoening left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

socket-security Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Muhammad-Moiz626 commented Aug 19, 2025 •

edited by Flix6x

Loading

read-the-docs-community Bot commented Aug 25, 2025 •

edited

Loading

socket-security Bot commented Jul 6, 2026 •

edited

Loading