Feature/signal data operation by jwasikpsnc · Pull Request #90 · iterorganization/IBEX

jwasikpsnc · 2026-06-22T06:58:02Z

adds /data/plot_data/ signal_operations parameter
adds signal operations description in /info/data_manipulation_methods endpoint response
accepts operations as list ["operation:<URI>", "..."] e.g. ["add:<uri>","mul:<uri>","div:<uri>"]
operations are executed in order determined by position in the list
allowed operations:
- add
- sub
- mul
- div
updated docs
if <base_uri> and <operand_uri> have the same shapes - > no interpolation needed
if <base_uri> and <operand_uri> have different shapes -> error is being raised

Waiting for #89 to be merged first

…signal_data_operation # Conflicts: # backend/ibex/endpoints/schemas/request_data_schemas.py

…nto feature/signal_data_operation # Conflicts: # backend/ibex/data_source/imas_python_source_utils.py

read-the-docs-community · 2026-06-22T06:59:30Z

Documentation build overview

📚 imas-ibex | 🛠️ Build #33724519 | 📁 Comparing 353af26 against latest (9fe753c)

🔍 Preview build

136 files changed · + 86 added · ± 50 modified

+ Added

± Modified

…re/signal_data_operation # Conflicts: # backend/ibex/core/data_manipulation_methods.py # backend/ibex/data_source/imas_python_source.py # backend/ibex/data_source/imas_python_source_utils.py # backend/ibex/endpoints/schemas/request_data_schemas.py # backend/tests/conftest.py # backend/tests/test_data_endpoints.py # backend/tests/test_data_manipulation.py # docs/source/developers_manual/backend_development/data_manipulation.rst

jwasikpsnc · 2026-06-30T10:07:01Z

@olivhoenen

rebased onto develop
fixed bug in pyproject.toml which made pip download https://pypi.org/project/ibex/ instead of imas-ibex when installing pip install -e .[all]

prasad-sawantdesai · 2026-07-16T12:52:49Z

+            #    b. Otherwise use raw signal data directly (same shape path)
+            # 5. Build a flat uri->array dict and apply all signal operations
+
+            if plot_data_query.signal_operations:


After Square / Multiplication operation shown below, should unit of the quantity remains same?

$ curl -G http://127.0.0.1:8000/data/plot_data --data-urlencode "uri=$BASE" --data-urlencode "interpolate_over=$OTHER" --data-urlencode "interpolation_method=exact_value" --data-urlencode "signal_operations=mul:$OTHER" | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 7880 100 7880 0 0 769k 0 --:--:-- --:--:-- --:--:-- 769k { "data": { "name": "b0", "unit": "T", "shape": [ 300 ], "downsampled_shape": [ 300 ],

I am not sure concerning physical interpretation of mentioned operations. If we multiply signal1 x signal2 the unit should be unit^2 ?

I am not sure concerning physical interpretation of mentioned operations. If we multiply signal1 x signal2 the unit should be unit^2 ?

If signal1 and signal2 have the same unit, yes, this is correct. Combining units is only valid with your mul and div operators thought, with add and sub you shall make sure that units are the same or throw an error.
You may check this short video: https://www.youtube.com/watch?v=2DJjV3I4xGc

DONE. ad0a37b

add and sub you shall make sure that units are the same or throw an error. #90 (comment)

I think we are not checking if units are the same. and it did not throw exception.

[sawantp1@98dci4-srv-1005 frontend]$ echo $ANOTHER imas:hdf5?path=/work/imas/shared/imasdb/ITER/3/134174/117#equilibrium/time_slice[0]/profiles_1d/psi [sawantp1@98dci4-srv-1005 frontend]$ echo $ANOTHER1 imas:hdf5?path=/work/imas/shared/imasdb/ITER/3/134174/117#equilibrium/time_slice[0]/profiles_1d/pressure $ curl -G http://127.0.0.1:8000/data/plot_data --data-urlencode "uri=$ANOTHER1" --data-urlencode "signal_operations=add:$ANOTHER" | jq | head -n 20 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 13737 100 13737 0 0 353k 0 --:--:-- --:--:-- --:--:-- 353k { "data": { "name": "pressure", "unit": "Pa", "shape": [ 299 ], "downsampled_shape": [ 299 ], "ndim": 1, "path": "#equilibrium/time_slice[0]/profiles_1d/pressure", "description": "Pressure", "coordinates": [ { "name": "psi", "target": "#equilibrium/time_slice[0]/profiles_1d/pressure", "unit": "Wb", "shape": [ 299

prasad-sawantdesai · 2026-07-16T12:57:14Z

+                        if (
+                            other_signal["data"]["shape"] != "irregular"
+                            and other_signal["data"]["shape"] == original_data_shape
+                        ):


other_signal["data"]["shape"] == original_data_shape if shape is same but coordinate has different time stamps, it won't ask for interpolation message and still can go for operation.

DONE. c218ca2
Now, signal operation fails due to time coordinates incompatibilities. Other signals should be chosen...

c218ca2 Should we check other coordinates if available for example dim1, dim2 or rho_tor_norm? May be its good to check dynamically..

Now all coordinates are checked for their mutual compatibility. d213449
Two coordinate are equal only when:

Both signals contain the same number of coordinates.

Each coordinate appears in the same position.

Corresponding coordinate names match exactly, such as "rho" == "rho".

Corresponding values match in shape and content.

NaN values at corresponding positions count as equal.

bpalak · 2026-07-20T11:47:13Z

@prasad-sawantdesai : Could you provide me info about your test files (BASE and OTHER) ? I would like to reproduce your tests using the same data you are using.

prasad-sawantdesai · 2026-07-20T11:51:27Z

@prasad-sawantdesai : Could you provide me info about your test files (BASE and OTHER) ? I would like to reproduce your tests using the same data you are using.
Here it is --

$ echo $OTHER
imas:hdf5?path=/work/imas/shared/imasdb/ITER/3/134174/117#equilibrium/vacuum_toroidal_field/b0
$ echo $BASE
imas:hdf5?path=/work/imas/shared/imasdb/ITER/3/53299/2#equilibrium/vacuum_toroidal_field/b0

…ture/signal_data_operation # Conflicts: # backend/ibex/data_source/imas_python_source.py

…nto feature/signal_data_operation

prasad-sawantdesai · 2026-07-22T09:38:33Z

+                    )
+
+            # ============= END signal operations =============
+


Do you know what I am doing wrong here?

import imas import numpy as np entries = [ ("/tmp/ibex-grid-first", [0.0, 0.5, 1.0], [10.0, 20.0, 30.0]), ("/tmp/ibex-grid-second", [0.0, 0.4, 0.8], [100.0, 200.0, 300.0]), ] for path, rho, temperature in entries: with imas.DBEntry(f"imas:hdf5?path={path}", mode="w") as entry: cp = entry.factory.core_profiles() cp.ids_properties.homogeneous_time = 1 cp.time = np.array([1.0]) cp.profiles_1d.resize(1) cp.profiles_1d[0].grid.rho_tor_norm = np.array(rho) cp.profiles_1d[0].electrons.temperature = np.array(temperature) entry.put(cp) # $ echo $FIRST # imas:hdf5?path=/tmp/ibex-grid-first#core_profiles:0/profiles_1d[:]/electrons/temperature # $ echo $SECOND # imas:hdf5?path=/tmp/ibex-grid-second#core_profiles:0/profiles_1d[:]/electrons/temperature # $ curl -sS -G http://127.0.0.1:8000/data/plot_data --data-urlencode "uri=$FIRST" --data-urlencode "interpolate_over=$FIRST" --data-urlencode "interpolation_method=exact_value" --data-urlencode "signal_operations=add:$SECOND" | jq # { # "message": "'interpolated_data'" # }

I am on it...

prasad-sawantdesai · 2026-07-22T10:01:37Z

-            axis_indices.append(lookup[value])
-
+        axis_indices = np.searchsorted(np.asarray(target), np.asarray(orig))
        target_indices.append(axis_indices)


Could we verify that target[indices] == orig after searchsorted and reject coordinates missing from the target grid?

import imas import numpy as np entries = [ ("/tmp/ibex-grid-first", [0.0, 0.5, 1.0], [10.0, 20.0, 30.0]), ("/tmp/ibex-grid-second", [0.0, 0.4, 0.8], [100.0, 200.0, 300.0]), ] for path, rho, temperature in entries: with imas.DBEntry(f"imas:hdf5?path={path}", mode="w") as entry: cp = entry.factory.core_profiles() cp.ids_properties.homogeneous_time = 1 cp.time = np.array([1.0]) cp.profiles_1d.resize(1) cp.profiles_1d[0].grid.rho_tor_norm = np.array(rho) cp.profiles_1d[0].electrons.temperature = np.array(temperature) entry.put(cp) # This works correctly curl -sS -G http://127.0.0.1:8000/data/plot_data --data-urlencode "uri=$FIRST" --data-urlencode "interpolate_over=$SECOND" --data-urlencode "interpolation_method=exact_value" --data-urlencode "signal_operations=add:$SECOND" | jq '.data | {coordinates, value}' "value": [ [ 110.0, null, null, null, null ] ] # but this one gives wrong result seems. $SECOND coordinates 0.4 and 0.8 do not exist in $FIRST’s grid, should raise an error? curl -sS -G http://127.0.0.1:8000/data/plot_data --data-urlencode "uri=$FIRST" --data-urlencode "interpolate_over=$FIRST" --data-urlencode "interpolation_method=exact_value" --data-urlencode "signal_operations=sub:$FIRST" --data-urlencode "signal_operations=add:$SECOND" | jq '.data | {coordinates, value}' "value": [ [ 100.0, 200.0, 300.0 ] ]

jwasikpsnc added 20 commits June 2, 2026 13:55

Add simple data operations

59008f7

Add subtraction. Check parameters before operation.

c86a6f2

Add operation priorities

e67f0f1

Fix broken test

fc9c0b3

Apply linter

1c7e665

Signal operations (WIP)

85c3a2b

Basic operations order (WIP)

a105b66

Fix wrong description

f5da1cf

Update docstring and test.

6639d45

Merge branch 'refs/heads/feature/binary_data_operation' into feature/…

ca41656

…signal_data_operation # Conflicts: # backend/ibex/endpoints/schemas/request_data_schemas.py

Signal data operations (WIP)

d6b6227

Update docs. Mini-fixes.

d79fac5

Signal operations (WIP)

4e3cec7

Signal data operations (WIP)

25a4b1d

Delete unused validator

1455de1

Merge remote-tracking branch 'origin/feature/signal_data_operation' i…

3dc3aad

…nto feature/signal_data_operation # Conflicts: # backend/ibex/data_source/imas_python_source_utils.py

Fix broken tests.

ae27462

Signal operations

93fd74d

Update tests. Fix bug.

3b8f226

Delete unused operations

2499794

jwasikpsnc added 3 commits June 30, 2026 11:38

Apply linter. Fix bugs.

56d3040

Add validator for signal operations.

8cc64a4

jwasikpsnc marked this pull request as ready for review June 30, 2026 10:05

jwasikpsnc requested review from olivhoenen and prasad-sawantdesai as code owners June 30, 2026 10:05

mfiaudrin-epsyl mentioned this pull request Jul 9, 2026

Interface for data operations #101

Open

prasad-sawantdesai reviewed Jul 16, 2026

View reviewed changes

Comment thread backend/ibex/data_source/imas_python_source_utils.py Outdated

prasad-sawantdesai reviewed Jul 16, 2026

View reviewed changes

Comment thread backend/ibex/data_source/imas_python_source.py

Fix plot data shape after interpolation

3d05817

Bartek Palak and others added 7 commits July 20, 2026 14:10

Preserve NaN values in signal operations

e4f6f2c

Propagate units through signal operations

ad0a37b

Matching time coordinates for signal operations

c218ca2

Code refactoring

405950c

Merge remote-tracking branch 'refs/remotes/upstream/develop' into fea…

1c2d9b4

…ture/signal_data_operation # Conflicts: # backend/ibex/data_source/imas_python_source.py

Removing NaN handling section from signal operations documentation.

556cdb5

Merge remote-tracking branch 'origin/feature/signal_data_operation' i…

8787daf

…nto feature/signal_data_operation

prasad-sawantdesai reviewed Jul 22, 2026

View reviewed changes

Bartek Palak and others added 2 commits July 22, 2026 15:20

Signal operations: checking compatibility of all coordinates of operands

d213449

refactor: simplify signal operand data preparation

e9a9537

bpalak force-pushed the feature/signal_data_operation branch from 066171e to c20ea80 Compare July 23, 2026 11:49

Reject add and sub operations with mismatched units

353af26

bpalak force-pushed the feature/signal_data_operation branch from c20ea80 to 353af26 Compare July 23, 2026 11:53

Uh oh!

Conversation

jwasikpsnc commented Jun 22, 2026

Uh oh!

read-the-docs-community Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

jwasikpsnc commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bpalak commented Jul 20, 2026

Uh oh!

prasad-sawantdesai commented Jul 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prasad-sawantdesai Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

read-the-docs-community Bot commented Jun 22, 2026 •

edited

Loading

jwasikpsnc commented Jun 30, 2026 •

edited

Loading

prasad-sawantdesai Jul 22, 2026 •

edited

Loading