Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
95bdd1a
adding beegfs globus endpoint to config.yaml for bl832
davramov Apr 21, 2026
5f54a3b
updating move.py to copy to beegfs
davramov Apr 21, 2026
8c0b727
adding tiled[client] to requirements
davramov Apr 21, 2026
9a1bf3e
adding beegfs globus endpoint to config.yaml for bl832
davramov Apr 21, 2026
03cd05e
adding orchestration/tiled.py for ingesting data on beegfs
davramov Apr 21, 2026
04c25c3
cleaning up
davramov Apr 21, 2026
104a134
Adding todo note
davramov Apr 21, 2026
a940ddd
fixing bad import statement
davramov Apr 21, 2026
0e70447
adding beegfs endpoint to unit tests
davramov Apr 21, 2026
2bbe6d1
pointing beegfs root_path to the correct directory
davramov Apr 24, 2026
8792ea6
adding tags to ingested data
davramov Apr 24, 2026
12b3def
removing TILED_API_KEY from tiled.py
davramov May 4, 2026
8fb836c
removing check tags logic from main, moving it to a new method (check…
davramov May 4, 2026
669dbbe
Removing requirements.txt, moving any missing deps to pyproject.toml,…
davramov May 4, 2026
a76742a
moving load_dotenv() to top of the module
davramov May 5, 2026
402a34d
moving _apply_tags() into a prefect task, logger.info -> logger.debug…
davramov May 5, 2026
e270fae
Adding register_file_to_tiled() call to move.py
davramov May 6, 2026
d3b8528
Adding calls to register_file_to_tiled() from nersc_recon_flow() for …
davramov May 6, 2026
308b167
Adding TILED_URI to .env.example
davramov May 6, 2026
eb52e4b
Adding BEAMLINE=<> to .env.example
davramov May 6, 2026
a42b959
Making register_file_to_tiled() sync to fix async/sync nesting. Examp…
davramov May 6, 2026
1be3639
adding beegfs copy to nersc.py
davramov May 6, 2026
819308e
create unit tests in test_tiled.py
davramov May 6, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
BEAMLINE=<832>
GLOBUS_CLIENT_ID=<globus_client_id>
GLOBUS_CLIENT_SECRET=<globus_client_secret>
PREFECT_API_URL=<url_of_prefect_server>
PREFECT_API_KEY=<prefect_client_secret>
PUSHGATEWAY_URL=<url_of_pushgateway_server>
JOB_NAME=<jobname_for_pushgateway>
INSTANCE_LABEL=<label_for_pushgateway>
INSTANCE_LABEL=<label_for_pushgateway>
TILED_URI=<url_of_tiled_server>
6 changes: 2 additions & 4 deletions .github/workflows/python-app.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,11 @@ jobs:
with:
python-version: 3.11
cache: 'pip'
cache-dependency-path: pyproject.toml
- name: Install dependencies
run: |
python -m pip install --no-cache-dir --upgrade pip
pip install --no-cache-dir flake8 pytest
pip install --no-cache-dir .
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
if [ -f requirements-dev.txt ]; then pip install -r requirements-dev.txt; fi
pip install -e . --group dev
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
Expand Down
6 changes: 6 additions & 0 deletions config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,12 @@ globus:
uuid: df82346e-9a15-11ea-b3c4-0ae144191ee3
name: nersc832

bl832-beegfs-raw:
root_path: /global/beegfs/beamlines/bl832/raw/
uri: beegfs.als.lbl.gov
uuid: d33b5d6e-1603-414e-93cb-bcb732b7914a
name: bl832-beegfs-raw

globus_apps:
als_transfer:
client_id: ${GLOBUS_CLIENT_ID}
Expand Down
3 changes: 3 additions & 0 deletions orchestration/_tests/test_globus_flow.py
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ def __init__(self) -> None:
MockSecret.for_endpoint("nersc832_alsdev_scratch")),
"alcf832_raw": MockEndpoint("mock_alcf832_raw_path", MockSecret.for_endpoint("alcf832_raw")),
"alcf832_scratch": MockEndpoint("mock_alcf832_scratch_path", MockSecret.for_endpoint("alcf832_scratch")),
"beegfs_raw": MockEndpoint("mock_beegfs_raw_path", MockSecret.for_endpoint("beegfs_raw"))
}

# Mock apps
Expand All @@ -169,6 +170,7 @@ def __init__(self) -> None:
self.data832_raw = self.endpoints["data832_raw"]
self.data832_scratch = self.endpoints["data832_scratch"]
self.nersc832_alsdev_scratch = self.endpoints["nersc832_alsdev_scratch"]
self.beegfs_raw = self.endpoints["beegfs_raw"]
self.scicat = config["scicat"]


Expand Down Expand Up @@ -250,6 +252,7 @@ def test_alcf_recon_flow(mocker: MockFixture):
"nersc832_alsdev_recon_scripts": mocker.MagicMock(),
"alcf832_raw": mocker.MagicMock(),
"alcf832_scratch": mocker.MagicMock(),
"bl832-beegfs-raw": mocker.MagicMock(),
}
)
mocker.patch(
Expand Down
Loading