From 903452c9a3e5ecfde14a2977a6d651b1ce0ba889 Mon Sep 17 00:00:00 2001 From: VsevolodX Date: Fri, 21 Aug 2026 18:15:39 -0700 Subject: [PATCH 01/10] docs: tutorial for running a custom Python script on materials Covers the custom_python_calculation.ipynb notebook: how the script reads the material and its own data files, how dependencies are declared, how the run proceeds, what the four workflow units do, and how the saved workflow is reused from the Jobs Designer. Navigation entries added to mkdocs.yml and mkdocs-guide.yml. Post-build link check passes with no broken internal links. Co-Authored-By: Claude Opus 5 (1M context) --- .../other/custom-python-calculation.md | 202 ++++++++++++++++++ mkdocs-guide.yml | 1 + mkdocs.yml | 1 + 3 files changed, 204 insertions(+) create mode 100644 lang/en/docs/tutorials/other/custom-python-calculation.md diff --git a/lang/en/docs/tutorials/other/custom-python-calculation.md b/lang/en/docs/tutorials/other/custom-python-calculation.md new file mode 100644 index 000000000..c8e94d692 --- /dev/null +++ b/lang/en/docs/tutorials/other/custom-python-calculation.md @@ -0,0 +1,202 @@ +# Running a Custom Python Script on One or More Materials + +The platform can run an arbitrary Python script against materials from an +account's [collection]({{ reference_url }}/accounts/collections/), with no +simulation engine involved. The script, its dependencies and any data files it +reads are uploaded to the account's object storage folder; a +[workflow]({{ reference_url }}/workflows/overview/) fetches them onto the +compute node next to the material, and whatever the script prints comes back as +the result. + +This page describes the flow through the `custom_python_calculation.ipynb` +notebook, which performs the upload, assembles the workflow, and creates one +[job]({{ reference_url }}/jobs/overview/) per material. For a script that needs +neither a material from the collection nor data files of its own, the general +Python flavor/template described in +[Python MLFF](../ml/run-mlff-python-workflows-mattersim.md#3-using-the-general-python-template) +is sufficient on its own. + + +## 1. Prepare the script + +The script runs on the compute node in a working directory that holds the +material and the uploaded files side by side, so every path in it is relative. +Nothing in the script is specific to the platform: it reads files, computes, and +prints. + +### 1.1. Read the material + +The job's material is written to `material.json` before the script starts, in +the [representation]({{ reference_url }}/materials/overview/) the platform +stores it in — a `lattice` object holding the cell lengths and angles, and a +`basis` object holding the elements and their coordinates. A script that reports +the number of atoms and the cell volume reads it as follows: + +```python title="USER_SCRIPT" +import json +import math + +material = json.load(open("material.json")) + +lattice = material["lattice"] +a, b, c = lattice["a"], lattice["b"], lattice["c"] +cos_alpha, cos_beta, cos_gamma = (math.cos(math.radians(lattice[key])) for key in ("alpha", "beta", "gamma")) +volume = ( + a * b * c + * math.sqrt(1 - cos_alpha**2 - cos_beta**2 - cos_gamma**2 + 2 * cos_alpha * cos_beta * cos_gamma) +) + +elements = [element["value"] for element in material["basis"]["elements"]] + +print(json.dumps({"elements": sorted(set(elements)), "n_atoms": len(elements), "volume": round(volume, 4)})) +``` + +This is the default script in the notebook, and it serves as the starting point +for a custom one. + +### 1.2. Print the results + +The standard output of the script is the result. The notebook parses each line +that contains a JSON object and renders the parsed values as a table, one row +per material, so a script that prints a single JSON object per run needs no +further formatting. Lines that are not JSON are shown as printed, which makes +`print()` usable for progress messages and debugging. + +### 1.3. Add data files + +A script that reads its own data — a table of radii, a set of parameters, a +pseudopotential — takes those files from the `uploads` folder of the JupyterLite +session. The files are placed there first, by dragging them into the file +browser, and named in the `USER_ASSET_FILES` parameter. The notebook then +uploads them next to the script, and the workflow fetches them into the same +working directory, where the script opens them by name. + +!!!warning "Reserved file names" + The names `script.py`, `requirements.txt` and `material.json` are written by + the workflow itself. An upload under one of those names is refused, since it + would be overwritten before the script runs. + +### 1.4. Declare dependencies + +Packages the script imports are listed in the `USER_REQUIREMENTS` parameter, in +the form used by `requirements.txt` — for example `["numpy<2"]`. The list is +installed into a Python virtual environment on the compute node before the +script starts. An empty list is valid, and is what the default script uses, +since it imports nothing beyond the standard library. + +!!!info "Shared virtual environment" + Virtual environments are shared across jobs and users. As long as the + contents of the requirement list are unchanged, the same environment is + reused, so the first job pays for `pip install` and later ones start + immediately. Versions that need to be exact should be pinned explicitly. + + +## 2. Run the notebook + +The notebook lives in the JupyterLite session that ships with the platform, and +it runs entirely in the browser until the moment it submits the jobs. + +### 2.1. Open the notebook + +First, open a +[JupyterLite session]({{ interface_url }}/jupyterlite/accessing-jupyterlite/) +from the account page. Then, in the file browser, open `made`, then `workflows`, +then `custom_python_calculation.ipynb`. + +### 2.2. Set the parameters + +The parameters sit in the cell under *1.2. Set parameters and configurations for +the workflow and job*, and the script sits on its own in the cell under *1.3. +Set the script to run*, so the two are edited independently. + +`MATERIAL_NAMES` lists the materials, one job per entry. Each name is looked up +in the `uploads` folder first and in the standard materials set that ships with +the notebook environment as a fallback, then saved to the account's collection if +an identical material is not there already. +`USER_ASSET_FILES` and `USER_REQUIREMENTS` are described in Section 1 above. +`MY_WORKFLOW_NAME` names the workflow, and `save_to_collection` decides whether +it is kept. The cluster, queue and processor count are set further down the same +cell. + +### 2.3. Run all cells + +Click **Run** > **Run All**. The notebook authenticates in the browser, uploads +the script and the data files, assembles the workflow, creates one job per +material, submits them, and polls until each one finishes. A run against a small +material takes a few minutes, most of it spent waiting for the queue and, on the +first run with a given requirement list, for `pip install`. + +### 2.4. View the results + +The last cell prints the standard output of each job and renders the parsed JSON +values as a table. The same output is available in the +[Job Viewer]({{ interface_url }}/jobs/ui/viewer/): the *Workflow* tab exposes +the standard output of the script unit, and the +[*Files* tab]({{ interface_url }}/jobs/ui/files-tab/) lists the uploaded files, +the material, and everything the script wrote. + + +## 3. Anatomy of the workflow + +The notebook does not build a workflow from nothing. It takes the *Custom Python +Script* workflow from the standard workflow set and fills in the three things +that differ per run. The same workflow is held in the +[Workflows Bank]({{ reference_url }}/workflows/bank/), where it can be inspected +in the web interface. It consists of four +[units]({{ reference_url }}/workflows/components/units/): + +- **I/O unit, object storage:** fetches the uploaded script and data files into +the working directory. The notebook sets its input list from the upload it has +just performed. +- **I/O unit, material:** fetches the job's materials from the job context. +- **Assignment unit:** takes the first material and assigns it to a global +variable named `MATERIAL`. +- **Execution unit:** runs a short runner script that writes `MATERIAL` to +`material.json` and executes the uploaded script in the same process. The +notebook sets the runner and the requirement list on this unit. + +The runner is what keeps the user's script untouched: the script is fetched as a +file rather than embedded in the workflow, so text in it that resembles a +template placeholder is never rendered. + + +## 4. Reuse the saved workflow + +With `save_to_collection` left at its default, the workflow is saved to the +account's collection under the name given by `MY_WORKFLOW_NAME`. It points at +the uploaded files, so it can be selected in the +[Jobs Designer]({{ interface_url }}/jobs-designer/overview/) like any other +workflow and [run]({{ interface_url }}/jobs/actions/run/) against a different +material without opening the notebook again. + +Re-running the notebook overwrites the uploaded files in place, under the same +names in the same folder. An edited script therefore reaches a saved workflow +without any change to the workflow itself. + + +## 5. Limits + +An upload travels inside the request body, which the platform caps at 50 MB, and +the JupyterLite session holds the file contents in memory before sending them. +Anything larger is uploaded through the +[Dropbox page]({{ interface_url }}/data-in-objectstorage/ui/dropbox-page/) of +the web interface instead, into the same folder, and named in +`USER_ASSET_FILES` all the same. + +The script is given one material per job, as `material.json`. A calculation over +a set of materials at once is a different shape of workflow, and is not what this +notebook builds. + + +## 6. Links + +The pages below cover the platform features this notebook builds on. + +- [Dropbox]({{ resources_url }}/data-in-objectstorage/dropbox/) — the object + storage folder the files are uploaded to. +- [Python]({{ reference_url }}/software-directory/scripting/python/overview/) — + the application the execution unit uses. +- [JupyterLite]({{ interface_url }}/jupyterlite/overview/) — the in-browser + notebook environment. +- [api-examples](https://github.com/mat3ra/api-examples) — the repository the + notebook is maintained in. diff --git a/mkdocs-guide.yml b/mkdocs-guide.yml index ea40f040f..5d22fbf29 100644 --- a/mkdocs-guide.yml +++ b/mkdocs-guide.yml @@ -266,6 +266,7 @@ nav: - Magnetic Moment on Atoms by Specie: tutorials/templating/set-magnetic-moment.md - 3.3. Tools and Environments: - Accessing the Platform: tutorials/platform-access.md + - Custom Python Calculation: tutorials/other/custom-python-calculation.md - Jupyter Notebook: tutorials/other/jupyter.md - Restart from Previous Job: tutorials/other/restart-job.md - TensorFlow (GPU): tutorials/general-functionality/tensorflow-gpu.md diff --git a/mkdocs.yml b/mkdocs.yml index 17e81bb45..b3bbc9986 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -137,6 +137,7 @@ nav: - Structural Relaxation: tutorials/dft/addons/structural-relaxation.md - General Functionality: - Accessing the Platform: tutorials/platform-access.md + - Custom Python Calculation: tutorials/other/custom-python-calculation.md - Jupyter Notebook: tutorials/other/jupyter.md - Restart from Previous Job: tutorials/other/restart-job.md - TensorFlow (GPU): tutorials/general-functionality/tensorflow-gpu.md From e8f893332448c8f553367fd39f487c8481499505 Mon Sep 17 00:00:00 2001 From: VsevolodX Date: Fri, 21 Aug 2026 18:28:55 -0700 Subject: [PATCH 02/10] docs: match the example script to the notebook default Co-Authored-By: Claude Opus 5 (1M context) --- .../tutorials/other/custom-python-calculation.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/lang/en/docs/tutorials/other/custom-python-calculation.md b/lang/en/docs/tutorials/other/custom-python-calculation.md index c8e94d692..623738170 100644 --- a/lang/en/docs/tutorials/other/custom-python-calculation.md +++ b/lang/en/docs/tutorials/other/custom-python-calculation.md @@ -38,17 +38,25 @@ import math material = json.load(open("material.json")) +# The platform stores the unit cell as lengths and angles, so the volume comes from those. lattice = material["lattice"] a, b, c = lattice["a"], lattice["b"], lattice["c"] cos_alpha, cos_beta, cos_gamma = (math.cos(math.radians(lattice[key])) for key in ("alpha", "beta", "gamma")) volume = ( - a * b * c + a + * b + * c * math.sqrt(1 - cos_alpha**2 - cos_beta**2 - cos_gamma**2 + 2 * cos_alpha * cos_beta * cos_gamma) ) elements = [element["value"] for element in material["basis"]["elements"]] -print(json.dumps({"elements": sorted(set(elements)), "n_atoms": len(elements), "volume": round(volume, 4)})) +print(json.dumps({ + "elements": sorted(set(elements)), + "n_atoms": len(elements), + "volume": round(volume, 4), + "atoms_per_volume": round(len(elements) / volume, 4), +})) ``` This is the default script in the notebook, and it serves as the starting point @@ -57,7 +65,7 @@ for a custom one. ### 1.2. Print the results The standard output of the script is the result. The notebook parses each line -that contains a JSON object and renders the parsed values as a table, one row +that is a JSON object and renders the parsed values as a table, one row per material, so a script that prints a single JSON object per run needs no further formatting. Lines that are not JSON are shown as printed, which makes `print()` usable for progress messages and debugging. From bc7d1074de7bd1d98616da13e82c2f29b33f3603 Mon Sep 17 00:00:00 2001 From: VsevolodX Date: Fri, 21 Aug 2026 18:37:48 -0700 Subject: [PATCH 03/10] docs: state the text-only upload contract and the route for other files The notebook uploads UTF-8 text, since an upload travels as a string in a JSON body. Section 1.3 says so, and Section 5 no longer implies that a file placed in the object storage folder by hand is covered by USER_ASSET_FILES - it is not, and the notebook's upload cell carries the lines that add such an object instead. Co-Authored-By: Claude Opus 5 (1M context) --- .../tutorials/other/custom-python-calculation.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/lang/en/docs/tutorials/other/custom-python-calculation.md b/lang/en/docs/tutorials/other/custom-python-calculation.md index 623738170..b9127044e 100644 --- a/lang/en/docs/tutorials/other/custom-python-calculation.md +++ b/lang/en/docs/tutorials/other/custom-python-calculation.md @@ -79,6 +79,10 @@ browser, and named in the `USER_ASSET_FILES` parameter. The notebook then uploads them next to the script, and the workflow fetches them into the same working directory, where the script opens them by name. +An upload travels as a string inside a JSON request body, so the files named +here are UTF-8 text. The notebook refuses anything else with a message pointing +at the route described in Section 5 below, which is where binary data belongs. + !!!warning "Reserved file names" The names `script.py`, `requirements.txt` and `material.json` are written by the workflow itself. An upload under one of those names is refused, since it @@ -186,10 +190,13 @@ without any change to the workflow itself. An upload travels inside the request body, which the platform caps at 50 MB, and the JupyterLite session holds the file contents in memory before sending them. -Anything larger is uploaded through the +Anything larger, and anything that is not UTF-8 text, goes to the same folder +through the [Dropbox page]({{ interface_url }}/data-in-objectstorage/ui/dropbox-page/) of -the web interface instead, into the same folder, and named in -`USER_ASSET_FILES` all the same. +the web interface instead. Such a file is not covered by `USER_ASSET_FILES`, +which names local files for the notebook to upload; the notebook's upload cell +carries the two lines that add an already-stored object to the list the workflow +fetches. The script is given one material per job, as `material.json`. A calculation over a set of materials at once is a different shape of workflow, and is not what this From 90c915d13beac41657a0f061add44c1b3ed0c9a2 Mon Sep 17 00:00:00 2001 From: VsevolodX Date: Mon, 24 Aug 2026 09:39:25 -0700 Subject: [PATCH 04/10] docs: section for the pseudopotential example inside the notebook An upload feeds any application: the notebook's closing example runs a QE band structure with an uploaded Si.upf. Limits and Links renumbered. Co-Authored-By: Claude Opus 5 (1M context) --- .../other/custom-python-calculation.md | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/lang/en/docs/tutorials/other/custom-python-calculation.md b/lang/en/docs/tutorials/other/custom-python-calculation.md index b9127044e..977a32dc8 100644 --- a/lang/en/docs/tutorials/other/custom-python-calculation.md +++ b/lang/en/docs/tutorials/other/custom-python-calculation.md @@ -186,7 +186,20 @@ names in the same folder. An edited script therefore reaches a saved workflow without any change to the workflow itself. -## 5. Limits +## 5. Feed an uploaded file to another application + +An upload is an ordinary platform file, so it is not limited to the custom script's workflow — +any workflow can fetch it. The notebook closes with a worked example: a pseudopotential +(`Si.upf` ships in the `uploads` folder) goes up through the same upload call, an `io` unit at +the head of the standard Quantum ESPRESSO band structure workflow downloads it into the job's +`pseudo` directory — where `pseudo_dir` in the inputs points — and the `ATOMIC_SPECIES` card of +each pw.x input is pointed at the file. The final cells print the cards the calculation consumed +and plot the band structure. The same upload can also be registered as a first-class +pseudopotential through the web interface, as described in +[Upload a custom pseudopotential](../dft/upload-pseudopotential.md). + + +## 6. Limits An upload travels inside the request body, which the platform caps at 50 MB, and the JupyterLite session holds the file contents in memory before sending them. @@ -203,7 +216,7 @@ a set of materials at once is a different shape of workflow, and is not what thi notebook builds. -## 6. Links +## 7. Links The pages below cover the platform features this notebook builds on. From 0242f1485d69f14d7b2b60f05c06fa622ed25a21 Mon Sep 17 00:00:00 2001 From: VsevolodX Date: Mon, 24 Aug 2026 14:29:26 -0700 Subject: [PATCH 05/10] docs: point the example section at the shell twin notebook The QE-with-uploaded-pseudo example runs through custom_shell_calculation.ipynb now; section 5 describes that flow instead of the in-notebook workflow surgery. Co-Authored-By: Claude Opus 5 (1M context) --- .../other/custom-python-calculation.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/lang/en/docs/tutorials/other/custom-python-calculation.md b/lang/en/docs/tutorials/other/custom-python-calculation.md index 977a32dc8..30b83c482 100644 --- a/lang/en/docs/tutorials/other/custom-python-calculation.md +++ b/lang/en/docs/tutorials/other/custom-python-calculation.md @@ -186,16 +186,16 @@ names in the same folder. An edited script therefore reaches a saved workflow without any change to the workflow itself. -## 5. Feed an uploaded file to another application - -An upload is an ordinary platform file, so it is not limited to the custom script's workflow — -any workflow can fetch it. The notebook closes with a worked example: a pseudopotential -(`Si.upf` ships in the `uploads` folder) goes up through the same upload call, an `io` unit at -the head of the standard Quantum ESPRESSO band structure workflow downloads it into the job's -`pseudo` directory — where `pseudo_dir` in the inputs points — and the `ATOMIC_SPECIES` card of -each pw.x input is pointed at the file. The final cells print the cards the calculation consumed -and plot the band structure. The same upload can also be registered as a first-class -pseudopotential through the web interface, as described in +## 5. The shell twin: run any node-side application + +A sibling notebook, `custom_shell_calculation.ipynb`, carries the same flow with a shell script +in place of the Python one. A shell script runs in a `module`-capable shell, so it can invoke any +application installed on the compute node. Its default example uploads a pseudopotential +(`Si.upf` ships in the `uploads` folder) through the same upload call, builds Quantum ESPRESSO +inputs from `material.json` with `pseudo_dir` set to the working directory, and runs an SCF and a +band structure step — Quantum ESPRESSO's own log then names the uploaded file as the +pseudopotential it read. The same upload can also be registered as a first-class pseudopotential +through the web interface, as described in [Upload a custom pseudopotential](../dft/upload-pseudopotential.md). From 1fdfb63e21b86daac61ce78346758c406f18a939 Mon Sep 17 00:00:00 2001 From: VsevolodX Date: Thu, 27 Aug 2026 09:59:51 -0700 Subject: [PATCH 06/10] docs: match the four-line default script Co-Authored-By: Claude Opus 5 (1M context) --- .../other/custom-python-calculation.md | 22 ++----------------- 1 file changed, 2 insertions(+), 20 deletions(-) diff --git a/lang/en/docs/tutorials/other/custom-python-calculation.md b/lang/en/docs/tutorials/other/custom-python-calculation.md index 30b83c482..fe1dfa884 100644 --- a/lang/en/docs/tutorials/other/custom-python-calculation.md +++ b/lang/en/docs/tutorials/other/custom-python-calculation.md @@ -30,33 +30,15 @@ The job's material is written to `material.json` before the script starts, in the [representation]({{ reference_url }}/materials/overview/) the platform stores it in — a `lattice` object holding the cell lengths and angles, and a `basis` object holding the elements and their coordinates. A script that reports -the number of atoms and the cell volume reads it as follows: +the number of atoms and the elements present reads it as follows: ```python title="USER_SCRIPT" import json -import math material = json.load(open("material.json")) - -# The platform stores the unit cell as lengths and angles, so the volume comes from those. -lattice = material["lattice"] -a, b, c = lattice["a"], lattice["b"], lattice["c"] -cos_alpha, cos_beta, cos_gamma = (math.cos(math.radians(lattice[key])) for key in ("alpha", "beta", "gamma")) -volume = ( - a - * b - * c - * math.sqrt(1 - cos_alpha**2 - cos_beta**2 - cos_gamma**2 + 2 * cos_alpha * cos_beta * cos_gamma) -) - elements = [element["value"] for element in material["basis"]["elements"]] -print(json.dumps({ - "elements": sorted(set(elements)), - "n_atoms": len(elements), - "volume": round(volume, 4), - "atoms_per_volume": round(len(elements) / volume, 4), -})) +print(json.dumps({"n_atoms": len(elements), "elements": sorted(set(elements))})) ``` This is the default script in the notebook, and it serves as the starting point From 1521ffee879cdc052dc9d72ca698276a366eb53c Mon Sep 17 00:00:00 2001 From: VsevolodX Date: Thu, 27 Aug 2026 19:25:01 -0700 Subject: [PATCH 07/10] docs: the shell example reads the platform pseudo library; nothing ships in uploads Co-Authored-By: Claude Opus 5 (1M context) --- .../tutorials/other/custom-python-calculation.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/lang/en/docs/tutorials/other/custom-python-calculation.md b/lang/en/docs/tutorials/other/custom-python-calculation.md index fe1dfa884..cd2e5f0b2 100644 --- a/lang/en/docs/tutorials/other/custom-python-calculation.md +++ b/lang/en/docs/tutorials/other/custom-python-calculation.md @@ -172,12 +172,13 @@ without any change to the workflow itself. A sibling notebook, `custom_shell_calculation.ipynb`, carries the same flow with a shell script in place of the Python one. A shell script runs in a `module`-capable shell, so it can invoke any -application installed on the compute node. Its default example uploads a pseudopotential -(`Si.upf` ships in the `uploads` folder) through the same upload call, builds Quantum ESPRESSO -inputs from `material.json` with `pseudo_dir` set to the working directory, and runs an SCF and a -band structure step — Quantum ESPRESSO's own log then names the uploaded file as the -pseudopotential it read. The same upload can also be registered as a first-class pseudopotential -through the web interface, as described in +application installed on the compute node. Its default example builds Quantum ESPRESSO inputs +from `material.json` and runs an SCF and a band structure step for silicon with a pseudopotential +from the platform's library. A pseudopotential of the user's own goes up through the same upload +call as any data file — placed in the `uploads` folder, listed in `USER_ASSET_FILES`, with the +script's `PSEUDO_DIR` pointed at the working directory — and Quantum ESPRESSO's own log then names +the uploaded file as the one it read. Such a file can also be registered as a first-class +pseudopotential through the web interface, as described in [Upload a custom pseudopotential](../dft/upload-pseudopotential.md). From 441c0a49726a116e13cc3b875025b4e02ddeb0de Mon Sep 17 00:00:00 2001 From: VsevolodX Date: Thu, 27 Aug 2026 20:38:13 -0700 Subject: [PATCH 08/10] docs: binary and large files go to storage through a signed URL Co-Authored-By: Claude Opus 5 (1M context) --- .../tutorials/other/custom-python-calculation.md | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/lang/en/docs/tutorials/other/custom-python-calculation.md b/lang/en/docs/tutorials/other/custom-python-calculation.md index cd2e5f0b2..4c553d02a 100644 --- a/lang/en/docs/tutorials/other/custom-python-calculation.md +++ b/lang/en/docs/tutorials/other/custom-python-calculation.md @@ -184,15 +184,10 @@ pseudopotential through the web interface, as described in ## 6. Limits -An upload travels inside the request body, which the platform caps at 50 MB, and -the JupyterLite session holds the file contents in memory before sending them. -Anything larger, and anything that is not UTF-8 text, goes to the same folder -through the -[Dropbox page]({{ interface_url }}/data-in-objectstorage/ui/dropbox-page/) of -the web interface instead. Such a file is not covered by `USER_ASSET_FILES`, -which names local files for the notebook to upload; the notebook's upload cell -carries the two lines that add an already-stored object to the list the workflow -fetches. +Text files travel inside the request body. Anything else — a model checkpoint, an archive, a file +too large for a request — is sent to a URL the platform signs for it and lands in storage directly, +so the only bound is what the browser can hold in memory while uploading. The `uploads` folder of +the JupyterLite session is the staging area in both cases. The script is given one material per job, as `material.json`. A calculation over a set of materials at once is a different shape of workflow, and is not what this From 7cc33bde8260669bf374030e6b55219cbc28c7ad Mon Sep 17 00:00:00 2001 From: VsevolodX Date: Fri, 28 Aug 2026 09:38:48 -0700 Subject: [PATCH 09/10] docs: explain the upload size limit and the direct-upload route Co-Authored-By: Claude Opus 5 (1M context) --- .../tutorials/other/custom-python-calculation.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/lang/en/docs/tutorials/other/custom-python-calculation.md b/lang/en/docs/tutorials/other/custom-python-calculation.md index 4c553d02a..2c926fb52 100644 --- a/lang/en/docs/tutorials/other/custom-python-calculation.md +++ b/lang/en/docs/tutorials/other/custom-python-calculation.md @@ -182,12 +182,14 @@ pseudopotential through the web interface, as described in [Upload a custom pseudopotential](../dft/upload-pseudopotential.md). -## 6. Limits - -Text files travel inside the request body. Anything else — a model checkpoint, an archive, a file -too large for a request — is sent to a URL the platform signs for it and lands in storage directly, -so the only bound is what the browser can hold in memory while uploading. The `uploads` folder of -the JupyterLite session is the staging area in both cases. +## 6. File size + +A file uploaded from the notebook or the web interface travels inside the request that carries it, +encoded as text, so its size is bounded: roughly 75 MB through the web interface and 37 MB through +the API. Anything larger is not sent through the platform at all — the platform signs a short-lived +upload URL and the file goes straight to object storage, with no size limit. The notebook chooses +between the two, so a script, a pseudopotential and a several-hundred-megabyte model checkpoint are +all listed the same way in `USER_ASSET_FILES`. The script is given one material per job, as `material.json`. A calculation over a set of materials at once is a different shape of workflow, and is not what this From cea93d970366c6e9b775673bdbb4060700305352 Mon Sep 17 00:00:00 2001 From: VsevolodX Date: Fri, 28 Aug 2026 12:02:15 -0700 Subject: [PATCH 10/10] docs: data files are no longer text-only Review: the page still said an asset must be UTF-8 text and that the notebook refuses anything else, pointing at a Section 5 that is now the shell notebook. Type stopped mattering when binary uploads landed; size is what decides the route, and that is section 6. Co-Authored-By: Claude Opus 5 (1M context) --- lang/en/docs/tutorials/other/custom-python-calculation.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lang/en/docs/tutorials/other/custom-python-calculation.md b/lang/en/docs/tutorials/other/custom-python-calculation.md index 2c926fb52..de36ae0a9 100644 --- a/lang/en/docs/tutorials/other/custom-python-calculation.md +++ b/lang/en/docs/tutorials/other/custom-python-calculation.md @@ -61,9 +61,9 @@ browser, and named in the `USER_ASSET_FILES` parameter. The notebook then uploads them next to the script, and the workflow fetches them into the same working directory, where the script opens them by name. -An upload travels as a string inside a JSON request body, so the files named -here are UTF-8 text. The notebook refuses anything else with a message pointing -at the route described in Section 5 below, which is where binary data belongs. +A data file may be of any type — a table, a pseudopotential, a model checkpoint. +Its size is what decides the route it takes, which +[Section 6](#6-file-size) covers. !!!warning "Reserved file names" The names `script.py`, `requirements.txt` and `material.json` are written by @@ -186,7 +186,7 @@ pseudopotential through the web interface, as described in A file uploaded from the notebook or the web interface travels inside the request that carries it, encoded as text, so its size is bounded: roughly 75 MB through the web interface and 37 MB through -the API. Anything larger is not sent through the platform at all — the platform signs a short-lived +the API — the same bound for a text file and a binary one. Anything larger is not sent through the platform at all — the platform signs a short-lived upload URL and the file goes straight to object storage, with no size limit. The notebook chooses between the two, so a script, a pseudopotential and a several-hundred-megabyte model checkpoint are all listed the same way in `USER_ASSET_FILES`.