Skip to content

fix: key generated files on the notebook, not a default pipeline name - #960

Open
Ya-shh wants to merge 1 commit into
kubeflow:mainfrom
Ya-shh:fix/module-names-per-notebook
Open

fix: key generated files on the notebook, not a default pipeline name#960
Ya-shh wants to merge 1 commit into
kubeflow:mainfrom
Ya-shh:fix/module-names-per-notebook

Conversation

@Ya-shh

@Ya-shh Ya-shh commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Fixes #942.

Two composition notebooks compiled in one directory with kale --nb overwrote each other's generated files.

Cause

Wider than composition. kale --nb documents that an explicit CLI argument beats the notebook's metadata:

If the same argument (e.g. pipeline_name) is provided both in the Notebook metadata and from CLI, the CLI parameter will take precedence.

But --pipeline_name carried default="kale-pipeline". The overrides dict is built from every argument that is not None, so the default was indistinguishable from a value the user typed, and the notebook's own pipeline_name could never win. Every notebook compiled on the CLI was called kale-pipeline, which is why the issue sees notebook-sequence and notebook-sequence-mixed ignored.

For a single notebook that meant two files were overwritten. A composition generates one module per referenced notebook, so it meant five.

Fix

  • --pipeline_name no longer has a default, restoring the documented precedence.
  • A notebook that declares no pipeline name is named after its file, so removing the default cannot turn into a validation failure. One notebook in the repo is in that position (examples/taxi-cab-classification/read_data.ipynb).
  • _module_name keys on the root notebook's file rather than its pipeline name. File names are necessarily unique within a directory; pipeline names are not, so with only the CLI fix two notebooks sharing a name would still collide.

Result

$ kale --nb main.ipynb
$ ls .kale/
kale_notebook_main_notebook_a.py  notebook-sequence.kale.py
kale_notebook_main_notebook_b.py  notebook-sequence.pipeline.yaml
kale_notebook_main_notebook_c.py

$ kale --nb main_mixed.ipynb
$ ls .kale/
kale_notebook_main_mixed_notebook_a.py  notebook-sequence-mixed.kale.py
kale_notebook_main_mixed_notebook_b.py  notebook-sequence-mixed.pipeline.yaml
kale_notebook_main_mixed_notebook_c.py  (plus main's, untouched)

Both notebooks now compile under the names their metadata declares, which the issue called out as being ignored.

Note on behaviour change

kale --nb some_notebook.ipynb no longer writes kale-pipeline.kale.py; it writes a file named after the notebook's pipeline. That is the documented behaviour rather than new behaviour, but it is a visible change for anyone who relied on the fixed name. --pipeline_name still overrides, unchanged.

Testing

@google-oss-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign ederign for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[backend] Generated sub-pipeline modules collide across notebooks on the kale --nb path

1 participant