feat(recipes): render value-aware Kustomize templates

[sttts]

Summary

• Add value-aware Kustomize Component templates selected explicitly with source and an overlay-relative path.
• Render templates with strict sandboxed Jinja, a generic resource index from the built Base (base.<kind>[name]), and template defaults overridden by matrix values.
• Reuse template-family files through a narrow convention: direct YAML files in the selected directory replace same-output-name files from its direct parent.
• Materialize each template as a normal checked-in local Component under kustomize/overlays/<variant>/components/<name>/, so users need neither Python nor Jinja.
• Replace the fixed aws-efa-p8d8, aws-efa-p8d16, and aws-efa-p16d16 Components with AWS instance-type selections. P5 adds its instance-affinity patch while P6 uses the family defaults.
• Select the existing disagg-workers/aws-efa Component directly and remove the redundant components/aws-efa wrapper.

Closes #13109.

Template model

A recipe matrix selects a concrete source and the public Component path:

templates:
  - source: ../../../kustomize/templates/aws-efa/p5.48xlarge
    path: components/efa

The selected directory and exactly its direct parent form the template bundle:

recipes/kustomize/templates/aws-efa/
├── kustomization.yaml.j2
├── dgd-worker-efa-resources-patch.yaml.j2
├── p5.48xlarge/
│   ├── kustomization.yaml.j2
│   └── values.yaml
└── p6-b200.48xlarge/
    └── values.yaml

Resolution is deterministic:

1. Direct parent *.yaml and *.yaml.j2 files provide defaults.
2. Direct selected-directory files replace defaults with the same rendered output name.
3. Removing one trailing .j2 determines that output name. Child x.yaml therefore replaces parent x.yaml.j2, and child x.yaml.j2 replaces parent x.yaml.
4. Defining both forms in one directory is an error.
5. values.yaml mappings are merged parent, selected directory, then matrix; later values win.
6. Grandparents, subdirectories, sibling templates, and non-YAML assets are not scanned.

There is no .kustomize-template.yaml, recursive inheritance, inferred Kustomize layer, or Python merge of Kustomize YAML. A template needing layers declares normal Kustomize components, resources, or patches in its files.

Both static kustomization.yaml and rendered kustomization.yaml.j2 are accepted. The result must be one normal Kustomize Component. External Kustomize paths are rebased from the source file that declared them.

unfold materializes the P5 selection as a flat, hand-written-looking Component:

kustomize/overlays/aws-p5.48xlarge/components/efa/
├── kustomization.yaml
└── dgd-worker-efa-resources-patch.yaml

P5 replaces the family Kustomization to add instance affinity and retains the inherited EFA resource patch. P6 is values-only and inherits both family YAML files. The P5 final manifest matches Erez's proposed recipe output; P6 remains unchanged.

Template context

Before rendering, the generator runs kustomize build for the matrix source. Every resulting resource is available by lowercase kind and name:

{% set dgd = base.dynamographdeployment | only %}
{{ base.configmap[values.CONFIG_NAME].data.model }}

The generic only filter requires exactly one resource of a kind. Templates can use arithmetic, loops, and conditionals against base and values; StrictUndefined turns missing inputs into generation errors. No provider-, DGD-, or worker-specific values are synthesized in Python.

Generated YAML files include a contributor warning and runnable regeneration command. Final deploy-*.yaml files preserve ordinary Base comments at their corresponding positions while omitting nested generated-file metadata comments.

User workflow

Users consume the checked-in artifacts directly. They do not run unfold or render:

kubectl apply -f deploy-<name>.yaml
kubectl apply -k kustomize/overlays/<name>
kustomize build kustomize/overlays/<name>

They can also write their own kustomization.yaml from the checked-in Base and Components, or compose an ad hoc overlay with:

scripts/kustomize-matrix.py compose <base> <component>...

Contributor workflow

Contributors edit the matrix, Base, static Components, and template sources. From the repository root, regenerate checked-in public overlays first and rendered manifests second:

scripts/kustomize-matrix.py unfold <matrix.yaml>
scripts/kustomize-matrix.py render <matrix.yaml>

unfold --check and the repository-wide check command compare expected output without overwriting contributor files. unfold --clean removes obsolete generated Components after template paths or variants move.

The Recipe Check CI job runs:

scripts/kustomize-matrix.py check

It validates generated template Components, public overlays, rendered manifests, stale artifacts, and the central OpenAPI schema generated from the operator CRDs.

Validation

• pre-commit run --files <changed files>
• uv run --no-project --with 'pytest>=8.0' --with 'jinja2==3.1.6' --with 'pyyaml>=6.0' --with 'pydantic>=2' --with 'pytest-asyncio>=1.0' --with 'pytest-benchmark>=4' python3 -m pytest --noconftest tests/test_kustomize_matrix.py tests/test_generate_kustomize_openapi.py (15 passed)
• scripts/kustomize-matrix.py check
• P5 rendered manifest compared byte-for-byte with Erez's proposed recipe output
• P6 rendered manifest compared with the pre-change PR output

[coderabbitai]

Walkthrough

Changes

The Kustomize matrix generator now supports value-aware Jinja templates, strict sandboxed rendering, base-resource lookup, generated Component materialization, stale-file cleanup, and validation. AWS EFA recipe overlays now use shared instance-type templates and generated Components.

Kustomize template generation

Layer / File(s)	Summary
Matrix template and value contracts scripts/kustomize-matrix.py, recipes/AGENTS.md, recipes/CONTRIBUTING.md, .github/workflows/pre-merge.yml	Matrix variants accept templates and values. Validation rejects duplicate templates and conflicting keys. Jinja2 is installed for recipe checks.
Base indexing and Jinja rendering scripts/kustomize-matrix.py	The generator indexes Kustomize resources and renders Components and assets with strict, immutable sandboxing and the only filter.
Generated Component lifecycle and tests scripts/kustomize-matrix.py, tests/test_kustomize_matrix.py	unfold generates, compares, writes, and removes template Components and assets. Tests cover rendering, Kustomize builds, stale checks, ambiguous resources, and undefined values.
AWS EFA template migration recipes/kustomize/templates/*, recipes/qwen3-235b-a22b-fp8/..., recipes/qwen3-32b/...	AWS EFA variants use shared instance-type templates. Generated Components set EFA resources from worker GPU counts. Generated-file guidance and source documentation were updated.

Estimated code review effort: 4 (Complex) | ~60 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The implementation, documentation, CI updates, generated artifacts, and tests address the requirements in issue [#13109].
Out of Scope Changes check	✅ Passed	The code, tests, documentation, CI dependency, and AWS EFA migrations are related to the linked issue objectives.
Description check	✅ Passed	The description covers the template model, implementation details, workflows, linked issue, and validation steps in sufficient detail.
Title check	✅ Passed	The title clearly summarizes the main change: rendering Kustomize templates that use matrix values.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@recipes/qwen3-235b-a22b-fp8/README.md`:
- Around line 66-70: Update the contributor regeneration instruction near the
existing `unfold` and `render` guidance to explicitly include shared template
sources alongside recipe-local Components and shared Components. Ensure
contributors are told to run both regeneration steps after editing a template,
while preserving the existing generated-artifact editing guidance.

In `@recipes/qwen3-32b/vllm/cloud-providers/README.md`:
- Around line 69-73: Update the regeneration commands in the README to reference
recipes/qwen3-32b/vllm/cloud-providers/.kustomize-matrix.yaml instead of
.kustomize-matrix.yaml for both unfold and render, while preserving the existing
command structure.

In `@tests/test_kustomize_matrix.py`:
- Around line 187-189: Add a pytest timeout marker to
test_unfold_materializes_template_component_with_base_and_variant_values, using
the repository’s established timeout-marker convention and a suitable limit for
its two run_matrix subprocesses and kustomize build.
- Around line 283-289: Update the subprocess invocation in the test build to use
the existing kustomize_command() helper instead of the bare "kustomize"
executable, while preserving the current overlay path, output capture, and
return-code assertion.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0433d04d-c43f-4746-b656-e2758ab0470e

📥 Commits

Reviewing files that changed from the base of the PR and between c67b0f1 and 37fd011.

📒 Files selected for processing (31)

• .github/workflows/pre-merge.yml
• recipes/AGENTS.md
• recipes/CONTRIBUTING.md
• recipes/kustomize/components/aws-efa-p16d16/kustomization.yaml
• recipes/kustomize/components/aws-efa-p16d16/patch-dgd.yaml
• recipes/kustomize/components/aws-efa-p8d16/kustomization.yaml
• recipes/kustomize/components/aws-efa-p8d16/patch-dgd.yaml
• recipes/kustomize/components/aws-efa-p8d8/kustomization.yaml
• recipes/kustomize/components/aws-efa-p8d8/patch-dgd.yaml
• recipes/kustomize/templates/aws-efa/p5.48xlarge/kustomization.yaml.j2
• recipes/kustomize/templates/aws-efa/p5.48xlarge/values.yaml
• recipes/kustomize/templates/aws-efa/p6-b200.48xlarge/kustomization.yaml.j2
• recipes/kustomize/templates/aws-efa/p6-b200.48xlarge/values.yaml
• recipes/qwen3-235b-a22b-fp8/README.md
• recipes/qwen3-235b-a22b-fp8/trtllm/disagg/blackwell/.kustomize-matrix.yaml
• recipes/qwen3-235b-a22b-fp8/trtllm/disagg/blackwell/deploy-aws-p6-b200.48xlarge.yaml
• recipes/qwen3-235b-a22b-fp8/trtllm/disagg/blackwell/kustomize/overlays/aws-p6-b200.48xlarge/generated-components/00-aws-efa-p6-b200.48xlarge/kustomization.yaml
• recipes/qwen3-235b-a22b-fp8/trtllm/disagg/blackwell/kustomize/overlays/aws-p6-b200.48xlarge/kustomization.yaml
• recipes/qwen3-235b-a22b-fp8/trtllm/disagg/blackwell/kustomize/overlays/gcp-roce/kustomization.yaml
• recipes/qwen3-235b-a22b-fp8/trtllm/disagg/blackwell/kustomize/overlays/generic/kustomization.yaml
• recipes/qwen3-235b-a22b-fp8/trtllm/disagg/blackwell/kustomize/overlays/nscale-ib/kustomization.yaml
• recipes/qwen3-32b/vllm/cloud-providers/.kustomize-matrix.yaml
• recipes/qwen3-32b/vllm/cloud-providers/README.md
• recipes/qwen3-32b/vllm/cloud-providers/kustomize/overlays/aks-ib/kustomization.yaml
• recipes/qwen3-32b/vllm/cloud-providers/kustomize/overlays/aws-p5.48xlarge/generated-components/00-aws-efa-p5.48xlarge/kustomization.yaml
• recipes/qwen3-32b/vllm/cloud-providers/kustomize/overlays/aws-p5.48xlarge/kustomization.yaml
• recipes/qwen3-32b/vllm/cloud-providers/kustomize/overlays/gke-roce/kustomization.yaml
• recipes/qwen3-32b/vllm/cloud-providers/kustomize/overlays/nebius-ib/kustomization.yaml
• recipes/qwen3-32b/vllm/cloud-providers/kustomize/overlays/nscale-ib/kustomization.yaml
• scripts/kustomize-matrix.py
• tests/test_kustomize_matrix.py

💤 Files with no reviewable changes (6)

• recipes/kustomize/components/aws-efa-p8d8/kustomization.yaml
• recipes/kustomize/components/aws-efa-p8d16/kustomization.yaml
• recipes/kustomize/components/aws-efa-p8d16/patch-dgd.yaml
• recipes/kustomize/components/aws-efa-p16d16/patch-dgd.yaml
• recipes/kustomize/components/aws-efa-p8d8/patch-dgd.yaml
• recipes/kustomize/components/aws-efa-p16d16/kustomization.yaml

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a03345-402e-72db-8851-96158926aeb9.docs.buildwithfern.com/dynamo/dev

[devin-ai-integration]

Devin Review found 2 potential issues.

[datadog-official]

🔄 Datadog auto-retried 1 job - 1 passed on retry

🎯 Code Coverage (details)
• Patch Coverage: 7.10%
• Overall Coverage: 49.39% (-8.69%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 9bb03a9 | Docs | Datadog PR Page | Give us feedback!}

[erezzarum]

Instead of duplicating the Jinja template across instance types, we should have a base Jinja template that allow a recipe matrix to select an instance directory, it will supply a local values and optional custom component logic while sharing the parent Jinja template across instance types.

Matrix file

templates:
  - source: ../../../kustomize/templates/aws-efa/p5.48xlarge
    path: components/efa

recipes/kustomize/templates/aws-efa/
├── kustomization.yaml.j2              # shared base Component
├── values.yaml                         # optional base defaults
├── p5.48xlarge/
│   ├── values.yaml                     # instance overrides
│   └── kustomization.yaml.j2           # optional instance Component
└── p6-b200.48xlarge/
    └── values.yaml

Generated component structured for an inherited P5 template example:

kustomize/overlays/aws-p5.48xlarge/components/efa/
├── kustomization.yaml
├── base/
│   └── kustomization.yaml
└── instance/
    └── kustomization.yaml  # generated only when local Jinja exists

The root generated Component is a wrapper:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
components:
  - base
  - instance # present only when the instance layer exists

We should also Keep the shared Jinja template focused on calculating EFA resource requests. It must not select recipes/kustomize/components/aws-efa.
Delete recipes/kustomize/components/aws-efa/ after replacing its only use with the direct disagg-workers/aws-efa matrix selection as we do with other kustomize components.

[yunzhoul-nv]

/ok to test 56e7a94

[jh-nv]

Thanks.

There is a concern of removing newly generated components, please take a look

[sttts]

/ok to test 4f6ce8f

[datadog-official]

⚠️ Warnings

🚦 18 Pipeline jobs failed

PR | DGDR Deploy Test / CPU / lifecycle

View in Datadog · View in GitHub Actions

AttributeError: module 'plotext' has no attribute 'plot_size' during profiling execution in rapid.py at line 394.

PR | DGDR Deploy Test / CPU / profiling

View in Datadog · View in GitHub Actions

Profiling job failed due to AttributeError: module 'plotext' has no attribute 'plot_size'.

PR | Operator Integration

View in Datadog · View in GitHub Actions

Profiling Job failed: AttributeError: module 'plotext' has no attribute 'plot_size'.

View all 18 failed jobs.

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 4f6ce8f | Docs | View more details | Give us feedback!}