fix(jupyterhub): make the chart configurable — unlock schema, expose 9 values, stop rotating hub secrets and rerunning the pre-pull hook on upgrade

[Gursewakzopdev]

Three fixes that together make JupyterHub configurable. The schema rejected every edit; once an edit was accepted, applying it broke the hub; and every apply needlessly reran the image pre-pull hook. The first two each leave the chart unusable on their own, so they belong together; the third (#300) was found while reviewing this PR and is the same class of bug as the second.

Chart 0.0.3 → 0.0.4.

Problem 1 — the schema rejected everything

Every scalar in values.schema.json was generated as enum: [<current value>] — 49 of 49 leaves. A single-value enum makes that value the only legal one:

Error: UPGRADE FAILED: values don't meet the specifications of the schema(s) in the following chart(s):
- at '/scheduling/userScheduler/replicas': value must be 2

Separately, helm-manager only emits a form field for a leaf carrying mutable: true (compare charts/litellm, whose form works). This chart had category on all six top-level objects but mutable on nothing, so its published form came back empty and the UI fell back to a generic field list.

Problem 2 — applying a change broke the hub

Found by running an actual helm upgrade on a live cluster, not just helm template. Install was fine; any config change gave:

api_request to proxy failed: HTTP 403: Forbidden   →  hub CrashLoopBackOff

Two causes compounding:

1. The password helpers looked the hub Secret up as jupyterhub.hub.fullname (<release>-jupyterhubhub) while templates/hub/secret.yaml creates it as jupyterhub-secrets. The lookup always missed and fell through to randAlphaNum, minting fresh secrets on every upgrade — measured auth_token changing fuOrDC8TfcdR → XP4hBshwGqRb across one apply. This hit all four helpers: auth_token, cookie_secret (logs every user out), CryptKeeper.keys (renders stored encrypted auth state undecryptable) and the service api_tokens.
2. templates/proxy/deployment.yaml's annotations: block contained only the comments describing a checksum/auth-token annotation — the annotation line itself was missing, so the proxy never restarted to pick up a changed token. Confirmed by the proxy pod keeping 0 restarts and the same name across an upgrade: hub rolled with a new token, proxy kept the old one.

Problem 3 — every upgrade reran the pre-pull hook (#300)

The same wrong-name pattern, one file over: image-puller/_helpers-daemonset.tpl looked the hub ConfigMap up as jupyterhub.hub.fullname (<release>-jupyterhubhub) while hub/configmap.yaml creates it as jupyterhub-configs.

That ConfigMap is where checksum_hook-image-puller is stored so the next release can tell whether the pre-pull DaemonSet actually changed. The lookup never found it, so $old_checksum was always "", never equalled the sha256 in $new_checksum, and prePuller.hook.pullOnlyOnChanges: true — the default — was a no-op. The hook Job, DaemonSet, RBAC and ServiceAccount were recreated and rerun on every helm upgrade, including one that changed nothing. Not a correctness bug like Problem 2, just an image-pull DaemonSet spun up on every apply.

Reproduced directly, comparing the two packaged charts under identical conditions — the rendered decision spells the miss out:

# "113fcf12f1e634672c3bcf16845adcb8a5eddb1cb59c7a6812a9435f762f424a" != ""
                                                                       ^^ stored checksum never read

Changes

Schema — describes exactly the 9 values worth exposing, each mutable: true with a description and a default matching values.yaml:

Group	Field	Type	Default
runtime	singleuser.image.name	string	quay.io/jupyterhub/k8s-singleuser-sample
runtime	singleuser.image.tag	string	4.1.1-0.dev.git.6957.h0e735928
runtime	singleuser.storage.capacity	string	10Gi
advanced	hub.config.JupyterHub.authenticator_class	string	dummy
advanced	hub.config.JupyterHub.admin_access	boolean	true
runtime	cull.enabled	boolean	true
runtime	cull.timeout	integer	3600
advanced	proxy.https.enabled	boolean	false
advanced	proxy.https.hosts	array	[]

Defaults are JSON-typed (true, 3600, []) to match how charts/litellm and charts/postgres already store non-string defaults, and each is read programmatically from values.yaml and asserted equal to it, so schema and values cannot drift.

Deliberately not exposed: scheduling.userScheduler.* (internal scheduler tuning — and this field being editable is what produced a failed release), hub.baseUrl (path-based routing only), singleuser.storage.type (static also needs pvcName, which isn't exposed — a half-usable option is worse than none), cull.every (polling interval; timeout is the knob), and ~35 internals (network policies, cloud-metadata iptables, uid/fsGid, prePuller). Omitting a value does not disable it: it keeps its values.yaml default and is still settable via --set — verified scheduling.userScheduler.replicas=1 still works after removal.

Templates — both bugs were places disagreeing about a hardcoded string, so each name now exists exactly once:

Helper	Value	Read by
jupyterhub.hub-secret.fullname	jupyterhub-secrets	hub/secret.yaml, its two consumers, all 4 password lookups
jupyterhub.hub-configmap.fullname	jupyterhub-configs	hub/configmap.yaml, the hub Deployment's volume, the image-puller lookup

Plus the missing checksum/auth-token annotation restored, as the safety net upstream intends.

Neither helper changes any resource's actual name — both resolve to the same literal that was hardcoded before. Confirmed by diffing helm template across the change: identical once the per-run randAlphaNum secrets are masked (2271 lines each), so existing releases see no resource churn.

Packaging

Per CONTRIBUTING.md steps 4–6, matching #295 / #290 — docs/ is the repo itself (GitHub Pages, CNAME → helm.zop.dev), so an unpackaged change publishes nothing:

helm package charts/jupyterhub --version v0.0.4 -d .
mv jupyterhub-v0.0.4.tgz docs/
cd docs && helm repo index . --url https://helm.zop.dev

Published history is v0.0.1–v0.0.3 plus v0.0.4.

Verification (live cluster, against the packaged tgz)

Schema + secret fixes — install then three upgrades, because the lookup bug only manifests once a prior Secret exists and a stale token only surfaces on the next apply:

INSTALL (defaults)          rc=0  hub 1/1  403s: 0
  defaults applied: timeout:3600  admin_access:true
                    authenticator_class:dummy  capacity:10Gi

UPDATE 1  7100 / 20Gi / 1   rc=0  hub 1/1  403s: 0   auth+cookie+cryptkeeper KEPT
UPDATE 2  7200 / 30Gi / 1   rc=0  hub 1/1  403s: 0   auth+cookie+cryptkeeper KEPT
UPDATE 3  3600 / 10Gi / 2   rc=0  hub 1/1  403s: 0   auth+cookie+cryptkeeper KEPT
  scheduler Deployment replicas moved 2 → 1 → 1 → 2

helm history: 4 revisions, all "complete", none failed

Pre-pull hook fix (#300) — the identical scenario run on the chart as it stood before this commit and after it, so the only variable is which ConfigMap name the lookup asks for. Counted as pre-pull hook resources the render decided to create, and as hook-image-awaiter Job events actually recorded in the namespace:

                                    before fix   after fix
unchanged upgrade                        9            0     ← the fix
upgrade changing singleuser.image.tag    9            9     ← still reruns when it should
client dry-run (lookup impossible)       9            9     ← control: the 0 above comes
                                                              from the lookup, not the flag

Then real upgrades, not dry-runs, on a release installed with the true default singleuser image:

install                        rc=0  hub 1/1  hook Job events: 2
upgrade, unchanged             rc=0  hub 1/1  hook events still 2   auth_token PRESERVED
upgrade, cull.timeout=7200
        + capacity=20Gi        rc=0  hub 1/1  restarts 0  403s: 0   auth_token PRESERVED
                               hook events still 2 — a config edit does not touch the
                               DaemonSet spec, so it correctly does not re-pull
  live values after: timeout: 7200   capacity: 20Gi

A config edit no longer drags an image-pull DaemonSet along with it — which is the case the UI actually generates.

Also checked: all 9 defaults present inside the packaged tgz (not just the source); index digest matches the tgz byte-for-byte (7ff775eacab9…); the index diff is exactly one digest, jupyterhub v0.0.4 — 29 charts and every other chart's version list byte-identical; helm lint clean.

Notes

• Scope: this PR touches templates, not only the schema. The three fixes are independent and could be split, but merging the schema change alone would ship a chart whose values still cannot be edited without breaking the hub.
• Version: repackaged as v0.0.4 rather than bumping, per review — v0.0.4 has never been published, so there is no released artifact to supersede.
• Pre-existing, not addressed: templates/proxy/service.yaml hardcodes type: LoadBalancer and is not values-configurable, so helm --wait cannot complete on a cluster with no LB controller; the local runs poll pod readiness directly instead. Fine on a cloud provider.
• Existing releases stay pinned to whatever version they were installed with, so they need repointing to 0.0.4 or reinstalling to pick this up. New installs get it automatically.

[arunesh-j]

Per CONTRIBUTING.md (Steps 4–6), chart changes need to be packaged and the repo index regenerated as part of the PR itself — this is how the last few chart PRs (#295, #294, #290) were merged, each committing the built .tgz + updated docs/index.yaml alongside the code change.

This PR currently only touches Chart.yaml and values.schema.json. Can you also:

helm package charts/jupyterhub --version 0.0.4
mv jupyterhub-0.0.4.tgz docs/
cd docs && helm repo index . --url https://helm.zop.dev

and commit docs/jupyterhub-0.0.4.tgz + the regenerated docs/index.yaml? Without that, merging to main won't actually publish the fixed schema — docs/index.yaml would still point to jupyterhub-0.0.3.tgz with the old all-enum-locked schema.

[Gursewakzopdev]

Good catch — you're right, and I'd misread how publishing works here. I'd assumed helm/chart-releaser-action in .github/workflows/release.yaml handled it; in fact docs/ is the repo (GitHub Pages, CNAME → helm.zop.dev), so an unpackaged change publishes nothing. Merging would have left docs/index.yaml pointing at jupyterhub-v0.0.3.tgz with the old all-enum-locked schema — exactly as you said.

Done in 84a09dc:

helm package charts/jupyterhub --version v0.0.4 -d .
mv jupyterhub-v0.0.4.tgz docs/
cd docs && helm repo index . --url https://helm.zop.dev

One deviation from your snippet: I packaged as v0.0.4 rather than 0.0.4, since every existing package in docs/ is v-prefixed (jupyterhub-v0.0.3.tgz, postgres-v0.0.12.tgz, …) and the index URLs match. Chart.yaml keeps the bare 0.0.4 — the same split every other chart here uses. Using 0.0.4 would have produced jupyterhub-0.0.4.tgz and broken the naming pattern.

Verified before pushing:

• docs/index.yaml now lists jupyterhub v0.0.4 → https://helm.zop.dev/jupyterhub-v0.0.4.tgz
• the packaged .tgz carries the fixed schema, not the old one: 9 fields, 0 single-value enum locks, all mutable, grouped runtime/advanced
• no other chart's version list changed — 29 charts before and after, jupyterhub the only entry gaining a version. The rest of the +190/−174 index diff is regenerated created timestamps and digests, which lines up with feat(clickhouse): restart pod on config change via stakater reloader #295 (+188/−173) and fix(clickhouse): load profile settings via users.d in customConfig #290 (+186/−171)

PR description updated with the packaging step and these checks. Thanks for the pointer to the CONTRIBUTING steps and the prior PRs — that made the expected shape unambiguous.

[arunesh-j]

Packaging looks good — verified the actual .tgz contents (not just the source diff): docs/jupyterhub-v0.0.4.tgz's digest matches the docs/index.yaml entry, the old v0.0.1–v0.0.3 entries are preserved, and helm lint/helm template pass against the extracted package, including the override that broke the old schema (scheduling.userScheduler.replicas=1).

Two smaller things worth addressing before merge, neither blocking:

1. Missing default on the 9 new mutable leaves. Each has a known, concrete values.yaml value (10Gi, dummy, 3600, false, [], etc.) but none carry a "default" key, so the settings form will render them blank instead of pre-filled. Compare charts/openobserve-standalone/values.schema.json's persistence.size — "default": "10Gi", "mutable": true — same shape as singleuser.storage.capacity here. Suggest adding "default" to all 9 to match.

[Gursewakzopdev]

Fixed in d3c9862 — all 9 leaves now carry a default.

Rather than hand-copying the values, the script reads each one out of charts/jupyterhub/values.yaml and asserts it equal to what lands in the schema, so the two can't drift:

Field	Default
singleuser.image.name	quay.io/jupyterhub/k8s-singleuser-sample
singleuser.image.tag	4.1.1-0.dev.git.6957.h0e735928
singleuser.storage.capacity	10Gi
hub.config.JupyterHub.authenticator_class	dummy
hub.config.JupyterHub.admin_access	true
cull.enabled	true
cull.timeout	3600
proxy.https.enabled	false
proxy.https.hosts	[]

One note on typing: I stored these JSON-typed (true, 3600, []) rather than stringified. openobserve-standalone only demonstrates the string case ("10Gi"), and I have booleans, an integer and an array here — charts/litellm and charts/postgres already store those unquoted (storeModelInDB: true, autoscaling.minReplicas: 1, replication.enabled: false), so I followed that. Happy to switch if the form expects strings.

Repackaged and re-indexed, since the .tgz is what the UI actually reads. Verified against the packaged artifact, not the source:

• all 9 leaves carry a default inside docs/jupyterhub-v0.0.4.tgz; 0 enum locks; all still mutable
• docs/index.yaml digest matches the rebuilt .tgz byte-for-byte (aa6e00f8…)
• 29 charts and the v0.0.1–v0.0.4 list unchanged
• helm template against the extracted package renders on defaults alone, and still accepts cull.timeout=7200 and scheduling.userScheduler.replicas=1

PR description updated with the defaults table.

One thing — you mentioned "two smaller things" but only item 1 made it into the comment; there's no item 2 in the body. Did the second point get cut off? Happy to pick it up in the same pass.

[arunesh-j]

Approving — the schema-lock bug is fixed and verified end-to-end (extracted the packaged `.tgz`, confirmed digest matches `docs/index.yaml`, `helm lint`/`helm template` pass on defaults and on the override that broke the old schema, all 9 exposed fields now carry matching `default`s).

Re: `hub.config.JupyterHub.authenticator_class` staying under `advanced` — acknowledged, keeping it there per discussion. Not a blocker.

[Gursewakzopdev]

Before merging I ran a real helm install and helm upgrade against the packaged tgz on a live cluster, and the upgrade path was broken — worth flagging since "edit a value and save" is the whole point of this PR. Fixed in 0f3ab2e (chart now 0.0.5).

Symptom: install was fine, but any config change left the hub in CrashLoopBackOff:

api_request to proxy failed: HTTP 403: Forbidden

Two bugs compounding, both pre-existing:

1. The password helpers looked the hub Secret up under the wrong name — jupyterhub.hub.fullname → <release>-jupyterhubhub, while templates/hub/secret.yaml creates it as jupyterhub-secrets. The lookup always missed and fell through to randAlphaNum, so every upgrade minted fresh values. Measured across one upgrade:

   auth_token before: fuOrDC8TfcdR
   auth_token after : XP4hBshwGqRb
   

   This hit all four helpers, not just the proxy token: cookie_secret (logs every user out), CryptKeeper.keys (makes stored encrypted auth state undecryptable) and the service api_tokens were rotated the same way.

2. templates/proxy/deployment.yaml had an annotations: block containing only the comments describing a checksum/auth-token annotation — the annotation line itself was missing. So the proxy never restarted to pick up a changed token; confirmed by the proxy pod keeping 0 restarts and the same name across an upgrade. Hub rolled with a new token, proxy kept the old one, everything between them 403'd.

Fix: a single jupyterhub.hub-secret.fullname helper as the source of truth for that Secret's name, used by the Secret, its two consumers and all four lookups — the bug was two places disagreeing about a hardcoded string, so the name now exists once. Plus the missing annotation restored as the safety net upstream intends.

Verified against the packaged tgz — install, then TWO successive upgrades:

install      rc=0   hub 1/1
upgrade #1   rc=0   hub 1/1, no 403      auth_token PRESERVED  cookie_secret PRESERVED
upgrade #2   rc=0   hub 1/1, no 403      auth_token PRESERVED  cookie_secret PRESERVED
values applied each time: timeout 7100 → 7200, replicas 1, capacity 20Gi
proxy pod now rolls with the hub instead of being left behind

Also dropped docs/jupyterhub-v0.0.4.tgz (e1ceccc). It only ever existed on this branch, was never published, and its upgrade path is exactly the broken one — no reason to publish a version we know breaks. Published history is now v0.0.1–v0.0.3 plus v0.0.5; index digest verified against the tgz; 29 charts unchanged.

This does mean the PR now touches templates, where it previously said "schema and version only" — description updated accordingly. Happy to split the upgrade fix into its own PR if you'd rather keep this one schema-only; the two are independent, though merging the schema change alone would ship a chart whose values still can't be edited without breaking the hub.

[Gursewakzopdev]

Final end-to-end verification, run against the tgz downloaded from this branch (raw.githubusercontent.com/.../docs/jupyterhub-v0.0.5.tgz, sha256 a2d43114c81b51c6…) rather than a local build — so this is the artifact a merge would publish.

Live single-node cluster, fresh namespace, cluster-scoped RBAC cleared first:

1. INSTALL (defaults only)
   rc=0   hub 1/1   403s in hub log: 0
   defaults applied: timeout:3600 admin_access:true authenticator_class:dummy capacity:10Gi

2.1 UPDATE  timeout=7100  capacity=20Gi  replicas=1
   rc=0   hub 1/1   403s: 0
   applied: timeout:7100 replicas:1 capacity:20Gi   scheduler deploy=1
   secrets: auth=PRESERVED cookie=PRESERVED cryptkeeper=PRESERVED

2.2 UPDATE  timeout=7200  capacity=30Gi  replicas=1
   rc=0   hub 1/1   403s: 0
   applied: timeout:7200 replicas:1 capacity:30Gi   scheduler deploy=1
   secrets: auth=PRESERVED cookie=PRESERVED cryptkeeper=PRESERVED

2.3 UPDATE  timeout=3600  capacity=10Gi  replicas=2
   rc=0   hub 1/1   403s: 0
   applied: timeout:3600 replicas:2 capacity:10Gi   scheduler deploy=2
   secrets: auth=PRESERVED cookie=PRESERVED cryptkeeper=PRESERVED

3. IDEMPOTENT re-apply (same values)
   rc=0   hub 1/1   403s: 0

4. helm history — 5 revisions, all "Upgrade complete", none failed

What each part is checking:

• Four upgrades, not one. A single upgrade could pass by luck; the lookup bug only shows once a prior Secret exists, and a stale token only surfaces on the next apply. Four consecutive applies with all three secrets unchanged is what actually demonstrates stability.
• replicas 2 → 1 → 1 → 2 confirms the value reaches the cluster, not just the rendered manifest — the previously enum-locked field now moves in both directions.
• 403s in hub log: 0 at every step is the direct check on the original failure; before the fix this was CrashLoopBackOff with api_request to proxy failed: HTTP 403.
• cryptkeeper preserved matters most for real data: rotating that key makes previously stored encrypted auth state undecryptable, which is silent and unrecoverable rather than merely noisy.

helm lint clean and lint-test green on the same commit.

One limitation, stated plainly: the local cluster has no load-balancer controller, so templates/proxy/service.yaml's hardcoded type: LoadBalancer stays <pending> and helm --wait can never return there. The runs above therefore poll pod readiness directly instead of using --wait. That's pre-existing and untouched by this PR; on a cloud provider the LB gets an address and --wait completes normally.

[arunesh-j]

Since there is fix it bumped from v0.0.4 to v0.0.5 ..

We can keep it has v0.0.4 only and use package that version again and do the helm index.

[arunesh-j]

Re-approving after the template changes (secret-rotation fix) — treating the prior approval as stale since scope materially grew from schema-only to 5 template files fixing a real bug.

Verified independently:

• The naming mismatch is real: `hub/secret.yaml` hardcodes the Secret name as `jupyterhub-secrets`, but all 4 `lookup` call sites in `hub/_helpers-passwords.tpl` looked it up via `jupyterhub.hub.fullname` (`-jupyterhubhub`) — grepped the pre-PR code to confirm these never matched.
• The fix (`jupyterhub.hub-secret.fullname` helper) is applied consistently everywhere a Secret name is needed — no leftover references to the old path anywhere in the template tree.
• Pulled the packaged `docs/jupyterhub-v0.0.4.tgz`, confirmed its digest matches `docs/index.yaml` byte-for-byte, and confirmed the fix is actually baked into the extracted templates (not just source).
• `helm lint`/`helm template` clean against the packaged artifact; `checksum/auth-token` renders correctly.
• CI (`lint-test`) passing.

One caveat: `lookup` is always empty under `helm template`, so I can't independently reproduce the secret-persists-across-upgrades behavior myself — that requires the live-cluster `helm upgrade` cycle already described in the PR's testing section.

Filing a separate follow-up issue for a related-but-out-of-scope finding: `image-puller/_helpers-daemonset.tpl` has the same lookup-name-mismatch pattern against a ConfigMap, which silently breaks the `prePuller.hook.pullOnlyOnChanges` optimization. Not a blocker for this PR.

[Gursewakzopdev]

Both open points are in f4d4658.

1. Kept as v0.0.4 rather than v0.0.5, as you asked — repackaged the same version and regenerated the index. v0.0.4 was only ever on this branch, so there's no published artifact being superseded. Published history stays v0.0.1–v0.0.3 plus v0.0.4; new digest 7ff775eacab9…, matched against the tgz byte-for-byte.

2. #300 fixed in the same commit — you were right that it's the same pattern. image-puller/_helpers-daemonset.tpl looked the hub ConfigMap up as jupyterhub.hub.fullname while hub/configmap.yaml creates it as jupyterhub-configs, so checksum_hook-image-puller was never read back. Fixed with a jupyterhub.hub-configmap.fullname helper, exactly as you suggested — and used it in the ConfigMap and the hub Deployment's volume too, since those were the other two places hardcoding the string.

I reproduced it before fixing rather than taking the reasoning on trust. Same scenario on the pre-fix and post-fix packages, only variable being the name the lookup asks for. The rendered decision states the miss outright:

# "113fcf12f1e634672c3bcf16845adcb8a5eddb1cb59c7a6812a9435f762f424a" != ""

Counting pre-pull hook resources the render decides to create:

                                    before fix   after fix
unchanged upgrade                        9            0     ← the fix
upgrade changing singleuser.image.tag    9            9     ← still reruns when it should
client dry-run (lookup impossible)       9            9     ← control: the 0 is the lookup
                                                              hitting, not the flag

The third row is the one that makes the second row mean something — without it, "0" could just as well be the hook switched off.

Then real upgrades on a release installed with the true default singleuser image, counting hook-image-awaiter Job events in the namespace:

install                     rc=0  hub 1/1   hook Job events: 2
upgrade, unchanged          rc=0  hub 1/1   still 2    auth_token PRESERVED
upgrade, cull.timeout=7200
        + capacity=20Gi     rc=0  hub 1/1   still 2    auth_token PRESERVED
                            restarts 0, 403s 0
  live values after: timeout: 7200   capacity: 20Gi

So a config edit — the case the UI actually generates — no longer drags an image-pull DaemonSet along with it.

Re-proved #299's guarantees, since this fix touches the ConfigMap the hub Deployment mounts and checksums: four consecutive upgrades on a second release, all three secrets (auth_token, cookie_secret, CryptKeeper.keys) preserved every time, 0 restarts, 0 403s, scheduler replicas moving 2 → 1 → 2 to confirm values reach the cluster. And the hook correctly reran on the one upgrade that changed the image tag (events 2 → 4), so it isn't stuck off.

No resource churn: neither helper changes any resource's real name — both return the literal that was hardcoded before. helm template across the change is identical once the per-run randAlphaNum secrets are masked (2271 lines either side), and the index diff is exactly one digest with all 29 charts' version lists untouched.

Description and title updated for the third fix.

Still no item 2 from your "two smaller things" — only item 1 (the missing defaults) made it into that comment. If #300 was item 2, then we're square; if it was something else, happy to pick it up.

[arunesh-j]

Re-approving — the PR now also includes the fix for #300 (image-puller ConfigMap lookup using the wrong name).

Verified:

• New `jupyterhub.hub-configmap.fullname` helper (literal `jupyterhub-configs`) is used consistently by the ConfigMap's own name, the hub Deployment's volume reference, and the `lookup` call in `image-puller/_helpers-daemonset.tpl:218` that previously targeted `jupyterhub.hub.fullname` — grepped the tree, no leftover references to the old broken path.
• Pulled the packaged `docs/jupyterhub-v0.0.4.tgz`, confirmed its digest matches `docs/index.yaml`, and confirmed the fix is baked into the extracted templates.
• `helm lint`/`helm template` clean against the packaged artifact.
• CI (`lint-test`) passing.

Ref: #300 (leaving that issue open per request, not closing it from here).