Skip to content

fix(charts): make the run-once init a Job, and make litellm boot - #302

Closed
Gursewakzopdev wants to merge 1 commit into
mainfrom
fix/init-jobs-and-litellm
Closed

fix(charts): make the run-once init a Job, and make litellm boot#302
Gursewakzopdev wants to merge 1 commit into
mainfrom
fix/init-jobs-and-litellm

Conversation

@Gursewakzopdev

Copy link
Copy Markdown
Contributor

What's broken

Every app chart that bundles postgres or mysql is uninstallable under the command the ZopDay installer actually runs:

helm upgrade --install <name> <chart> -n <ns> --create-namespace --atomic --timeout 10m --wait

Both datastore subcharts create their run-once init step as a bare kind: Pod. A Pod that finishes its work reports Succeeded / Ready=False / PodCompleted and never goes Ready — so --wait sits for its full timeout and --atomic then rolls back a release that had installed correctly.

Measured on a clean minikube, straight from helm.zop.dev:

chart version result
superset v0.0.8 context deadline exceeded, rolled back at 10m01s
wordpress v0.0.8 ❌ same
outline v0.0.9 ❌ same
litellm v0.0.1 ❌ same

Isolating it, rather than guessing

Installing superset without --atomic reaches a fully healthy state in 3m31s. Running --wait against that already-healthy release still times out. The readiness dump names the only object with no controller:

postgres-superset-superset-init-job   Succeeded  Ready=False  PodCompleted  owner=<none — BARE POD>
superset-superset-init-db-rd5vq       Succeeded  Ready=False  PodCompleted  owner=Job
superset-postgres-0                   Running    Ready=True                 owner=StatefulSet

Helm resolves Job-owned pods through the Job, which is why init-db finishing is fine. A bare Pod is evaluated as a Pod, so it can never satisfy the readiness check.

The change

  • charts/postgres 0.0.12 → 0.0.13 and charts/mysql 0.0.17 → 0.0.18 — the init step becomes a batch/v1 Job (backoffLimit: 6, ttlSecondsAfterFinished: 600). Same container, same restartPolicy: OnFailure.
  • charts/litellm 0.0.1 → 0.0.2 — needed a second, unrelated fix (below).
  • superset 0.0.9 · wordpress 0.0.9 · outline 0.0.10 — repin the fixed subchart, no other change.

Subcharts are version-bumped, not mutated, so the ~29 other dependents keep resolving postgres 0.0.12 / mysql 0.0.17 until they choose to repin. Nothing else moves.

Both deliberate pins are preserved with their comments:

  • redis stays at 0.0.1 — from 0.0.5 its alerts.yaml PrometheusRule collides by name with postgres's, and two same-named objects in one release make helm refuse to install.
  • wordpress keeps service at 0.0.17 — 0.0.18+ moves env into a ConfigMap, and the kubelet only expands $(VAR) inside env:, so WordPress receives the literal $(DB_HOST):$(DB_PORT).

litellm needed more than the Job fix

With only the Job change, litellm still failed — its proxy pod crash-looped with 6 restarts. That's the app, not the init step: 512Mi cannot hold the boot spike, because the litellm-database image runs Prisma migrations (a Node process) while Python imports ~100 provider SDKs.

0.0.2 also closes the gap that made litellm unusable from the ZopDay UI — there was no way to supply provider API keys or a fixed master key from values:

  • a chart-owned apiKeys Secret mounted via envFrom, so api_key: os.environ/OPENAI_API_KEY resolves without pre-creating a secret by hand
  • masterkey.value for a reproducible key, still preserved across upgrades when left empty
  • replicaCount, ignored when autoscaling.enabled so helm and the HPA don't fight over the field
  • boot-safe resources (1Gi request / 2Gi limit)
  • values.schema.json entries so all of it is editable from the UI

Verified

Installed and then upgraded a value on minikube with the installer's exact command, from the packaged artifacts in this PR:

chart install value change result
superset v0.0.9 ✅ 54s resources.requests.cpu=300m ✅ revision 2
wordpress v0.0.9 ✅ 28s service.minCPU=150m ✅ revision 2
outline v0.0.10 ✅ 62s service.minCPU=150m ✅ revision 2
litellm v0.0.2 ✅ 42s replicaCount=2 ✅ revision 2, 2 live replicas

Also confirmed the change is present in the shipped tarballs (kind: Job in the vendored subchart, zero bare Pods) and that helm lint passes on both subcharts.

Separately, without --wait, the three currently-published charts do deploy and accept value updates — only litellm v0.0.1 never becomes ready. So this PR is what makes them work under --wait, and what makes litellm work at all.

Two things for the reviewer

  1. This edits two shared subcharts. That's unavoidable — the bare Pod is in charts/postgres and charts/mysql, and it is what blocks every dependent. Mitigated by version-bumping instead of mutating, so adoption is opt-in per chart.
  2. The four apps' Chart.lock digests are stale. helm dep update can't resolve postgres v0.0.13 until it's published, so I built the subchart locally and vendored it to package. The digest isn't verified at install time and the next helm dep update after merge corrects it — flagging it rather than hand-faking a digest.

Every app chart that bundles postgres or mysql was uninstallable under the
ZopDay installer's command (helm upgrade --install --atomic --timeout 10m
--wait). The datastore subcharts create their init step as a bare `kind: Pod`.
A Pod that finishes reports Succeeded / Ready=False / PodCompleted and never
goes Ready, so --wait sat for its full timeout and --atomic then rolled back a
release that had in fact installed correctly.

Isolated rather than inferred: installing superset without --atomic reaches a
fully healthy state in 3m31s, and running --wait against that healthy release
still times out. The readiness dump names the one object without a controller:

  postgres-…-init-job   Succeeded  Ready=False  PodCompleted  owner=<none>
  superset-…-init-db    Succeeded  Ready=False  PodCompleted  owner=Job
  superset-postgres-0   Running    Ready=True                 owner=StatefulSet

Helm tracks a Job's completion, so the Job-owned pod is fine; the bare Pod is
evaluated as a Pod and can never satisfy the check. Both init steps are now
batch/v1 Jobs, with backoffLimit and ttlSecondsAfterFinished so the record is
kept briefly and then reaped.

litellm needed a second, unrelated fix: its proxy pod crash-looped even with
the Job change, because 512Mi cannot hold the boot spike — the
litellm-database image runs Prisma migrations while Python imports ~100
provider SDKs. It also had no way to supply provider API keys or a fixed master
key from values, which is what the ZopDay UI needs. 0.0.2 adds a chart-owned
apiKeys Secret, masterkey.value, replicaCount, boot-safe resources, and the
schema entries that make all of it editable from the UI.

Subcharts are version-bumped rather than mutated, so the ~29 other dependents
keep resolving postgres 0.0.12 / mysql 0.0.17 until they choose to repin. The
two deliberate pins are untouched: redis stays at 0.0.1 (from 0.0.5 its
alerts.yaml PrometheusRule collides by name with postgres's) and wordpress
keeps service 0.0.17 (0.0.18+ moves env into a ConfigMap, so $(DB_HOST) stops
expanding).

Verified on minikube with the installer's exact command — all four install and
then accept a values upgrade:

  superset  v0.0.9   install 54s   resources.requests.cpu=300m -> revision 2
  wordpress v0.0.9   install 28s   service.minCPU=150m         -> revision 2
  outline   v0.0.10  install 62s   service.minCPU=150m         -> revision 2
  litellm   v0.0.2   install 42s   replicaCount=2              -> revision 2

Before this change all four failed at the 10-minute timeout and rolled back.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant