fix(charts): make the run-once init a Job, and make litellm boot - #302
Closed
Gursewakzopdev wants to merge 1 commit into
Closed
fix(charts): make the run-once init a Job, and make litellm boot#302Gursewakzopdev wants to merge 1 commit into
Gursewakzopdev wants to merge 1 commit into
Conversation
Every app chart that bundles postgres or mysql was uninstallable under the ZopDay installer's command (helm upgrade --install --atomic --timeout 10m --wait). The datastore subcharts create their init step as a bare `kind: Pod`. A Pod that finishes reports Succeeded / Ready=False / PodCompleted and never goes Ready, so --wait sat for its full timeout and --atomic then rolled back a release that had in fact installed correctly. Isolated rather than inferred: installing superset without --atomic reaches a fully healthy state in 3m31s, and running --wait against that healthy release still times out. The readiness dump names the one object without a controller: postgres-…-init-job Succeeded Ready=False PodCompleted owner=<none> superset-…-init-db Succeeded Ready=False PodCompleted owner=Job superset-postgres-0 Running Ready=True owner=StatefulSet Helm tracks a Job's completion, so the Job-owned pod is fine; the bare Pod is evaluated as a Pod and can never satisfy the check. Both init steps are now batch/v1 Jobs, with backoffLimit and ttlSecondsAfterFinished so the record is kept briefly and then reaped. litellm needed a second, unrelated fix: its proxy pod crash-looped even with the Job change, because 512Mi cannot hold the boot spike — the litellm-database image runs Prisma migrations while Python imports ~100 provider SDKs. It also had no way to supply provider API keys or a fixed master key from values, which is what the ZopDay UI needs. 0.0.2 adds a chart-owned apiKeys Secret, masterkey.value, replicaCount, boot-safe resources, and the schema entries that make all of it editable from the UI. Subcharts are version-bumped rather than mutated, so the ~29 other dependents keep resolving postgres 0.0.12 / mysql 0.0.17 until they choose to repin. The two deliberate pins are untouched: redis stays at 0.0.1 (from 0.0.5 its alerts.yaml PrometheusRule collides by name with postgres's) and wordpress keeps service 0.0.17 (0.0.18+ moves env into a ConfigMap, so $(DB_HOST) stops expanding). Verified on minikube with the installer's exact command — all four install and then accept a values upgrade: superset v0.0.9 install 54s resources.requests.cpu=300m -> revision 2 wordpress v0.0.9 install 28s service.minCPU=150m -> revision 2 outline v0.0.10 install 62s service.minCPU=150m -> revision 2 litellm v0.0.2 install 42s replicaCount=2 -> revision 2 Before this change all four failed at the 10-minute timeout and rolled back.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's broken
Every app chart that bundles
postgresormysqlis uninstallable under the command the ZopDay installer actually runs:Both datastore subcharts create their run-once init step as a bare
kind: Pod. A Pod that finishes its work reportsSucceeded / Ready=False / PodCompletedand never goes Ready — so--waitsits for its full timeout and--atomicthen rolls back a release that had installed correctly.Measured on a clean minikube, straight from
helm.zop.dev:context deadline exceeded, rolled back at 10m01sIsolating it, rather than guessing
Installing superset without
--atomicreaches a fully healthy state in 3m31s. Running--waitagainst that already-healthy release still times out. The readiness dump names the only object with no controller:Helm resolves Job-owned pods through the Job, which is why
init-dbfinishing is fine. A bare Pod is evaluated as a Pod, so it can never satisfy the readiness check.The change
charts/postgres0.0.12 → 0.0.13 andcharts/mysql0.0.17 → 0.0.18 — the init step becomes abatch/v1Job (backoffLimit: 6,ttlSecondsAfterFinished: 600). Same container, samerestartPolicy: OnFailure.charts/litellm0.0.1 → 0.0.2 — needed a second, unrelated fix (below).Subcharts are version-bumped, not mutated, so the ~29 other dependents keep resolving
postgres 0.0.12/mysql 0.0.17until they choose to repin. Nothing else moves.Both deliberate pins are preserved with their comments:
redisstays at 0.0.1 — from 0.0.5 itsalerts.yamlPrometheusRule collides by name with postgres's, and two same-named objects in one release make helm refuse to install.serviceat 0.0.17 — 0.0.18+ movesenvinto a ConfigMap, and the kubelet only expands$(VAR)insideenv:, so WordPress receives the literal$(DB_HOST):$(DB_PORT).litellm needed more than the Job fix
With only the Job change, litellm still failed — its proxy pod crash-looped with 6 restarts. That's the app, not the init step:
512Micannot hold the boot spike, because thelitellm-databaseimage runs Prisma migrations (a Node process) while Python imports ~100 provider SDKs.0.0.2 also closes the gap that made litellm unusable from the ZopDay UI — there was no way to supply provider API keys or a fixed master key from values:
apiKeysSecret mounted viaenvFrom, soapi_key: os.environ/OPENAI_API_KEYresolves without pre-creating a secret by handmasterkey.valuefor a reproducible key, still preserved across upgrades when left emptyreplicaCount, ignored whenautoscaling.enabledso helm and the HPA don't fight over the fieldresources(1Gi request / 2Gi limit)values.schema.jsonentries so all of it is editable from the UIVerified
Installed and then upgraded a value on minikube with the installer's exact command, from the packaged artifacts in this PR:
resources.requests.cpu=300mservice.minCPU=150mservice.minCPU=150mreplicaCount=2Also confirmed the change is present in the shipped tarballs (
kind: Jobin the vendored subchart, zero bare Pods) and thathelm lintpasses on both subcharts.Separately, without
--wait, the three currently-published charts do deploy and accept value updates — only litellm v0.0.1 never becomes ready. So this PR is what makes them work under--wait, and what makes litellm work at all.Two things for the reviewer
charts/postgresandcharts/mysql, and it is what blocks every dependent. Mitigated by version-bumping instead of mutating, so adoption is opt-in per chart.Chart.lockdigests are stale.helm dep updatecan't resolvepostgres v0.0.13until it's published, so I built the subchart locally and vendored it to package. The digest isn't verified at install time and the nexthelm dep updateafter merge corrects it — flagging it rather than hand-faking a digest.