Fix: leaf smokes seed world-writable /work dirs for the non-root harness

[pdettori]

Summary

Fixes the leaf-work PVC permission mismatch (#39): the leaf-orchestrator pod runs as root and created /work/<run> and /work/<run>/results as 0755 root:root, but the harness ksvc (and the leaf-worker job) run as uid 65532 with readOnlyRootFilesystem. So runLeaf's write of result_ref hit EACCES. Only fast paths that write no result (bad_inputs, workspace-isolation, scale-to-zero) passed, masking the bug.

Fix (issue option 1, factored to a shared helper)

Add seed_work_dirs to deploy/knative/lib.sh — it creates the run dirs via the orchestrator and chmod 777s them, so the non-root harness UID can create result subdirs and write result_ref.

On Kind the ksvc's fsGroup: 65532 is not applied to the orchestrator-created dirs (local-path/hostPath PVC) — that's exactly why the bug exists — so an explicit chmod, not fsGroup, is the reliable fix. Options 2/3 (orchestrator-as-65532 / fsGroup contract) are noted in #39 but don't hold on Kind: option 2 also breaks the orchestrator's apk add.

• lib.sh: add seed_work_dirs
• leaf-smoke.sh, leaf-async-smoke.sh: use seed_work_dirs
• leaf-cron-smoke.sh: seed /work/nightly world-writable (the leaf-worker creates the fire-stamped results subdir under it)
• leaf-gate-smoke.sh: fold the three inline mkdir -p + chmod -R 0777 blocks into seed_work_dirs (DRY; behavior unchanged)

Also: unrelated fan-out hang in leaf-smoke.sh

Surfaced during verification. The parallel fan-out used a bare wait, which also blocks on the background port-forward started by ensure_port_forward (and the sampler) — so on a clean session with no pre-existing port-forward the smoke hung forever at Claim 1. Now it waits only on the dispatch PIDs.

Verification (live, Kind sh-knative)

• Isolated repro of the mechanism: a uid-65532 probe pod gets EACCES on a 0755 root dir, and writes OK after chmod 777.
• Full LEAF_LIVE_SMOKE=1 leaf-smoke.sh → 9 passed, 0 failed — LEAF SMOKE PASS.
• result_ref files written by uid 65532 with correct verdicts (i1=FLAGGED, i2=CLEAR, i3=CLEAR).

Closes #39

Assisted-By: Claude Code