Sync main with reliability fixes - #14
Merged
Merged
Conversation
…ble cgo during build process
…rlink-docker-plugin into 2-light-version-no-gpu
use just dind
Signed-off-by: Diego Ciangottini <dciangot@host-10-25-127-213.wired.local>
Signed-off-by: Diego Ciangottini <diego.ciangottini@pg.infn.it>
enable listen on socket
…n into 2-light-version-no-gpu
Signed-off-by: Giulio Bianchini <giulio.bianchini@hotmail.it>
…llback
Create.go
- Do not touch the ResponseWriter after the create handler has returned. The
asynchronous container-creation goroutine called HandleErrorAndRemoveData,
which writes to w long after the response was sent; that races with the http
server and can corrupt the next response on a keep-alive connection. Split
the cleanup out into cleanupPodData, which never touches w, and use it from
the goroutine. HandleErrorAndRemoveData stays for the request goroutine.
- Recover panics in that goroutine. net/http does not recover panics in bare
goroutines spawned by a handler, so any nil deref there took the whole
plugin down along with the bookkeeping of every other running pod.
- Bound the mesh_ready wait. When mesh.sh fails the sentinel is never written,
and the poll loop in containers_command.sh spun forever: the pod stayed in
Waiting with no diagnostics and its DIND stayed claimed until a human
deleted it. The loop now times out after 300s, dumps the overlay container
logs and exits non-zero, and the docker exec itself runs under a 10m context
so it cannot leak the goroutine for any other reason. Script output is
logged on failure.
- Select the network-overlay container by name instead of assuming it is
containers[0]. The startup path was gated on the mesh annotation, a weaker
condition than the one under which the overlay is actually built; when the
two disagreed a workload container was started as the overlay and every
other container waited on a sentinel nothing would write. It also panicked
on an empty container list. The mismatch is now logged as a warning and the
containers start directly.
- Scope envVars and fpgaArgs to a single container. Declared outside the loop,
every container inherited the -e/-v flags and --device entries of the
containers processed before it.
Status.go
- Do not fail a whole status batch because one pod has no DIND container.
interLink asks for every pod on the node in one call, so a 404 for a single
missing DIND left all the other pods unreconciled. That pod is now skipped
with a warning, which is also safer than inventing a terminal status: the
DIND is legitimately absent between the claim and the rename. An all-skipped
response marshals to [] rather than null.
DindHandler.go
- Tear the DIND down before returning its subnet to the pool. The failure
paths freed the subnet while the container and its network still existed, so
the next network create for that range failed with "Pool overlaps with other
one on this address space" — permanently, since the reaper cannot remove a
network whose container is alive. rollbackDind removes container, then
network, then frees the subnet, and deliberately leaks the subnet with a
loud error if the network cannot be removed.
- Check the exit code of docker network create. go-execute returns a nil error
for a non-zero exit, so a rejected create was treated as a success and only
surfaced later as an unrelated docker run failure.
AMDHandler.go
- Track FPGA assignees as a list instead of a single ContainerID, so sharing
under FPGA_DISABLE_BOOKKEEPING keeps every assignee and Release can find
each one instead of only the last. Assign/GetAvailableFPGAs no longer
double-lock FPGASpecsMutex (added assignLocked/getAvailableFPGAsLocked), the
fallback sort runs on a copy rather than reordering the shared list under no
lock, and the same physical device is no longer returned twice in one call.
Create_test.go
- Unit tests for selectNetworkOverlay, including the no-overlay regression and
the empty-list panic.
Brings the reliability work from the light branch onto main without adopting its removal of GPU support. pkg/docker/gpustrategies/ is kept unchanged, and the GPU wiring the branch had deleted is restored on top of the merged code: the gpuManager init in cmd/main.go, SidecarHandler.GpuManager, the nvidia.com/gpu request block in Create.go, and the GPU release in Delete.go. CGO_ENABLED stays at 1 in Makefile and .goreleaser.yaml: the branch could set it to 0 only because it had dropped go-nvml, which does not build without cgo. What the merge brings in: - The ResponseWriter is no longer written after the create handler returned, and the asynchronous container-creation goroutine now recovers panics instead of taking the whole plugin down with it. - A pod whose DIND container is missing is skipped in the status response rather than failing the whole batch with a 404, which used to leave every other pod on the node unreconciled. - envVars, fpgaArgs and gpuArgs are scoped to a single container; they previously leaked from one container to the next. - DIND subnet pool: rollback on failed creates, an orphan-network reaper, and the missing exit-code check on docker network create. - The wait for the mesh_ready sentinel is bounded, and the network-overlay container is selected by name instead of assumed to be containers[0]. - FPGA assignment tracks every assignee, no longer double-locks its mutex, and no longer sorts the shared list without holding one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.