feat: parallelize bundle/image processing in prepare-restricted-environment.sh

[Fortune-Ndlovu]

This PR significantly improves the execution speed of prepare-restricted-environment.sh by parallelizing bundle and image processing operations that were previously sequential. The core changes introduce a FIFO-based global semaphore (sem_init/sem_acquire/sem_release) that enforces MAX_PARALLEL as a true global cap across all concurrent image operations, replacing the previous per-level PID-counting approach that could allow up to MAX_PARALLEL² simultaneous operations.

Bundle processing (process_single_bundle, process_single_bundle_from_dir) and image mirroring now run concurrently with proper backpressure, while registry URL lookups (buildRegistryUrl) are cached at startup to eliminate hundreds of redundant subshell forks and potential oc API calls in hot loops.

Additional performance wins come from inlining replaceInternalRegIfNeeded and extract_last_two_elements in hot paths via a fork-free nameref helper (_last_two), removing ~400+ unnecessary fork()/exec() cycles per run.

The PR also fixes several correctness issues: quoting $TMPDIR in EXIT traps to handle paths with spaces, guarding trap cleanup commands with || true to prevent failures under set -euo pipefail, and correcting a tag-loss bug in
mirror_extra_images_from_dir where ${extraImg%:*} was stripping the tag from targetImg instead of preserving it.

Which issue(s) does this PR fix or relate to

Resolves: https://redhat.atlassian.net/browse/RHDHBUGS-3432

PR acceptance criteria

• Tests
• Documentation

How to test changes / Special notes to the reviewer

# Export to dir
bash .rhdh/scripts/prepare-restricted-environment.sh \
  --to-dir /tmp/rhdh-export-test \
  --filter-versions '1.10' \
  --install-operator false

# Push from dir to OCP internal registry and install
/tmp/rhdh-export-test/install.sh --from-dir /tmp/rhdh-export-test/ --install-operator true

Signed-off-by: Fortune-Ndlovu fndlovu@redhat.com
Assisted-by: Claude (claude-opus-4-6)

[Fortune-Ndlovu]

/agentic_review

[rhdh-qodo-merge]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials

✅ Compliance rules (platform): 18 rules

1. Unquoted $TMPDIR in EXIT trap ✓ Resolved 📘 Rule violation ≡ Correctness

Description

The EXIT trap runs rm -fr $TMPDIR, which leaves $TMPDIR unquoted and subject to word
splitting/globbing. This can fail on paths with spaces or delete unintended paths during trap-driven
cleanup.

Code

.rhdh/scripts/prepare-restricted-environment.sh[437]

+  trap "rm -fr $TMPDIR || true; jobs -p | xargs -r kill 2>/dev/null; wait 2>/dev/null" EXIT

Relevance

⭐⭐⭐ High

PR #1305 moved TMPDIR cleanup trap to quoted "$TMPDIR" form; team regularly accepts quoting-in-trap
fixes.

PR-#1305

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 2 requires quoting all path variables used in EXIT-trap cleanup commands. The added
EXIT trap includes rm -fr $TMPDIR, where $TMPDIR is unquoted while being used as an rm
argument.

Rule 2: Quote variables in EXIT traps performing filesystem cleanup in Bash
.rhdh/scripts/prepare-restricted-environment.sh[435-438]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The EXIT trap performs filesystem cleanup using an unquoted path variable (`rm -fr $TMPDIR`), which can break on spaces or expand globs.

## Issue Context
This script runs with `set -euo pipefail`, and `$TMPDIR` is used as a filesystem path. Cleanup in traps must quote path variables.

## Fix Focus Areas
- .rhdh/scripts/prepare-restricted-environment.sh[435-438]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Tag lost on replay ✓ Resolved 🐞 Bug ≡ Correctness

Description

In mirror_extra_images_from_dir, tag_* extra images compute targetImg by stripping the tag
(${extraImg%:*}), so skopeo pushes to an untagged/default-tag destination instead of the
intended tag. This can overwrite the wrong tag (often :latest) and makes --from-dir replays of
tag-based extra images incorrect.

Code

.rhdh/scripts/prepare-restricted-environment.sh[R738-751]

+  for tag_dir in "${tag_dirs[@]}"; do
+    local relative_path=${tag_dir#"$BASE_DIR/"}
+    local tag_hash=${tag_dir##*/tag_}
+    local parent_path
+    parent_path=$(dirname "$relative_path")
+    local lastTwo
+    lastTwo=$(extract_last_two_elements "${parent_path}")
+    local extraImg="${lastTwo}:${tag_hash}"
+    if [[ -n "$TO_REGISTRY" ]]; then
+      local targetImg
+      targetImg="$(buildRegistryUrl)/${extraImg%:*}"
+      throttle_parallel pids
+      push_image_from_archive "$tag_dir" "$targetImg" &
+      pids+=($!)

Relevance

⭐⭐ Medium

No clear historical review precedent found about preserving tags in from-dir replay target image
computation.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The tag_* replay path explicitly strips the tag when building targetImg, unlike the
non-from_dir extra-image mirroring path which appends :${imgTag}.

.rhdh/scripts/prepare-restricted-environment.sh[738-755]
.rhdh/scripts/prepare-restricted-environment.sh[639-655]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`mirror_extra_images_from_dir` constructs `extraImg="${lastTwo}:${tag_hash}"` but then sets `targetImg` using `${extraImg%:*}`, which removes the tag. This causes pushed images to lose their original tag.

### Issue Context
The non-`from_dir` path (`mirror_extra_images`) preserves tags when building `targetImg`, so the `from_dir` path should mirror that behavior.

### Fix Focus Areas
- .rhdh/scripts/prepare-restricted-environment.sh[738-751]

### Implementation notes
- For `tag_*` directories, set `targetImg="$(buildRegistryUrl)/${extraImg}"` (or `"$(buildRegistryUrl)/${lastTwo}:${tag_hash}"`) instead of using `${extraImg%:*}`.
- Keep the digest path unchanged.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Parallelism exceeds MAX_PARALLEL ✓ Resolved 🐞 Bug ☼ Reliability

Description

--max-parallel/MAX_PARALLEL is enforced independently for (a) bundle workers and (b) each
bundle’s inner related-image workers, so total concurrent image operations can reach roughly
MAX_PARALLEL * MAX_PARALLEL (plus extra-images), not MAX_PARALLEL. This can overload
registries/hosts and contradicts the advertised meaning of the flag.

Code

.rhdh/scripts/prepare-restricted-environment.sh[R812-849]

+      local csv_sed_file="${sed_commands_dir}/${digest}_csv.sed"
+      : > "$csv_sed_file"
+      local inner_pids=()
+
+      for relatedImage in "${all_related_images[@]}"; do
+        local imgDir="./images/"
+        local lastTwo targetImg internalTargetImg
+        if [[ "$relatedImage" == *"@sha256:"* ]]; then
+          local relatedImageDigest="${relatedImage##*@sha256:}"
+          imgDir+="${relatedImage%@*}/sha256_$relatedImageDigest"
+          lastTwo=$(extract_last_two_elements "${relatedImage%@*}")
+          targetImg="$(buildRegistryUrl)/${lastTwo}:$relatedImageDigest"
+          internalTargetImg="$(buildRegistryUrl "internal")/${lastTwo}:$relatedImageDigest"
+        elif [[ "$relatedImage" == *":"* ]]; then
+          local relatedImageTag="${relatedImage##*:}"
+          imgDir+="${relatedImage%:*}/tag_$relatedImageTag"
+          lastTwo=$(extract_last_two_elements "${relatedImage%:*}")
+          targetImg="$(buildRegistryUrl)/${lastTwo}:$relatedImageTag"
+          internalTargetImg="$(buildRegistryUrl "internal")/${lastTwo}:$relatedImageTag"
+        else
+          imgDir+="${relatedImage}/tag_latest"
+          lastTwo=$(extract_last_two_elements "${relatedImage}")
+          targetImg="$(buildRegistryUrl)/${lastTwo}:latest"
+          internalTargetImg="$(buildRegistryUrl "internal")/${lastTwo}:latest"
+        fi

-            if [[ -n "$TO_REGISTRY" ]]; then
-              mirror_image_to_registry "$relatedImage" "$targetImg"
-              debugf "replacing $relatedImage in file '${file}' => $internalTargetImg"
-              sed -i 's#'"$relatedImage"'#'"$internalTargetImg"'#g' "$file"
-            else
-              if [ ! -d "$imgDir" ]; then
-                mkdir -p "${imgDir}"
-                mirror_image_to_archive "$relatedImage" "$imgDir"
-              fi
-            fi
-          done
+        if [[ -n "$TO_REGISTRY" ]]; then
+          echo "s#${relatedImage}#${internalTargetImg}#g" >> "$csv_sed_file"
+          throttle_parallel inner_pids
+          mirror_image_to_registry "$relatedImage" "$targetImg" &
+          inner_pids+=($!)
+        else
+          if [ ! -d "$imgDir" ]; then
+            mkdir -p "${imgDir}"
+            throttle_parallel inner_pids
+            mirror_image_to_archive "$relatedImage" "$imgDir" &
+            inner_pids+=($!)
+          fi

Relevance

⭐⭐ Medium

Repo uses MAX_PARALLEL throttling (PR #2870), but no evidence reviewers require a single global cap
vs nested limits.

PR-#2870

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

process_bundles starts up to MAX_PARALLEL bundle workers, and each process_single_bundle can
start up to MAX_PARALLEL related-image mirror jobs simultaneously via its own inner_pids
throttle; the limits do not share a global cap.

.rhdh/scripts/prepare-restricted-environment.sh[875-904]
.rhdh/scripts/prepare-restricted-environment.sh[812-853]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
Concurrency limiting is currently hierarchical: outer bundle processing throttles bundle jobs, while each bundle job also throttles its own related-image mirroring. This allows the total number of simultaneous `skopeo` operations to multiply beyond `MAX_PARALLEL`.

### Issue Context
`MAX_PARALLEL` is documented as the maximum number of parallel image operations, but separate PID lists (`pids` vs `inner_pids`) each allow up to `MAX_PARALLEL` active jobs concurrently.

### Fix Focus Areas
- .rhdh/scripts/prepare-restricted-environment.sh[875-904]
- .rhdh/scripts/prepare-restricted-environment.sh[812-853]

### Implementation notes
Choose one approach:
- **Global semaphore**: implement a simple global token bucket (FIFO/`mkfifo`) and acquire/release tokens around *every* backgrounded skopeo/umoci/push operation.
- **Single shared PID list**: pass a single shared PID array (or a lock-protected file of PIDs) to `throttle_parallel` for all image operations, not one per nesting level.
- If you want two tiers, introduce two explicit knobs (e.g., `MAX_BUNDLE_PARALLEL` and `MAX_IMAGE_PARALLEL`) and document the combined effect.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. EXIT trap kill/wait unguarded ✓ Resolved 📘 Rule violation ☼ Reliability

Description

The EXIT trap runs jobs -p | xargs -r kill ...; wait ... under set -euo pipefail without
guarding failures, making cleanup brittle. If kill or wait returns non-zero (including benign
cases like already-exited jobs), trap execution can be interrupted or propagate an unintended
failure status during shutdown.

Code

.rhdh/scripts/prepare-restricted-environment.sh[R432-437]

+  trap "jobs -p | xargs -r kill 2>/dev/null; wait 2>/dev/null" EXIT
+  trap "exit 1" INT TERM
else
  TMPDIR=$(mktemp -d)
  # shellcheck disable=SC2064
-  trap "rm -fr $TMPDIR || true" EXIT
+  trap "rm -fr $TMPDIR || true; jobs -p | xargs -r kill 2>/dev/null; wait 2>/dev/null" EXIT

Relevance

⭐⭐⭐ High

PR #2870 accepted making EXIT/INT/TERM traps best-effort under set -e (guard kill/wait failures).

PR-#2870

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 3 requires EXIT-trap cleanup to be guarded so that failures in best-effort shutdown
logic do not cause the script to fail under set -e. The script runs in strict mode (`set -euo
pipefail) and installs EXIT traps that execute a jobs -p | xargs -r kill ...` pipeline and a
subsequent wait without any neutralizing guard such as || true/|| :, a { ...; } || true
wrapper, or a local set +e, so any non-zero status from kill or wait can affect trap
completion and overall exit status.

Rule 3: Cleanup commands in EXIT traps must not cause script failure
.rhdh/scripts/prepare-restricted-environment.sh[429-438]
.rhdh/scripts/prepare-restricted-environment.sh[8-9]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The script’s EXIT-trap cleanup executes a `jobs -p | xargs -r kill ...` pipeline and a final `wait` under global `set -euo pipefail` without guarding their exit statuses. Cleanup should be best-effort so benign non-zero returns (e.g., attempting to kill jobs that already exited, or `wait` returning non-zero) do not abort trap execution or cause the script to exit with an unintended failure.

## Issue Context
The script enables strict mode globally (`set -euo pipefail`) and then installs EXIT traps that run `jobs | xargs kill; wait`. Per policy (PR Compliance ID 3), EXIT-trap cleanup commands must be guarded so their failure does not cause or propagate failures during shutdown; use `|| true` / `|| :`, wrap the cleanup block as best-effort (`{ ...; } || true`), or temporarily disable `-e` inside the trap with `set +e` (restoring prior state if needed).

## Fix Focus Areas
- .rhdh/scripts/prepare-restricted-environment.sh[429-439]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[Fortune-Ndlovu]

/agentic_review

[rhdh-qodo-merge]

Qodo is busy working

Check back in a few minutes. Qodo's code review agents are on it.

[Fortune-Ndlovu]

/agentic_review

[rhdh-qodo-merge]

Code Review by Qodo

Sorry, something went wrong

We weren't able to complete the code review on our side. Please try again

[Fortune-Ndlovu]

/agentic_review

[rhdh-qodo-merge]

Code Review by Qodo

Sorry, something went wrong

We weren't able to complete the code review on our side. Please try again

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Fortune-Ndlovu]

/agentic_review

[rhdh-qodo-merge]

Qodo is busy working

Check back in a few minutes. Qodo's code review agents are on it.