Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 119 additions & 0 deletions internal/scaffold/fullsend-repo/scripts/pre-retro-test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
#!/usr/bin/env bash
# pre-retro-test.sh — Test pre-retro.sh GH_TOKEN validation.
#
# Uses a mock gh command to simulate auth success/failure.
# Run from the repo root: bash internal/scaffold/fullsend-repo/scripts/pre-retro-test.sh

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PRE_SCRIPT="${SCRIPT_DIR}/pre-retro.sh"
FAILURES=0

TMPDIR="$(mktemp -d)"
trap 'rm -rf "${TMPDIR}"' EXIT

# --- Helpers ---

# build_mock creates a mock gh binary.
# Arguments:
# $1 — exit code for "gh auth status" (0 = valid, 1 = invalid)
build_mock() {
local auth_exit="$1"
local mock_bin="${TMPDIR}/bin"

rm -rf "${mock_bin}"
mkdir -p "${mock_bin}"

cat > "${mock_bin}/gh" <<MOCKEOF
#!/usr/bin/env bash
if [[ "\$1" == "auth" && "\$2" == "status" ]]; then
exit ${auth_exit}
fi
MOCKEOF

chmod +x "${mock_bin}/gh"
echo "${mock_bin}"
}

run_test_stdout() {
local test_name="$1"
local auth_exit="$2"
local expected_stdout="$3"
local expect_exit="$4"
local extra_env="${5:-}"

local mock_bin
mock_bin="$(build_mock "${auth_exit}")"

local env_cmd=(
env
PATH="${mock_bin}:${PATH}"
ORIGINATING_URL="https://github.com/test-org/test-repo/pull/42"
)

if [[ -n "${extra_env}" ]]; then
while IFS= read -r kv; do
[[ -n "${kv}" ]] && env_cmd+=("${kv}")
done <<< "${extra_env}"
fi

local exit_code=0
"${env_cmd[@]}" bash "${PRE_SCRIPT}" > "${TMPDIR}/stdout.log" 2>&1 || exit_code=$?

if [[ ${exit_code} -ne ${expect_exit} ]]; then
echo "FAIL: ${test_name} — expected exit ${expect_exit}, got ${exit_code}"
cat "${TMPDIR}/stdout.log"
FAILURES=$((FAILURES + 1))
return
fi

if ! grep -qF "${expected_stdout}" "${TMPDIR}/stdout.log" 2>/dev/null; then
echo "FAIL: ${test_name} — expected stdout '${expected_stdout}' not found"
echo "Actual stdout:"
cat "${TMPDIR}/stdout.log"
FAILURES=$((FAILURES + 1))
return
fi

echo "PASS: ${test_name}"
}

# --- Test cases ---

# GH_TOKEN valid → pass through with success message.
run_test_stdout "valid-token-passes" \
0 \
"GH_TOKEN validated successfully." \
0 \
"GH_TOKEN=fake-valid-token"

# GH_TOKEN invalid → exit 1 with error.
run_test_stdout "invalid-token-fails" \
1 \
"GH_TOKEN is invalid" \
1 \
"GH_TOKEN=bad-token"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] test-adequacy

Test 'no-token-warns' checks for warning message and exit 0 but does not verify the script runs to completion (unlike 'valid-token-continues-to-completion' which checks for 'Pre-retro validation complete.').

# GH_TOKEN unset → warn, don't fail.
run_test_stdout "no-token-warns" \
0 \
"GH_TOKEN is not set" \
0 \
"GH_TOKEN="

# GH_TOKEN valid → still shows retro target after validation.
run_test_stdout "valid-token-continues-to-completion" \
0 \
"Pre-retro validation complete." \
0 \
"GH_TOKEN=fake-valid-token"

# --- Summary ---

echo ""
if [[ ${FAILURES} -gt 0 ]]; then
echo "${FAILURES} test(s) failed"
exit 1
fi
echo "All tests passed"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] test-adequacy

Missing edge case: whitespace-only GH_TOKEN (e.g., ' ') passes the -n check and would attempt token validation. No test covers this.

15 changes: 14 additions & 1 deletion internal/scaffold/fullsend-repo/scripts/pre-retro.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,15 @@
# pre-retro.sh — Validate inputs for the retro agent.
#
# Runs on the host via the harness pre_script mechanism. Validates the
# originating URL (PR or issue) and logs the trigger context.
# originating URL (PR or issue), checks GH_TOKEN validity, and logs
# the trigger context.
#
# Required env vars:
# ORIGINATING_URL — HTML URL of the PR or issue that triggered retro
#
# Optional env vars:
# RETRO_COMMENT — The /retro comment text (empty for automatic triggers)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] contract-mismatch

GH_TOKEN is listed as 'Optional' and the else branch warns-and-continues when unset. But post-retro.sh line 17 hard-fails with : ${GH_TOKEN:?GH_TOKEN is required}. The warning path wastes compute — sandbox + model inference runs then post-retro fails.

Suggested fix: Make GH_TOKEN required in pre-retro.sh with : "${GH_TOKEN:?GH_TOKEN is required}" to match the post-retro.sh contract and fail fast.

# GH_TOKEN — GitHub token for API access

set -euo pipefail

Expand All @@ -20,6 +22,17 @@ if [[ ! "${ORIGINATING_URL}" =~ ^https://github\.com/[a-zA-Z0-9._-]+/[a-zA-Z0-9.
exit 1
fi

# Validate GH_TOKEN before starting the sandbox.
if [[ -n "${GH_TOKEN:-}" ]]; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[high] correctness

gh auth status produces false negatives — rejects tokens that work for API access. Verified: gh api /rate_limit succeeds (4972 remaining) while gh auth status returns exit 1 with 'The token in GH_TOKEN is invalid' on this runner (gh 2.93.0). This would block valid retro runs, violating issue #511 validation criterion 4.

Suggested fix: Replace gh auth status with a lightweight API call: if ! gh api /rate_limit >/dev/null 2>&1; then echo '::error::GH_TOKEN is invalid'; exit 1; fi

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] error-handling

2>/dev/null on the validation call suppresses diagnostic stderr. Network failures, DNS errors, and TLS issues are indistinguishable from invalid tokens — operators see only the generic ::error:: message.

Suggested fix: Let stderr through to the step log, or capture and include in the error message on failure.

if ! gh auth status 2>/dev/null; then
echo "::error::GH_TOKEN is invalid — retro agent requires GitHub API access"
exit 1
fi
echo "GH_TOKEN validated successfully."
else
echo "::warning::GH_TOKEN is not set — retro agent will have limited functionality"
fi

echo "::notice::Retro target: ${ORIGINATING_URL}"

if [[ -n "${RETRO_COMMENT:-}" ]]; then
Expand Down
Loading