Skip to content

Finalize terminal PR orchestration in the single-job flow #501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
sepo-agent-app wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Finalize terminal PR orchestration in the single-job flow #501

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion .agent/docs/architecture/agent-orchestrator.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,14 +145,25 @@ the same compact table style while preserving their hidden durable markers.
If terminal child metadata is found but rejected by trust checks or cannot be
safely updated, the dispatcher posts a compact stop comment on the current
terminal issue or PR with a hidden dedupe marker. Ordinary terminal PR stops
without sub-orchestrator metadata remain silent.
without sub-orchestrator metadata finalize one visible note. The dispatcher
merges the outcome into the current planner progress comment when available;
otherwise it updates or creates a trusted comment carrying the existing
`sepo-agent-orchestrate-stop` marker. The note includes the planner summary,
source outcome, target, round, reason, run ID, and a requester mention only for
human GitHub logins.
If the resumed parent planner decides there is no next child or action, the
parent run posts a terminal stop comment on the parent issue with the source
conclusion, target, round, reason, and hidden `sepo-agent-orchestrate-stop`
marker. Exact trusted duplicates are skipped on reruns.
When the planner returns `blocked` with `user_message` or
`clarification_request`, that same terminal comment surfaces the planner's
question directly and the chain pauses without dispatching an `answer` route.
After a validated terminal `review`/`SHIP`, `agent-self-approve`/`approved`, or
`agent-self-merge`/`merged` or `auto_merge_enabled` outcome, the dispatcher
best-effort minimizes older trusted review synthesis, rubrics review, fix-pr,
and completed handoff comments from the PR conversation. It keeps the final
note and pending handoffs visible, skips cleanup for every other outcome, and
warns without hiding the final note when cleanup fails.

Initial user-launched `/orchestrate` requests validate that the requester has
access to the delegated route capability set before dispatching work. When
Expand Down
8 changes: 8 additions & 0 deletions .agent/docs/usage/supported-workflows.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,10 @@ memory and rubrics read-only so automated control-flow planning can use steering
context without mutating those state branches. Orchestration stops when target
state indicates no safe next action, a route fails, a duplicate handoff marker
is found, the planner stops or blocks, or the max-round budget is exhausted.
Terminal PR stops reuse the current run's progress comment when available, or
update one trusted `sepo-agent-orchestrate-stop` marker comment otherwise. The
final note includes the planner summary and outcome metadata and mentions the
original requester only when it is a human GitHub login.

When a child issue reaches a terminal stop, the handoff dispatcher resolves the
trusted child metadata from the issue body or an agent-authored child issue
Expand Down Expand Up @@ -122,6 +126,10 @@ HTML markers for robust matching, with heading/text fallbacks for older
comments. Rubrics reviews match the `## Rubrics Review` heading, and
orchestrator handoffs match their hidden handoff marker. This keeps the latest
generated status prominent while leaving older generated comments expandable.
Successful terminal PR orchestration also best-effort minimizes older matching
conversation comments while leaving the final note and pending handoffs
visible. Blocked, clarification, failed, malformed-planner, and other
non-success outcomes never trigger this terminal cleanup.
Set `AGENT_COLLAPSE_OLD_REVIEWS=false` to skip this cleanup and leave prior
generated comments visible.

Expand Down
306 changes: 303 additions & 3 deletions .agent/src/__tests__/orchestrate-handoff-cli.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,10 @@ if [ "\${1-}" = "pr" ] && [ "\${2-}" = "view" ]; then
exit 1
fi
if [[ "$*" == *"body"* ]]; then
if [ "\${FAKE_PR_BODY_MODE-}" = "error" ]; then
printf 'pull request body unavailable\\n' >&2
exit 1
fi
printf '{"body":"%s"}\\n' "\${FAKE_PR_BODY-}"
exit 0
fi
Expand Down Expand Up @@ -90,6 +94,15 @@ if [ "\${1-}" = "api" ] && [ "\${2-}" = "--paginate" ] && [ "\${3-}" = "--slurp"
exit 0
fi

if [ "\${1-}" = "api" ] && [[ "\${2-}" == repos/*/issues/comments/* ]] && [ "\${3-}" = "--jq" ] && [ "\${4-}" = ".body" ]; then
if [ "\${FAKE_PROGRESS_BODY_MODE-}" = "error" ]; then
printf 'progress comment unavailable\\n' >&2
exit 1
fi
printf '%s\\n' "\${FAKE_PROGRESS_BODY-}"
exit 0
fi

if [ "\${1-}" = "api" ] && [ "\${2-}" = "--paginate" ] && [[ "\${3-}" == repos/*/issues/*/sub_issues ]]; then
if [ "\${FAKE_SUB_ISSUES_MODE-}" = "error" ]; then
printf 'sub-issues unavailable\\n' >&2
Expand Down Expand Up @@ -151,6 +164,14 @@ if [ "\${1-}" = "api" ] && [ "\${2-}" = "--method" ] && [ "\${3-}" = "PATCH" ] &
exit 0
fi

if [ "\${1-}" = "pr" ] && [ "\${2-}" = "comment" ]; then
if [ "\${FAKE_PR_COMMENT_MODE-}" = "error" ]; then
printf 'pull request comment unavailable\\n' >&2
exit 1
fi
exit 0
fi

if [ "\${1-}" = "api" ] && [ "\${2-}" = "-X" ] && [ "\${3-}" = "POST" ] && [[ "\${4-}" == repos/*/actions/workflows/*/dispatches ]]; then
cat > "$FAKE_DISPATCH_PAYLOAD"
exit 0
Expand Down Expand Up @@ -1501,7 +1522,7 @@ test("heuristics parent orchestrate stops do not post final comments", () => {
assert.equal(run.dispatchPayload, null);
});

test("agent parent orchestrate stops for pull requests do not post final comments", () => {
test("agent parent orchestrate stops for pull requests publish a final note", () => {
const run = runOrchestrateHandoff({
SOURCE_ACTION: "orchestrate",
SOURCE_CONCLUSION: "done",
Expand All @@ -1517,12 +1538,291 @@ test("agent parent orchestrate stops for pull requests do not post final comment
assert.equal(run.status, 0, run.stderr || run.stdout);
assert.equal(run.outputs.get("decision"), "stop");
assert.equal(run.outputs.get("reason"), "pull request is closed");
assert.doesNotMatch(run.ghLog, /api --method POST repos\/self-evolving\/repo\/issues\/76\/comments/);
assert.doesNotMatch(run.ghLog, /<!-- sepo-agent-orchestrate-stop -->/);
assert.match(run.ghLog, /pr comment 76/);
assert.match(run.ghLog, /Sepo orchestration stopped after `orchestrate` concluded `done`\./);
assert.match(run.ghLog, /> Requested by @lolipopshock\./);
assert.match(run.ghLog, /<!-- sepo-agent-orchestrate-stop -->/);
assert.doesNotMatch(run.ghLog, /actions\/workflows\//);
assert.equal(run.dispatchPayload, null);
});

test("parent resolution failures do not suppress the terminal PR note", () => {
const run = runOrchestrateHandoff({
SOURCE_ACTION: "review",
SOURCE_CONCLUSION: "SHIP",
TARGET_KIND: "pull_request",
TARGET_NUMBER: "128",
AUTOMATION_MODE: "heuristics",
AGENT_COLLAPSE_OLD_REVIEWS: "false",
FAKE_PR_BODY_MODE: "error",
});

assert.equal(run.status, 0, run.stderr || run.stdout);
assert.match(run.stderr, /Failed to report terminal sub-orchestration state/);
assert.match(run.ghLog, /pr comment 128/);
assert.match(run.ghLog, /<!-- sepo-agent-orchestrate-stop -->/);
});

test("terminal PR publication failures fail the handoff step", () => {
const run = runOrchestrateHandoff({
SOURCE_ACTION: "review",
SOURCE_CONCLUSION: "SHIP",
TARGET_KIND: "pull_request",
TARGET_NUMBER: "128",
AUTOMATION_MODE: "heuristics",
AGENT_COLLAPSE_OLD_REVIEWS: "false",
FAKE_PR_COMMENT_MODE: "error",
});

assert.equal(run.status, 1, run.stderr || run.stdout);
assert.match(run.stderr, /Failed to publish terminal pull request note/);
});

test("successful terminal PR stop merges its summary into the progress note", () => {
const run = runOrchestrateHandoff({
SOURCE_ACTION: "review",
SOURCE_CONCLUSION: "SHIP",
TARGET_KIND: "pull_request",
TARGET_NUMBER: "128",
AUTOMATION_MODE: "agent",
AUTOMATION_CURRENT_ROUND: "4",
AUTOMATION_MAX_ROUNDS: "8",
SOURCE_RUN_ID: "review-run-123",
AGENT_COLLAPSE_OLD_REVIEWS: "false",
AGENT_PROGRESS_COMMENT_ID: "777",
AGENT_PROGRESS_FINAL_COMMENT_MODE: "merge",
MODEL_DISPLAY: "`codex` | `gpt-test`",
FAKE_PROGRESS_BODY: [
"### Sepo is working",
"",
"Latest planner activity",
"",
"<!-- sepo-progress:run-123 -->",
].join("\n"),
FAKE_PLANNER_RESPONSE: JSON.stringify({
decision: "stop",
reason: "The reviewed implementation is ready.",
user_message: "Implementation and review completed successfully.",
}),
});

assert.equal(run.status, 0, run.stderr || run.stdout);
assert.equal(run.outputs.get("decision"), "stop");
assert.match(run.ghLog, /api repos\/self-evolving\/repo\/issues\/comments\/777 --jq \.body/);
assert.match(run.ghLog, /api --method PATCH repos\/self-evolving\/repo\/issues\/comments\/777/);
assert.match(run.ghLog, /Sepo orchestration finished successfully after `review` concluded `SHIP`\./);
assert.match(run.ghLog, /Implementation and review completed successfully\./);
assert.match(run.ghLog, /> Requested by @lolipopshock\./);
assert.match(run.ghLog, /<!-- sepo-agent-orchestrate-stop -->/);
assert.match(run.ghLog, /<!-- sepo-progress:run-123 -->/);
assert.match(run.ghLog, /`codex` \| `gpt-test`/);
assert.doesNotMatch(run.ghLog, /pr comment 128/);
});

test("terminal PR fallback updates the trusted marker note and suppresses bot mentions", () => {
const run = runOrchestrateHandoff({
SOURCE_ACTION: "review",
SOURCE_CONCLUSION: "SHIP",
TARGET_KIND: "pull_request",
TARGET_NUMBER: "128",
AUTOMATION_MODE: "agent",
AUTOMATION_CURRENT_ROUND: "4",
AGENT_COLLAPSE_OLD_REVIEWS: "false",
REQUESTED_BY: "sepo-agent-app[bot]",
FAKE_ISSUE_COMMENTS_JSON: JSON.stringify([{
id: "existing-final",
body: "Old final note\n\n<!-- sepo-agent-orchestrate-stop -->",
created_at: "2026-08-10T00:00:00Z",
user: { login: "app/sepo-agent-app" },
}]),
FAKE_PLANNER_RESPONSE: JSON.stringify({
decision: "stop",
reason: "The reviewed implementation is ready.",
user_message: "Implementation and review completed successfully.",
}),
});

assert.equal(run.status, 0, run.stderr || run.stdout);
assert.match(run.ghLog, /api --method PATCH repos\/self-evolving\/repo\/issues\/comments\/existing-final/);
assert.doesNotMatch(run.ghLog, /pr comment 128/);
assert.doesNotMatch(run.ghLog, /Requested by/);
assert.match(run.stdout, /Updated orchestrator final comment\./);
});

test("successful terminal PR cleanup minimizes only older conversation artifacts", () => {
const generated = (id: string, databaseId: number, body: string) => ({
id,
databaseId,
body,
isMinimized: false,
author: { login: "sepo-agent-app[bot]" },
});
const run = runOrchestrateHandoff({
SOURCE_ACTION: "review",
SOURCE_CONCLUSION: "SHIP",
TARGET_KIND: "pull_request",
TARGET_NUMBER: "128",
AUTOMATION_MODE: "agent",
AGENT_PROGRESS_COMMENT_ID: "777",
AGENT_PROGRESS_FINAL_COMMENT_MODE: "merge",
FAKE_PROGRESS_BODY: "### Sepo is working\n\n<!-- sepo-progress:run-123 -->",
FAKE_GRAPHQL_PR_COMMENTS: JSON.stringify([
generated("review-summary", 700, "## AI Review Synthesis\nold"),
generated("rubrics-review", 701, "## Rubrics Review\nold"),
generated("fix-status", 702, "<!-- sepo-agent-fix-pr-status -->"),
generated("handoff", 703, "<!-- sepo-agent-handoff state:dispatched created:123 base64:aGFuZG9m -->"),
generated("pending", 704, "<!-- sepo-agent-handoff state:pending created:456 base64:cGVuZGluZw -->"),
generated("old-final", 705, "Old final\n<!-- sepo-agent-orchestrate-stop -->"),
generated("current-final", 777, "Current final\n<!-- sepo-agent-orchestrate-stop -->"),
generated("newer-review", 800, "## AI Review Synthesis\nnewer"),
{ ...generated("human", 706, "## AI Review Synthesis\nhuman"), author: { login: "lolipopshock" } },
]),
FAKE_PLANNER_RESPONSE: JSON.stringify({
decision: "stop",
reason: "The reviewed implementation is ready.",
user_message: "Implementation and review completed successfully.",
}),
});

assert.equal(run.status, 0, run.stderr || run.stdout);
for (const id of ["review-summary", "rubrics-review", "fix-status", "handoff", "old-final"]) {
assert.match(run.ghLog, new RegExp(`id=${id} -f classifier=OUTDATED`));
}
assert.doesNotMatch(run.ghLog, /id=pending -f classifier=OUTDATED/);
assert.doesNotMatch(run.ghLog, /id=current-final -f classifier=OUTDATED/);
assert.doesNotMatch(run.ghLog, /id=newer-review -f classifier=OUTDATED/);
assert.doesNotMatch(run.ghLog, /id=human -f classifier=OUTDATED/);
assert.doesNotMatch(run.ghLog, /PullRequestReviewSummaries/);
assert.match(run.stdout, /Collapsed 5 previous orchestration artifact comment\(s\)\./);
});

for (const [sourceAction, sourceConclusion] of [
["agent-self-approve", "approved"],
["agent-self-merge", "merged"],
["agent-self-merge", "auto_merge_enabled"],
] as const) {
test(`${sourceAction} ${sourceConclusion} is eligible for terminal PR cleanup`, () => {
const run = runOrchestrateHandoff({
SOURCE_ACTION: sourceAction,
SOURCE_CONCLUSION: sourceConclusion,
TARGET_KIND: "pull_request",
TARGET_NUMBER: "128",
AUTOMATION_MODE: "heuristics",
AGENT_PROGRESS_COMMENT_ID: "777",
AGENT_PROGRESS_FINAL_COMMENT_MODE: "merge",
FAKE_PROGRESS_BODY: "### Sepo is working\n\n<!-- sepo-progress:run-123 -->",
FAKE_GRAPHQL_PR_COMMENTS: "[]",
});

assert.equal(run.status, 0, run.stderr || run.stdout);
assert.match(run.ghLog, /PullRequestReviewSummaryComments/);
});
}

for (const [name, env] of [
["blocked planner", {
SOURCE_ACTION: "review",
SOURCE_CONCLUSION: "SHIP",
AUTOMATION_MODE: "agent",
FAKE_PLANNER_RESPONSE: JSON.stringify({
decision: "blocked",
reason: "Need maintainer input.",
clarification_request: "Should this ship now?",
}),
}],
["planner stop without summary", {
SOURCE_ACTION: "review",
SOURCE_CONCLUSION: "SHIP",
AUTOMATION_MODE: "agent",
FAKE_PLANNER_RESPONSE: JSON.stringify({
decision: "stop",
reason: "No cumulative summary was produced.",
}),
}],
["malformed planner", {
SOURCE_ACTION: "review",
SOURCE_CONCLUSION: "SHIP",
AUTOMATION_MODE: "agent",
FAKE_PLANNER_RESPONSE: "not json",
}],
["failed source", {
SOURCE_ACTION: "review",
SOURCE_CONCLUSION: "failed",
AUTOMATION_MODE: "agent",
FAKE_PLANNER_RESPONSE: JSON.stringify({
decision: "stop",
reason: "The review failed.",
user_message: "Review did not complete.",
}),
}],
] as const) {
test(`${name} does not trigger terminal PR cleanup`, () => {
const run = runOrchestrateHandoff({
...env,
TARGET_KIND: "pull_request",
TARGET_NUMBER: "128",
AGENT_PROGRESS_COMMENT_ID: "777",
AGENT_PROGRESS_FINAL_COMMENT_MODE: "merge",
FAKE_PROGRESS_BODY: "### Sepo is working\n\n<!-- sepo-progress:run-123 -->",
FAKE_GRAPHQL_PR_COMMENTS: JSON.stringify([{
id: "review-summary",
body: "## AI Review Synthesis\nold",
isMinimized: false,
author: { login: "sepo-agent-app[bot]" },
}]),
});

assert.equal(run.status, 0, run.stderr || run.stdout);
assert.doesNotMatch(run.ghLog, /classifier=OUTDATED/);
});
}

test("enabled downstream automation prevents terminal cleanup on a budget stop", () => {
const run = runOrchestrateHandoff({
SOURCE_ACTION: "review",
SOURCE_CONCLUSION: "SHIP",
TARGET_KIND: "pull_request",
TARGET_NUMBER: "128",
AUTOMATION_MODE: "heuristics",
AUTOMATION_CURRENT_ROUND: "5",
AUTOMATION_MAX_ROUNDS: "5",
AGENT_ALLOW_SELF_APPROVE: "true",
FAKE_GRAPHQL_PR_COMMENTS: JSON.stringify([{
id: "review-summary",
body: "## AI Review Synthesis\nold",
isMinimized: false,
author: { login: "sepo-agent-app[bot]" },
}]),
});

assert.equal(run.status, 0, run.stderr || run.stdout);
assert.equal(run.outputs.get("reason"), "automation round budget exhausted");
assert.doesNotMatch(run.ghLog, /classifier=OUTDATED/);
});

test("terminal PR cleanup failure is visible and non-fatal", () => {
const run = runOrchestrateHandoff({
SOURCE_ACTION: "review",
SOURCE_CONCLUSION: "SHIP",
TARGET_KIND: "pull_request",
TARGET_NUMBER: "128",
AUTOMATION_MODE: "agent",
AGENT_PROGRESS_COMMENT_ID: "777",
AGENT_PROGRESS_FINAL_COMMENT_MODE: "merge",
FAKE_PROGRESS_BODY: "### Sepo is working\n\n<!-- sepo-progress:run-123 -->",
FAKE_GRAPHQL_MODE: "error",
FAKE_PLANNER_RESPONSE: JSON.stringify({
decision: "stop",
reason: "The reviewed implementation is ready.",
user_message: "Implementation and review completed successfully.",
}),
});

assert.equal(run.status, 0, run.stderr || run.stdout);
assert.match(run.ghLog, /api --method PATCH repos\/self-evolving\/repo\/issues\/comments\/777/);
assert.match(run.stderr, /Failed to collapse previous orchestration artifacts/);
});

test("terminal child result reports to parent and preserves terminal reruns", () => {
const childBody = "<!-- sepo-sub-orchestrator parent:76 stage:stage-1 state:running parent_round:2 -->";
const run = runOrchestrateHandoff({
Expand Down
Loading
Loading