ci(ff): support feature freeze automation by fr4nc1sc0-r4m0n · Pull Request #15433 · elastic/elastic-agent

fr4nc1sc0-r4m0n · 2026-07-09T09:35:52Z

What does this PR do?

This PR migrates the Elastic Agent release automation from Makefile-based scripts to a pure Go implementation using Mage, eliminating external tool dependencies for the feature-freeze workflow.

Key features

8 Mage commands for release workflow automation
Pure Go implementation (no external CLI tools such as hub, gh, sed, yq, or Python)
DRY_RUN mode for safe testing without pushing changes or creating branches
Unit tests for the release tooling
Documentation in RELEASE.md and dev-tools/mage/release/README.md

Idempotency

Release workflow steps are safe to re-run without failing or creating duplicate side effects:

Step	Behavior on re-run
`UpdateVersion`	No-op when `version/version.go` already has the target version
`UpdateDocs`	Skips files whose image tags already match (existing)
`UpdateMergify`	Skips when the backport rule already exists (existing)
`CreateBranch`	Checks out the existing branch instead of failing
`CommitAll`	Skips commit when the worktree is clean
`Push`	Treats already-up-to-date remote as success (existing)
`CreatePR`	Returns the existing open PR for the same head/base

This makes mage release:runMajorMinor and mage release:runPatch resilient to CI retries and partial failures.

This branch is rebased on current main with merge conflicts resolved in go.mod, go.sum, and .mergify.yml.

Why is it important?

Supports feature-freeze automation as part of the broader release workflow effort in #12711.

Idempotent steps reduce operational risk when automation is retried after transient failures (network, auth, or partial workflow completion).

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

Disruptive User Impact

No user-facing impact. This affects release automation tooling used by maintainers.

How to test this PR locally

Use a valid X.Y.Z release version (for example, the next planned minor release). Versions such as 9.5.0-test are rejected by the tooling.

export PROJECT_OWNER="your-user"
export CURRENT_RELEASE="9.6.0"
export GITHUB_TOKEN=$(gh auth token)
export DRY_RUN=true

mage release:runMajorMinor
git diff

go test ./dev-tools/mage/release/... -count=1

To verify idempotency, run the same workflow twice; the second run should complete without errors and without duplicate branches, commits, or PRs.

Related issues

Supersedes ci(ff): support feature freeze automation #13138
Relates add DRA version bump pipeline #12711

Related issue: https://github.com/elastic/observability-robots/issues/3404

this should help with manipulating the YAML using the YAML golang library

Resolve conflicts in .mergify.yml, go.mod, and go.sum by taking upstream/main as the base and retaining PR-specific release tooling dependencies (go-git, go-github).

infra-vault-gh-plugin-prod · 2026-07-09T09:35:56Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

infra-vault-gh-plugin-prod · 2026-07-09T09:35:57Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

Validate and allowlist repository paths before writing release files, and update tests to use supported relative manifest paths.

Allow re-running release workflows without duplicate branches, empty commits, duplicate PRs, or false version-update errors.

mallendem · 2026-07-10T09:15:48Z

+	}
+
+	// Pattern: docker.elastic.co/elastic-agent/elastic-agent:X.Y.Z
+	re := regexp.MustCompile(`(docker\.elastic\.co/elastic-agent/elastic-agent:)[0-9]+\.[0-9]+\.[0-9]+`)


Is there a way we can generalize? or at least surface the pattern to the magefile?

mallendem · 2026-07-10T09:19:27Z

+			return fmt.Errorf("failed to checkout existing branch: %w", err)
+		}
+
+		fmt.Printf("✓ Checked out existing branch: %s\n", branchName)


Do we want this? If someone creates a branch beforehand, we will checkout that and continue working, which could inject unwanted code.

mallendem · 2026-07-10T09:28:00Z

+		return err
+	}
+
+	dryRun := os.Getenv("DRY_RUN") == "true"


Suggested change

dryRun := os.Getenv("DRY_RUN") == "true"

var dryRun bool

if v, err := strconv.ParseBool(os.Getenv("DRY_RUN")); err == nil {

dryRun = v

}

maybe? or at least

Suggested change

dryRun := os.Getenv("DRY_RUN") == "true"

dryRun := strings.ToLower(os.Getenv("DRY_RUN")) == "true"

Replace invalid 9.5.0-test examples with 9.6.0 and document the required version format for local release workflow testing.

Use go-git CheckoutOptions.Keep so major/minor release workflow can prepare version files before checking out the release branch.

Replace go-git AddWithOptions(All: true) with per-path staging from worktree status so CommitAll includes every changed path (including submodule gitlinks like beats) without reading submodule directories.

Match the Beats release automation by requiring GITHUB_TOKEN for push operations and authenticating via HTTP basic auth with an explicit branch refspec.

swiatekm · 2026-07-10T16:44:08Z

buildkite test this

infra-vault-gh-plugin-prod · 2026-07-10T16:53:54Z

💔 Build Failed

Buildkite Build
Commit: e1b7898

Failed CI Steps

History

💛 Build #42705 was flaky d069e51

cc @fr4nc1sc0-r4m0n

github-actions · 2026-07-10T17:25:08Z

TL;DR

The Windows unit-test failures are a real regression in the new release tooling, independent of the Beats submodule mismatch already reported above. validateRepoRelativePath cleans paths with the host OS separator, but writeRepoFile allowlists only /-separated strings; on Windows version/version.go becomes version\version.go and is rejected as unsupported.

Remediation

Make the allowlist comparison platform-independent, for example by comparing filepath.ToSlash(safePath) with slash-separated constants, or by constructing the allowlist entries with filepath.Join.
Add/run the release package tests on Windows (especially TestUpdateVersion, TestUpdateDocs, TestUpdateVersionInFile, and TestPrepareMajorMinorRelease) and rerun the Windows matrix.

Investigation details

Root Cause

In dev-tools/mage/release/release.go:18-31, validateRepoRelativePath returns filepath.Clean(path). In writeRepoFile (release.go:33-47), the subsequent switch compares that cleaned value to literals such as version/version.go and deploy/kubernetes/elastic-agent-managed-kubernetes.yaml. On Windows, filepath.Clean produces version\version.go, so every write through this allowlist fails. The same path is passed by UpdateVersion and updateVersionInFile, causing the eight release-test failures.

The tests change into a temporary directory before invoking the functions (dev-tools/mage/release/release_test.go:18-49, 99-149, and 167-223), so the failure is in the production path validation rather than test fixture setup.

Evidence

Build: https://buildkite.com/elastic/elastic-agent/builds/42775
Windows logs: TestUpdateVersion fails with failed to write version\version.go: unsupported file path: version\version.go (lines 103-126); TestUpdateDocs and TestUpdateVersionInFile fail with the same separator mismatch (lines 115-125).
The identical eight release failures occur on Windows 11, Windows 11 arm64, and fips140=only Windows 11 arm64.
The ESS cleanup failure reports No artifacts found for downloading after the FIPS setup/package path failed; it is a downstream cleanup symptom, not an additional root cause.

Verification

Not reproduced locally because this checkout does not contain the PR branch’s new dev-tools/mage/release package and the failing Buildkite jobs are Windows-specific.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

v1v and others added 11 commits March 12, 2026 15:13

ci(ff): support feature freeze automation phase 1

90a603e

ci(ff): support feature freeze automation phase 2

831bfd3

ci(ff): support feature freeze automation phase 3

043f9eb

fix linting

79cf391

Merge branch 'main' into feature/release-scripts-for-ff

842cd1f

go mod tidy

9533815

remove

e91354e

dry-run

11d42f6

mergify: reorder

14dc49c

this should help with manipulating the YAML using the YAML golang library

Merge branch 'main' into feature/release-scripts-for-ff

ea53a5b

Merge upstream/main into feature/release-scripts-for-ff

d069e51

Resolve conflicts in .mergify.yml, go.mod, and go.sum by taking upstream/main as the base and retaining PR-specific release tooling dependencies (go-git, go-github).

fr4nc1sc0-r4m0n requested a review from a team as a code owner July 9, 2026 09:35

fr4nc1sc0-r4m0n added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog backport-active-all Automated backport with mergify to all the active branches labels Jul 9, 2026

fr4nc1sc0-r4m0n requested review from lorienhu and swiatekm July 9, 2026 09:35

fr4nc1sc0-r4m0n added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog backport-active-all Automated backport with mergify to all the active branches labels Jul 9, 2026

fr4nc1sc0-r4m0n self-assigned this Jul 9, 2026

fr4nc1sc0-r4m0n added the skip-ci label Jul 9, 2026

fr4nc1sc0-r4m0n mentioned this pull request Jul 9, 2026

ci(ff): support feature freeze automation #13138

Closed

8 tasks

fr4nc1sc0-r4m0n added 4 commits July 9, 2026 11:50

fix(release): satisfy gosec path traversal checks in release tooling

f3886bb

Validate and allowlist repository paths before writing release files, and update tests to use supported relative manifest paths.

Make release automation steps idempotent

a3f8375

Allow re-running release workflows without duplicate branches, empty commits, duplicate PRs, or false version-update errors.

Merge branch 'main' into feature/release-scripts-for-ff

e57a65e

fix(release): fix goimports formatting in github_test.go

15b86c9

mallendem reviewed Jul 10, 2026

View reviewed changes

fr4nc1sc0-r4m0n added 5 commits July 10, 2026 13:55

Merge branch 'main' into feature/release-scripts-for-ff

89180f5

docs(release): use valid X.Y.Z version in local testing examples

cddbe54

Replace invalid 9.5.0-test examples with 9.6.0 and document the required version format for local release workflow testing.

Keep unstaged changes when creating release branch

b295b36

Use go-git CheckoutOptions.Keep so major/minor release workflow can prepare version files before checking out the release branch.

Stage all worktree changes without recursing into submodules

7e55b0f

Replace go-git AddWithOptions(All: true) with per-path staging from worktree status so CommitAll includes every changed path (including submodule gitlinks like beats) without reading submodule directories.

Use GITHUB_TOKEN for authenticated git push in release automation

e1b7898

Match the Beats release automation by requiring GITHUB_TOKEN for push operations and authenticating via HTTP basic auth with an explicit branch refspec.

This comment has been minimized.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci(ff): support feature freeze automation#15433

ci(ff): support feature freeze automation#15433
fr4nc1sc0-r4m0n wants to merge 20 commits into
elastic:mainfrom
fr4nc1sc0-r4m0n:feature/release-scripts-for-ff

fr4nc1sc0-r4m0n commented Jul 9, 2026 •

edited

Loading

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 9, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 9, 2026

Uh oh!

mallendem Jul 10, 2026

Uh oh!

mallendem Jul 10, 2026

Uh oh!

mallendem Jul 10, 2026

Uh oh!

swiatekm commented Jul 10, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 10, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

github-actions Bot commented Jul 10, 2026

Root Cause

Evidence

Verification

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

-	dryRun := os.Getenv("DRY_RUN") == "true"
+	var dryRun bool
+	if v, err := strconv.ParseBool(os.Getenv("DRY_RUN")); err == nil {
+		dryRun = v
+	}

	dryRun := os.Getenv("DRY_RUN") == "true"
	dryRun := strings.ToLower(os.Getenv("DRY_RUN")) == "true"

Uh oh!

Conversation

fr4nc1sc0-r4m0n commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Key features

Idempotency

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 9, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 9, 2026

Uh oh!

mallendem Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

mallendem Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

mallendem Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

swiatekm commented Jul 10, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

This comment has been minimized.

github-actions Bot commented Jul 10, 2026

TL;DR

Remediation

Root Cause

Evidence

Verification

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fr4nc1sc0-r4m0n commented Jul 9, 2026 •

edited

Loading

infra-vault-gh-plugin-prod Bot commented Jul 10, 2026 •

edited

Loading