Automate releases from main [SPRE-183]

[osevcik1]

Summary

Move exodus-rsync closer to continuous deployment by creating a GitHub release automatically for every change merged to main.

This replaces the old manual/tag-driven release workflow with a push-to-main release workflow that:

• calculates the next patch version from the latest vX.Y.Z tag
• builds the exodus-rsync binary with BUILDVERSION set to the release tag
• generates release notes from commits since the previous release
• creates an annotated tag
• creates a GitHub release
• attaches:
  • exodus-rsync
  • exodus-rsync.sha256

Also updates local/CI checks:

• make make check portable on macOS and Linux
• update obsolete GitHub Actions versions
• skip the glibc symbol-version check on non-Linux platforms

Why

exodus-rsync is stable, but dependency updates still need regular releases. The previous manual monthly release process was easy to ignore, so new merges should generate releases automatically.

Testing

• go test ./...
• make check
• go build ./...
• make exodus-rsync BUILDVERSION=v1.12.4-dryrun
• make all
• go run github.com/rhysd/actionlint/cmd/actionlint@latest .github/workflows/go.yml .github/workflows/release.yml
• local non-publishing release dry-run:
  • resolved latest tag: v1.12.3
  • calculated next tag: v1.12.4
  • built binary
  • generated checksum
  • generated release notes
  • did not create a tag or GitHub release

Notes

Downstream version bump automation for c3i-images, rhsm-pulp, and pub is handled in separate GitLab merge requests.

The only remaining operational check is to confirm the GitLab pipeline schedules and EXODUS_GITLAB_TOKEN credentials are configured and active in those repositories.

[negillett]

Thank you, @osevcik1, for getting to work on this. Though, I have to push back on the current approach.

Every merge will create a tag and release, building unnecessary binaries and generating a lot of noise; not just here but also in rcm-pub and rhsm-pulp-playbooks merge requests. Those pub and pulp version bumps will still require a human to deploy, so invariably we will end up with a stack of MRs on those repos. Because there's no batching, just this past week would've generated 4 exodus-rsync versions/releases (potentially 12 MRs opened on rhsm-pulp-playbooks).

Also, this only deals in patches, meaning that larger changes worthy of minor or major bump never get the opportunity to receive proper semver, they're automatically Z. Workaround: we would very quickly have to interrupt the automation post-merge for such changes and perform the tagging + push manually. Fighting our own automation is not a sustainable process.

I'm sure there are edge cases too that haven't been fully considered, such as partial failures/releases.

Suggestion:
Make tagging more robust, covering X, Y, and Z. Inspiration: https://github.com/rohanpm/pidiff +
Release commit batches, perhaps on a schedule with a monthly interval +
Reincorporate CHANGELOG (current approach would obsolete it, but we can't discard it either) +
Cover edge cases; e.g., make all fails, code lands, but no release is created, or gh release create fails and we end up with a tag and no release +
A .sha256 is good but it should be documented and actually used to verify downstream (pulp could verify the checksum it generates to block invalid releases)

We also want to ensure all consumers are prepared to actually deal with the releases/MRs. This automates the easy part. The real bottleneck, IMO, is deployment; and that's certainly within scope since SPRE-183 aims to retire the deployment template.
Consumers: pub & pulp, but also also double check our new pipeline-as-a-service.

I'll stop there, I think that's everything worth mentioning now.
@crungehottman, anything you'd add? Or maybe something you'd correct me on/disagree with?

[osevcik1]

Thanks, that’s helpful context.

I think I understand the concern now. My implementation focused on automating release creation itself, but it doesn’t really address the downstream deployment workflow and would likely create too much noise if every merge resulted in a new release and corresponding pub/pulp MRs.

Looking at pidiff, it seems closer to a batched release model where changes accumulate and a release is created intentionally, with proper semver handling and changelog management. That also sounds like a better fit for downstream consumers.

I’ll take another look at the problem from that perspective and review how pub, rhsm-pulp and Pipeline-as-a-Service would consume releases. It sounds like the real goal is to automate the entire promotion path rather than just the release step itself.

[negillett]

Yeah, exactly. The Jira issue wants to do away with the whole exodus-rsync release and deployment process, not only releasing. It's important we make this robust so that little to no human participation is required.

I'm currently working through that deployment template/process now (https://redhat.atlassian.net/browse/RHELDST-39083), so this is fresh for me if you have any questions.

I've also confirmed that our pipeline-as-a-service will consume pulp images from https://gitlab.cee.redhat.com/exd-guild-distribution/container-images as opposed to https://gitlab.cee.redhat.com/c3i/c3i-images, so that's another point the release automation will need to touch. There should already be automation that's targeting c3i-images so it should be trivial.

The current bump automation for rhsm-pulp images and pub are probably the right solution for those consumers; release creates MRs, human hits the merge button. We just might need something to drive us to push said button, as MRs are easy to ignore.
The difficult part of this, as I see it, is automating the pulp deployment process, but should be doable. Maybe check out our one-click deployment process for exodus-gw / exodus-cdn for inspiration.

And as for pidiff, I don't believe it's being maintained, and it wouldn't be appropriate for exodus-rsync anyway (Go, not Python/PIPy). What's interesting about it is the semver suggestions. And I only brought it up because I happened to know about it and not another tool. However, I just found this: https://github.com/bobg/modver, which should be very helpful.

[osevcik1]

Thanks, I spent some time reviewing the current implementation and your feedback in more detail.

I have a proposal for reworking the release flow:

• move away from release-on-every-merge
• use scheduled releases with manual workflow_dispatch support
• support patch/minor/major version selection, with modver used as a suggestion mechanism
• keep CHANGELOG.md as part of the release process
• perform a no-op when there are no changes since the last release
• keep the existing pub/rhsm-pulp MR flow but align it with batched releases
• add the Pipeline-as-a-Service consumer through container-images
• improve failure handling around tag/release creation and checksum verification

Before I start implementing this redesign, I’d like to confirm that this direction matches what you have in mind.

[negillett]

Yeah, @osevcik1, that sounds like the right direction. I'd imagine the scheduled workflow would generate a semver, update changelog, bump version, and open a PR with those changes for approval. Merging that would trigger another workflow that creates the release/tag, starting the chain updates to piaas, pub, and pulp.

[osevcik1]

@negillett could you add me as contributor? So I can add GH actions var and secrets?
And into RHSM-PULP as Maintainer so I can create pipeline trigger token fot that repo

[negillett]

@negillett could you add me as contributor? So I can add GH actions var and secrets? And into RHSM-PULP as Maintainer so I can create pipeline trigger token fot that repo

Looks like you're assigned to rhsm-pulp-playbooks via ldap (role: developer) and here you don't seem to be a member of release-engineering and so can't be added to this repo.

What I would recommend is continue to open requests with your changes and I can help get tokens/secrets in place when the time comes to activate the actions/workflows.

For now, could you collapse the real/desired changes here into a single commit with an informative commit message (not just GitHub comment)? Right now it's not clear what's being implemented because there are no commit messages and the initial PR comment is outdated.

[codecov]

Codecov Report

❌ Patch coverage is 32.72251% with 257 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.56%. Comparing base (4ae3bd0) to head (f60dbb9).

Files with missing lines	Patch %	Lines
tools/releaseprep/main.go	32.72%	250 Missing and 7 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##              main     #619       +/-   ##
============================================
- Coverage   100.00%   85.56%   -14.44%     
============================================
  Files           22       23        +1     
  Lines         1398     1780      +382     
============================================
+ Hits          1398     1523      +125     
- Misses           0      250      +250     
- Partials         0        7        +7     

Flag	Coverage Δ
unit-tests	85.56% <32.72%> (-14.44%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.