8385552: Improve devkit/sysroot creation

[vidmik]

Background (from JBS)

The devkit creation makefiles under make/devkit have been more or less untouched since they were first introduced. While they have served us well there are some key challenges which are slowly becoming more urgent to address.

One such issue relates to the logic for creating the sysroot. Currently, it is implemented using wget with a set of pattern to download packages directly from the upstream yum repo. One problem with this is that it will download all packages which happen to match the pattern, no matter if those packages are actually needed or not. More problematically though, using wget like this does not (necessarily) work for more recent linux distribution versions/repositories.

The dnf tool is the native solution for downloading packages, including dependencies etc.

The sysroot logic could also benefit from being split out into its own file, making the fairly involved Tools.gmk slightly less complex.

A secondary goal of this cleanup is to make it possible to reuse the logic (especially the sysroot logic) for creating devkit equivalents for building project Detroit.

About the change

The goal of this change was, initially, to change the sysroot logic to use dnf. In making the required changes I relatively quickly I realized that the Tools.gmk file was long overdue for cleanup. I chose to split out the sysroot logic to a separate file but couldn't resist making some fairly significant, (mostly) cosmetic changes to Tools.gmk while at it, moving it closer to the general style of the JDK build system. That said, a lot of the logic in Tools.gmk remains "old-style" even after this change.

Some specific things worth highlighting:

• I've tried to adjust formatting to closer match the JDK build system style, but I'm sure there are places/cases where it's off.
• I updated versions and URLs to reflect the latest reality, and adjusted the documentation in Makefile accordingly.
• To avoid having to call the Sysroot.gmk file twice (first to download packages and then to create the actual sysroot) I updated the logic to not depend on parse time globing. Specifically, what used to be RPM_FILE_LIST is now handled in the recipe instead.
• I fixed some missing dependencies which were most likely not actual problems, unless running with parallelism (which is not needed - Tools.gmk sets BUILDPAR where relevant). Even with these fixes parallelism is unlikely to work well.
• I dropped some packages which no longer exist (assuming they did in the first place, many years ago). Specifying invalid/non-existent packages to wget would just result in no packages being downloaded. With dnf it's a hard failure, which I argue is a good thing.
• I dropped the ffi header links - AFAICT that is no longer a problem in any of the distributions/versions we use.
• I added --disable-libgomp for GCC. I see that others have run into the same issue and I have no reason to believe it's needed for the JDK.
• I also added --disable-libsanitizer for riscv64 to work around a GCC issue. If somebody else things it's important and wants to have a look I'd appreciate the help.

Testing

I generated devkits for aarch64, x64 (OL) and armhfp, ppc64le, riscv64, s390x (Fedora). I then used those devkits to build the (mainline) JDK. All worked with two exceptions:

• I ran into a build problem with armhfp but AFAICT it is pre-existing, unrelated to my changes. As a matter of fact, I was unable to build the armhfp devkit using the mainline devkit version (without my changes)
• On the same general note, the mainline version is broken for riscv64 (the repo URL is dead). I located an alternative version which works.

I will want to do additional testing but would appreciate a review or two first. TYIA!

---

• I confirm that I make this contribution in accordance with the OpenJDK Interim AI Policy.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8385552: Improve devkit/sysroot creation (Enhancement - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/31303/head:pull/31303
$ git checkout pull/31303

Update a local copy of the PR:
$ git checkout pull/31303
$ git pull https://git.openjdk.org/jdk.git pull/31303/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 31303

View PR using the GUI difftool:
$ git pr show -t 31303

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/31303.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back mikael! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@vidmik Please do not rebase or force-push to an active PR as it invalidates existing review comments. Note for future reference, the bots always squash all changes into a single commit automatically as part of the integration. See OpenJDK Developers’ Guide for more information.

[openjdk]

@vidmik The following label will be automatically applied to this pull request:

• build

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (ee85bdeb)
• 00: Full (3568adad)

[erikj79]

See https://openjdk.org/groups/build/doc/code-conventions.html for reference.

[YaSuenag]

This PR seems to be overwrapped with #31227. How should I treat #31227? I can close it if this PR can cover the problem in it.

I've created PR #31227 for fixing the problem on x64. I saw the error which related to multilib when I built devkit on Rocky Linux and Fedora, but this PR does not seem to fix it (--disable-multilib is not specified specified for GCC).

In #31227, @erikj79 commented he will discuss about it internally, thus I guess the direction of fixing should follow that conclusion.

[YaSuenag]

Do we need to update building.md?

You can for example create and use a devkit with GCC 7.3 and a Fedora 12 sysroot environment (with glibc 2.11) on Ubuntu 14.04

This sentence can be read we can make devkit for Fedora 12 on Ubuntu 14.04.
(each versions is too old now!)

This PR completely depends on dnf, but it wouldn't be provided by Ubuntu.
building.md does not clarify the OS which is a builder of devkit. I guess it is different from BASE_OS because BASE_OS means sysroot.

Actually I could make devkit for OEL6 on Rocky Linux 9, but it couldn't with this change because baseos repository does not exist on it.

[erikj79]

This PR seems to be overwrapped with #31227. How should I treat #31227? I can close it if this PR can cover the problem in it.

I've created PR #31227 for fixing the problem on x64. I saw the error which related to multilib when I built devkit on Rocky Linux and Fedora, but this PR does not seem to fix it (--disable-multilib is not specified specified for GCC).

In #31227, @erikj79 commented he will discuss about it internally, thus I guess the direction of fixing should follow that conclusion.

This PR does not aim to change 32bit support. I think that should still be a separate change, because this change is likely going to be backported so we can produce devkits with 32bit support for our internal update releases.

[vidmik]

@erikj79 thank you for the review and feedback. I tried to address it all in the most recent commit, have a look and let me know.

Meanwhile I'll do another round of testing to make sure it all still works.

[vidmik]

Do we need to update building.md?

You can for example create and use a devkit with GCC 7.3 and a Fedora 12 sysroot environment (with glibc 2.11) on Ubuntu 14.04

This sentence can be read we can make devkit for Fedora 12 on Ubuntu 14.04. (each versions is too old now!)

Indeed, that information is definitely old. That said, it's not immediately clear to me that it's worth keeping it up to date with the latest exact versions (Fedora, Ubuntu, glibc). After all, it's just trying to outline what the purpose and value is of the devkit/sysroot.

I'm taking suggestions.

This PR completely depends on dnf, but it wouldn't be provided by Ubuntu. building.md does not clarify the OS which is a builder of devkit. I guess it is different from BASE_OS because BASE_OS means sysroot.

The new logic does indeed depend on dnf, by design. I don't have easy access to try it, but according to Ubuntu it looks like building the devkit on Ubuntu could work. If somebody happens to be able to test that out let me know.

Actually I could make devkit for OEL6 on Rocky Linux 9, but it couldn't with this change because baseos repository does not exist on it.

(It's just "OL" - no 'E')

OL6 reached end-of-life several years ago so I chose to bump it to OL7 (same as aarch64). If it's important to keep OL6 support it should be fairly straightforward to add back.

[YaSuenag]

it's not immediately clear to me that it's worth keeping it up to date with the latest exact versions (Fedora, Ubuntu, glibc). After all, it's just trying to outline what the purpose and value is of the devkit/sysroot.

IMHO it does not need to update everytime when devkit is updated, but it should be updated when minimum version is updated.

OL6 reached end-of-life several years ago so I chose to bump it to OL7 (same as aarch64). If it's important to keep OL6 support it should be fairly straightforward to add back.

According to building.md, OL6 should be supported for Linux x64 at least.