Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -550,9 +550,11 @@ See [`docs/validator.md`](docs/validator.md) for details.
### Dense kernel sweeps and freshness

One cloud image samples a kernel series at a single release. The sweep lane
installs exact kernel releases (from the distro archive pool, indexed by
[falcosecurity/kernel-crawler](https://github.com/falcosecurity/kernel-crawler))
inside the guest and reboots into them before validating:
installs exact kernel releases indexed by
[falcosecurity/kernel-crawler](https://github.com/falcosecurity/kernel-crawler)
inside the guest and reboots into them before validating. Debian/RHEL-family
profiles use pinned archive packages; Amazon Linux resolves the exact signed
kernel from its vendor repository (including AL2023's versioned repositories):

```bash
./bin/bpfcompat kernel-sweep --profile ubuntu-22.04-5.15 --count 4
Expand Down
26 changes: 15 additions & 11 deletions cmd/bpfcompat/kernel_sweep.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import (
func runKernelSweep(args []string) int {
fs := flag.NewFlagSet("kernel-sweep", flag.ContinueOnError)
fs.SetOutput(os.Stderr)
profileID := fs.String("profile", "", "Base profile id (debian- and rhel-family profiles)")
profileID := fs.String("profile", "", "Base profile id (debian-, rhel-, or amazon-family profile)")
count := fs.Int("count", 4, "Number of most-recent kernel releases to include")
series := fs.String("series", "", "Kernel release prefix (default: from the baseline mapping, else <kernel_family>.0-)")
crawlerTarget := fs.String("target", "", "kernel-crawler target flavor (default: from the baseline mapping, else the distro default)")
Expand Down Expand Up @@ -67,7 +67,7 @@ func runKernelSweep(args []string) int {
}
family := vm.KernelInstallFamily(base.Distro)
if family == "" {
fmt.Fprintf(os.Stderr, "kernel-sweep supports debian- and rhel-family profiles only (got distro %q)\n", base.Distro)
fmt.Fprintf(os.Stderr, "kernel-sweep supports debian-, rhel-, and amazon-family profiles only (got distro %q)\n", base.Distro)
return runner.ExitToolError
}
if base.InstallKernel != "" {
Expand Down Expand Up @@ -134,15 +134,19 @@ func runKernelSweep(args []string) int {
written := 0
for _, entry := range entries {
release := entry.KernelRelease
// Direct pool URLs, because the package indexes only carry the
// current ABI: superseded releases stay downloadable but are not
// installable by name.
var debs []string
// Direct pool URLs are required for Ubuntu and RHEL-family releases
// that have left package indexes. Amazon's repositories retain recent
// exact kernels, so those profiles deliberately install by signed
// package name (and AL2023 selects releasever=latest in the guest).
var packages []string
var err error
if family == vm.KernelFamilyRHEL {
debs, err = freshness.RHELKernelRPMs(entry)
} else {
debs, err = freshness.UbuntuKernelDebs(entry)
switch family {
case vm.KernelFamilyRHEL:
packages, err = freshness.RHELKernelRPMs(entry)
case vm.KernelFamilyDebian:
packages, err = freshness.UbuntuKernelDebs(entry)
case vm.KernelFamilyAmazon:
packages = nil
}
if err != nil {
fmt.Fprintf(os.Stderr, "skip %s: %v\n", release, err)
Expand All @@ -152,7 +156,7 @@ func runKernelSweep(args []string) int {
derived.ID = sweepProfileID(base.ID, release)
derived.KernelFamily = sweepKernelFamily(release, base.KernelFamily)
derived.InstallKernel = release
derived.KernelPackages = debs
derived.KernelPackages = packages

payload, err := yaml.Marshal(derived)
if err != nil {
Expand Down
2 changes: 1 addition & 1 deletion docs/falco-parity.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ multi-architecture runners, and release matrix publication.
| Firecracker executable backend | `--runner firecracker`, generated initramfs validator execution, serial result extraction, `make acceptance-firecracker-dev-one`, `.github/workflows/firecracker-preflight.yml`, and `docs/firecracker-backend.md`. |
| Catalog maintenance | `.github/workflows/profile-catalog-maintenance.yml`, `make profile-catalog-audit`, `make matrix-readiness`. |
| kernel-crawler freshness signal | `bpfcompat kernel-freshness` + `vm/kernel-baselines.yaml` + `.github/workflows/kernel-freshness.yml` compare validated kernels against falcosecurity/kernel-crawler's weekly inventory. |
| Dense per-release kernel sweep | `bpfcompat kernel-sweep` generates `install_kernel`/`kernel_packages` profiles that install exact kernel releases (archive-pool .debs from kernel-crawler URLs) inside the guest and reboot into them before validation. |
| Dense per-release kernel sweep | `bpfcompat kernel-sweep` generates `install_kernel` profiles that install exact kernel releases (archive-pool packages for Debian/RHEL families; exact signed vendor-repository packages for Amazon Linux) inside the guest and reboot into them before validation. |
| Release matrix publishing | `.github/workflows/compatibility-matrix-publish.yml`, `scripts/publish-compatibility-site.sh`, optional GitHub Pages deployment, tag release attachment. |
| Project adapter template | `adapters/generic-ebpf-suite/`. |

Expand Down
16 changes: 9 additions & 7 deletions docs/image-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,13 +207,15 @@ The crawler mapping (distro key, target flavor, release prefix, and the
than `ubuntu-generic`. `--target` and `--series` override it.

Supported families: **Debian** (`apt`/`dpkg`, boot selected via grub menu
titles) and **RHEL** — AlmaLinux, Rocky, CentOS Stream — (`dnf`, boot
selected with `grubby --set-default`, which matches on the vmlinuz path
rather than a menu string). Deliberately unsupported: Oracle Linux (UEK
ships from its own repositories, not BaseOS), Amazon Linux (kernels are
not published in a browsable pool the crawler maps to), and the
immutable/image-based systems (RHCOS, Fedora CoreOS, Flatcar, Bottlerocket,
Talos), whose kernels are part of the image rather than packages.
titles), **RHEL** — AlmaLinux, Rocky, CentOS Stream, and Oracle UEK — (`dnf`,
boot selected with `grubby --set-default`), and **Amazon Linux**. Amazon
Linux installs the exact signed `kernel-<release>` package from the vendor
repository; AL2 uses its rolling `yum` repository, while AL2023 explicitly
uses `dnf --releasever=latest` so an older image's deterministic repository
lock cannot hide the current kernel. Every lane verifies the exact `uname -r`
after reboot before validation. Immutable/image-based systems (RHCOS,
Fedora CoreOS, Flatcar, Bottlerocket, Talos) remain deliberately unsupported
because their kernels are part of the image rather than ordinary packages.

## Adding a profile (checklist)

Expand Down
4 changes: 3 additions & 1 deletion docs/support-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ pinned see [image-pipeline.md](image-pipeline.md).

| Category | Count | Meaning |
|---|---|---|
| Runnable (auto-download) | 57 | Supported transport and a public vendor image URL — runs anywhere with KVM, no manual setup. |
| Runnable (auto-download) | 59 | Supported transport and a public vendor image URL — runs anywhere with KVM, no manual setup. |
| Manual image required | 8 | Supported transport, but the image is licensed or has no public URL; the operator imports it (see `make import-required-images`). |
| Generated lane | 0 | No vendor image at all — the kernel is built/booted at run time (virtme-ng upstream, Firecracker). |
| Cataloged, not runnable here | 4 | Present in the catalog but not bootable on the current SSH/cloud-init executor (immutable images, executor limits). Marked non-blocking in matrices. |
Expand All @@ -33,7 +33,9 @@ image is fetched on first use and its sha256 is recorded.
| almalinux-9-5.14-k5.14.0-687.29.1.el9_8 | almalinux | 9 | 5.14 | x86_64 | ssh | auto-downloads vendor cloud image on first run |
| almalinux-9-5.14-k5.14.0-687.36.1.el9_8 | almalinux | 9 | 5.14 | x86_64 | ssh | auto-downloads vendor cloud image on first run |
| amazon-linux-2-5.10 | amazon-linux | 2 | 5.10 | x86_64 | ssh | auto-downloads vendor cloud image on first run |
| amazon-linux-2-5.10-k5.10.260-259.1061.amzn2 | amazon-linux | 2 | 5.10 | x86_64 | ssh | auto-downloads vendor cloud image on first run |
| amazon-linux-2023-6.1 | amazon-linux | 2023 | 6.1 | x86_64 | ssh | auto-downloads vendor cloud image on first run |
| amazon-linux-2023-6.1-k6.1.177-224.371.amzn2023 | amazon-linux | 2023 | 6.1 | x86_64 | ssh | auto-downloads vendor cloud image on first run |
| centos-stream-10-6.12 | centos-stream | 10 | 6.12 | x86_64 | ssh | auto-downloads vendor cloud image on first run |
| centos-stream-9-5.14 | centos-stream | 9 | 5.14 | x86_64 | ssh | auto-downloads vendor cloud image on first run |
| centos-stream-9-5.14-k5.14.0-725.el9 | centos-stream | 9 | 5.14 | x86_64 | ssh | auto-downloads vendor cloud image on first run |
Expand Down
12 changes: 8 additions & 4 deletions internal/vm/profile.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,9 @@ type Profile struct {
// InstallKernel installs a specific kernel release inside the guest
// before validation and reboots into it, so one base image yields a
// dense per-release matrix instead of only the kernel the image
// shipped with. Debian-family (apt/grub) and RHEL-family (dnf/grubby)
// profiles are supported; the release must be a package-exact string
// shipped with. Debian-family (apt/grub), RHEL-family (dnf/grubby), and
// Amazon Linux (yum or versioned dnf/grubby) profiles are supported; the
// release must be a package-exact string
// such as "5.15.0-118-generic" or "5.14.0-687.26.1.el9_8.x86_64".
InstallKernel string `yaml:"install_kernel,omitempty"`
// KernelPackages lists direct package URLs (.deb or .rpm) to install
Expand Down Expand Up @@ -92,11 +93,12 @@ func LoadProfile(path string) (Profile, error) {

// Kernel install families. A profile's distro determines how a specific
// kernel release is installed inside the guest and how the next boot is
// selected: apt plus grub menu titles on the Debian family, dnf plus grubby
// on the RHEL family.
// selected: apt plus grub menu titles on Debian, dnf plus grubby on RHEL,
// and yum/versioned dnf plus grubby on Amazon Linux.
const (
KernelFamilyDebian = "debian"
KernelFamilyRHEL = "rhel"
KernelFamilyAmazon = "amazon"
)

// KernelInstallFamily maps a profile distro to its kernel install family, or
Expand All @@ -111,6 +113,8 @@ func KernelInstallFamily(distro string) string {
return KernelFamilyDebian
case "almalinux", "rocky", "centos-stream", "rhel", "oracle":
return KernelFamilyRHEL
case "amazon-linux", "amazonlinux", "amzn", "amzn2":
return KernelFamilyAmazon
default:
return ""
}
Expand Down
35 changes: 29 additions & 6 deletions internal/vm/qemu.go
Original file line number Diff line number Diff line change
Expand Up @@ -566,14 +566,14 @@ func runGuestCommand(ctx context.Context, req ExecutionRequest, target sshTarget
// overlay with the requested kernel selected. Returns the new QEMU command;
// the caller re-establishes SSH and verifies uname -r.
//
// Debian- and RHEL-family only by validation: the release string is
// Debian-, RHEL-, and Amazon-family only by validation: the release string is
// package-exact and the boot-default selection is distro specific (grub menu
// titles on Ubuntu, grubby on RHEL).
// titles on Ubuntu, grubby on RHEL and Amazon Linux).
func installGuestKernelAndReboot(ctx context.Context, result *ExecutionResult, req ExecutionRequest, target sshTarget, qemuCmd *exec.Cmd,
overlayPath, serialLogPath, qemuLogPath string, sshPort int, seedMode seedDeliveryMode, seedURL, seedDir, seedImagePath string) (*exec.Cmd, error) {
release := req.Profile.InstallKernel

installCmd := guestKernelInstallCmd(req.Profile.Distro, release, req.Profile.KernelPackages)
installCmd := guestKernelInstallCmd(req.Profile.Distro, req.Profile.Version, release, req.Profile.KernelPackages)
if err := sshRun(ctx, target, installCmd); err != nil {
return nil, fmt.Errorf("install kernel %s in guest: %w", release, err)
}
Expand Down Expand Up @@ -604,11 +604,15 @@ func installGuestKernelAndReboot(ctx context.Context, result *ExecutionResult, r
// timeout rides out cloud-init/unattended-upgrades holding the apt lock
// right after first boot. All interpolated values are validated at profile
// load (validKernelRelease / validKernelPackageURL), so they are shell-safe.
func guestKernelInstallCmd(distro, release string, packageURLs []string) string {
if KernelInstallFamily(distro) == KernelFamilyRHEL {
func guestKernelInstallCmd(distro, version, release string, packageURLs []string) string {
switch KernelInstallFamily(distro) {
case KernelFamilyRHEL:
return guestKernelInstallCmdRHEL(release, packageURLs)
case KernelFamilyAmazon:
return guestKernelInstallCmdAmazon(version, release)
default:
return guestKernelInstallCmdDebian(release, packageURLs)
}
return guestKernelInstallCmdDebian(release, packageURLs)
}

func guestKernelInstallCmdDebian(release string, packageURLs []string) string {
Expand Down Expand Up @@ -663,6 +667,25 @@ func guestKernelInstallCmdRHEL(release string, packageURLs []string) string {
return b.String()
}

// guestKernelInstallCmdAmazon installs an exact repository kernel and selects
// its vmlinuz for the next boot. AL2 follows the rolling yum repository model;
// AL2023 images are intentionally locked to the repository version they were
// built from, so --releasever=latest is required to test the current vendor
// kernel instead of silently reinstalling the image-era package set. Package
// signatures remain enforced by yum/dnf.
func guestKernelInstallCmdAmazon(version, release string) string {
var b strings.Builder
b.WriteString("set -e; ")
packageName := "kernel-" + release
if version == "2023" {
fmt.Fprintf(&b, "sudo dnf -y --releasever=latest install %s; ", shellQuote(packageName))
} else {
fmt.Fprintf(&b, "sudo yum -y install %s; ", shellQuote(packageName))
}
fmt.Fprintf(&b, "sudo grubby --set-default %s", shellQuote("/boot/vmlinuz-"+release))
return b.String()
}

// waitProcessExit waits for the guest-initiated QEMU exit (-no-reboot turns
// the reboot into a clean process exit). The exit status is irrelevant.
func waitProcessExit(cmd *exec.Cmd, timeout time.Duration) error {
Expand Down
35 changes: 35 additions & 0 deletions internal/vm/qemu_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -540,6 +540,41 @@ func TestShellQuote(t *testing.T) {
}
}

func TestGuestKernelInstallCmdAmazon(t *testing.T) {
al2Release := "5.10.260-259.1061.amzn2.x86_64"
al2 := guestKernelInstallCmd("amazon-linux", "2", al2Release, nil)
for _, want := range []string{
"sudo yum -y install 'kernel-" + al2Release + "'",
"sudo grubby --set-default '/boot/vmlinuz-" + al2Release + "'",
} {
if !strings.Contains(al2, want) {
t.Fatalf("AL2 install command %q missing %q", al2, want)
}
}
if strings.Contains(al2, "--releasever=latest") {
t.Fatalf("AL2 install command must not use AL2023 release versioning: %q", al2)
}

al2023Release := "6.1.177-224.371.amzn2023.x86_64"
al2023 := guestKernelInstallCmd("amazon-linux", "2023", al2023Release, nil)
for _, want := range []string{
"sudo dnf -y --releasever=latest install 'kernel-" + al2023Release + "'",
"sudo grubby --set-default '/boot/vmlinuz-" + al2023Release + "'",
} {
if !strings.Contains(al2023, want) {
t.Fatalf("AL2023 install command %q missing %q", al2023, want)
}
}
}

func TestKernelInstallFamilyIncludesAmazon(t *testing.T) {
for _, distro := range []string{"amazon-linux", "amazonlinux", "amzn", "amzn2"} {
if got := KernelInstallFamily(distro); got != KernelFamilyAmazon {
t.Fatalf("KernelInstallFamily(%q) = %q, want %q", distro, got, KernelFamilyAmazon)
}
}
}

func TestGuestCommandLine(t *testing.T) {
got := guestCommandLine(
"$BPFCOMPAT_BIN --obj $BPFCOMPAT_ARTIFACT",
Expand Down
10 changes: 8 additions & 2 deletions internal/vm/validate.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,13 +46,19 @@ func ValidateProfile(p Profile) error {
}
family := KernelInstallFamily(p.Distro)
if family == "" {
return fmt.Errorf("profile.install_kernel is only supported for debian- and rhel-family profiles (got distro %q)", p.Distro)
return fmt.Errorf("profile.install_kernel is only supported for debian-, rhel-, and amazon-family profiles (got distro %q)", p.Distro)
}
if family == KernelFamilyAmazon && p.Version != "2" && p.Version != "2023" {
return fmt.Errorf("amazon profile.install_kernel requires profile.version %q or %q (got %q)", "2", "2023", p.Version)
}
if family == KernelFamilyAmazon && len(p.KernelPackages) > 0 {
return fmt.Errorf("amazon profile.install_kernel resolves the exact signed package from vendor repositories; profile.kernel_packages must be empty")
}
if !validKernelRelease(p.InstallKernel) {
return fmt.Errorf("profile.install_kernel must match [A-Za-z0-9._+-]+ (got %q)", p.InstallKernel)
}
wantExt := ".deb"
if family == KernelFamilyRHEL {
if family == KernelFamilyRHEL || family == KernelFamilyAmazon {
wantExt = ".rpm"
}
for _, pkg := range p.KernelPackages {
Expand Down
28 changes: 28 additions & 0 deletions internal/vm/validate_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,34 @@ func TestValidateProfileInstallKernel(t *testing.T) {
t.Fatalf("valid Oracle UEK install_kernel rejected: %v", err)
}

amazon2 := ubuntuVMProfile()
amazon2.Distro = "amazon-linux"
amazon2.Version = "2"
amazon2.InstallKernel = "5.10.260-259.1061.amzn2.x86_64"
if err := ValidateProfile(amazon2); err != nil {
t.Fatalf("valid Amazon Linux 2 install_kernel rejected: %v", err)
}

amazon2023 := ubuntuVMProfile()
amazon2023.Distro = "amazon-linux"
amazon2023.Version = "2023"
amazon2023.InstallKernel = "6.1.177-224.371.amzn2023.x86_64"
if err := ValidateProfile(amazon2023); err != nil {
t.Fatalf("valid Amazon Linux 2023 install_kernel rejected: %v", err)
}

unknownAmazon := amazon2023
unknownAmazon.Version = "future"
if err := ValidateProfile(unknownAmazon); err == nil {
t.Fatal("expected unknown Amazon version with install_kernel to fail")
}

amazonWithURL := amazon2
amazonWithURL.KernelPackages = []string{"https://example.com/kernel.rpm"}
if err := ValidateProfile(amazonWithURL); err == nil {
t.Fatal("expected Amazon install_kernel with direct package URL to fail")
}

nonUbuntu := ubuntuVMProfile()
nonUbuntu.Distro = "debian"
nonUbuntu.InstallKernel = "6.1.0-30-cloud-amd64"
Expand Down
6 changes: 6 additions & 0 deletions matrices/kernel-sweep-amazon-linux-2-5.10.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Generated by `bpfcompat kernel-sweep --profile amazon-linux-2-5.10`; regenerate rather than edit.
name: kernel-sweep-amazon-linux-2-5.10
profiles:
- id: amazon-linux-2-5.10-k5.10.260-259.1061.amzn2
required: true
timeout: 20m
6 changes: 6 additions & 0 deletions matrices/kernel-sweep-amazon-linux-2023-6.1.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Generated by `bpfcompat kernel-sweep --profile amazon-linux-2023-6.1`; regenerate rather than edit.
name: kernel-sweep-amazon-linux-2023-6.1
profiles:
- id: amazon-linux-2023-6.1-k6.1.177-224.371.amzn2023
required: true
timeout: 20m
4 changes: 2 additions & 2 deletions matrices/pre-campaign-enterprise-refresh.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ profiles:
- id: rocky-9-5.14-k5.14.0-687.36.1.el9_8
required: true
timeout: 20m
- id: amazon-linux-2-5.10
- id: amazon-linux-2-5.10-k5.10.260-259.1061.amzn2
required: true
timeout: 20m
- id: amazon-linux-2023-6.1
- id: amazon-linux-2023-6.1-k6.1.177-224.371.amzn2023
required: true
timeout: 20m
- id: oracle-linux-9-uek7-5.15
Expand Down
6 changes: 6 additions & 0 deletions vm/kernel-baselines.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,19 @@ baselines:
distro: amazonlinux2
target: amazonlinux2
release_prefix: 5.10.
- profile: amazon-linux-2-5.10-k5.10.260-259.1061.amzn2
kernel: 5.10.260-259.1061.amzn2.x86_64
recorded: "2026-08-09"
- profile: amazon-linux-2023-6.1
kernel: 6.1.176-223.369.amzn2023.x86_64
recorded: "2026-07-30"
crawler:
distro: amazonlinux2023
target: amazonlinux2023
release_prefix: 6.1.
- profile: amazon-linux-2023-6.1-k6.1.177-224.371.amzn2023
kernel: 6.1.177-224.371.amzn2023.x86_64
recorded: "2026-08-09"
- profile: centos-stream-10-6.12
kernel: 6.12.0-226.el10.x86_64
recorded: "2026-05-19"
Expand Down
17 changes: 17 additions & 0 deletions vm/profiles/amazon-linux-2-5.10-k5.10.260-259.1061.amzn2.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Generated by `bpfcompat kernel-sweep --profile amazon-linux-2-5.10`; regenerate rather than edit.
id: amazon-linux-2-5.10-k5.10.260-259.1061.amzn2
distro: amazon-linux
version: "2"
kernel_family: "5.10"
arch: x86_64
install_kernel: 5.10.260-259.1061.amzn2.x86_64
image:
source_url: https://cdn.amazonlinux.com/os-images/2.0.20260727.0/kvm/amzn2-kvm-2.0.20260727.0-x86_64.xfs.gpt.qcow2
local_path: vm/cache/amazon-linux-2-5.10.qcow2
boot:
memory_mb: 1024
cpus: 1
validator:
path: /usr/local/bin/bpfcompat-validator
capabilities:
expected_btf: true
Loading
Loading