diff --git a/CHANGELOG.md b/CHANGELOG.md index 76e35f5..7c6ad02 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -67,6 +67,10 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). without changing the centrally managed review-agent credential contract. ### Security +- Bound archive-member enumeration before ZIP object-table allocation and stream + gzip/tar physical headers before semantic parsing, enforcing 10,000 members, + 512 MiB expanded tar, and 1 MiB extension-header limits while rejecting + unnecessary ZIP64, multi-disk, sparse, link, device, FIFO, and special forms. - Preflight every direct release-SBOM wheel or source archive as one regular file with a finite 256 MiB compressed-byte ceiling before ZIP or gzip/tar parsing, metadata reads, or artifact hashing. Symlinks, directories, devices, diff --git a/docs/sbom-release-evidence.md b/docs/sbom-release-evidence.md index 8c2233f..ec166ae 100644 --- a/docs/sbom-release-evidence.md +++ b/docs/sbom-release-evidence.md @@ -11,50 +11,80 @@ and package URLs (purls). This is a read-only evidence foundation. It does not authorize a pull-request branch to execute release logic with write credentials. The only permitted future integration source is a protected-main or organization-level reusable workflow -whose source is immutable before receiving OIDC or attestation permissions. -No SLSA Build level is claimed merely because an SBOM or attestation exists. +whose source is immutable before receiving OIDC or attestation +permissions. No provenance, signing, publication, attestation, or SLSA Build +level follows merely from direct SBOM generation. ## Normative evidence contract `scripts/ci/generate_release_sbom.py` treats every archive, manifest, and lock file as untrusted input and never imports EgressWeave. It must: -1. accept only a wheel or gzip source distribution; -2. inspect the caller-supplied final release-artifact path without resolving that - final component, require a regular file, enforce a 256 MiB compressed-byte - ceiling, open it without following a final symbolic link where the platform - supports that flag, and require the opened descriptor to retain the accepted - device and inode before any parser; -3. parse and hash only that bound descriptor, keep every parser-visible read and - seek live-bounded by the same 256 MiB ceiling, bracket metadata parsing with - finite SHA-256 reads, and fail if the archive bytes change during verification; -4. reject unsafe or duplicate paths, links, devices, excessive member counts, - ambiguous metadata, oversized metadata, and malformed archives; -5. check the declared wheel metadata size before decompression; -6. read exactly one wheel `METADATA` or root source `PKG-INFO` member; -7. verify package identity, license expression, and complete direct runtime - requirement declarations against the reviewed manifest; -8. verify every dependency version, SHA-256, and environment marker against the - executable hash-locked subset in `requirements-ci.txt`, while rejecting - dependency extras that could activate packages outside the reviewed graph; -9. validate identities, SPDX license identifiers, purls, graph references, - relationships, reachability, and acyclicity; -10. compute the artifact SHA-256 without trusting its filename; and -11. emit sorted UTF-8 CycloneDX 1.7 JSON without timestamps or random identifiers. - -The direct generator normalizes missing, uninspectable, symbolic-link, directory, -device, FIFO, socket, replaced, and other non-regular artifact inputs to -`release artifact is missing or unsafe`. The final artifact component remains -unresolved until no-follow validation binds the accepted path to its descriptor. -Inputs above the compressed-byte ceiling fail with -`release artifact exceeds the compressed-byte safety bound`; the parser-facing -wrapper rechecks the live regular descriptor before and after reads and seeks, so -an initially accepted archive that grows past the ceiling fails before the parser -can consume the expanded input. Bytes that change across the descriptor-bound -metadata pass fail with `release artifact changed during verification`. -Accepted-size archives remain subject to all member-count, path, link/device, -metadata-size, decompression, identity, dependency, and digest controls; the -compressed-input check does not replace those independent defenses. +1. accept only a canonical wheel or gzip source distribution; +2. inspect the caller-supplied final artifact path without resolving its final + component, require a regular file, enforce a 256 MiB compressed-byte ceiling, + open without following a final symbolic link where supported, and bind the + opened descriptor to the accepted device and inode before any parser runs; +3. parse and hash only that descriptor, keep every parser-visible read and seek + live-bounded by the same compressed-byte ceiling, bracket metadata parsing + with finite SHA-256 reads, and reject bytes that change during verification; +4. count wheel central-directory records before `zipfile.ZipFile` allocates a + complete `ZipInfo` table, permit at most 10,000 members, and require the + physical record count, record boundaries, directory size, directory offset, + and classic end-of-central-directory counts to agree exactly; +5. reject multi-disk wheels, ZIP64 wheels, truncated or inconsistent central + directories, and malformed extra fields because those formats are not needed + by the bounded canonical release contract; +6. stream the gzip/tar physical headers before semantic parsing, permit at most + 10,000 physical members, enforce an aggregate expanded-tar ceiling of + 512 MiB, and permit at most 1 MiB for each PAX or GNU extension-header payload; +7. reject malformed or truncated tar headers, checksum errors, links, devices, + FIFOs, sparse forms, unsupported special forms, and nonzero trailing data; +8. use sequential `tarfile` parsing without `getmembers()`, retain only the + bounded seen-name set and one root `PKG-INFO` payload, and preserve the same + semantic path, duplicate, type, and member-count checks after preflight; +9. reject unsafe or duplicate paths, ambiguous metadata, malformed archives, + and metadata larger than 1 MiB, checking declared wheel metadata size before + decompression and bounding source metadata extraction; +10. read exactly one wheel `METADATA` or root source `PKG-INFO` member; +11. verify package identity, license expression, and complete direct runtime + requirement declarations against the reviewed manifest; +12. verify every dependency version, SHA-256, and environment marker against the + executable hash-locked subset in `requirements-ci.txt`, while rejecting + dependency extras that could activate packages outside the reviewed graph; +13. validate identities, reviewed SPDX license identifiers, purls, graph + references, relationships, reachability, and acyclicity; +14. compute the artifact SHA-256 without trusting its filename; and +15. emit sorted UTF-8 CycloneDX 1.7 JSON without timestamps or random identifiers. + +## Resource boundaries and failure behavior + +The controls are deliberately layered rather than interchangeable: + +| Boundary | Exact limit | Enforced before | Purpose | +|---|---:|---|---| +| Compressed release artifact | 256 MiB | hashing or archive parsing | Bounds descriptor-visible input and concurrent growth | +| Archive members | 10,000 | `ZipFile` table creation and semantic tar materialization | Prevents member-table and parser-object amplification | +| Expanded gzip/tar stream | 512 MiB | physical payload skipping or semantic tar parsing | Bounds decompression and aggregate tar processing | +| PAX/GNU extension payload | 1 MiB per header | retaining extension bytes | Bounds parser metadata controlled by archive authors | +| Core package metadata | 1 MiB | email metadata parsing | Bounds `METADATA` and `PKG-INFO` processing | + +The direct generator normalizes missing, uninspectable, symbolic-link, +directory, device, FIFO, socket, replaced, and other non-regular artifact inputs +to `release artifact is missing or unsafe`. Inputs above the compressed-byte +ceiling fail with `release artifact exceeds the compressed-byte safety bound`. +The parser-facing wrapper rechecks the live regular descriptor before and after +reads and seeks, so an initially accepted archive that grows past the ceiling +fails before the parser consumes the expanded input. Bytes that differ across +the descriptor-bound metadata pass fail with +`release artifact changed during verification`. + +Canonical wheels and ordinary gzip-compressed source distributions remain +accepted. Multi-disk or ZIP64 wheels and sparse or special tar forms fail closed +because they add parser complexity without serving the release profile. +Accepted compressed size does not imply safe expansion: member, expanded-tar, +extension-header, metadata, path, type, dependency, and digest controls remain +independent defenses. The root component uses a digest-derived `bom-ref`, preventing different artifacts from sharing evidence identity. The dependency graph is the union @@ -65,10 +95,9 @@ The reviewed manifest is `scripts/ci/release_runtime_dependencies.json`. Its versions, hashes, and markers must match `requirements-ci.txt`. Lock entries with PEP 508 extras are invalid for SBOM parity because an extra can introduce additional runtime -requirements that are absent from the reviewed component graph. This prevents -buyer-facing evidence from describing one dependency set while CI executes -another. A dependency change is incomplete until the lock, manifest, tests, -license evidence, and SBOM semantics are reviewed together. +requirements absent from the reviewed component graph. A dependency change is +incomplete until the lock, manifest, tests, license evidence, and SBOM semantics +are reviewed together. ## Generate and verify locally @@ -94,24 +123,32 @@ mismatch must fail closed. Never edit generated evidence to fit existing artifacts; correct the reviewed inputs, rebuild, and regenerate every evidence file. +Run direct generation from an isolated, read-only exact-artifact directory. +Descriptor identity, bounded reads, physical archive preflight, and digest +bracketing reduce ordinary pathname-replacement, growth, mutation, and resource +amplification risks. They do not make mutable storage immutable: a privileged +writer able to alter and restore the same inode entirely between observations +remains a residual risk. + ## Offline operator verification An operator evaluating an acquired or air-gapped release should: -1. obtain each distribution, CycloneDX JSON, `SHA256SUMS`, and signed - attestation bundle through independently authenticated media; +1. obtain each distribution, CycloneDX JSON, `SHA256SUMS`, and any separately + produced signed attestation bundle through independently authenticated media; 2. verify `SHA256SUMS` before parsing an archive; 3. confirm the SBOM root hash equals the artifact hash; 4. validate the document against the CycloneDX 1.7 JSON schema; -5. verify attestation subject, repository, immutable workflow source, exact - commit, and predicate bytes; +5. when attestations exist, independently verify their subject, repository, + immutable workflow source, exact commit, and predicate bytes; 6. inventory purls, versions, SPDX licenses, markers, and relationships; and 7. reject any digest, identity, schema, workflow, signature, or graph mismatch. An SBOM is inventory evidence, not proof that a dependency is vulnerability -free, correctly licensed for every use, or benign. Vulnerability assessment, -legal review, provenance verification, and deployment policy remain separate -controls. +free, correctly licensed for every use, benign, reproducibly built, or produced +by a trusted builder. Vulnerability assessment, legal review, provenance +verification, deployment policy, and reproducible-build evidence remain +separate controls. ## Protected release integration @@ -128,11 +165,11 @@ current permission contract during protected integration. Before public GitHub Release publication, verify each attestation against the exact artifact SHA-256, repository identity, immutable workflow source, exact -protected-main commit, CycloneDX predicate bytes, and release tag. +protected-main commit, CycloneDX predicate bytes, and release tag. Public release +must fail closed on any mismatch. -Public release must fail closed on any mismatch. -A branch must never add a temporary job that publishes, moves refs, writes contents, -pushes to a pull-request branch, self-modifies workflows, or executes +A branch must never add a temporary job that publishes, moves refs, writes +contents, pushes to a pull-request branch, self-modifies workflows, or executes model-modified source under a write credential. ## Threats, failure, and recovery @@ -140,30 +177,22 @@ model-modified source under a write credential. These controls address omitted inventory, evidence bound to the wrong artifact, filename substitution, path replacement between inspection and parsing, manifest-versus-lock drift, undeclared dependency extras, mutable dependency -resolution, nondeterministic evidence, unsafe archives, compressed-input resource -exhaustion, metadata decompression, stale or wrong-workflow attestations, and -publication before exact verification. - -Descriptor identity, live parser bounds, and digest bracketing close ordinary -final-symlink, pathname-replacement, unbounded-growth, and in-place mutation races -during parsing. They do not convert a writable build host into an immutable-storage -system: a privileged writer able to alter and restore the same inode entirely -between verification observations remains a residual mutable-storage risk. Run -evidence generation from an isolated, read-only exact-artifact directory, and -rely on the later sealed-evidence descriptor, digest, and post-publication checks -before any credential-bearing use. No provenance or SLSA claim follows from these -direct-generator controls. - -These controls do not detect every compromised upstream source, malicious but -correctly hashed package, license obligation, build-host compromise, or -undisclosed vulnerability. Those risks require provenance, reproducible builds, -vulnerability management, legal review, and hardened runners. +resolution, nondeterministic evidence, unsafe archives, member-table exhaustion, +compressed-input and decompression resource exhaustion, oversized extension or +package metadata, stale or wrong-workflow attestations, and publication before +exact verification. + +They do not detect every compromised upstream source, malicious but correctly +hashed package, license obligation, build-host compromise, undisclosed +vulnerability, or privileged mutable-storage attack. Those risks require +provenance, reproducible builds, vulnerability management, legal review, +hardened runners, and sealed or read-only artifact storage. On any generator, digest, semantic, manifest, lock, or attestation failure, -publish nothing. Correct the source through normal review and regenerate from the -exact accepted commit. Never replace published bytes under an existing version. -If protected main advances, discard the stale attempt and rebuild. Partial -publication requires a new version and transparent changelog entry. +publish nothing. Correct the source through normal review and regenerate from +the exact accepted commit. Never replace published bytes under an existing +version. If protected main advances, discard the stale attempt and rebuild. +Partial publication requires a new version and transparent changelog entry. ## SLSA statement @@ -186,9 +215,24 @@ https://github.com/actions/attest MITRE. (2026). *CWE-400: Uncontrolled resource consumption.* Common Weakness Enumeration. https://cwe.mitre.org/data/definitions/400.html +MITRE. (2026). *CWE-409: Improper handling of highly compressed data (data +amplification).* Common Weakness Enumeration. +https://cwe.mitre.org/data/definitions/409.html + +MITRE. (2026). *CWE-770: Allocation of resources without limits or throttling.* +Common Weakness Enumeration. https://cwe.mitre.org/data/definitions/770.html + +Python Software Foundation. (n.d.). *tarfile—Read and write tar archive files.* +Python 3.13 documentation. Retrieved August 7, 2026, from +https://docs.python.org/3.13/library/tarfile.html + +Python Software Foundation. (n.d.). *zipfile—Work with ZIP archives.* Python +3.13 documentation. Retrieved August 7, 2026, from +https://docs.python.org/3.13/library/zipfile.html + Python Software Foundation. (n.d.). *zipfile—Work with ZIP archives: -Decompression pitfalls.* Python 3 documentation. Retrieved August 6, 2026, from -https://docs.python.org/3/library/zipfile.html#decompression-pitfalls +Decompression pitfalls.* Python 3.13 documentation. Retrieved August 7, 2026, +from https://docs.python.org/3.13/library/zipfile.html#decompression-pitfalls Python Packaging Authority. (n.d.). *Core metadata specifications.* Python Packaging User Guide. Retrieved August 5, 2026, from diff --git a/pyproject.toml b/pyproject.toml index d00eb13..334a561 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -71,7 +71,7 @@ testpaths = ["tests"] [tool.coverage.run] branch = true -source = ["egressweave"] +source_dirs = ["src/egressweave", "scripts/ci"] [tool.coverage.report] fail_under = 100 diff --git a/scripts/ci/generate_release_sbom.py b/scripts/ci/generate_release_sbom.py index fab8e9a..bd96e93 100644 --- a/scripts/ci/generate_release_sbom.py +++ b/scripts/ci/generate_release_sbom.py @@ -8,14 +8,17 @@ from __future__ import annotations import argparse +import gzip import hashlib import io import json import os import re import stat +import struct import tarfile import zipfile +import zlib from email.message import Message from email.parser import BytesParser from email.policy import default @@ -32,6 +35,14 @@ MAX_METADATA_BYTES = MAX_MANIFEST_BYTES = 1_048_576 MAX_ARCHIVE_MEMBERS = 10_000 MAX_RELEASE_ARTIFACT_BYTES = 256 * 1024 * 1024 +MAX_EXPANDED_TAR_BYTES = 512 * 1024 * 1024 +MAX_TAR_EXTENSION_BYTES = 1 * 1024 * 1024 +ZIP_EOCD_SIGNATURE = b"PK\x05\x06" +ZIP64_EOCD_LOCATOR_SIGNATURE = b"PK\x06\x07" +ZIP64_EOCD_LOCATOR_SIZE = 20 +ZIP_CENTRAL_SIGNATURE = b"PK\x01\x02" +ZIP_EOCD = struct.Struct("<4s4H2LH") +ZIP_CENTRAL_HEADER = struct.Struct("<4s6H3L5H2L") NAME_SEPARATORS = re.compile(r"[-_.]+") SHA256 = re.compile(r"[0-9a-f]{64}") REVIEWED_SPDX_LICENSE_IDS = frozenset( @@ -212,9 +223,226 @@ def _parse_metadata(payload: bytes, source: str) -> Message: return BytesParser(policy=default).parsebytes(payload) +def _read_exact(stream: BinaryIO, size: int, error_message: str) -> bytes: + """Read exactly ``size`` bytes or reject a truncated untrusted archive.""" + payload = stream.read(size) + if len(payload) != size: + raise SystemExit(error_message) + return payload + + +def _find_zip_eocd(stream: BinaryIO) -> tuple[int, tuple[int, ...]]: + """Locate one canonical single-disk ZIP end record with a bounded tail read.""" + invalid = "release wheel is not a valid ZIP archive" + archive_size = _require_live_artifact_descriptor(stream) + tail_size = min(archive_size, ZIP_EOCD.size + 65_535) + stream.seek(archive_size - tail_size) + tail = _read_exact(stream, tail_size, invalid) + candidate = tail.rfind(ZIP_EOCD_SIGNATURE) + while candidate >= 0: + if candidate + ZIP_EOCD.size <= len(tail): + record = ZIP_EOCD.unpack_from(tail, candidate) + if candidate + ZIP_EOCD.size + record[-1] == len(tail): + return archive_size - tail_size + candidate, record[1:] + candidate = tail.rfind(ZIP_EOCD_SIGNATURE, 0, candidate) + raise SystemExit(invalid) + + +def _zip_extra_uses_zip64(extra: bytes) -> bool: + """Return whether a central-directory extra field declares ZIP64 data.""" + cursor = 0 + while cursor < len(extra): + if cursor + 4 > len(extra): + raise SystemExit("release wheel is not a valid ZIP archive") + field_id, field_size = struct.unpack_from(" len(extra): + raise SystemExit("release wheel is not a valid ZIP archive") + if field_id == 0x0001: + return True + cursor += field_size + return False + + +def _preflight_wheel_members(stream: BinaryIO) -> None: + """Count canonical ZIP members before ``ZipFile`` allocates ``ZipInfo`` objects.""" + invalid = "release wheel is not a valid ZIP archive" + eocd_offset, fields = _find_zip_eocd(stream) + disk_number, directory_disk, disk_entries, total_entries, size, offset, _ = fields + if ( + disk_number != 0 + or directory_disk != 0 + or disk_entries != total_entries + or ZIP64_EOCD_LOCATOR_SIGNATURE + in _zip_tail_before(stream, eocd_offset, ZIP64_EOCD_LOCATOR_SIZE) + ): + raise SystemExit(invalid) + if ( + total_entries == 0xFFFF + or size == 0xFFFFFFFF + or offset == 0xFFFFFFFF + or offset + size != eocd_offset + ): + raise SystemExit(invalid) + if total_entries > MAX_ARCHIVE_MEMBERS: + raise SystemExit("wheel exceeds the archive-member safety bound") + + stream.seek(offset) + consumed = 0 + actual_entries = 0 + while consumed < size: + fixed = _read_exact(stream, ZIP_CENTRAL_HEADER.size, invalid) + consumed += len(fixed) + values = ZIP_CENTRAL_HEADER.unpack(fixed) + if values[0] != ZIP_CENTRAL_SIGNATURE: + raise SystemExit(invalid) + compressed_size, uncompressed_size = values[8], values[9] + name_size, extra_size, comment_size = values[10], values[11], values[12] + start_disk, local_offset = values[13], values[16] + variable_size = name_size + extra_size + comment_size + if consumed + variable_size > size: + raise SystemExit(invalid) + variable = _read_exact(stream, variable_size, invalid) + consumed += variable_size + extra = variable[name_size : name_size + extra_size] + if ( + start_disk != 0 + or compressed_size == 0xFFFFFFFF + or uncompressed_size == 0xFFFFFFFF + or local_offset == 0xFFFFFFFF + or _zip_extra_uses_zip64(extra) + ): + raise SystemExit(invalid) + actual_entries += 1 + if actual_entries > MAX_ARCHIVE_MEMBERS: + raise SystemExit("wheel exceeds the archive-member safety bound") + if consumed != size or actual_entries != total_entries: + raise SystemExit(invalid) + stream.seek(0) + + +def _zip_tail_before(stream: BinaryIO, offset: int, size: int) -> bytes: + """Read at most ``size`` bytes immediately before one ZIP structure.""" + start = max(0, offset - size) + stream.seek(start) + return _read_exact( + stream, + offset - start, + "release wheel is not a valid ZIP archive", + ) + + +def _tar_number(field: bytes) -> int: + """Parse a canonical non-negative POSIX tar octal number.""" + if field and field[0] & 0x80: + raise SystemExit("release source distribution is not a valid gzip tar") + stripped = field.rstrip(b"\x00 ").lstrip(b" ") + if not stripped: + return 0 + if any(byte < ord("0") or byte > ord("7") for byte in stripped): + raise SystemExit("release source distribution is not a valid gzip tar") + return int(stripped, 8) + + +def _require_tar_checksum(header: bytes) -> None: + """Require the stored POSIX tar checksum to match one physical header.""" + expected = _tar_number(header[148:156]) + observed = sum(header[:148]) + (8 * ord(" ")) + sum(header[156:]) + if observed != expected: + raise SystemExit("release source distribution is not a valid gzip tar") + + +def _read_expanded( + stream: BinaryIO, + size: int, + consumed: int, + *, + retain: bool = True, +) -> tuple[bytes, int]: + """Read or skip bounded expanded tar bytes without one large allocation.""" + invalid = "release source distribution is not a valid gzip tar" + if size < 0 or consumed + size > MAX_EXPANDED_TAR_BYTES: + raise SystemExit("source distribution exceeds the expanded-tar safety bound") + chunks: list[bytes] = [] + remaining = size + while remaining: + chunk = stream.read(min(remaining, 1_048_576)) + if not chunk: + raise SystemExit(invalid) + if retain: + chunks.append(chunk) + remaining -= len(chunk) + return b"".join(chunks), consumed + size + + +def _preflight_sdist_members(stream: BinaryIO) -> None: + """Bound gzip/tar expansion and physical headers before semantic parsing.""" + invalid = "release source distribution is not a valid gzip tar" + stream.seek(0) + consumed = 0 + members = 0 + zero_headers = 0 + try: + with gzip.GzipFile(fileobj=stream, mode="rb") as expanded: + while zero_headers < 2: + header, consumed = _read_expanded(expanded, 512, consumed) + if header == b"\x00" * 512: + zero_headers += 1 + continue + zero_headers = 0 + _require_tar_checksum(header) + members += 1 + if members > MAX_ARCHIVE_MEMBERS: + raise SystemExit( + "source distribution exceeds the archive-member safety bound" + ) + size = _tar_number(header[124:136]) + type_flag = header[156:157] + if type_flag in {b"1", b"2", b"3", b"4", b"6", b"7", b"S"}: + raise SystemExit( + "source distribution contains a link or special file" + ) + if type_flag not in {b"\x00", b"0", b"5", b"x", b"g", b"L"}: + raise SystemExit( + "source distribution contains an unsupported tar form" + ) + if type_flag in {b"x", b"g", b"L"} and size > MAX_TAR_EXTENSION_BYTES: + raise SystemExit( + "source distribution extension header exceeds the safety bound" + ) + padded_size = (size + 511) // 512 * 512 + payload, consumed = _read_expanded( + expanded, + padded_size, + consumed, + retain=type_flag in {b"x", b"g"}, + ) + if type_flag in {b"x", b"g"} and b"GNU.sparse." in payload[:size]: + raise SystemExit( + "source distribution contains a sparse archive form" + ) + while True: + trailing = expanded.read(1_048_576) + if not trailing: + break + if consumed + len(trailing) > MAX_EXPANDED_TAR_BYTES: + raise SystemExit( + "source distribution exceeds the expanded-tar safety bound" + ) + if trailing.strip(b"\x00"): + raise SystemExit(invalid) + consumed += len(trailing) + except SystemExit: + raise + except (OSError, EOFError, zlib.error) as error: + raise SystemExit(invalid) from error + finally: + stream.seek(0) + + def _wheel_metadata(stream: BinaryIO) -> Message: """Read the sole bounded wheel METADATA member from the bound archive.""" - stream.seek(0) + _preflight_wheel_members(stream) try: with zipfile.ZipFile(stream) as archive: members = archive.infolist() @@ -232,31 +460,72 @@ def _wheel_metadata(stream: BinaryIO) -> Message: def _sdist_metadata(stream: BinaryIO) -> Message: - """Read the sole bounded root PKG-INFO member from the bound archive.""" - stream.seek(0) + """Read one root PKG-INFO while retaining only bounded streaming state.""" + _preflight_sdist_members(stream) + selected_payload: bytes | None = None + seen_names: set[str] = set() + member_count = 0 try: - with tarfile.open(fileobj=stream, mode="r:gz") as archive: - members = archive.getmembers() - _check_archive_names([item.name for item in members], "source distribution") - if any(item.issym() or item.islnk() or item.isdev() for item in members): - raise SystemExit("source distribution contains a link or device") - selected = [ - item - for item in members - if item.isfile() - and len(PurePosixPath(item.name).parts) == 2 - and PurePosixPath(item.name).name == "PKG-INFO" - ] - if len(selected) != 1: - raise SystemExit("source distribution must contain one root PKG-INFO") - if selected[0].size > MAX_METADATA_BYTES: - raise SystemExit("source distribution metadata exceeds the safety bound") - extracted = archive.extractfile(selected[0]) - if extracted is None: - raise SystemExit("source distribution metadata could not be read") - return _parse_metadata(extracted.read(), "source distribution") + with tarfile.open(fileobj=stream, mode="r|gz") as archive: + for member in archive: + member_count += 1 + if member_count > MAX_ARCHIVE_MEMBERS: + raise SystemExit( + "source distribution exceeds the archive-member safety bound" + ) + if member.name in seen_names: + raise SystemExit( + "source distribution contains duplicate archive paths" + ) + seen_names.add(member.name) + if not _safe_archive_name(member.name): + raise SystemExit( + "source distribution contains an unsafe archive path" + ) + if ( + member.issym() + or member.islnk() + or member.isdev() + or member.isfifo() + ): + raise SystemExit( + "source distribution contains a link or special file" + ) + if member.issparse(): + raise SystemExit( + "source distribution contains a sparse archive form" + ) + if not (member.isfile() or member.isdir()): + raise SystemExit( + "source distribution contains an unsupported tar form" + ) + path = PurePosixPath(member.name) + if ( + member.isfile() + and len(path.parts) == 2 + and path.name == "PKG-INFO" + ): + if selected_payload is not None: + raise SystemExit( + "source distribution must contain one root PKG-INFO" + ) + if member.size > MAX_METADATA_BYTES: + raise SystemExit( + "source distribution metadata exceeds the safety bound" + ) + extracted = archive.extractfile(member) + if extracted is None: + raise SystemExit( + "source distribution metadata could not be read" + ) + selected_payload = extracted.read(MAX_METADATA_BYTES + 1) + if selected_payload is None: + raise SystemExit("source distribution must contain one root PKG-INFO") + return _parse_metadata(selected_payload, "source distribution") except tarfile.TarError as error: - raise SystemExit("release source distribution is not a valid gzip tar") from error + raise SystemExit( + "release source distribution is not a valid gzip tar" + ) from error def _artifact_metadata(stream: BinaryIO, filename: str) -> Message: diff --git a/tests/test_quality_contracts.py b/tests/test_quality_contracts.py index f123cf0..65b45d2 100644 --- a/tests/test_quality_contracts.py +++ b/tests/test_quality_contracts.py @@ -70,7 +70,7 @@ def test_pyproject_enforces_complete_statement_and_branch_coverage() -> None: assert coverage["run"] == { "branch": True, - "source": ["egressweave"], + "source_dirs": ["src/egressweave", "scripts/ci"], } assert coverage["report"] == { "fail_under": 100, diff --git a/tests/test_release_sbom_member_enumeration_bound.py b/tests/test_release_sbom_member_enumeration_bound.py new file mode 100644 index 0000000..c29da0c --- /dev/null +++ b/tests/test_release_sbom_member_enumeration_bound.py @@ -0,0 +1,368 @@ +"""Regression tests for bounded archive-member enumeration.""" + +from __future__ import annotations + +import gzip +import importlib.util +import io +import struct +import tarfile +import zipfile +from pathlib import Path +from types import ModuleType + +import pytest + +REPOSITORY_ROOT = Path(__file__).resolve().parents[1] +GENERATOR_PATH = REPOSITORY_ROOT / "scripts" / "ci" / "generate_release_sbom.py" +MANIFEST_PATH = REPOSITORY_ROOT / "scripts" / "ci" / "release_runtime_dependencies.json" +EXPECTED_MAX_ARCHIVE_MEMBERS = 10_000 + + +def _load_generator() -> ModuleType: + """Load the standalone generator without importing the package under test.""" + specification = importlib.util.spec_from_file_location( + "egressweave_generate_release_sbom_member_bound", + GENERATOR_PATH, + ) + assert specification is not None and specification.loader is not None + module = importlib.util.module_from_spec(specification) + specification.loader.exec_module(module) + return module + + +def _write_overmember_wheel(path: Path) -> None: + """Create a compact wheel-shaped ZIP with one member beyond the policy.""" + with zipfile.ZipFile(path, mode="w", compression=zipfile.ZIP_STORED) as archive: + for index in range(EXPECTED_MAX_ARCHIVE_MEMBERS + 1): + archive.writestr(f"payload/member-{index:05d}", b"") + + +def _write_overmember_sdist(path: Path) -> None: + """Create a compact gzip tar with one member beyond the policy.""" + with tarfile.open(path, mode="w:gz") as archive: + for index in range(EXPECTED_MAX_ARCHIVE_MEMBERS + 1): + member = tarfile.TarInfo(f"payload/member-{index:05d}") + member.size = 0 + archive.addfile(member, io.BytesIO()) + + +def test_wheel_member_bound_precedes_zipfile_materialization( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Reject an overmember wheel before ZipFile builds its complete table.""" + generator = _load_generator() + wheel_path = tmp_path / "egressweave-0.3.0-py3-none-any.whl" + _write_overmember_wheel(wheel_path) + + def unexpected_zip_parser(*args: object, **kwargs: object) -> object: + pytest.fail("ZipFile materialized members before the repository bound") + + monkeypatch.setattr(generator.zipfile, "ZipFile", unexpected_zip_parser) + + with pytest.raises(SystemExit, match="archive-member safety bound"): + generator.build_sbom(wheel_path, MANIFEST_PATH) + + +def test_sdist_member_bound_does_not_materialize_getmembers( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Stop streaming tar enumeration without calling getmembers().""" + generator = _load_generator() + sdist_path = tmp_path / "egressweave-0.3.0.tar.gz" + _write_overmember_sdist(sdist_path) + + def unexpected_getmembers(*args: object, **kwargs: object) -> object: + pytest.fail("TarFile.getmembers materialized members before the bound") + + monkeypatch.setattr(generator.tarfile.TarFile, "getmembers", unexpected_getmembers) + + with pytest.raises(SystemExit, match="archive-member safety bound"): + generator.build_sbom(sdist_path, MANIFEST_PATH) + + +def _metadata() -> bytes: + """Return minimal metadata matching the reviewed release manifest.""" + return ( + b"Metadata-Version: 2.4\n" + b"Name: egressweave\n" + b"Version: 0.3.0\n" + b"License-Expression: Apache-2.0\n" + b"Requires-Dist: httpcore<2.0,>=1.0\n" + b"Requires-Dist: httpx<0.29,>=0.28\n" + b"Requires-Dist: idna<4,>=3.18\n\n" + ) + + +def _write_valid_wheel(path: Path) -> None: + """Write one canonical wheel accepted by both preflight and ZipFile.""" + with zipfile.ZipFile(path, mode="w") as archive: + archive.writestr("egressweave-0.3.0.dist-info/METADATA", _metadata()) + + +def _write_valid_sdist(path: Path) -> None: + """Write one canonical gzip tar accepted by both parser stages.""" + payload = _metadata() + member = tarfile.TarInfo("egressweave-0.3.0/PKG-INFO") + member.size = len(payload) + with tarfile.open(path, mode="w:gz") as archive: + archive.addfile(member, io.BytesIO(payload)) + + +def _rewrite_eocd(path: Path, **changes: int) -> None: + """Replace selected EOCD integer fields in one small ZIP fixture.""" + payload = bytearray(path.read_bytes()) + offset = payload.rfind(b"PK\x05\x06") + assert offset >= 0 + fields = list(struct.unpack_from("<4s4H2LH", payload, offset)) + indexes = { + "disk_number": 1, + "directory_disk": 2, + "disk_entries": 3, + "total_entries": 4, + "directory_size": 5, + "directory_offset": 6, + } + for name, value in changes.items(): + fields[indexes[name]] = value + struct.pack_into("<4s4H2LH", payload, offset, *fields) + path.write_bytes(payload) + + +def _central_offset(payload: bytes) -> int: + """Return the first central-directory offset from a small ZIP EOCD.""" + eocd = payload.rfind(b"PK\x05\x06") + assert eocd >= 0 + return struct.unpack_from("<4s4H2LH", payload, eocd)[6] + + +def _write_raw_sdist( + path: Path, + records: list[tuple[tarfile.TarInfo, bytes]], + trailing: bytes = b"", +) -> None: + """Write explicit physical tar records inside one gzip member.""" + raw = bytearray() + for member, payload in records: + member.size = len(payload) + raw.extend(member.tobuf(format=tarfile.PAX_FORMAT)) + raw.extend(payload) + raw.extend(b"\x00" * ((-len(payload)) % 512)) + raw.extend(b"\x00" * 1024) + raw.extend(trailing) + path.write_bytes(gzip.compress(bytes(raw), mtime=0)) + + +def test_new_archive_bounds_are_exact() -> None: + """Lock every pre-materialization resource constant to its reviewed value.""" + generator = _load_generator() + + assert generator.MAX_ARCHIVE_MEMBERS == EXPECTED_MAX_ARCHIVE_MEMBERS + assert generator.MAX_EXPANDED_TAR_BYTES == 512 * 1024 * 1024 + assert generator.MAX_TAR_EXTENSION_BYTES == 1 * 1024 * 1024 + assert generator.ZIP64_EOCD_LOCATOR_SIZE == 20 + + +def test_canonical_wheel_and_sdist_remain_compatible(tmp_path: Path) -> None: + """Accept canonical release archives after bounded preflight.""" + generator = _load_generator() + wheel = tmp_path / "egressweave-0.3.0-py3-none-any.whl" + sdist = tmp_path / "egressweave-0.3.0.tar.gz" + _write_valid_wheel(wheel) + _write_valid_sdist(sdist) + + assert generator.build_sbom(wheel, MANIFEST_PATH)["bomFormat"] == "CycloneDX" + assert generator.build_sbom(sdist, MANIFEST_PATH)["bomFormat"] == "CycloneDX" + + +def test_zip_comment_signature_does_not_hide_the_real_eocd(tmp_path: Path) -> None: + """Ignore EOCD-like bytes inside the bounded ZIP comment.""" + generator = _load_generator() + wheel = tmp_path / "comment.whl" + with zipfile.ZipFile(wheel, mode="w") as archive: + archive.writestr("egressweave-0.3.0.dist-info/METADATA", _metadata()) + archive.comment = b"comment-PK\x05\x06-not-an-end-record" + with wheel.open("rb") as stream: + generator._preflight_wheel_members(stream) + + +@pytest.mark.parametrize( + "changes", + [ + {"disk_number": 1}, + {"directory_disk": 1}, + {"disk_entries": 0}, + {"total_entries": 0xFFFF}, + {"directory_size": 0xFFFFFFFF}, + {"directory_offset": 0xFFFFFFFF}, + {"directory_offset": 1}, + ], +) +def test_zip_eocd_inconsistencies_fail_closed( + tmp_path: Path, + changes: dict[str, int], +) -> None: + """Reject multi-disk, ZIP64-sentinel, count, and offset drift.""" + generator = _load_generator() + wheel = tmp_path / "bad.whl" + _write_valid_wheel(wheel) + _rewrite_eocd(wheel, **changes) + with wheel.open("rb") as stream, pytest.raises(SystemExit, match="valid ZIP"): + generator._preflight_wheel_members(stream) + + +def test_zip_central_structure_and_zip64_forms_fail_closed(tmp_path: Path) -> None: + """Reject malformed central records and unnecessary ZIP64 structures.""" + generator = _load_generator() + + wheel = tmp_path / "signature.whl" + _write_valid_wheel(wheel) + payload = bytearray(wheel.read_bytes()) + central = _central_offset(payload) + payload[central : central + 4] = b"NOPE" + wheel.write_bytes(payload) + with wheel.open("rb") as stream, pytest.raises(SystemExit, match="valid ZIP"): + generator._preflight_wheel_members(stream) + + wheel = tmp_path / "length.whl" + _write_valid_wheel(wheel) + payload = bytearray(wheel.read_bytes()) + central = _central_offset(payload) + struct.pack_into(" None: + """Stop on the first physical record over the configured bound.""" + generator = _load_generator() + wheel = tmp_path / "two.whl" + with zipfile.ZipFile(wheel, mode="w") as archive: + archive.writestr("one", b"") + archive.writestr("two", b"") + _rewrite_eocd(wheel, total_entries=1, disk_entries=1) + monkeypatch.setattr(generator, "MAX_ARCHIVE_MEMBERS", 1) + with wheel.open("rb") as stream, pytest.raises(SystemExit, match="archive-member"): + generator._preflight_wheel_members(stream) + + +def test_tar_preflight_bounds_and_special_forms( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Reject over-budget expansion, extensions, and unsafe tar forms.""" + generator = _load_generator() + + extension = tmp_path / "extension.tar.gz" + member = tarfile.TarInfo("pax") + member.type = b"x" + _write_raw_sdist(extension, [(member, b"x")]) + monkeypatch.setattr(generator, "MAX_TAR_EXTENSION_BYTES", 0) + with extension.open("rb") as stream, pytest.raises(SystemExit, match="extension"): + generator._preflight_sdist_members(stream) + + ordinary = tmp_path / "ordinary.tar.gz" + _write_valid_sdist(ordinary) + monkeypatch.setattr(generator, "MAX_TAR_EXTENSION_BYTES", 1_048_576) + monkeypatch.setattr(generator, "MAX_EXPANDED_TAR_BYTES", 511) + with ordinary.open("rb") as stream, pytest.raises(SystemExit, match="expanded-tar"): + generator._preflight_sdist_members(stream) + + monkeypatch.setattr(generator, "MAX_EXPANDED_TAR_BYTES", 512 * 1024 * 1024) + for index, (type_flag, payload, message) in enumerate( + [ + (b"2", b"", "link or special"), + (b"Z", b"", "unsupported tar form"), + (b"x", b"20 GNU.sparse.name=x\n", "sparse archive form"), + ] + ): + path = tmp_path / f"special-{index}.tar.gz" + member = tarfile.TarInfo("entry") + member.type = type_flag + _write_raw_sdist(path, [(member, payload)]) + with path.open("rb") as stream, pytest.raises(SystemExit, match=message): + generator._preflight_sdist_members(stream) + + +def test_tar_preflight_normalizes_corruption_and_rewinds(tmp_path: Path) -> None: + """Normalize gzip and deflate corruption and rewind the descriptor.""" + generator = _load_generator() + + corrupt = tmp_path / "corrupt.tar.gz" + corrupt.write_bytes(b"not gzip") + with corrupt.open("rb") as stream: + with pytest.raises(SystemExit, match="not a valid gzip tar"): + generator._preflight_sdist_members(stream) + assert stream.tell() == 0 + + damaged = tmp_path / "damaged-deflate.tar.gz" + damaged_payload = bytearray(gzip.compress(b"\x00" * 4096, mtime=0)) + damaged_payload[15:60] = b"\xff" * 45 + damaged.write_bytes(damaged_payload) + with damaged.open("rb") as stream: + with pytest.raises(SystemExit, match="not a valid gzip tar"): + generator._preflight_sdist_members(stream) + assert stream.tell() == 0 + + trailing = tmp_path / "trailing.tar.gz" + _write_raw_sdist(trailing, [], trailing=b"X") + with trailing.open("rb") as stream, pytest.raises( + SystemExit, + match="not a valid gzip tar", + ): + generator._preflight_sdist_members(stream) + + +def test_streaming_sdist_preserves_semantic_archive_checks(tmp_path: Path) -> None: + """Reject duplicate, unsafe, missing, and oversized package metadata.""" + generator = _load_generator() + + duplicate = tmp_path / "duplicate.tar.gz" + one = tarfile.TarInfo("root/file") + two = tarfile.TarInfo("root/file") + _write_raw_sdist(duplicate, [(one, b""), (two, b"")]) + with duplicate.open("rb") as stream, pytest.raises(SystemExit, match="duplicate"): + generator._sdist_metadata(stream) + + unsafe = tmp_path / "unsafe.tar.gz" + member = tarfile.TarInfo("../outside") + _write_raw_sdist(unsafe, [(member, b"")]) + with unsafe.open("rb") as stream, pytest.raises(SystemExit, match="unsafe"): + generator._sdist_metadata(stream) + + missing = tmp_path / "missing.tar.gz" + member = tarfile.TarInfo("root/file") + _write_raw_sdist(missing, [(member, b"")]) + with missing.open("rb") as stream, pytest.raises(SystemExit, match="PKG-INFO"): + generator._sdist_metadata(stream) + + oversized = tmp_path / "oversized.tar.gz" + member = tarfile.TarInfo("root/PKG-INFO") + _write_raw_sdist( + oversized, + [(member, b"x" * (generator.MAX_METADATA_BYTES + 1))], + ) + with oversized.open("rb") as stream, pytest.raises(SystemExit, match="metadata"): + generator._sdist_metadata(stream)