diff --git a/.github/workflows/README.md b/.github/workflows/README.md index d28097881f..3c0dc0ba7b 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -63,6 +63,11 @@ Production artifacts are produced via nix builds in a separate CI workflow. - `ci:+miri` - Run Miri checks - `ci:+wasm` - Run the WASM build check - `ci:+concurrency` - Run Shuttle and Loom tests +- `ci:+debug-images` - Also build and push the core viewer, DAP debugger, and + syscall tracer images. They are built on main, in the merge queue, and on + dispatch regardless, and `ci:+merge-ready` turns them on too, since + `ci-gate` treats that label as enabling every gate; this is for when the + build itself needs debugging - `ci:+cross` - Build all cross-platform containers - `ci:+cross/full` - Also run the workspace test suite under qemu-user, on the two aarch64 musl legs. Gated like every other job, so the merge queue and @@ -104,6 +109,8 @@ If those queue failures stop being rare, the phasing is worth revisiting. - Coverage: `debug` by default; `fuzz` on deep runs - Miri: required on deep runs; opt-in on pull requests with `ci:+miri` - Containers: debug/release for dataplane and FRR; release for validator +- Debug images (core viewer, DAP debugger, syscall tracer): deep runs only, + or on a pull request with `ci:+debug-images` - VLAB configurations: spine-leaf fabric mode, L2VNI/L3VNI VPC modes, with gateway enabled @@ -111,6 +118,83 @@ If those queue failures stop being rare, the phasing is worth revisiting. - Container images pushed to GitHub Container Registry (GHCR) - Release containers published on tag pushes via `just push` +- `ghcr.io/githedgehog/dataplane/core-viewer` opens a core file from the lab. + It carries gdb plus the unstripped binaries and sources for the matching + `ghcr.io/githedgehog/dataplane` build. + Pull the tag matching the build the core came from; symbols only line up with + the exact version and profile that produced it. + The entrypoint takes the core as its only argument: + + ```console + docker run --rm -it -v /path/to/cores:/cores \ + ghcr.io/githedgehog/dataplane/core-viewer:TAG /cores/core.1234 + ``` + +- `ghcr.io/githedgehog/dataplane/dev-debugger` debugs a live dataplane from an + editor. It carries bugstalker, which understands Rust's std collections and + enum layouts, and listens for a Debug Adapter Protocol client on port 4711. + Publish the port and point the editor's DAP client at it: + + ```console + docker run --rm -p 127.0.0.1:4711:4711 ghcr.io/githedgehog/dataplane/dev-debugger:TAG + ``` + + Connecting does not by itself start anything. In remote-DAP mode bugstalker + waits for the client's `launch` request to name the program, so the editor + has to send `program`, and any dataplane arguments as `args`. A request + without `program` is rejected with `launch: missing arguments.program`. + For VS Code, in `.vscode/launch.json`. `type` has to match whatever debug + type the BugStalker extension you installed registers -- it is not a name we + choose, and it differs between extensions, so check the one you have rather + than copying this field blind: + + ```json + { + "type": "bs", + "request": "launch", + "name": "dataplane (container)", + "debugServer": 4711, + "program": "/bin/dataplane", + "args": [] + } + ``` + + For `nvim-dap`, where the first line names the adapter itself, so `type = "bs"` + below is our own label rather than an extension's: + + ```lua + dap.adapters.bs = { type = "server", host = "127.0.0.1", port = 4711 } + dap.configurations.rust = { + { + type = "bs", + request = "launch", + name = "dataplane (container)", + program = "/bin/dataplane", + args = {}, + }, + } + ``` + +- `ghcr.io/githedgehog/dataplane/syscall-tracer` records what the dataplane + asks the kernel for, as JSON, using lurk. + It carries the same stripped binaries the release image ships, since nothing + here symbolizes, so it is smaller than the other two -- though not by as much + as that suggests: like them it ships the source tree, which the entrypoint + makes the working directory. Only the debug symbols and the debuggers + themselves are absent. + + ```console + docker run --rm ghcr.io/githedgehog/dataplane/syscall-tracer:TAG > trace.jsonl + ``` + + The stream is one JSON object per line, except that tracing child threads + makes lurk announce each one with a bare `Attaching to child ` line. + Filter those out if the consumer needs strict JSONL: + + ```console + jq -R 'fromjson? // empty' < trace.jsonl + ``` + - Coverage reports from each `coverage/` job, kept for 7 days: - `coverage-html-.tar.gz` - `llvm-cov` HTML report, including the per-branch counts that Codecov does not render. Unpack and open @@ -121,6 +205,53 @@ If those queue failures stop being rare, the phasing is worth revisiting. Both upload unarchived, so they download as the named file rather than wrapped in a zip. +### Debugging locally + +The published images debug what CI built. To debug what you are building, `just +debug` builds the matching image and runs a workspace binary or a single test +inside it, at whatever `profile`, `platform`, `instrument`, and `sanitize` you +pass. Symbols only line up when those match the build the problem appeared in, +which is the whole reason to go through the image rather than a system gdb. + +```console +just debug # pick from a list +just debug bugstalker # pick, then wait for an editor +just debug lurk dataplane # trace syscalls, streams JSON, runs to exit +just debug gdb dataplane # gdbserver on 2345, waits for a client +just profile=fuzz debug gdb test_parse_interface args +just debug-list # print the same list without running anything +``` + +Name nothing and everything is offered through `skim`, which the dev shell +provides. Name a filter matching one test and it runs without asking; name one +matching several and those are offered. A filter matching nothing is an error +rather than a guess, and so is an ambiguous one when there is no terminal to +ask at, which is what makes this safe to call from a script. + +The third argument narrows which archive is searched, so +`just debug gdb some_test args` builds only `args`' tests. It is worth passing: +the default builds every test in the workspace, which is a long wait if all you +wanted was to pick from a short list. + +`gdb` and `bugstalker` block until you disconnect and interrupt them; that is +the point. `gdb` prints the `target remote` line to use. `bugstalker` prints a +`.zed/debug.json` entry ready to paste, because in remote-DAP mode it takes the +program from the client's launch request rather than from its own command line, +so connecting an editor is only half of it. The `tcp_connection` field in that +entry is what stops the editor spawning a second debugger of its own. + +A test runs with its package directory as the working directory, the way +nextest runs it, so relative paths behave the same as under `just test`. + +To open a core file: + +```console +just inspect-core /path/to/core.1234 +``` + +Pass the same build settings that produced the binary that dumped +(`just profile=release inspect-core ...`), for the same reason. + --- ## Linting and Validation Workflows for Pull Requests diff --git a/.github/workflows/dev.yml b/.github/workflows/dev.yml index 5cb2a33d5c..acd5c20e4f 100644 --- a/.github/workflows/dev.yml +++ b/.github/workflows/dev.yml @@ -68,6 +68,7 @@ jobs: outputs: container_profiles: "${{ steps.container-profiles.outputs.value }}" parallel: "${{ steps.parallel.outputs.value }}" + container_targets: "${{ steps.container-targets.outputs.value }}" profiles: "${{ steps.profiles.outputs.value }}" concurrency: "${{ steps.concurrency.outputs.value }}" cross: "${{ steps.cross.outputs.value }}" @@ -132,6 +133,16 @@ jobs: on-value: '["debug", "release", "fuzz"]' off-value: '["debug"]' + # Debug images are opt-in on pull requests because of their size. + - id: "container-targets" + uses: *gate + with: + labels: "debug-images" + # Keep this on one line: `ci-gate` writes the value to GITHUB_OUTPUT + # with a plain printf, which a multi-line value would corrupt. + on-value: '["frr.dataplane", "dataplane", "dataplane-core-viewer", "dataplane-dev-debugger", "dataplane-syscall-tracer", "validator"]' + off-value: '["frr.dataplane", "dataplane", "validator"]' + # Lab jobs require release images but not other release/fuzz checks. - id: "container-profiles" uses: *gate @@ -425,10 +436,7 @@ jobs: fail-fast: false max-parallel: ${{ fromJSON(needs.plan.outputs.parallel) }} matrix: - nix-target: - - frr.dataplane - - dataplane - - validator + nix-target: "${{ fromJSON(needs.plan.outputs.container_targets) }}" # TODO: enable cfi and safe-stack on release when possible profile: "${{ fromJSON(needs.plan.outputs.container_profiles) }}" exclude: @@ -443,6 +451,17 @@ jobs: with: recipe: "ci::push-container" recipe_args: "${{ matrix.nix-target }} ${{ matrix.profile }} ${{ needs.version.outputs.version }}" + + # A debug image that builds is not a debug image that works. Two of + # these shipped with entrypoints that could not do what the README + # documents, and building them said nothing about it. + - name: "smoke" + if: "${{ startsWith(matrix.nix-target, 'dataplane-') }}" + uses: *just + with: + recipe: "ci::smoke-container" + recipe_args: "${{ matrix.nix-target }} ${{ matrix.profile }}" + - *verify-clean-tree - *tmate diff --git a/CLAUDE.md b/CLAUDE.md index 62e214e5fb..9116ee28da 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,3 +4,21 @@ Follow the [development guide](development/README.md). If you make a design decision or do a code review, try to cite the section of the development guide you are following. + +## Commit messages + +A commit message signposts the diff; it does not restate it. Rationale belongs where it +constrains the code -- in the code, its doc comments, or the development guide -- because a +second copy in the message is a second thing to keep correct, and the two will drift. + +A message carries what the diff cannot: + +- the "why" of the commit, +- non-obvious process facts, + +It does not re-summarize the change, narrate how it was made, or repeat an argument the +commit already puts in the tree. For a docs commit, the file is the content; the message +points at it. + +Keep `git log` scannable: a few lines by default, longer only where the diff is opaque and +the message is what makes the commit reviewable. diff --git a/ci.just b/ci.just index f4c5406f8f..7f659c5ecc 100644 --- a/ci.just +++ b/ci.just @@ -78,6 +78,10 @@ cross platform libc +args: cross-test platform libc: NEXTEST_PROFILE=cross-qemu just {{ _lab }} platform={{ platform }} libc={{ libc }} profile=debug test +# Verify a debug image's entrypoint actually does its job. +smoke-container target profile: + just {{ _lab }} profile={{ profile }} platform=x86-64-v3 smoke-container {{ target }} + # Publish both content-derived and discoverable per-commit tags. [script] push-container target profile version: diff --git a/config/Cargo.toml b/config/Cargo.toml index 4c6ace6c8e..e2a04255c4 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -5,6 +5,10 @@ license.workspace = true publish.workspace = true version.workspace = true +[features] +# Generators for the configuration types, for property tests in this crate and downstream. +bolero = ["dep:bolero", "lpm/bolero"] + [dependencies] # internal common = { workspace = true } @@ -16,6 +20,7 @@ net = { workspace = true } # external arc-swap = { workspace = true } +bolero = { workspace = true, optional = true, default-features = false, features = ["alloc"] } chrono = { workspace = true, features = ["alloc", "std"] } derive_builder = { workspace = true, features = [] } ipnet = { workspace = true } diff --git a/config/src/converters/k8s/config/expose.rs b/config/src/converters/k8s/config/expose.rs index 719ecf2e91..fc4cc946f5 100644 --- a/config/src/converters/k8s/config/expose.rs +++ b/config/src/converters/k8s/config/expose.rs @@ -491,7 +491,8 @@ mod test { "10.0.4.0/24".parse::().unwrap(), ), ]); - let expose_gen = k8s_intf::bolero::expose::LegalValueExposeGenerator::new(&subnets); + // One expose at a time here, so any slot will do. + let expose_gen = k8s_intf::bolero::expose::AnyExposeGenerator::new(0, &subnets); bolero::check!() .with_generator(expose_gen) .for_each(|k8s_expose| { @@ -549,6 +550,7 @@ mod test { }) .unwrap_or(vec![]); k8s_nots.sort(); + k8s_nots.dedup(); let k8s_subnets = k8s_expose .ips .as_ref() @@ -565,7 +567,13 @@ mod test { }) .unwrap_or(vec![]); k8s_ips.extend(k8s_subnets); + // Sorted *and* deduplicated, because the conversion collects into a set-like + // structure: a prefix written twice in one expose means the same as writing it once, + // and comes out once. This only came up when the generators started producing + // repeats -- the previous ones drew each prefix from a uniqueness-preserving + // generator, so the question never arose. k8s_ips.sort(); + k8s_ips.dedup(); let k8s_as = k8s_expose.r#as.as_ref().map(|r#as| { let mut ret = r#as @@ -574,6 +582,7 @@ mod test { .map(|r#as| r#as.cidr.as_ref().unwrap().clone()) .collect::>(); ret.sort(); + ret.dedup(); ret }); @@ -584,6 +593,7 @@ mod test { .map(|r#as| r#as.not.as_ref().unwrap().clone()) .collect::>(); ret.sort(); + ret.dedup(); ret }); diff --git a/config/src/converters/k8s/config/peering.rs b/config/src/converters/k8s/config/peering.rs index 45ae4a8e29..40c6e9dac9 100644 --- a/config/src/converters/k8s/config/peering.rs +++ b/config/src/converters/k8s/config/peering.rs @@ -91,6 +91,7 @@ mod test { use k8s_intf::bolero::peering::{ LegalValuePeeringsGenerator, LegalValuePeeringsPeeringGenerator, }; + use k8s_intf::bolero::{AddressFamily, NatFlavour}; use lpm::prefix::Prefix; use crate::converters::k8s::config::{SubnetMap, VpcSubnetMap}; @@ -98,7 +99,12 @@ mod test { #[test] fn test_vpc_manifest_conversion() { let subnets = SubnetMap::new(); // Let this be empty since we are test subnet conversion elsewhere - let generator = LegalValuePeeringsPeeringGenerator::new(&subnets); + // any flavour, one family, a couple of exposes: this is testing the conversion, not the + // rules the generators satisfy + let flavours = NatFlavour::all(); + let generator = + // One manifest at a time here, so vpc zero: there is nothing to keep it disjoint from. + LegalValuePeeringsPeeringGenerator::new(&subnets, &flavours, AddressFamily::V4, 3, 0); bolero::check!() .with_generator(generator) .for_each(|peering| { @@ -169,7 +175,11 @@ mod test { ]), ), ]); - let generator = LegalValuePeeringsGenerator::new(&subnets).unwrap(); + let flavours = NatFlavour::all(); + let families = AddressFamily::all(); + let groups = vec!["gwgroup-0".to_string()]; + let generator = + LegalValuePeeringsGenerator::new(&subnets, &flavours, &families, 3, &groups).unwrap(); bolero::check!() .with_generator(generator) .for_each(|peering| { diff --git a/config/src/errors.rs b/config/src/errors.rs index bbc6509359..03058e2e95 100644 --- a/config/src/errors.rs +++ b/config/src/errors.rs @@ -75,8 +75,33 @@ pub enum ConfigError { #[error("Invalid ACL configuration: {0}")] InvalidAcl(String), // NAT-specific - #[error("Mismatched prefixes sizes for static NAT: {0:?} and {1:?}")] + /// The two sides of a static NAT expose cover different numbers of address-port pairs. + // + // The sizes are rendered with `Debug` because `PrefixWithPortsSize` is a 145-bit bnum type + // with no `Display`, and Debug pads it out to a run of digits that reads as gibberish. Hence + // leading with what to change and leaving the numbers to the end. + #[error( + "Mismatched sizes for static NAT: the exposed prefixes and the range they translate to \ + must cover the same number of address-port pairs (they cover {0:?} and {1:?})" + )] MismatchedPrefixSizes(PrefixWithPortsSize, PrefixWithPortsSize), + /// The two sides of a port-forwarding expose have prefixes of different lengths. + /// + /// Distinct from [`ConfigError::MismatchedPrefixSizes`], which compares addresses times ports. + /// That product can match while the lengths do not -- a `/32` carrying 100 ports and a `/30` + /// carrying 25 both come to 100 -- so reporting it as a size mismatch would name two numbers + /// that are equal. A port-forwarding rule maps addresses one for one, so it is the lengths + /// that have to agree. + #[error( + "Mismatched prefix lengths for port forwarding: /{private} exposed and /{public} \ + translated to; a rule maps addresses one for one, so the two must be the same length" + )] + MismatchedPrefixLengths { private: u8, public: u8 }, + #[error( + "Mismatched port range sizes for port forwarding: {private} ports exposed and {public} \ + translated to; a rule maps ports one for one, so the two must be the same size" + )] + MismatchedPortRangeSizes { private: usize, public: usize }, #[error("Peering {0} has manifests using incompatible NAT modes")] IncompatibleNatModes(String), #[error("Vpc {0} has a peering with no exposes")] diff --git a/config/src/external/overlay/validation_tests.rs b/config/src/external/overlay/validation_tests.rs index 571811ebc8..6ee8432513 100644 --- a/config/src/external/overlay/validation_tests.rs +++ b/config/src/external/overlay/validation_tests.rs @@ -636,6 +636,11 @@ mod test { } // Port forwarding: mismatched sizes rejected + // + // Reported as the prefix lengths differing rather than as `MismatchedPrefixSizes`. Port + // forwarding compares the two lengths and the two port counts directly now, instead of the + // product of the two, because a product accepts pairings a rule cannot express -- see + // `contract::tests::compensating_sizes_do_not_make_a_valid_expose`. #[test] fn test_port_forwarding_mismatched_sizes_rejected() { let expose = VpcExpose::empty() @@ -646,7 +651,13 @@ mod test { .unwrap(); let result = expose.validate(); assert!( - matches!(result, Err(ConfigError::MismatchedPrefixSizes(_, _))), + matches!( + result, + Err(ConfigError::MismatchedPrefixLengths { + private: 24, + public: 25 + }) + ), "{result:?}", ); } diff --git a/config/src/external/overlay/vpcpeering.rs b/config/src/external/overlay/vpcpeering.rs index d5f641a26c..09336ae3a6 100644 --- a/config/src/external/overlay/vpcpeering.rs +++ b/config/src/external/overlay/vpcpeering.rs @@ -432,8 +432,8 @@ impl VpcExpose { // - we have no exclusion prefixes (note: we could relax this constraint now that we // collapse exclusion prefixes early) // - we have a single prefix on each side (private and public addresses) - // - we have the same number of addresses on each side - // - the list of associated port ranges also has the same size on each side + // - a port range is present on each side + // - the two prefixes are the same length, and the two port ranges the same size if collapsed_expose.has_port_forwarding() { if !self.nots.is_empty() || !self.not_as_or_empty().is_empty() { return Err(ConfigError::Forbidden( @@ -446,12 +446,6 @@ impl VpcExpose { "Port forwarding requires a single prefix on each side", )); } - if ips_sizes != as_range_sizes { - return Err(ConfigError::MismatchedPrefixSizes( - ips_sizes, - as_range_sizes, - )); - } // For port forwarding, ensure that a port range is always present. Lack of port range would imply // all ports, which is not allowed since port 0 is forbidden in the implementation for prefixes in [collapsed_expose.ips(), collapsed_expose.as_range_or_empty()] { @@ -461,6 +455,43 @@ impl VpcExpose { )); } } + + // Matched lengths and matched port counts, rather than the matched *totals* static NAT + // asks for just above. + // + // A total is addresses times ports, so it is equally satisfied by a `/32` carrying 100 + // ports opposite a `/30` carrying 25. A port-forwarding rule cannot express that: it + // maps one prefix onto another address for address, and one port range onto another + // positionally. `PortFwEntry::is_valid` duly refuses such a rule -- but it does so + // while the configuration is being *applied*, at the last of the NAT stages, by which + // point the kernel interfaces, the flow filter, the ACLs, the static NAT tables and + // the masquerade allocator have all been committed. The apply then fails and rolls + // back, which restores the configuration but not the masquerade flows that rebuilding + // the allocator has already torn down. Refusing the same shape here costs nothing. + // + // Single prefixes carrying port ranges, both established above. + let internal = collapsed_expose + .ips() + .first() + .unwrap_or_else(|| unreachable!()); + let external = collapsed_expose + .as_range_or_empty() + .first() + .unwrap_or_else(|| unreachable!()); + if internal.prefix().length() != external.prefix().length() { + return Err(ConfigError::MismatchedPrefixLengths { + private: internal.prefix().length(), + public: external.prefix().length(), + }); + } + let internal_ports = internal.ports().unwrap_or_else(|| unreachable!()); + let external_ports = external.ports().unwrap_or_else(|| unreachable!()); + if internal_ports.len() != external_ports.len() { + return Err(ConfigError::MismatchedPortRangeSizes { + private: internal_ports.len(), + public: external_ports.len(), + }); + } } // For masquerade, we don't support port ranges @@ -1005,3 +1036,554 @@ impl VpcPeeringTable { .filter(move |p| p.left.name == vpc || p.right.name == vpc) } } + +#[cfg(any(test, feature = "bolero"))] +pub mod contract { + //! Generators for the configuration types. + //! + //! These exist so that the stages downstream of configuration -- NAT's static, masquerade and + //! port-forwarding tables -- can be driven by generated configurations rather than by a + //! handful of hand-written ones. + + use super::{VpcExpose, VpcExposeNatConfig, VpcManifest, VpcPeering, VpcPeeringTable}; + use crate::ConfigError; + use crate::external::overlay::vpc::{Vpc, VpcTable}; + use crate::external::overlay::{Overlay, ValidatedOverlay}; + use bolero::{Driver, ValueGenerator}; + use lpm::prefix::{ + IpPrefix, Ipv4Prefix, Ipv6Prefix, L4Protocol, PortRange, Prefix, PrefixWithOptionalPorts, + }; + use std::net::{Ipv4Addr, Ipv6Addr}; + use std::ops::Bound::Included; + use std::time::Duration; + + /// The widest span of addresses either side of a generated expose covers, as host bits. + /// + /// Kept small because the two sides are matched address for address: nothing here needs a + /// large prefix to be interesting, and a large one only makes what consumes it slower. + const MAX_HOST_BITS: u8 = 8; + + /// The widest port range either side covers. Same reasoning. + const MAX_PORTS: u16 = 1024; + + /// Generates [`VpcExpose`]s that use port forwarding and that [`VpcExpose::validate`] accepts. + /// + /// Valid by construction rather than by generate-and-reject, so every case reaches the code + /// under test. The rules being satisfied, each of which `validate` enforces: + /// + /// * exactly one prefix on each side, and no exclusion prefixes; + /// * both sides of one address family; + /// * neither prefix overlapping a special-use block -- hence drawing from `10.0.0.0/8` and + /// `172.16.0.0/12` for v4 and from `2001:db8::/32` for v6, which are not reserved here; + /// * a port range present on each side, since a missing one means every port and port 0 is + /// forbidden. Note that [`PrefixWithOptionalPorts::new`] *drops* a range covering all ports, + /// turning it into the missing case, so the ranges here are always bounded; + /// * equal total size on the two sides, where size counts addresses times ports. + /// + /// The last of those used to be a product -- addresses times ports -- which a `/32` carrying + /// 100 ports opposite a `/30` carrying 25 satisfies while being a pairing no port-forwarding + /// rule can express. Validation compares the two lengths and the two port counts directly now, + /// so matched sides are the whole legal space rather than a corner of it that this generator + /// was staying inside. See `tests::compensating_sizes_do_not_make_a_valid_expose`. + #[derive(Debug, Clone, Copy, Default)] + pub struct PortForwardingExpose; + + impl ValueGenerator for PortForwardingExpose { + type Output = VpcExpose; + + fn generate(&self, driver: &mut D) -> Option { + let host_bits = driver.gen_u8(Included(&0), Included(&MAX_HOST_BITS))?; + let (internal, external) = if driver.produce::()? { + v4_pair(driver, host_bits)? + } else { + v6_pair(driver, host_bits)? + }; + + // One port count for both sides: equal address counts and equal port counts is what + // makes the two totals agree, and is what `PortFwEntry` accepts. + let count = driver.gen_u16(Included(&1), Included(&MAX_PORTS))?; + let internal_ports = port_range(driver, count)?; + let external_ports = port_range(driver, count)?; + + let proto = match driver.gen_u8(Included(&0), Included(&2))? { + 0 => L4Protocol::Tcp, + 1 => L4Protocol::Udp, + _ => L4Protocol::Any, + }; + let idle_timeout = match driver.gen_u8(Included(&0), Included(&2))? { + 0 => None, + 1 => Some(Duration::from_secs(5)), + _ => Some(Duration::from_mins(5)), + }; + + VpcExpose::empty() + .make_port_forwarding(idle_timeout, Some(proto)) + .ok()? + .ip(PrefixWithOptionalPorts::new(internal, Some(internal_ports))) + .as_range(PrefixWithOptionalPorts::new(external, Some(external_ports))) + .ok() + } + } + + /// Generates [`VpcExpose`]s that masquerade and that [`VpcExpose::validate`] accepts. + /// + /// Looser than [`PortForwardingExpose`], because masquerade is: several prefixes are allowed on + /// each side and their sizes need not agree, which is the point of masquerade -- many private + /// addresses behind few public ones. What it does forbid is port ranges, on either side. + /// + /// Prefixes within a side are carved so as not to overlap, since a manifest rejects + /// overlapping ones, and the two sides are drawn from separate blocks. + #[derive(Debug, Clone, Copy, Default)] + pub struct MasqueradeExpose; + + impl ValueGenerator for MasqueradeExpose { + type Output = VpcExpose; + + fn generate(&self, driver: &mut D) -> Option { + let v4 = driver.produce::()?; + let privates = driver.gen_u8(Included(&1), Included(&3))?; + let publics = driver.gen_u8(Included(&1), Included(&2))?; + let base = driver.produce::()?; + let idle_timeout = match driver.gen_u8(Included(&0), Included(&2))? { + 0 => None, + 1 => Some(Duration::from_secs(30)), + _ => Some(Duration::from_mins(2)), + }; + + let mut expose = VpcExpose::empty().make_masquerade(idle_timeout).ok()?; + for index in 0..privates { + expose = expose.ip(PrefixWithOptionalPorts::new( + block(v4, Side::Private, base.wrapping_add(index))?, + None, + )); + } + for index in 0..publics { + expose = expose + .as_range(PrefixWithOptionalPorts::new( + block(v4, Side::Public, base.wrapping_add(index))?, + None, + )) + .ok()?; + } + Some(expose) + } + } + + #[derive(Clone, Copy)] + enum Side { + Private, + Public, + } + + // One non-overlapping block per index, from a range that is not special-use. + fn block(v4: bool, side: Side, index: u8) -> Option { + if v4 { + // 10..0.0/24 and 172.16..0/24. + let bits = match side { + Side::Private => 0x0A00_0000 | (u32::from(index) << 16), + Side::Public => 0xAC10_0000 | (u32::from(index) << 8), + }; + prefix_v4(bits, 24) + } else { + // 2001:db8:0:::/64 and 2001:db8:1:::/64. + let selector = match side { + Side::Private => 0u128, + Side::Public => 1, + }; + let bits = (0x2001_0db8u128 << 96) | (selector << 80) | (u128::from(index) << 64); + prefix_v6(bits, 64) + } + } + + /// Generates [`VpcExpose`]s that use static NAT and that [`VpcExpose::validate`] accepts. + /// + /// The interesting rule for static NAT is that the two sides must be the same total size while + /// being free to have completely different shapes: a `/26` on one side can be answered by four + /// `/28`s on the other. That fragmenting is what `RangeBuilder` exists to work out, so the + /// generator produces it deliberately -- one total, split independently into a different set + /// of prefixes per side. + /// + /// Sizes stay small so that a property can enumerate every address on both sides rather than + /// sampling. Parts are placed largest first from an aligned base, which keeps every prefix + /// aligned to its own size and keeps them from overlapping. + /// + /// No port ranges yet: static NAT permits them, and they take the mapping down a second path + /// (`PortAddrTranslationValue` rather than `AddrTranslationValue`) that carries its own + /// unfinished work. That path wants a generator of its own. + #[derive(Debug, Clone, Copy, Default)] + pub struct StaticNatExpose; + + /// The largest total either side covers, as a power of two. Small enough to enumerate. + const MAX_TOTAL_LOG: u8 = 6; + + impl ValueGenerator for StaticNatExpose { + type Output = VpcExpose; + + fn generate(&self, driver: &mut D) -> Option { + let v4 = driver.produce::()?; + let total_log = driver.gen_u8(Included(&0), Included(&MAX_TOTAL_LOG))?; + + let privates = place(v4, Side::Private, &split(driver, total_log)?)?; + let publics = place(v4, Side::Public, &split(driver, total_log)?)?; + + let mut expose = VpcExpose::empty().make_static_nat().ok()?; + for prefix in privates { + expose = expose.ip(PrefixWithOptionalPorts::new(prefix, None)); + } + for prefix in publics { + expose = expose + .as_range(PrefixWithOptionalPorts::new(prefix, None)) + .ok()?; + } + Some(expose) + } + } + + // Split 2^total_log into powers of two, largest first. Halving a part keeps the total the + // same, which is what lets the two sides be split independently and still agree. + fn split(driver: &mut D, total_log: u8) -> Option> { + let mut parts = vec![total_log]; + for _ in 0..driver.gen_u8(Included(&0), Included(&3))? { + let splittable: Vec = parts + .iter() + .enumerate() + .filter(|(_, log)| **log > 0) + .map(|(index, _)| index) + .collect(); + if splittable.is_empty() { + break; + } + let choice = usize::from(driver.gen_u8( + Included(&0), + Included(&u8::try_from(splittable.len() - 1).ok()?), + )?); + let log = parts.swap_remove(splittable[choice]); + parts.push(log - 1); + parts.push(log - 1); + } + parts.sort_unstable_by(|a, b| b.cmp(a)); + Some(parts) + } + + // Lay the parts out from the side's base, largest first, so each lands on a multiple of its + // own size and none of them overlap. + // + // Each part is followed by a gap of its own size. Placed end to end they would be aligned + // siblings, and validation normalizes those back into one prefix -- so the shape the generator + // worked out to differ between the sides would be collapsed away before anything saw it. + fn place(v4: bool, side: Side, parts: &[u8]) -> Option> { + let mut cursor = if v4 { + u128::from(match side { + Side::Private => 0x0A00_0000u32, + Side::Public => 0xAC10_0000, + }) + } else { + let selector = match side { + Side::Private => 0u128, + Side::Public => 1, + }; + (0x2001_0db8u128 << 96) | (selector << 80) + }; + + let mut out = Vec::with_capacity(parts.len()); + for &log in parts { + let prefix = if v4 { + prefix_v4(u32::try_from(cursor).ok()?, 32 - log)? + } else { + prefix_v6(cursor, 128 - log)? + }; + out.push(prefix); + cursor += 2u128 << log; + } + Some(out) + } + + /// The VNI of the VPC offering the expose in [`overlay_offering`]. + pub const LOCAL_VNI: u32 = 100; + /// The VNI of the peer it is offered to. + pub const REMOTE_VNI: u32 = 200; + + /// A two-VPC overlay whose local side offers `expose`. + /// + /// The remote side exposes an unrelated prefix, because a manifest with no exposes is + /// rejected, and one of the same address family, because a peering's two manifests must agree + /// on that. + /// + /// # Errors + /// + /// Returns whatever validating the resulting overlay returns. A generator from this module + /// produces exposes that pass, so a caller driving one can treat an error as a failure. + pub fn overlay_offering(expose: VpcExpose) -> Result { + overlay_with(expose)?.validate() + } + + /// The same two-VPC overlay as [`overlay_offering`], before validation. + /// + /// Separate because a caller assembling a whole [`crate::ExternalConfig`] has to hand it an + /// unvalidated overlay and validate the lot -- validating the overlay alone skips every check + /// that spans the underlay and the overlay together. + /// + /// # Errors + /// + /// Returns an error only if the fixed vpcs and peering this builds cannot be assembled, which + /// would be a bug here rather than anything about `expose`. + pub fn overlay_with(expose: VpcExpose) -> Result { + overlay_with_exposes(vec![expose]) + } + + /// The same, with several exposes on the local side. + /// + /// Worth having separately because one expose at a time only reaches the checks that look at an + /// expose on its own. The interesting rejections -- and the interesting things for whatever + /// consumes the result to get wrong -- are between exposes. + /// + /// # Errors + /// + /// As [`overlay_with`]. Note that *validating* the result may still fail for reasons that are + /// about the combination, such as two exposes covering overlapping prefixes, which is a + /// legitimate rejection rather than a defect. + pub fn overlay_with_exposes(exposes: Vec) -> Result { + let remote_prefix = match exposes + .first() + .and_then(|expose| expose.ips.first().map(PrefixWithOptionalPorts::prefix)) + { + Some(Prefix::IPV6(_)) => "2001:db8:ffff::/64", + _ => "3.3.3.0/24", + }; + + let mut vpc_table = VpcTable::new(); + vpc_table.add(Vpc::new("VPC-1", "AAAAA", LOCAL_VNI)?)?; + vpc_table.add(Vpc::new("VPC-2", "BBBBB", REMOTE_VNI)?)?; + + let local = exposes + .into_iter() + .fold(VpcManifest::new("VPC-1"), VpcManifest::exposing); + let remote = VpcManifest::new("VPC-2").exposing( + VpcExpose::empty().ip(remote_prefix + .parse::() + .unwrap_or_else(|_| unreachable!()) + .into()), + ); + let mut peerings = VpcPeeringTable::new(); + peerings.add(VpcPeering::with_default_group( + "VPC-1--VPC-2", + local, + remote, + ))?; + + Ok(Overlay::new(vpc_table, peerings)) + } + + /// Which NAT flavour a generated expose uses, for a caller that wants to branch on it. + #[must_use] + pub fn nat_config(expose: &VpcExpose) -> Option<&VpcExposeNatConfig> { + expose.nat_config() + } + + // A private and a public prefix of the same length, from blocks that are not special-use. + fn v4_pair(driver: &mut D, host_bits: u8) -> Option<(Prefix, Prefix)> { + let len = 32 - host_bits; + let mask = u32::MAX.checked_shl(u32::from(host_bits)).unwrap_or(0); + // 10.0.0.0/8 and 172.16.0.0/12; masking only clears host bits, so the blocks survive. + let internal = (0x0A00_0000 | (driver.produce::()? & 0x00FF_FFFF)) & mask; + let external = (0xAC10_0000 | (driver.produce::()? & 0x000F_FFFF)) & mask; + Some((prefix_v4(internal, len)?, prefix_v4(external, len)?)) + } + + // 2001:db8::/32, with the two sides separated so they cannot be the same prefix. + const INTERNAL_BASE: u128 = 0x2001_0db8_0000_0000_0000_0000_0000_0000; + const EXTERNAL_BASE: u128 = 0x2001_0db8_0001_0000_0000_0000_0000_0000; + + fn v6_pair(driver: &mut D, host_bits: u8) -> Option<(Prefix, Prefix)> { + let len = 128 - host_bits; + let mask = u128::MAX.checked_shl(u32::from(host_bits)).unwrap_or(0); + let internal = (INTERNAL_BASE | u128::from(driver.produce::()?)) & mask; + let external = (EXTERNAL_BASE | u128::from(driver.produce::()?)) & mask; + Some((prefix_v6(internal, len)?, prefix_v6(external, len)?)) + } + + fn prefix_v4(bits: u32, len: u8) -> Option { + Ipv4Prefix::new(Ipv4Addr::from_bits(bits), len) + .ok() + .map(Prefix::from) + } + + fn prefix_v6(bits: u128, len: u8) -> Option { + Ipv6Prefix::new(Ipv6Addr::from_bits(bits), len) + .ok() + .map(Prefix::from) + } + + // A range of exactly `count` ports, never starting at 0 and never covering every port. + fn port_range(driver: &mut D, count: u16) -> Option { + // Parenthesised: `start + count - 1` would add before subtracting, and the sum reaches + // 65536 at the top of the range. + let last_start = u16::MAX - (count - 1); + let start = driver.gen_u16(Included(&1), Included(&last_start))?; + PortRange::new(start, start + (count - 1)).ok() + } + + #[cfg(test)] + mod tests { + use super::*; + + /// Everything this generator produces is something `validate` accepts. + /// + /// The generator exists to reach the code past validation, so a case that does not get + /// there is a case wasted -- and silently, since a rejected configuration still counts as + /// a run. + #[test] + fn every_generated_expose_validates() { + bolero::check!() + .with_generator(PortForwardingExpose) + .for_each(|expose: &VpcExpose| { + let validated = expose.validate(); + assert!( + validated.is_ok(), + "generated expose was rejected: {expose} -- {:?}", + validated.err() + ); + }); + } + + // A port-forwarding expose from a prefix and an inclusive port range on each side. + fn forwarding(internal: (&str, u16, u16), external: (&str, u16, u16)) -> VpcExpose { + let side = |(prefix, first, last): (&str, u16, u16)| { + PrefixWithOptionalPorts::new( + prefix.into(), + Some(PortRange::new(first, last).unwrap_or_else(|_| unreachable!())), + ) + }; + VpcExpose::empty() + .make_port_forwarding(None, None) + .unwrap_or_else(|_| unreachable!()) + .ip(side(internal)) + .as_range(side(external)) + .unwrap_or_else(|_| unreachable!()) + } + + /// Sides of different prefix lengths are refused, however their totals work out. + /// + /// A total counts addresses times ports, so a `/32` with 100 ports and a `/30` with 25 have + /// the same one -- which is what a size check alone accepts. A port-forwarding rule maps + /// address for address and port for port, so it cannot express that pairing, and + /// `PortFwEntry` refuses it. It used to refuse it during the apply, after the earlier + /// stages had committed and with a rollback to follow that restores the configuration but + /// not the flows already torn down. Refused here, none of that happens. + #[test] + fn compensating_sizes_do_not_make_a_valid_expose() { + let expose = forwarding(("10.0.0.0/32", 1000, 1099), ("172.16.0.0/30", 2000, 2024)); + assert!( + matches!( + expose.validate(), + Err(ConfigError::MismatchedPrefixLengths { + private: 32, + public: 30 + }) + ), + "a /32 with 100 ports opposite a /30 with 25 was accepted: {:?}", + expose.validate() + ); + } + + /// Matched prefixes with port ranges of different sizes are refused too. + #[test] + fn port_ranges_of_different_sizes_do_not_make_a_valid_expose() { + let expose = forwarding(("10.0.0.0/32", 1000, 1099), ("172.16.0.0/32", 2000, 2049)); + assert!( + matches!( + expose.validate(), + Err(ConfigError::MismatchedPortRangeSizes { + private: 100, + public: 50 + }) + ), + "100 ports opposite 50 was accepted: {:?}", + expose.validate() + ); + } + + /// And the matched case still passes, so the two guards are about mismatch rather than + /// about port forwarding having stopped validating at all. + #[test] + fn matched_sides_still_make_a_valid_expose() { + let expose = forwarding(("10.0.0.0/30", 1000, 1099), ("172.16.0.0/30", 2000, 2099)); + assert!(expose.validate().is_ok(), "{:?}", expose.validate()); + } + + /// The masquerade generator's cases validate too, and stay masquerade. + /// + /// Masquerade's own rule is that neither side carries a port range, which is easy to break + /// by reusing the port-forwarding shape. + #[test] + fn every_generated_masquerade_expose_validates() { + bolero::check!() + .with_generator(MasqueradeExpose) + .for_each(|expose: &VpcExpose| { + let validated = expose.validate().unwrap_or_else(|e| { + panic!("generated expose was rejected: {expose} -- {e:?}") + }); + assert!(validated.has_masquerade()); + assert!(!validated.ips().is_empty()); + assert!(!validated.as_range_or_empty().is_empty()); + }); + } + + /// A generated expose can be dropped into an overlay that validates. + /// + /// The helper is what every downstream property is built on, so a case it cannot place is + /// a case none of them see. + #[test] + fn a_generated_expose_can_be_offered_in_an_overlay() { + bolero::check!() + .with_generator(MasqueradeExpose) + .cloned() + .for_each(|expose: VpcExpose| { + let shown = expose.to_string(); + assert!( + overlay_offering(expose).is_ok(), + "could not build an overlay around {shown}" + ); + }); + } + + /// The static NAT generator's cases validate, and the two sides really do differ in shape. + /// + /// The second half is the part worth asserting: a generator that always produced one + /// prefix per side would pass the first half while never exercising the fragmenting that + /// `RangeBuilder` exists for. + #[test] + fn every_generated_static_nat_expose_validates() { + let mut shapes_differed = false; + bolero::check!() + .with_generator(StaticNatExpose) + .for_each(|expose: &VpcExpose| { + let validated = expose.validate().unwrap_or_else(|e| { + panic!("generated expose was rejected: {expose} -- {e:?}") + }); + assert!(validated.has_static_nat()); + if validated.ips().len() != validated.as_range_or_empty().len() { + shapes_differed = true; + } + }); + assert!( + shapes_differed, + "no generated expose had a different number of prefixes on each side, so the \ + mapping was never asked to fragment" + ); + } + + /// And it is port forwarding that comes out the other side, with both sides intact. + #[test] + fn a_generated_expose_survives_validation_as_port_forwarding() { + bolero::check!() + .with_generator(PortForwardingExpose) + .for_each(|expose: &VpcExpose| { + let validated = expose.validate().unwrap_or_else(|e| panic!("{e:?}")); + assert!(validated.has_port_forwarding()); + assert_eq!(validated.ips().len(), 1); + assert_eq!(validated.as_range_or_empty().len(), 1); + }); + } + } +} diff --git a/default.nix b/default.nix index 301dba5408..6eaefdc632 100644 --- a/default.nix +++ b/default.nix @@ -159,10 +159,15 @@ let cargo-deny cargo-depgraph cargo-edit + cargo-expand cargo-llvm-cov + cargo-mutants cargo-nextest + cargo-show-asm commitlint-rs + deno direnv + duvet gateway-crd gettext jq @@ -182,6 +187,7 @@ let qemu-user rust-toolchain shellcheck + skim # the `just debug` picker skopeo wasmtime wget @@ -431,6 +437,8 @@ let # Keep debug paths stable across revisions. Source readers # must resolve this relative prefix from the workspace root. "--remap-path-prefix==${src-prefix}" + # Keep debug outputs from retaining the complete Rust toolchain. + "--remap-path-prefix=${pkgs.rust-toolchain}/lib/rustlib/src/rust=${pkgs.rust-toolchain.passthru.availableComponents.rust-src}/lib/rustlib/src/rust" ] ) else @@ -470,6 +478,13 @@ let mkdir -p $debug/bin for f in $out/bin/*; do mv "$f" "$debug/bin/$(basename "$f")" + # Trade index for size. gdb has consumed `.debug_names` + # as a real DWARF-5 index since 14, and this ships 17.2, + # so dropping it is not free -- gdb rebuilds an index on + # each start instead. The section is large enough on + # these binaries that the image is worth more than the + # startup, and bugstalker does not read it at all. + ${objcopy} --remove-section=.debug_names "$debug/bin/$(basename "$f")" ${strip} --strip-debug "$debug/bin/$(basename "$f")" -o "$f" ${objcopy} --add-gnu-debuglink="$debug/bin/$(basename "$f")" "$f" done @@ -962,32 +977,169 @@ let }).overrideAttrs source-volatile; - containers.dataplane-debugger = + # Shared runtime and unstripped binaries for the debugger images. + debug-image-paths = [ + pkgs.pkgsBuildHost.coreutils + pkgs.pkgsBuildHost.bashInteractive + pkgs.pkgsHostHost.dockerTools.usrBinEnv + + pkgs.pkgsHostHost.libc.debug + workspace.cli.debug + workspace.dataplane.debug + workspace.init.debug + ]; + + # Copy Rust's gdb helpers without retaining rustc as a runtime dependency. + rust-gdb-printers = pkgs.runCommand "rust-gdb-printers" { } '' + mkdir -p "$out/lib/rustlib/etc" + for f in gdb_load_rust_pretty_printers.py gdb_lookup.py gdb_providers.py rust_types.py; do + cp -L "${pkgs.rust-toolchain}/lib/rustlib/etc/$f" "$out/lib/rustlib/etc/$f" + done + ''; + + # Opens dataplane core files with matching symbols and sources. + containers.dataplane-core-viewer = (pkgs.dockerTools.buildLayeredImage { - name = "ghcr.io/githedgehog/dataplane/debugger"; + name = "ghcr.io/githedgehog/dataplane/core-viewer"; inherit tag; - contents = pkgs.buildEnv { - name = "dataplane-debugger-env"; - pathsToLink = [ - "/bin" - "/etc" - "/var" - "/lib" + contents = + (pkgs.buildEnv { + name = "dataplane-core-viewer-env"; + pathsToLink = [ + "/bin" + "/etc" + "/var" + "/lib" + ]; + paths = [ + pkgs.pkgsBuildHost.gdb + rust-gdb-printers + ] + ++ debug-image-paths; + }).overrideAttrs + source-volatile; + # gdb needs a writable HOME for logs and its index cache. + extraCommands = '' + # The remapped prefix is relative, so a debugger resolves source against + # its working directory rather than an absolute path. Ship the tree at + # /src and start there. Referencing ${src} is also what keeps it in the + # image closure: with the remap no longer naming a store path, nothing + # else retains it. + ln -s "${src}" src + mkdir -p tmp + chmod 1777 tmp + ''; + config = { + WorkingDir = "/src"; + Entrypoint = [ + "/bin/gdb" + "--directory=/lib/rustlib/etc" + "-iex" + "add-auto-load-safe-path /lib/rustlib/etc" + "-iex" + "source /lib/rustlib/etc/gdb_load_rust_pretty_printers.py" + "/bin/dataplane" ]; - paths = [ - pkgs.pkgsBuildHost.gdb - pkgs.pkgsBuildHost.rr - pkgs.pkgsBuildHost.coreutils - pkgs.pkgsBuildHost.bashInteractive - pkgs.pkgsBuildHost.iproute2 - pkgs.pkgsBuildHost.ethtool - pkgs.pkgsHostHost.dockerTools.usrBinEnv + Env = [ "HOME=/tmp" ]; + }; + }).overrideAttrs + source-volatile; - pkgs.pkgsHostHost.libc.debug - workspace.cli.debug - workspace.dataplane.debug - workspace.init.debug + # Exposes bugstalker's DAP server for live debugging. + containers.dataplane-dev-debugger = + (pkgs.dockerTools.buildLayeredImage { + name = "ghcr.io/githedgehog/dataplane/dev-debugger"; + inherit tag; + contents = + (pkgs.buildEnv { + name = "dataplane-dev-debugger-env"; + pathsToLink = [ + "/bin" + "/etc" + "/var" + "/lib" + ]; + paths = [ pkgs.pkgsBuildHost.bugstalker ] ++ debug-image-paths; + }).overrideAttrs + source-volatile; + # bugstalker needs a writable HOME for its keymap and history. + extraCommands = '' + # The remapped prefix is relative, so a debugger resolves source against + # its working directory rather than an absolute path. Ship the tree at + # /src and start there. Referencing ${src} is also what keeps it in the + # image closure: with the remap no longer naming a store path, nothing + # else retains it. + ln -s "${src}" src + mkdir -p tmp + chmod 1777 tmp + ''; + config = { + WorkingDir = "/src"; + Entrypoint = [ + "/bin/bs" + # Bind the published interface rather than container-local loopback. + "--dap-remote=0.0.0.0:4711" + # rustc is absent, so bugstalker cannot infer this path. + "--std-lib-path=${pkgs.rust-toolchain.passthru.availableComponents.rust-src}/lib/rustlib/src/rust" + # No debuggee here on purpose. In `--dap-remote` mode bugstalker + # ignores the CLI debuggee and waits for the client's `launch` request + # to name one, so a path here would be silently dead and would imply + # that connecting alone starts the dataplane. The editor supplies + # `program` instead; see .github/workflows/README.md. + ]; + Env = [ "HOME=/tmp" ]; + ExposedPorts = { + "4711/tcp" = { }; + }; + }; + }).overrideAttrs + source-volatile; + + # Traces the release binaries' syscalls as JSON with lurk. + containers.dataplane-syscall-tracer = + (pkgs.dockerTools.buildLayeredImage { + name = "ghcr.io/githedgehog/dataplane/syscall-tracer"; + inherit tag; + contents = + (pkgs.buildEnv { + name = "dataplane-syscall-tracer-env"; + pathsToLink = [ + "/bin" + "/etc" + "/var" + "/lib" + ]; + paths = [ + pkgs.pkgsBuildHost.lurk + pkgs.pkgsHostHost.dockerTools.fakeNss + pkgs.pkgsHostHost.busybox + pkgs.pkgsHostHost.dockerTools.usrBinEnv + workspace.cli + workspace.dataplane + workspace.init + ]; + }).overrideAttrs + source-volatile; + extraCommands = '' + # The remapped prefix is relative, so a debugger resolves source against + # its working directory rather than an absolute path. Ship the tree at + # /src and start there. Referencing ${src} is also what keeps it in the + # image closure: with the remap no longer naming a store path, nothing + # else retains it. + ln -s "${src}" src + mkdir -p tmp + chmod 1777 tmp + ''; + config = { + WorkingDir = "/src"; + Entrypoint = [ + "/bin/lurk" + "--json" + # Include the worker threads where the dataplane does its work. + "--follow-forks" + "/bin/dataplane" ]; + Env = [ "HOME=/tmp" ]; }; }).overrideAttrs source-volatile; @@ -1006,6 +1158,7 @@ let # pkgs.wireshark-cli pkgs.bashInteractive + pkgs.bugstalker pkgs.coreutils pkgs.curl pkgs.debianutils diff --git a/flow-entry/src/flow_table/mod.rs b/flow-entry/src/flow_table/mod.rs index a8d246a881..10307d6ace 100644 --- a/flow-entry/src/flow_table/mod.rs +++ b/flow-entry/src/flow_table/mod.rs @@ -9,7 +9,7 @@ pub mod table; mod concurrent_fuzz; pub use nf_lookup::FlowLookup; -pub use table::{FlowTable, FlowTableReadGuard}; +pub use table::{FlowTable, FlowTableReadGuard, Insertion}; pub use net::flows::atomic_instant::AtomicInstant; pub use net::flows::flow_info::*; diff --git a/flow-entry/src/flow_table/table.rs b/flow-entry/src/flow_table/table.rs index 27695322f7..0f15439e01 100644 --- a/flow-entry/src/flow_table/table.rs +++ b/flow-entry/src/flow_table/table.rs @@ -19,6 +19,16 @@ pub enum FlowTableError { CapacityExceeded, } +/// What [`FlowTable::insert_if_absent`] did. +#[derive(Debug)] +pub enum Insertion { + /// The flow is in the table, put there by this call. + Installed, + /// Nothing was inserted: a live flow already holds the key. That flow is carried here, since a + /// caller that lost the key generally wants to go on with whatever won it. + Occupied(Arc), +} + type Table = DashMap, RandomState>; #[derive(Debug)] @@ -212,25 +222,52 @@ impl FlowTable { }); } + /// Whether a flow may go into a table that is at its limit. + /// + /// Reject new flows when at capacity. Exception: always admit the second half of a + /// related pair (e.g. the reverse NAT flow) to avoid leaving a one-sided entry. + fn admit(&self, table: &Table, val: &Arc) -> Result<(), FlowTableError> { + if table.len() < self.capacity.load(Ordering::Relaxed) { + return Ok(()); + } + let has_related_in_table = val + .related + .as_ref() + .and_then(Weak::upgrade) + .is_some_and(|rel| rel.is_active()); + + if has_related_in_table { + Ok(()) + } else { + Err(FlowTableError::CapacityExceeded) + } + } + + /// Retire a flow that has just been displaced from the table, and the other half of its pair. + /// + /// The displaced flow is unreachable by key from here on. The half of its pair that is still + /// in the table is unreachable in a worse way: it stays live under a key of its own, mapping a + /// translation whose allocation goes back to the pool as soon as the displaced half is + /// collected. Return traffic matching that key once the public pair has been handed out again + /// would then be translated for whoever held it before. So the pair goes together. + fn displace(old: Option<&Arc>) { + let Some(old) = old else { + return; + }; + old.update_status(FlowStatus::Detached); + old.token.cancel(); + if let Some(related) = old.related.as_ref().and_then(Weak::upgrade) { + debug!("insert: invalidating the partner of a displaced flow"); + related.invalidate(); + } + } + fn insert_common(&self, val: &Arc) -> Result>, FlowTableError> { let table = self.table.read(); - let capacity = self.capacity.load(Ordering::Relaxed); let flow_key = val.flowkey(); debug!("insert: inserting flow {flow_key}"); - // Reject new flows when at capacity. Exception: always admit the second half of a - // related pair (e.g. the reverse NAT flow) to avoid leaving a one-sided entry. - if table.len() >= capacity { - let has_related_in_table = val - .related - .as_ref() - .and_then(Weak::upgrade) - .is_some_and(|rel| rel.is_active()); - - if !has_related_in_table { - return Err(FlowTableError::CapacityExceeded); - } - } + self.admit(&table, val)?; let result = table.insert(*flow_key, val.clone()); // Set Active only after the insert so that the invariant holds: Active iff in the @@ -243,10 +280,7 @@ impl FlowTable { #[cfg(not(any(feature = "shuttle", feature = "loom")))] Self::start_timer(self.table.clone(), val.clone()); - if let Some(old) = result.as_ref() { - old.update_status(FlowStatus::Detached); - old.token.cancel(); - } + Self::displace(result.as_ref()); let Some(ret) = result else { return Ok(None); @@ -259,6 +293,80 @@ impl FlowTable { Ok(Some(ret)) } + /// Add a flow entry to the table, unless a live flow already holds its key. + /// + /// Two packets of one new flow can reach a NAT stage at the same time. Packets of a 5-tuple + /// usually land on one core, but nothing guarantees that, and each packet builds a pair of its + /// own. With a plain insert whoever gets there second displaces the other's forward flow while + /// its reverse stays in the table: reverse keys carry the allocation that made them, no two + /// allocations agree, so the two never collide and the loser's reverse is never displaced with + /// its partner. That orphan outlives the allocation it maps, and return traffic for a public + /// pair handed out again later reaches whoever held it before. + /// + /// Arbitrating on one key is enough to prevent it, because racing packets of a single flow + /// share their forward key by construction: only the caller who wins it goes on to insert a + /// reverse. + /// + /// A flow that is present but no longer live is displaced as usual. It is a corpse its timer + /// has not swept yet, and standing aside for one would drop a packet that could have replaced + /// it. + /// + /// # Returns + /// + /// [`Insertion::Installed`] if the flow went in, or [`Insertion::Occupied`] with the live flow + /// that holds the key if it did not. + /// + /// # Panics + /// + /// Panics if this thread already holds the read lock on the table. + /// + /// # Errors + /// + /// Returns [`FlowTableError::CapacityExceeded`] when the table has reached its hard limit. + pub fn insert_if_absent(&self, val: &Arc) -> Result { + let table = self.table.read(); + let flow_key = val.flowkey(); + debug!("insert: inserting flow {flow_key} unless it is already held"); + + self.admit(&table, val)?; + + // The entry guard holds this key's shard, so testing the incumbent and standing aside + // cannot race another insert of the same key. It has to end before the table guard does, + // hence resolving to a value here rather than returning from inside the match. + let displaced = match table.entry(*flow_key) { + dashmap::Entry::Occupied(mut occupied) => { + if occupied.get().is_active() { + Err(occupied.get().clone()) + } else { + Ok(Some(occupied.insert(val.clone()))) + } + } + dashmap::Entry::Vacant(vacant) => { + vacant.insert(val.clone()); + Ok(None) + } + }; + let displaced = match displaced { + Ok(displaced) => displaced, + Err(held) => { + drop(table); + debug!("insert: flow {flow_key} is already held by a live flow"); + return Ok(Insertion::Occupied(held)); + } + }; + + // Active only after the insert, as in `insert_common`. + val.update_status(FlowStatus::Active); + drop(table); + + #[cfg(not(any(feature = "shuttle", feature = "loom")))] + Self::start_timer(self.table.clone(), val.clone()); + + Self::displace(displaced.as_ref()); + + Ok(Insertion::Installed) + } + /// Lookup a flow in the table. /// /// # Panics @@ -412,6 +520,7 @@ mod tests { #[concurrency_mode(std)] mod std_tests { + use net::flows::FlowInfoFlags; use std::time::Instant; use tracing_test::traced_test; @@ -634,6 +743,108 @@ mod tests { assert_eq!(flow_table.active_len().unwrap(), 0); } + fn key_for(src_port: u16) -> FlowKey { + FlowKey::new( + Some(VpcDiscriminant::VNI(Vni::new_checked(1).unwrap())), + "1.2.3.4".parse::().unwrap(), + "4.5.6.7".parse::().unwrap(), + IpProtoKey::Tcp(TcpProtoKey { + src_port: TcpPort::new_checked(src_port).unwrap(), + dst_port: TcpPort::new_checked(2048).unwrap(), + }), + ) + } + + /// A live flow keeps its key against a second insertion. + /// + /// This is what stops two packets of one new flow from each installing a pair. Without it + /// the second one displaces the first's forward flow, and the first's reverse -- which + /// carries an allocation of its own and so collides with nothing -- is left in the table + /// mapping a translation whose allocation is about to go back to the pool. + #[tokio::test] + async fn an_active_flow_holds_its_key_against_a_second_insertion() { + let flow_table = FlowTable::default(); + let key = key_for(1025); + let far_future = Instant::now() + Duration::from_hours(1); + + let first = Arc::new(FlowInfo::new(key, far_future)); + assert!(matches!( + flow_table.insert_if_absent(&first).unwrap(), + Insertion::Installed + )); + + let second = Arc::new(FlowInfo::new(key, far_future)); + let outcome = flow_table.insert_if_absent(&second).unwrap(); + let Insertion::Occupied(held) = outcome else { + panic!("a live flow was displaced by a second insertion: {outcome:?}"); + }; + assert!(Arc::ptr_eq(&held, &first), "the wrong flow was reported"); + + // The incumbent is still the one serving the key, and the newcomer never went in. + let found = flow_table.lookup(&key).expect("the key is still served"); + assert!(Arc::ptr_eq(&found, &first)); + assert_ne!(second.status(), FlowStatus::Active); + } + + /// A flow that is no longer live does not hold its key. + /// + /// Standing aside for one would drop a packet that could have replaced it: the entry is a + /// corpse whose timer has not swept it yet. + #[tokio::test] + async fn a_flow_that_is_not_live_is_displaced() { + let flow_table = FlowTable::default(); + let key = key_for(1026); + let far_future = Instant::now() + Duration::from_hours(1); + + let first = Arc::new(FlowInfo::new(key, far_future)); + flow_table.insert_if_absent(&first).unwrap(); + first.invalidate(); + + let second = Arc::new(FlowInfo::new(key, far_future)); + assert!( + matches!( + flow_table.insert_if_absent(&second).unwrap(), + Insertion::Installed + ), + "a flow that was no longer live held its key" + ); + let found = flow_table.lookup(&key).expect("the key is served"); + assert!(Arc::ptr_eq(&found, &second)); + } + + /// Displacing a flow takes the other half of its pair with it. + /// + /// The two halves have different keys, so nothing displaces the partner in its own right. + /// Left behind it stays live, mapping a translation whose allocation returns to the pool + /// with the half that was displaced. + #[tokio::test] + async fn displacing_a_flow_invalidates_its_partner() { + let flow_table = FlowTable::default(); + let (forward_key, reverse_key) = (key_for(1027), key_for(1028)); + let far_future = Instant::now() + Duration::from_hours(1); + + let (forward, reverse) = FlowInfo::related_pair( + far_future, + forward_key, + FlowInfoFlags::INITIATOR, + reverse_key, + FlowInfoFlags::default(), + ) + .expect("related_pair should succeed for distinct keys"); + flow_table.insert_from_arc(&forward).unwrap(); + flow_table.insert_from_arc(&reverse).unwrap(); + assert!(reverse.is_active()); + + // The unconditional path still displaces, and has to clean up after itself. + let replacement = Arc::new(FlowInfo::new(forward_key, far_future)); + flow_table.insert_from_arc(&replacement).unwrap(); + + assert!( + !reverse.is_active(), + "the partner of a displaced flow was left live in the table" + ); + } + #[tokio::test] async fn test_flow_table_capacity_exceeded() { let flow_table = FlowTable::default(); diff --git a/justfile b/justfile index 6eed433ff8..45cb7d752b 100644 --- a/justfile +++ b/justfile @@ -114,7 +114,9 @@ oci_insecure := "" oci_name := "githedgehog/dataplane" oci_frr_prefix := "githedgehog/dataplane/frr" oci_image_dataplane := oci_repo + "/" + oci_name + ":" + version -oci_image_dataplane_debugger := oci_repo + "/" + oci_name + "/debugger:" + version +oci_image_dataplane_core_viewer := oci_repo + "/" + oci_name + "/core-viewer:" + version +oci_image_dataplane_dev_debugger := oci_repo + "/" + oci_name + "/dev-debugger:" + version +oci_image_dataplane_syscall_tracer := oci_repo + "/" + oci_name + "/syscall-tracer:" + version oci_image_dataplane_validator := oci_repo + "/" + oci_name + "/validator:" + version oci_image_frr_dataplane := oci_repo + "/" + oci_frr_prefix + ":" + version oci_image_frr_host := oci_repo + "/" + oci_frr_prefix + "-host:" + version @@ -256,6 +258,339 @@ setup-roots *args: {{ args }} done +# Ports the debug helpers listen on. Override for a second session. +gdb_port := "2345" + +bs_port := "4711" + +# Workspace binaries the debug images already carry, spelled as the images +# spell them. The nix attribute is keyed by directory (`workspace.init`) but +# the binary it installs is `dataplane-init`, and it is the binary name that has +# to appear here -- `debug` passes it straight through as `/bin/`. +[private] +_debug_binaries := "dataplane cli dataplane-init" + +# Build settings to forward when a recipe has to re-enter `just` rather than +# depend on it. A debug session is only useful against the same build the +# problem showed up in. +[private] +_forward := "jobs=" + jobs + " cores=" + cores + " debug_justfile=" + debug_justfile \ + + " profile=" + profile + " platform=" + platform + " libc=" + libc \ + + " features=" + features + " default_features=" + default_features \ + + " instrument=" + instrument + " sanitize=" + sanitize + " nightly=" + nightly + +# List what `just debug` can run at the current profile: the workspace binaries +# the images carry, and every test in a nextest archive. +[script] +debug-list package="tests.all": (build (if package == "tests.all" { "tests.all" } else { "tests.pkg." + package })) + {{ _just_debuggable_ }} + declare -r suite="{{ if package == "tests.all" { "tests.all" } else { "tests.pkg." + package } }}" + echo "binaries:" + for b in {{ _debug_binaries }}; do printf ' %s\n' "${b}"; done + echo "tests:" + cargo nextest list --archive-file results/"${suite}"/*.tar.zst --workspace-remap "$(pwd)" + +# Run a workspace binary or a single test under a debug helper, in the image +# that carries it, built at the current profile and instrumentation. +# +# just debug # pick from a list +# just debug bugstalker # pick, then wait for an editor +# just debug lurk dataplane # trace syscalls, streams JSON +# just debug gdb dataplane # gdbserver, waits for gdb +# just profile=fuzz debug gdb test_parse_interface args +# +# `target` is one of the workspace binaries, or a nextest filter. Leave it out +# and everything is offered; give a filter matching one test and it is used +# without asking; give one matching several and those are offered. `package` +# narrows which archive is built to find them, so naming one is much quicker +# than the default of building every test in the workspace. gdb and bugstalker +# wait for a client instead of running to completion. +# +# The remapped source prefix is relative, so a client shows source only if its +# working directory is the root of the tree the binary was built from. This +# passes `-w` for that in every case; a local `gdb` needs you to be standing in +# the right place yourself. +[script] +debug tool="gdb" target="" package="tests.all" *args: (build-container (if tool == "gdb" { "dataplane-core-viewer" } else if tool == "bugstalker" { "dataplane-dev-debugger" } else if tool == "lurk" { "dataplane-syscall-tracer" } else { error("debug: unknown tool '" + tool + "'; expected gdb, bugstalker, or lurk") })) + {{ _just_debuggable_ }} + declare -xr DOCKER_HOST="${DOCKER_HOST:-unix://{{ docker_sock }}}" + + declare image + case "{{ tool }}" in + gdb) image="{{ oci_image_dataplane_core_viewer }}" ;; + bugstalker) image="{{ oci_image_dataplane_dev_debugger }}" ;; + lurk) image="{{ oci_image_dataplane_syscall_tracer }}" ;; + *) + # Unreachable: the dependency above rejects anything else before + # this body runs. Kept so `set -u` cannot meet an unset `image`. + >&2 echo "debug: unknown tool '{{ tool }}'" + exit 1 + ;; + esac + declare -r image + + # Resolve the target to a program, its arguments, and a working directory. + declare program workdir + declare -a program_args=() mounts=() + if [[ -n '{{ target }}' && " {{ _debug_binaries }} " == *" {{ target }} "* ]]; then + # The images carry these already, so nothing needs mounting. `/src` is + # where they ship the sources, and the remapped prefix is relative, so + # the debugger only resolves source if it starts there. + program="/bin/{{ target }}" + workdir="/src" + else + # A test, or nothing yet. Either way the archive has to exist before + # there is anything to name or to choose between. + declare -r suite="{{ if package == "tests.all" { "tests.all" } else { "tests.pkg." + package } }}" + just {{ _forward }} build "${suite}" + declare -r extract="${PWD}/results/debug-extract" + rm -rf -- "${extract}" + mkdir -p -- "${extract}" + declare errors candidates + errors="$(mktemp)" + candidates="$(mktemp)" + declare -r errors candidates + trap 'rm -f -- "${errors}" "${candidates}"' EXIT + # Keep nextest's diagnostics: it writes progress to stderr and JSON to + # stdout, and discarding the former turns "cargo is not on PATH" into + # an empty result that looks like "no such test". + declare listing + if ! listing="$(cargo nextest list --archive-file results/"${suite}"/*.tar.zst \ + --workspace-remap "$(pwd)" --extract-to "${extract}" --extract-overwrite \ + --message-format json \ + {{ if target == "" { "" } else { "-E 'test(/" + target + "/)'" } }} \ + 2>"${errors}")"; then + >&2 echo "::error::could not list the tests in ${suite}" + >&2 cat -- "${errors}" + exit 1 + fi + declare -r listing + + # label \t binary \t workdir \t test, so a picker can show the label + # and the caller can read the rest back off the same line. + if [ -z '{{ target }}' ]; then + for b in {{ _debug_binaries }}; do + printf '%s (binary)\t/bin/%s\t/\t\n' "${b}" "${b}" + done >>"${candidates}" + fi + jq -r ' + ."rust-suites" | to_entries[] | .value as $s + | ($s.testcases // {} | to_entries[] + | select(."value"."filter-match"."status" == "matches") | .key) as $t + | "\($s."binary-id") \($t)\t\($s."binary-path")\t\($s.cwd)\t\($t)" + ' <<<"${listing}" >>"${candidates}" + + declare -i found + found="$(wc -l <"${candidates}")" + declare chosen + if [ "${found}" -eq 0 ]; then + >&2 echo "::error::nothing in ${suite} matches '{{ target }}'" + exit 1 + elif [ "${found}" -eq 1 ]; then + chosen="$(cat -- "${candidates}")" + elif [ -t 0 ]; then + # Several matches and someone to ask. Showing only the label keeps + # the store paths out of the list without losing them. + if ! command -v sk >/dev/null 2>&1; then + >&2 echo "::error::sk (skim) is not on PATH; use the dev shell, or name a target exactly" + exit 1 + fi + chosen="$(sk --delimiter '\t' --with-nth 1 \ + --prompt "{{ tool }} > " --height 40% --reverse <"${candidates}")" + if [ -z "${chosen}" ]; then + >&2 echo "debug: nothing picked" + exit 1 + fi + else + if [ -n '{{ target }}' ]; then + >&2 echo "::error::'{{ target }}' matches ${found} in ${suite}; name one exactly, or pick from a terminal:" + else + >&2 echo "::error::not a terminal, so nothing to ask; name one of:" + fi + >&2 cut -f1 -- "${candidates}" + exit 1 + fi + declare -r chosen + + program="$(cut -f2 <<<"${chosen}")" + # nextest gives a test its package root as the working directory, and + # anything reading a relative path depends on that. + workdir="$(cut -f3 <<<"${chosen}")" + declare test_name + test_name="$(cut -f4 <<<"${chosen}")" + declare -r test_name + if [ -n "${test_name}" ]; then + program_args=(--exact "${test_name}" --nocapture) + # Built outside the image, so it still resolves its loader and its + # libraries through the store; the store has to come along. + mounts=(-v /nix/store:/nix/store:ro -v "${extract}":"${extract}":ro -v "${PWD}":"${PWD}":ro) + fi + fi + declare -r program workdir + + case "{{ tool }}" in + lurk) + docker run --rm -i "${mounts[@]}" -w "${workdir}" --entrypoint /bin/lurk \ + "${image}" --json --follow-forks "${program}" "${program_args[@]}" {{ args }} + ;; + gdb) + >&2 echo "gdbserver on {{ gdb_port }}. Connect with:" + >&2 echo " gdb -ex 'target remote 127.0.0.1:{{ gdb_port }}' '${program}'" + # Randomization stays on: docker's default seccomp answers + # personality(ADDR_NO_RANDOMIZE) with EPERM, and gdbserver would + # otherwise open with an alarming warning about a benign failure. + # Loopback, not `0.0.0.0`: gdbserver runs whatever a client asks + # it to, with no authentication, and a bare `-p` would offer that + # to anything that can reach this machine. + docker run --rm -i -p "127.0.0.1:{{ gdb_port }}:{{ gdb_port }}" "${mounts[@]}" -w "${workdir}" \ + --entrypoint /bin/gdbserver "${image}" --no-disable-randomization \ + ":{{ gdb_port }}" "${program}" "${program_args[@]}" {{ args }} + ;; + bugstalker) + # It takes the program from the client's launch request, not from + # here, so connecting an editor to this is only half of it. Print + # the other half ready to paste into .zed/debug.json: `tcp_connection` + # is what stops the editor spawning a `bs` of its own. + >&2 echo "bugstalker DAP on {{ bs_port }}. Give the editor:" + >&2 jq -n --arg l "container: {{ tool }} {{ target }}" --arg p "${program}" \ + --arg w "${workdir}" --argjson port "{{ bs_port }}" \ + '{label: $l, adapter: "bugstalker-dap", request: "launch", + program: $p, args: $ARGS.positional, cwd: $w, + tcp_connection: {host: "127.0.0.1", port: $port}}' \ + --args "${program_args[@]}" {{ args }} + # The container end is 4711 whatever `bs_port` says: the image's + # entrypoint hardcodes `--dap-remote=0.0.0.0:4711`, so publishing + # `bs_port:bs_port` only worked while `bs_port` was 4711. Loopback + # for the same reason as gdbserver -- a DAP `launch` request runs + # an arbitrary program. + docker run --rm -i -p "127.0.0.1:{{ bs_port }}:4711" "${mounts[@]}" -w "${workdir}" "${image}" + ;; + esac + +# Open a core file in a gdb matched to the build that produced it. +# +# Symbols only line up with the exact version and profile that dumped, so pass +# the same build settings that produced the binary. +[script] +inspect-core core *args: (build-container "dataplane-core-viewer") + {{ _just_debuggable_ }} + declare -xr DOCKER_HOST="${DOCKER_HOST:-unix://{{ docker_sock }}}" + if [ ! -r '{{ core }}' ]; then + >&2 echo "inspect-core: cannot read '{{ core }}'" + exit 1 + fi + declare core_dir core_file + core_dir="$(cd "$(dirname -- '{{ core }}')" && pwd)" + core_file="$(basename -- '{{ core }}')" + declare -r core_dir core_file + docker run --rm -it -v "${core_dir}":/cores:ro \ + "{{ oci_image_dataplane_core_viewer }}" "/cores/${core_file}" {{ args }} + +# Check that the debug images actually do what the README says they do. +# +# Building an image proves it links; it does not prove the entrypoint runs. +# Both of these shipped broken: the tracer's documented `docker run` produced a +# well-formed JSON trace of its own child failing to start, and still exited 0. +[script] +smoke-container target: (build-container (if target == "dataplane-core-viewer" { target } else if target == "dataplane-dev-debugger" { target } else if target == "dataplane-syscall-tracer" { target } else { error("smoke-container: no smoke test for '" + target + "'; expected dataplane-core-viewer, dataplane-dev-debugger, or dataplane-syscall-tracer") })) + {{ _just_debuggable_ }} + declare -xr DOCKER_HOST="${DOCKER_HOST:-unix://{{ docker_sock }}}" + case "{{ target }}" in + "dataplane-syscall-tracer") + # A trace runs to megabytes, so keep it in a file rather than a + # variable: `declare -x` would put it in the environment, and every + # child process then fails to exec with E2BIG, which reads exactly + # like a failed trace. Here-strings are fine at this size -- bash + # backs them with a pipe or temp file -- and the checks below use + # them on captured output. + declare trace + trace="$(mktemp)" + declare -r trace + # Name the container and remove it by name. `timeout` signals the + # docker CLI, and the daemon -- not the CLI -- owns the container's + # lifetime, so on the timeout path `--rm` alone can leave a `lurk` + # tracing something for as long as the runner lives. + declare cid + cid="smoke-syscall-tracer-$$" + declare -r cid + trap 'rm -f -- "${trace}"; docker rm -f "${cid}" >/dev/null 2>&1 || true' EXIT + # No seccomp relaxation on purpose: this is the documented command. + timeout 60 docker run --rm --name "${cid}" \ + "{{ oci_image_dataplane_syscall_tracer }}" \ + >"${trace}" 2>&1 || true + if grep -q "Unable to set ADDR_NO_RANDOMIZE" "${trace}"; then + >&2 echo "::error::lurk could not disable ASLR, so the tracee never ran" + exit 1 + fi + # The tracee has to actually execute, not merely be attached to. + if ! grep -q '"syscall":"execve"' "${trace}"; then + >&2 echo "::error::no execve in the trace: the traced program never started" + >&2 head -20 "${trace}" + exit 1 + fi + printf 'syscall-tracer: traced %s syscalls\n' "$(grep -c '"type":"SYSCALL"' "${trace}")" + ;; + "dataplane-dev-debugger") + # Only that it comes up and listens. Driving a DAP session from + # CI means carrying a protocol client for a contract better + # exercised by pointing a real editor at `just debug bugstalker`. + declare cid + # Let docker pick the host port and read it back. A fixed one + # collides: these run on a shared daemon, and `container_profiles` + # can put two of these jobs on the same node at once, where the + # second bind fails and reads as "the listener never came up". + cid="$(docker run -d --rm -p 127.0.0.1::4711 "{{ oci_image_dataplane_dev_debugger }}")" + declare -r cid + trap 'docker kill "${cid}" >/dev/null 2>&1 || true' EXIT + declare -i waited=0 + declare port + port="$(docker port "${cid}" 4711/tcp | head -1)" + port="${port##*:}" + declare -r port + if [ -z "${port}" ]; then + >&2 echo "::error::docker published no host port for 4711/tcp" + exit 1 + fi + until timeout 1 bash -c "/dev/null; do + if [ "${waited}" -ge 60 ]; then + >&2 echo "::error::dev-debugger never listened on 4711" + >&2 docker logs "${cid}" 2>&1 | tail -20 + exit 1 + fi + sleep 1 + waited=$(( waited + 1 )) + done + echo "dev-debugger: DAP listener up" + ;; + "dataplane-core-viewer") + # The Rust pretty-printers are the reason this image exists, and + # what registers them is the entrypoint's own `source` flag -- so + # drive the real entrypoint rather than invoking gdb directly, + # which would only test a copy of it. + declare out + out="$(printf 'info pretty-printer\nquit\n' \ + | timeout 120 docker run --rm -i "{{ oci_image_dataplane_core_viewer }}" 2>&1)" + if grep -qiE "traceback|no module named" <<<"${out}"; then + >&2 echo "::error::gdb could not load the rust pretty-printers" + >&2 printf '%s\n' "${out}" + exit 1 + fi + # A registered printer set, not merely a clean start. + for want in StdString StdVec StdHashMap; do + if ! grep -q "${want}" <<<"${out}"; then + >&2 echo "::error::rust pretty-printer ${want} is not registered" + exit 1 + fi + done + echo "core-viewer: rust pretty-printers registered" + ;; + *) + # Unreachable: the dependency above rejects anything else first. + >&2 echo "::error::no smoke test defined for {{ target }}" + exit 1 + ;; + esac + # Build the dataplane container image [script] build-container target="dataplane" *args: (build (if target == "dataplane" { "dataplane.tar" } else if target == "validator" { "workspace.validator" } else { "containers." + target }) args) @@ -279,10 +614,20 @@ build-container target="dataplane" *args: (build (if target == "dataplane" { "da docker tag "${img}" "{{oci_image_dataplane}}" echo "imported {{ oci_image_dataplane }} (${docker_platform})" ;; - "dataplane-debugger") - docker load < ./results/containers.dataplane-debugger - docker tag "ghcr.io/githedgehog/dataplane/debugger:{{version}}" "{{oci_image_dataplane_debugger}}" - echo "imported {{ oci_image_dataplane_debugger }}" + "dataplane-core-viewer") + docker load < ./results/containers.dataplane-core-viewer + docker tag "ghcr.io/githedgehog/dataplane/core-viewer:{{version}}" "{{oci_image_dataplane_core_viewer}}" + echo "imported {{ oci_image_dataplane_core_viewer }}" + ;; + "dataplane-dev-debugger") + docker load < ./results/containers.dataplane-dev-debugger + docker tag "ghcr.io/githedgehog/dataplane/dev-debugger:{{version}}" "{{oci_image_dataplane_dev_debugger}}" + echo "imported {{ oci_image_dataplane_dev_debugger }}" + ;; + "dataplane-syscall-tracer") + docker load < ./results/containers.dataplane-syscall-tracer + docker tag "ghcr.io/githedgehog/dataplane/syscall-tracer:{{version}}" "{{oci_image_dataplane_syscall_tracer}}" + echo "imported {{ oci_image_dataplane_syscall_tracer }}" ;; "debug-tools") # Uses nix only to produce a base image with the runtime closure (glibc, bash, etc.) @@ -386,8 +731,14 @@ push-container target="dataplane" *args: (build-container target args) && versio "dataplane") push_image "{{ oci_image_dataplane }}" ;; - "dataplane-debugger") - push_image "{{ oci_image_dataplane_debugger }}" + "dataplane-core-viewer") + push_image "{{ oci_image_dataplane_core_viewer }}" + ;; + "dataplane-dev-debugger") + push_image "{{ oci_image_dataplane_dev_debugger }}" + ;; + "dataplane-syscall-tracer") + push_image "{{ oci_image_dataplane_syscall_tracer }}" ;; "debug-tools") >&2 echo "do not push the debug tools!" @@ -421,7 +772,14 @@ push-container target="dataplane" *args: (build-container target args) && versio [script] push: {{ _just_debuggable_ }} - for container in dataplane frr.dataplane validator; do + # Debug images must match the release they inspect. + for container in \ + dataplane \ + dataplane-core-viewer \ + dataplane-dev-debugger \ + dataplane-syscall-tracer \ + frr.dataplane \ + validator; do if [ "${container}" = "validator" ]; then platform="wasm32-wasip1" else @@ -747,9 +1105,49 @@ coverage *args: cargo llvm-cov --no-report --branch nextest {{ args }} mkdir -p "${out}" cargo llvm-cov report --branch --html --output-dir="${out}" + cargo llvm-cov report --branch --lcov --output-path="${out}/lcov.info" cargo llvm-cov report --branch --codecov --output-path="${out}/codecov.json" cargo llvm-cov report --branch --summary-only +# Report specification compliance. See development/code/spec-compliance.md +[script] +duvet *args: + {{ _just_debuggable_ }} + duvet report {{ args }} + +# Fail if the compliance snapshot is out of date +[script] +duvet-check: + {{ _just_debuggable_ }} + # `duvet report` takes milliseconds and is bit-for-bit deterministic, so unlike mutation + # testing this can be a gate, and it is the cheapest correctness check in the repo. It is + # one because the snapshot had already drifted two commits after being introduced: a + # regenerate-by-hand rule is one nobody runs. + duvet report + if ! git diff --quiet -- .duvet/snapshot.txt; then + echo "error: .duvet/snapshot.txt is stale; run \`just duvet\` and commit the result" >&2 + git --no-pager diff -- .duvet/snapshot.txt >&2 + exit 1 + fi + +# Mutation-test a crate or a diff. See development/code/mutation-testing.md +[script] +mutants *args: + {{ _just_debuggable_ }} + # Deliberately not a gate, and deliberately not the whole workspace by default: a full + # sweep is hours, and the product is the list of survivors rather than the score. Scope + # it, as in `just mutants -p dataplane-nat` or `just mutants --in-diff <(git diff main)`. + cargo mutants --test-tool nextest {{ args }} + +# Check that each `type=test` citation tests its `type=implementation` citation +[script] +spec-interlock *args: + {{ _just_debuggable_ }} + # The cross-check duvet cannot do alone: mutate only the cited implementation region, run + # only the cited tests, and report a citation whose test notices nothing as decorative. + # See development/code/spec-compliance.md. + ./scripts/spec-interlock.ts {{ args }} + # Use Nix-built archives so local and CI coverage report the same binaries. [script] coverage-archive package="tests.all" *args: diff --git a/k8s-intf/src/bolero/acl.rs b/k8s-intf/src/bolero/acl.rs new file mode 100644 index 0000000000..e658431925 --- /dev/null +++ b/k8s-intf/src/bolero/acl.rs @@ -0,0 +1,297 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +use std::ops::Bound; + +use bolero::{Driver, ValueGenerator}; + +use crate::gateway_agent_crd::{ + GatewayAgentPeeringsAcl, GatewayAgentPeeringsAclDefault, GatewayAgentPeeringsAclRules, + GatewayAgentPeeringsAclRulesAction, GatewayAgentPeeringsAclRulesMatch, + GatewayAgentPeeringsAclRulesMatchDst, GatewayAgentPeeringsAclRulesMatchSrc, + GatewayAgentPeeringsAclRulesScope, GatewayAgentPeeringsPeering, +}; + +/// The most rules one generated ACL carries. Small, for the usual reasons. +const MAX_RULES: u8 = 3; + +/// What one side of a peering offers, as far as an ACL rule has to care. +/// +/// An ACL rule's `match` is checked against the manifests it sits beside: the **source** prefixes +/// have to intersect the *from* side's native addresses, and the **destination** prefixes the *to* +/// side's advertised ones. A rule whose match intersects neither is refused outright, so a generator +/// that draws prefixes freely produces rules that are thrown away. These are read back off the +/// manifests the peering generator has just built, so a rule can name something that is really +/// there. +#[derive(Debug, Clone)] +pub struct SideFacts { + /// The vpc this side is, for `from` and `to`. + pub vpc: String, + /// The addresses this side exposes natively -- what a rule's `src` is checked against when this + /// is the *from* side. + pub native: Vec, + /// The addresses this side is reachable at from outside -- the translation range where the + /// expose has one, and the native prefix where it does not. A rule's `dst` is checked against + /// this when this is the *to* side. + pub advertised: Vec, + /// Whether any expose restricts ports. + /// + /// Port forwarding attaches port ranges to its prefixes, and coverage is checked over addresses + /// *and* ports together -- so a rule naming ports outside those ranges intersects nothing. Where + /// this is set, rules on this side leave ports alone. + pub restricts_ports: bool, + /// Whether *every* expose on this side uses masquerade or port forwarding. + /// + /// `scope: flow` needs one side of the peering to be entirely stateful, since a flow-scoped rule + /// has nothing to attach to for connections that never reach the flow table. + pub all_stateful: bool, +} + +impl SideFacts { + /// Read the facts off a generated manifest. + #[must_use] + pub fn of(vpc: &str, manifest: &GatewayAgentPeeringsPeering) -> Self { + let exposes = manifest.expose.as_deref().unwrap_or(&[]); + let mut native = Vec::new(); + let mut advertised = Vec::new(); + let mut restricts_ports = false; + let mut all_stateful = !exposes.is_empty(); + + for expose in exposes { + let ips: Vec = expose + .ips + .iter() + .flatten() + .filter_map(|ip| ip.cidr.clone()) + .collect(); + let translations: Vec = expose + .r#as + .iter() + .flatten() + .filter_map(|entry| entry.cidr.clone()) + .collect(); + + native.extend(ips.iter().cloned()); + // reachable from outside at the translation range if there is one, at the native prefix + // otherwise + if translations.is_empty() { + advertised.extend(ips); + } else { + advertised.extend(translations); + } + + let nat = expose.nat.as_ref(); + if nat.is_some_and(|nat| nat.port_forward.is_some()) { + restricts_ports = true; + } + if !nat.is_some_and(|nat| nat.masquerade.is_some() || nat.port_forward.is_some()) { + all_stateful = false; + } + } + + Self { + vpc: vpc.to_string(), + native, + advertised, + restricts_ports, + all_stateful, + } + } +} + +/// Generates peering-scoped ACLs that validation accepts. +/// +/// Valid by construction, like the expose generator beside it. The rules an ACL has to satisfy: +/// +/// * `from` and `to` name the peering's two vpcs, in either order. One may be omitted, since the +/// converter completes it from the other -- which is worth generating, because that completion is +/// itself code. +/// * a rule's source prefixes intersect the *from* side's native addresses, and its destination +/// prefixes the *to* side's advertised ones. An empty list means "everything in this direction", +/// which is always accepted. +/// * source and destination prefixes are of one address family. +/// * only TCP and UDP support port matching. Any other protocol, and any-protocol, must carry no +/// ports at all. +/// * `scope: flow` needs one side of the peering to use masquerade or port forwarding for *all* of +/// its exposes. +#[derive(Debug, Clone)] +pub struct AclGenerator { + left: SideFacts, + right: SideFacts, +} + +impl AclGenerator { + #[must_use] + pub fn new(left: SideFacts, right: SideFacts) -> Self { + Self { left, right } + } + + /// A port range, as the CRD writes them: a single port or a range. + fn ports(d: &mut D) -> Option> { + let count = d.gen_u8(Bound::Included(&1), Bound::Included(&2))?; + let mut out = Vec::with_capacity(usize::from(count)); + for _ in 0..count { + let first = d.gen_u16(Bound::Included(&1), Bound::Included(&65535))?; + if d.produce::()? { + out.push(format!("{first}")); + } else { + let second = d.gen_u16(Bound::Included(&1), Bound::Included(&65535))?; + out.push(format!("{}-{}", first.min(second), first.max(second))); + } + } + Some(out) + } + + /// One prefix of `choices`, or `None` to match everything in the rule's direction. + /// + /// Naming a prefix the manifest really has is what makes the coverage check pass. No ports go + /// with it: coverage compares addresses *and* ports, so a prefix that carries ports of its own in + /// the manifest would be intersected against whatever this named, and a mismatch means the rule + /// matches nothing and is refused. + fn prefix(d: &mut D, choices: &[String]) -> Option { + if choices.is_empty() { + return None; + } + let index = d.gen_usize(Bound::Included(&0), Bound::Excluded(&choices.len()))?; + Some(choices[index].clone()) + } + + fn rule(&self, d: &mut D, index: u8) -> Option { + // either direction across the peering + let (from, to) = if d.produce::()? { + (&self.left, &self.right) + } else { + (&self.right, &self.left) + }; + + // Sometimes leave one end out: the converter fills it in from the other, and that + // completion is code worth running. + let (from_field, to_field) = match d.gen_u8(Bound::Included(&0), Bound::Included(&2))? { + 0 => (Some(from.vpc.clone()), None), + 1 => (None, Some(to.vpc.clone())), + _ => (Some(from.vpc.clone()), Some(to.vpc.clone())), + }; + + // TCP and UDP are the only protocols that may carry ports. + let (proto, may_use_ports) = match d.gen_u8(Bound::Included(&0), Bound::Included(&3))? { + 0 => (None, false), + 1 => (Some("tcp".to_string()), true), + 2 => (Some("udp".to_string()), true), + _ => ( + Some( + d.gen_u8(Bound::Included(&1), Bound::Included(&254))? + .to_string(), + ), + false, + ), + }; + + let src_ports = may_use_ports && !from.restricts_ports && d.produce::()?; + let dst_ports = may_use_ports && !to.restricts_ports && d.produce::()?; + + // A src or dst entry may name a prefix, may name ports, or may be left out entirely -- but an + // entry with neither says nothing, so it is only emitted when it has one or the other. + let src_prefix = if d.produce::()? { + Self::prefix(d, &from.native) + } else { + None + }; + let dst_prefix = if d.produce::()? { + Self::prefix(d, &to.advertised) + } else { + None + }; + + let src = if src_prefix.is_some() || src_ports { + Some(vec![GatewayAgentPeeringsAclRulesMatchSrc { + cidr: src_prefix, + ports: if src_ports { + Self::ports(d)?.into() + } else { + None + }, + vpc_subnet: None, + }]) + } else { + None + }; + let dst = if dst_prefix.is_some() || dst_ports { + Some(vec![GatewayAgentPeeringsAclRulesMatchDst { + cidr: dst_prefix, + ports: if dst_ports { + Self::ports(d)?.into() + } else { + None + }, + vpc_subnet: None, + }]) + } else { + None + }; + + let r#match = if src.is_some() || dst.is_some() || proto.is_some() { + Some(GatewayAgentPeeringsAclRulesMatch { dst, proto, src }) + } else { + None + }; + + // `scope: flow` needs one side of the peering to use masquerade or port forwarding for all + // of its exposes, since a flow-scoped rule has nothing to attach to for connections that + // never reach the flow table. + // + // Note what omitting the field means. The CRD says the scope "can be either 'flow' (default + // if empty) or 'packet'" -- so leaving it out is not "unspecified", it is *flow*, and it is + // refused in exactly the cases an explicit `flow` would be. An earlier version of this drew + // three ways and let the no-flow case fall through to omitting the field, which asked for + // flow by another name. + let flow_allowed = self.left.all_stateful || self.right.all_stateful; + let scope = if flow_allowed && d.produce::()? { + // both spellings of flow: named, and left to the default + if d.produce::()? { + Some(GatewayAgentPeeringsAclRulesScope::Flow) + } else { + None + } + } else { + Some(GatewayAgentPeeringsAclRulesScope::Packet) + }; + + Some(GatewayAgentPeeringsAclRules { + action: if d.produce::()? { + GatewayAgentPeeringsAclRulesAction::Allow + } else { + GatewayAgentPeeringsAclRulesAction::Deny + }, + from: from_field, + log: Some(d.produce::()?), + r#match, + name: Some(format!("rule{index}")), + scope, + to: to_field, + }) + } +} + +impl ValueGenerator for AclGenerator { + type Output = GatewayAgentPeeringsAcl; + + fn generate(&self, d: &mut D) -> Option { + // At least one: an ACL with no rules is refused, since it says nothing that the peering's + // default action does not already say. + let count = d.gen_u8(Bound::Included(&1), Bound::Included(&MAX_RULES))?; + let mut rules = Vec::with_capacity(usize::from(count)); + for index in 0..count { + rules.push(self.rule(d, index)?); + } + + Some(GatewayAgentPeeringsAcl { + default: match d.gen_u8(Bound::Included(&0), Bound::Included(&2))? { + 0 => GatewayAgentPeeringsAclDefault::Deny, + 1 => GatewayAgentPeeringsAclDefault::DenyUnlessExposed, + // the empty string, which the CRD accepts and means the default + _ => GatewayAgentPeeringsAclDefault::KopiumEmpty, + }, + rules: Some(rules).filter(|r| !r.is_empty()), + }) + } +} diff --git a/k8s-intf/src/bolero/crd.rs b/k8s-intf/src/bolero/crd.rs index 92fc069b5a..957017de74 100644 --- a/k8s-intf/src/bolero/crd.rs +++ b/k8s-intf/src/bolero/crd.rs @@ -6,7 +6,8 @@ use std::ops::Bound; use bolero::{Driver, TypeGenerator, ValueGenerator, produce}; use kube::core::ObjectMeta; -use crate::bolero::LegalValue; +use crate::bolero::spec::{GatewayAgentSpecs, SpecBuilder}; +use crate::bolero::{AddressFamily, LegalValue, NatFlavour}; use crate::gateway_agent_crd::{GatewayAgent, GatewayAgentSpec}; const HOSTNAME_BASE: &str = "host-"; @@ -23,21 +24,115 @@ fn simple_hostname(d: &mut D) -> Option { ) } -/// Generate a random legal `GatewayAgent` value +/// Put this gateway into some of its own gateway groups. /// -/// Is not exhaustive due to hostname generation -/// Coverage of values is subject to limitations of the `GatewayAgentSpec` `TypeGenerator` as well -impl TypeGenerator for LegalValue { - fn generate(d: &mut D) -> Option { - Some(LegalValue(GatewayAgent { +/// Without this, **the entire peering half of the routing configuration is never built.** +/// `build_routing_config_peer` -- which produces every import prefix list, advertise prefix list and +/// route-map, and the VRF imports -- runs only for a peering whose gateway group lists *this* +/// gateway, by name. Group members were drawn as arbitrary strings, and an arbitrary string is never +/// the generated hostname, so that condition was false in essentially every configuration ever +/// generated. `internal.rs` sat at 42% region coverage and the reason was this one line. +/// +/// The gateway *replaces* a group's existing member rather than joining it: a group generated here +/// holds at most one member, so replacing cannot collide on a name or an address, both of which +/// validation refuses. +/// +/// Only some groups, drawn: a peering pointed at a group this gateway does not belong to is a real +/// configuration, and it is the one that legitimately renders nothing. +fn join_own_groups(d: &mut D, name: &str, spec: &mut GatewayAgentSpec) -> Option<()> { + for group in spec.groups.iter_mut().flatten().map(|(_, group)| group) { + if !d.produce::()? { + continue; + } + if let Some(member) = group.members.iter_mut().flatten().next() { + member.name = name.to_string(); + } + } + Some(()) +} + +/// Draws `GatewayAgent` custom resources, as configured by a [`GatewayAgentBuilder`]. +/// +/// This is the generator to reach for when a property wants to *aim*: at one NAT flavour, at one +/// address family, at a fabric of a particular size. The `TypeGenerator` impl below is this with +/// default knobs. +#[derive(Debug, Clone, Default)] +pub struct GatewayAgents(GatewayAgentSpecs); + +impl ValueGenerator for GatewayAgents { + type Output = GatewayAgent; + + fn generate(&self, d: &mut D) -> Option { + let name = simple_hostname(d)?; + let generation = d.gen_i64(Bound::Excluded(&0), Bound::Unbounded)?; + let mut spec = self.0.generate(d)?; + join_own_groups(d, &name, &mut spec)?; + Some(GatewayAgent { metadata: ObjectMeta { - name: Some(simple_hostname(d)?), - generation: Some(d.gen_i64(Bound::Excluded(&0), Bound::Unbounded)?), + name: Some(name), + generation: Some(generation), namespace: Some("default".to_string()), ..Default::default() }, - spec: d.produce::>()?.take(), + spec, status: None, // Add when we build a generator and converter for status - })) + }) + } +} + +/// Knobs for [`GatewayAgents`]. +/// +/// A thin wrapper over [`SpecBuilder`], since everything worth steering is in the spec. +#[derive(Debug, Clone, Default)] +pub struct GatewayAgentBuilder(SpecBuilder); + +impl GatewayAgentBuilder { + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Restrict the NAT flavours the exposes may use. + #[must_use] + pub fn flavours(mut self, flavours: Vec) -> Self { + self.0 = self.0.flavours(flavours); + self + } + + /// Restrict the address families the exposes may use. + #[must_use] + pub fn families(mut self, families: Vec) -> Self { + self.0 = self.0.families(families); + self + } + + /// The most vpcs, peerings, exposes per peering, and subnets per vpc. + #[must_use] + pub fn sizes(mut self, vpcs: u8, peerings: u8, exposes: u8, subnets: u8) -> Self { + self.0 = self + .0 + .max_vpcs(vpcs) + .max_peerings(peerings) + .max_exposes(exposes) + .max_subnets(subnets); + self + } + + /// The generator these knobs describe. + #[must_use] + pub fn build(self) -> GatewayAgents { + GatewayAgents(self.0.build()) + } +} + +/// Generate a random legal `GatewayAgent` value +/// +/// Is not exhaustive due to hostname generation +/// Coverage of values is subject to limitations of the `GatewayAgentSpec` `TypeGenerator` as well +/// +/// Delegates to [`GatewayAgents`] with default knobs; use [`GatewayAgentBuilder`] to aim. +impl TypeGenerator for LegalValue { + fn generate(d: &mut D) -> Option { + Some(LegalValue(GatewayAgents::default().generate(d)?)) } } diff --git a/k8s-intf/src/bolero/expose.rs b/k8s-intf/src/bolero/expose.rs index 8750ed9c29..0f3343d8f9 100644 --- a/k8s-intf/src/bolero/expose.rs +++ b/k8s-intf/src/bolero/expose.rs @@ -3,10 +3,10 @@ use std::ops::Bound; -use bolero::{Driver, TypeGenerator, ValueGenerator}; +use bolero::{Driver, ValueGenerator}; -use crate::bolero::support::generate_prefixes; -use crate::bolero::{LegalValue, SubnetMap}; +use crate::bolero::support::blocks; +use crate::bolero::{AddressFamily, NatFlavour, SubnetMap}; use crate::gateway_agent_crd::{ GatewayAgentPeeringsPeeringExpose, GatewayAgentPeeringsPeeringExposeAs, GatewayAgentPeeringsPeeringExposeIps, GatewayAgentPeeringsPeeringExposeNat, @@ -17,175 +17,357 @@ use crate::gateway_agent_crd::{ GatewayAgentPeeringsPeeringExposeNatStatic, }; -/// Generate a legal value for `GatewayAgentPeeringsPeeringExpose` +/// The most prefixes either side of an expose carries, where the flavour allows more than one. /// -/// This is not exhaustive over all legal values due to the complexity of doing this. For example, -/// the CIDR generators are not exhaustive; and we use a single port range for all CIDRs rather than -/// trying different combinations. -pub struct LegalValueExposeGenerator<'a> { +/// Small on purpose. Nothing here needs a long list to be interesting, and a long one costs +/// throughput and makes a counterexample unreadable. +const MAX_PREFIXES: u8 = 3; + +/// The widest port range port forwarding maps. Same reasoning. +const MAX_PORTS: u16 = 1024; + +/// Generates exposes that [`crate::gateway_agent_crd::GatewayAgent`] validation accepts. +/// +/// **Valid by construction rather than by generate-and-reject**, so every case reaches the code +/// under test. That matters more than it sounds: the generator this replaced drew the prefixes first +/// and chose a NAT flavour afterwards, and since each flavour constrains the shape, essentially no +/// expose it produced could be accepted. Peerings were generated in their tens of thousands and none +/// ever survived validation, so nothing downstream of it -- the NAT tables, the ACLs, the internal +/// config builder -- had ever seen one. +/// +/// The rules each flavour has to satisfy, all of which `VpcExpose::validate` enforces: +/// +/// * the private list is non-empty, and non-empty again once exclusions are applied; +/// * every prefix in the expose is of one address family, since NAT46 and NAT64 are unsupported; +/// * no prefix overlaps a special-use range, hence [`blocks`]; +/// * a flavour that translates has a non-empty translation range; +/// * **static NAT**: the two sides cover the same number of address-port pairs, which one prefix of +/// equal length on each side satisfies; +/// * **port forwarding**: exactly one prefix per side of equal length, a port range on each, the two +/// ranges of equal size, and no exclusion prefixes at all. +#[derive(Debug, Clone)] +pub struct ExposeGenerator<'a> { + flavour: NatFlavour, + family: AddressFamily, + which: Which, subnets: &'a SubnetMap, } -impl<'a> LegalValueExposeGenerator<'a> { +/// Which expose this is, at the two levels that matter for keeping prefixes apart. +/// +/// `slot` is the block slot the prefixes come from, and it is unique across the *whole* +/// configuration -- see [`blocks::expose_slot`], which explains why the vpc has to be part of it. +/// Every prefix this generator writes out sits inside that slot, so no two exposes anywhere can +/// overlap. +/// +/// `index` and `count` are this expose's place within its own manifest, which the slot does not +/// capture. They are needed because a *named* subnet contributes its prefix just as surely as a +/// written-out one does, and the subnets live in slots of their own: two exposes of one manifest +/// naming the same subnet share a prefix. So the manifest's subnets are dealt out round-robin, +/// `index` taking every `count`th one. +#[derive(Debug, Clone, Copy)] +pub struct Which { + pub slot: u8, + pub index: u8, + pub count: u8, +} + +impl Which { + /// A lone expose, for a caller generating one at a time. + #[must_use] + pub fn only(slot: u8) -> Self { + Self { + slot, + index: 0, + count: 1, + } + } + + /// Expose `index` of `count` in a manifest, drawing from `slot`. #[must_use] - pub fn new(subnets: &'a SubnetMap) -> Self { - Self { subnets } + pub fn nth(slot: u8, index: u8, count: u8) -> Self { + Self { + slot, + index, + count: count.max(index.saturating_add(1)), + } } } -impl ValueGenerator for LegalValueExposeGenerator<'_> { - type Output = GatewayAgentPeeringsPeeringExpose; +impl<'a> ExposeGenerator<'a> { + #[must_use] + pub fn new( + flavour: NatFlavour, + family: AddressFamily, + which: Which, + subnets: &'a SubnetMap, + ) -> Self { + Self { + flavour, + family, + which, + subnets, + } + } - fn generate(&self, d: &mut D) -> Option { - let num_ips = d.gen_u16(Bound::Included(&1), Bound::Included(&16))?; - let num_nots = d.gen_u16(Bound::Included(&0), Bound::Included(&16))?; - let num_subnets = std::cmp::max( - self.subnets.len(), - d.gen_usize(Bound::Included(&0), Bound::Included(&16))?, - ); - - let num_as = d.gen_u16(Bound::Included(&0), Bound::Included(&16))?; - let num_as_not = d.gen_u16(Bound::Included(&0), Bound::Included(&16))?; - - let num_v4_ips = d.gen_u16(Bound::Included(&0), Bound::Included(&num_ips))?; - let num_v6_ips = num_ips - num_v4_ips; - let num_v4_nots = d.gen_u16(Bound::Included(&0), Bound::Included(&num_nots))?; - let num_v6_nots = num_nots - num_v4_nots; - - let num_v4_as = d.gen_u16(Bound::Included(&0), Bound::Included(&num_as))?; - let num_v6_as = num_as - num_v4_as; - let num_v4_not_as = d.gen_u16(Bound::Included(&0), Bound::Included(&num_as_not))?; - let num_v6_not_as = num_as_not - num_v4_not_as; - - let ips = generate_prefixes(d, num_v4_ips, num_v6_ips)? - .into_iter() - .map(|p| GatewayAgentPeeringsPeeringExposeIps { - cidr: Some(p), - not: None, - vpc_subnet: None, - }) - .collect::>(); - let nots = generate_prefixes(d, num_v4_nots, num_v6_nots)? - .into_iter() - .map(|p| GatewayAgentPeeringsPeeringExposeIps { - cidr: None, - not: Some(p), - vpc_subnet: None, + /// A prefix length that fits inside `at`. + /// + /// The floor is the sub-slot's own length, not the block's: an expose splitting its slot between + /// several prefixes has less room for each, so it has to draw longer ones. + fn length(&self, d: &mut D, at: blocks::At) -> Option { + d.gen_u8( + Bound::Included(&blocks::min_len_at(self.family, at)), + Bound::Included(&blocks::max_len(self.family)), + ) + } + + /// The names of this vpc's subnets that are of the expose's family, and that are this expose's + /// to name. + /// + /// Two filters. Family, because a named subnet contributes its own prefix, so naming one of the + /// other family makes the expose mixed just as surely as writing the prefix out would. And the + /// round-robin share, because a subnet named by two exposes of one manifest is a prefix those two + /// exposes have in common, which is the overlap the whole slot scheme exists to prevent. + fn matching_subnets(&self) -> Vec<&'a String> { + self.subnets + .iter() + .filter(|(_, prefix)| prefix.is_ipv4() == self.family.is_v4()) + .map(|(name, _)| name) + .enumerate() + .filter(|(index, _)| { + index % usize::from(self.which.count) == usize::from(self.which.index) }) - .collect::>(); - let r#as = generate_prefixes(d, num_v4_as, num_v6_as)? - .into_iter() - .map(|p| GatewayAgentPeeringsPeeringExposeAs { - cidr: Some(p), - not: None, - }); - let not_as = generate_prefixes(d, num_v4_not_as, num_v6_not_as)? - .into_iter() - .map(|p| GatewayAgentPeeringsPeeringExposeAs { - cidr: None, - not: Some(p), - }); + .map(|(_, name)| name) + .collect() + } - let mut subnets = Vec::new(); - let mut subnet_iter = self.subnets.iter(); - for _ in 0..num_subnets { - let Some((name, _)) = subnet_iter.next() else { - break; - }; - subnets.push(GatewayAgentPeeringsPeeringExposeIps { - cidr: None, - not: None, - vpc_subnet: Some(name.clone()), - }); + /// An exclusion strictly inside `parent`, so it cannot remove all of it. + /// + /// Excluding a prefix from itself leaves nothing, and an expose whose private list is empty + /// after exclusions is refused. A longer prefix inside the parent always leaves something. + fn exclusion( + &self, + d: &mut D, + parent: &str, + at: blocks::At, + private: bool, + ) -> Option { + let (_, len) = parent.split_once('/')?; + let len: u8 = len.parse().ok()?; + let max = blocks::max_len(self.family); + if len >= max { + return None; } + let longer = d.gen_u8(Bound::Excluded(&len), Bound::Included(&max))?; + // The parent's own sub-slot, so an exclusion can only ever shrink the prefix it belongs + // to and never eats into a sibling's. + if private { + blocks::private(d, self.family, at, longer) + } else { + blocks::public(d, self.family, at, longer) + } + } - let mut final_ips = Vec::with_capacity(ips.len() + nots.len() + subnets.len()); - final_ips.extend(ips); - final_ips.extend(nots); - final_ips.extend(subnets); - - let mut final_as = Vec::with_capacity(r#as.len() + not_as.len()); - final_as.extend(r#as); - final_as.extend(not_as); - let has_as = !final_as.is_empty(); - - Some(GatewayAgentPeeringsPeeringExpose { - r#as: Some(final_as).filter(|f| !f.is_empty()), - ips: Some(final_ips).filter(|f| !f.is_empty()), - default: None, - nat: if has_as { - Some( - d.produce::>()? - .take(), - ) - } else { - None - }, - }) + /// A port range, and a second of the same size, for port forwarding. + fn port_pair(d: &mut D) -> Option<(String, String)> { + let size = d.gen_u16(Bound::Included(&1), Bound::Included(&MAX_PORTS))?; + // port 0 is forbidden on either side, so both starts are at least one + let first_start = d.gen_u16(Bound::Included(&1), Bound::Included(&(65535 - size + 1)))?; + let second_start = d.gen_u16(Bound::Included(&1), Bound::Included(&(65535 - size + 1)))?; + // `start + (size - 1)`, not `start + size - 1`: the latter groups as `(start + size) - 1` + // and overflows `u16` for a range that ends exactly at 65535. + Some(( + format!("{first_start}-{}", first_start + (size - 1)), + format!("{second_start}-{}", second_start + (size - 1)), + )) } -} -// This is not exhaustive as it does not generate all possible time -// strings, just 0 to 2*3600 seconds. -// -impl TypeGenerator for LegalValue { - fn generate(d: &mut D) -> Option { - let nat_mode = d.produce::()? % 3; - let idle_timeout_secs = d.gen_u64(Bound::Included(&0), Bound::Included(&(2 * 3600)))?; - let idle_timeout = std::time::Duration::from_secs(idle_timeout_secs); - match nat_mode { - 0 => Some(LegalValue(GatewayAgentPeeringsPeeringExposeNat { + /// The NAT block for a flavour that translates. + /// + /// Only called when [`NatFlavour::needs_translation`] holds, so the `None` arm cannot be reached; + /// returning the driver's `None` there rather than panicking costs nothing and keeps the return + /// type a single `Option`. + fn translation(&self, d: &mut D) -> Option { + let idle_secs = d.gen_u64(Bound::Included(&0), Bound::Included(&(2 * 3600)))?; + let idle = std::time::Duration::from_secs(idle_secs); + Some(match self.flavour { + NatFlavour::None => return None, + NatFlavour::Masquerade => GatewayAgentPeeringsPeeringExposeNat { masquerade: Some(GatewayAgentPeeringsPeeringExposeNatMasquerade { - idle_timeout: Some(idle_timeout.into()), + idle_timeout: Some(idle.into()), }), port_forward: None, r#static: None, - })), - 1 => Some(LegalValue(GatewayAgentPeeringsPeeringExposeNat { + }, + NatFlavour::Static => GatewayAgentPeeringsPeeringExposeNat { masquerade: None, port_forward: None, r#static: Some(GatewayAgentPeeringsPeeringExposeNatStatic {}), - })), - 2 => { - // Generate a valid port range - let bound1 = d.gen_u16(Bound::Included(&1), Bound::Included(&65535))?; - let bound2 = d.gen_u16(Bound::Included(&1), Bound::Included(&65535))?; - let start = bound1.min(bound2); - let end = bound1.max(bound2); - let port_range = format!("{start}-{end}"); - - // Generate another valid port range of the same size - let port_range_size = (end - start) as usize + 1; - let max_new_start = u16::try_from(65536 - port_range_size).unwrap(); - let new_bound = d.gen_u16(Bound::Included(&0), Bound::Included(&max_new_start))?; - let new_port_range = format!( - "{new_bound}-{}", - new_bound + u16::try_from(port_range_size - 1).unwrap() - ); - - Some(LegalValue(GatewayAgentPeeringsPeeringExposeNat { + }, + NatFlavour::PortForward => { + let (port, r#as) = Self::port_pair(d)?; + GatewayAgentPeeringsPeeringExposeNat { masquerade: None, port_forward: Some(GatewayAgentPeeringsPeeringExposeNatPortForward { - idle_timeout: Some(idle_timeout.into()), + idle_timeout: Some(idle.into()), ports: Some(vec![GatewayAgentPeeringsPeeringExposeNatPortForwardPorts { - r#as: Some(new_port_range), - port: Some(port_range), - proto: match d.produce::()? % 3 { + r#as: Some(r#as), + port: Some(port), + proto: match d.gen_u8(Bound::Included(&0), Bound::Included(&2))? { 0 => Some( GatewayAgentPeeringsPeeringExposeNatPortForwardPortsProto::Tcp, ), 1 => Some( GatewayAgentPeeringsPeeringExposeNatPortForwardPortsProto::Udp, ), - 2 => None, - _ => unreachable!(), + _ => None, }, }]), }), r#static: None, - })) + } + } + }) + } +} + +impl ValueGenerator for ExposeGenerator<'_> { + type Output = GatewayAgentPeeringsPeeringExpose; + + fn generate(&self, d: &mut D) -> Option { + // Static NAT and port forwarding both need one prefix per side of equal length. The other + // flavours may carry several, and may name vpc subnets alongside them. + let paired = matches!(self.flavour, NatFlavour::Static | NatFlavour::PortForward); + + let mut ips = Vec::new(); + let mut translations = Vec::new(); + + if paired { + let at = blocks::At::whole(self.which.slot); + let len = self.length(d, at)?; + let private = blocks::private(d, self.family, at, len)?; + let public = blocks::public(d, self.family, at, len)?; + ips.push(GatewayAgentPeeringsPeeringExposeIps { + cidr: Some(private), + not: None, + vpc_subnet: None, + }); + translations.push(GatewayAgentPeeringsPeeringExposeAs { + cidr: Some(public), + not: None, + }); + } else { + // The count first, then a sub-slot per prefix: an expose's own prefixes have to be + // disjoint from each other, not just from the other exposes'. + let count = d.gen_u8(Bound::Included(&1), Bound::Included(&MAX_PREFIXES))?; + for sub in 0..count { + let at = blocks::At::nth(self.which.slot, sub, count); + let len = self.length(d, at)?; + ips.push(GatewayAgentPeeringsPeeringExposeIps { + cidr: Some(blocks::private(d, self.family, at, len)?), + not: None, + vpc_subnet: None, + }); + } + if self.flavour.needs_translation() { + let count = d.gen_u8(Bound::Included(&1), Bound::Included(&MAX_PREFIXES))?; + for sub in 0..count { + let at = blocks::At::nth(self.which.slot, sub, count); + let len = self.length(d, at)?; + translations.push(GatewayAgentPeeringsPeeringExposeAs { + cidr: Some(blocks::public(d, self.family, at, len)?), + not: None, + }); + } + } + + // Naming some of the vpc's own subnets is the ordinary way to write an expose, so draw a + // prefix of them. Only those of this expose's family. + let named = self.matching_subnets(); + if !named.is_empty() { + let take = d.gen_usize(Bound::Included(&0), Bound::Included(&named.len()))?; + for name in named.into_iter().take(take) { + ips.push(GatewayAgentPeeringsPeeringExposeIps { + cidr: None, + not: None, + vpc_subnet: Some(name.clone()), + }); + } + } + } + + // Exclusions, where the flavour allows them. Each sits strictly inside a prefix already in + // the list, so it can never remove all of it. + if self.flavour.allows_exclusions() && d.produce::()? { + let parents: Vec = ips.iter().filter_map(|e| e.cidr.clone()).collect(); + let first = + blocks::At::nth(self.which.slot, 0, u8::try_from(parents.len()).unwrap_or(1)); + if let Some(parent) = parents.first() + && let Some(exclusion) = self.exclusion(d, parent, first, true) + { + ips.push(GatewayAgentPeeringsPeeringExposeIps { + cidr: None, + not: Some(exclusion), + vpc_subnet: None, + }); + } + let parents: Vec = translations.iter().filter_map(|e| e.cidr.clone()).collect(); + let first = + blocks::At::nth(self.which.slot, 0, u8::try_from(parents.len()).unwrap_or(1)); + if let Some(parent) = parents.first() + && let Some(exclusion) = self.exclusion(d, parent, first, false) + { + translations.push(GatewayAgentPeeringsPeeringExposeAs { + cidr: None, + not: Some(exclusion), + }); } - _ => unreachable!(), } + + Some(GatewayAgentPeeringsPeeringExpose { + r#as: Some(translations).filter(|t| !t.is_empty()), + ips: Some(ips).filter(|i| !i.is_empty()), + default: None, + nat: if self.flavour.needs_translation() { + Some(self.translation(d)?) + } else { + None + }, + }) + } +} + +/// Draws exposes of any flavour and family, for a caller that does not want to choose either. +/// +/// [`ExposeGenerator`] takes both because they have to be settled before the prefixes are drawn, and +/// because a property aiming at one flavour needs to say which. A caller that just wants "any legal +/// expose" -- a converter test, say -- can use this instead. +#[derive(Debug, Clone)] +pub struct AnyExposeGenerator<'a> { + which: Which, + subnets: &'a SubnetMap, +} + +impl<'a> AnyExposeGenerator<'a> { + #[must_use] + pub fn new(slot: u8, subnets: &'a SubnetMap) -> Self { + Self { + which: Which::only(slot), + subnets, + } + } +} + +impl ValueGenerator for AnyExposeGenerator<'_> { + type Output = GatewayAgentPeeringsPeeringExpose; + + fn generate(&self, d: &mut D) -> Option { + let flavours = NatFlavour::all(); + let families = AddressFamily::all(); + let flavour = + flavours[d.gen_usize(Bound::Included(&0), Bound::Excluded(&flavours.len()))?]; + let family = + families[d.gen_usize(Bound::Included(&0), Bound::Excluded(&families.len()))?]; + ExposeGenerator::new(flavour, family, self.which, self.subnets).generate(d) } } diff --git a/k8s-intf/src/bolero/mod.rs b/k8s-intf/src/bolero/mod.rs index 078ef00a18..87525f548c 100644 --- a/k8s-intf/src/bolero/mod.rs +++ b/k8s-intf/src/bolero/mod.rs @@ -1,6 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // Copyright Open Network Fabric Authors +pub mod acl; pub mod bgp; pub mod crd; pub mod expose; @@ -8,7 +9,10 @@ pub mod gateway; pub mod gwgroups; pub mod interface; pub mod logs; +pub mod mutate; pub mod peering; +pub mod permute; +pub mod reduce; pub mod spec; pub mod support; pub mod vpc; @@ -17,6 +21,86 @@ use std::collections::BTreeMap; use lpm::prefix::Prefix; +/// Which flavour of NAT an expose uses, if any. +/// +/// The flavour has to be chosen *before* the expose's prefixes are drawn, because each one imposes +/// its own rules on the shape and none of them can be satisfied afterwards. Static NAT needs both +/// sides to cover the same number of address-port pairs; port forwarding needs exactly one prefix +/// per side of equal length, matched port ranges and no exclusion prefixes at all; masquerade needs +/// a non-empty translation range and nothing else. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub enum NatFlavour { + /// No translation: the expose offers its prefixes as they are. + None, + /// Many private addresses behind fewer public ones. + Masquerade, + /// One private address per public address. + Static, + /// One prefix and port range mapped onto another, positionally. + PortForward, +} + +impl NatFlavour { + /// Every flavour, which is what a generator draws from unless told otherwise. + #[must_use] + pub fn all() -> Vec { + vec![ + Self::None, + Self::Masquerade, + Self::Static, + Self::PortForward, + ] + } + + /// Whether this flavour permits exclusion prefixes. + /// + /// Port forwarding does not, and static NAT's matched-size rule is easiest to satisfy without + /// them -- an exclusion has to be mirrored on both sides to keep the sizes equal. + #[must_use] + pub fn allows_exclusions(self) -> bool { + matches!(self, Self::None | Self::Masquerade) + } + + /// Whether this flavour needs a translation range at all. + #[must_use] + pub fn needs_translation(self) -> bool { + !matches!(self, Self::None) + } + + /// Whether this flavour keeps per-flow state. + /// + /// The two manifests of a peering may not *both* use a stateful flavour: masquerade opposite + /// masquerade, masquerade opposite port forwarding, and port forwarding opposite port forwarding + /// are all refused. No NAT and static NAT are compatible with anything. + #[must_use] + pub fn is_stateful(self) -> bool { + matches!(self, Self::Masquerade | Self::PortForward) + } +} + +/// Which address family an expose is built in. +/// +/// One family per expose, always: an expose mixing the two is refused, since NAT46 and NAT64 are +/// not supported. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub enum AddressFamily { + V4, + V6, +} + +impl AddressFamily { + /// Both families. + #[must_use] + pub fn all() -> Vec { + vec![Self::V4, Self::V6] + } + + #[must_use] + pub fn is_v4(self) -> bool { + matches!(self, Self::V4) + } +} + /// A type on which implement `bolero::TypeGenerator` for legal values of `T` /// /// Generally, `bolero` type generators should generate all possible values of `T` so that it is possible to test validation logic, etc. @@ -70,9 +154,9 @@ where // This is distinct from the SubnetMap in config/converters/k8s // since this type is only for the test library. It should be // compatible with the SubnetMap in config/converters/k8s -type SubnetMap = BTreeMap; +pub(crate) type SubnetMap = BTreeMap; // This is distinct from the VpcSubnetMap in config/converters/k8s // since this type is only for the test library. It should be // compatible with the SubnetMap in config/converters/k8s -type VpcSubnetMap = BTreeMap; +pub(crate) type VpcSubnetMap = BTreeMap; diff --git a/k8s-intf/src/bolero/mutate.rs b/k8s-intf/src/bolero/mutate.rs new file mode 100644 index 0000000000..1c18cb40b7 --- /dev/null +++ b/k8s-intf/src/bolero/mutate.rs @@ -0,0 +1,687 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +//! Configurations that are syntactically valid and semantically doubtful. +//! +//! Everything else in this module generates configurations that are legal *by construction*, which +//! exercises everything downstream of validation and nothing of validation itself. This generates +//! the other kind: a legal configuration with **one** rule deliberately broken, so the result lands +//! just outside the legal space rather than far outside it. +//! +//! Near-miss rather than arbitrary on purpose. A configuration wrong in one way is far more likely +//! to slip past a validator than one wrong in twenty, and slipping past is the failure that matters: +//! the validator ships as a wasm module in a separate process, which blesses a configuration and +//! writes it to Kubernetes. The dataplane runs the same validator later but has no way to report a +//! problem to anyone -- by then it is too late. So **anything the validator accepts has to be +//! enactable**, and a validator that is merely too strict is a nuisance while one that is too +//! permissive is unrecoverable. + +use std::ops::Bound; + +use bolero::{Driver, ValueGenerator}; + +use crate::bolero::crd::GatewayAgents; +use crate::gateway_agent_crd::{ + GatewayAgent, GatewayAgentPeeringsAclRulesScope, GatewayAgentPeeringsPeering, + GatewayAgentPeeringsPeeringExpose, GatewayAgentPeeringsPeeringExposeAs, + GatewayAgentPeeringsPeeringExposeIps, GatewayAgentPeeringsPeeringExposeNatMasquerade, +}; + +/// One way of breaking an otherwise-legal configuration. +/// +/// Each names a rule the validator is supposed to enforce. Where a rule is enforced only further +/// downstream -- by a table builder during apply, say -- the mutation that breaks it will be +/// *accepted*, and that is exactly the gap worth finding. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Mutation { + /// Leave it alone. A control: the properties should hold on an unmutated configuration too. + None, + /// Give the two sides of a port-forwarding expose prefixes of different length. + /// + /// The shape of a real bug: sides with equal address-port *totals* and unequal lengths pass a + /// product comparison and cannot be expressed as a port-forwarding rule. + MismatchPortForwardPrefixes, + /// Give the two sides of a static-NAT expose prefixes of different length. + MismatchStaticNatPrefixes, + /// Add an exclusion prefix to an expose that uses port forwarding, which forbids them. + ExcludeFromPortForwarding, + /// Put a prefix of the other address family into an expose. + MixAddressFamilies, + /// Replace a prefix with one that overlaps a special-use range. + UseReservedPrefix, + /// Empty an expose's private prefix list. + EmptyPrivatePrefixes, + /// Remove the translation range from an expose that translates. + DropTranslationRange, + /// Make both manifests of a peering use masquerade, which may not both be stateful. + MakeBothSidesStateful, + /// Point the peering at a gateway group that does not exist. + NameAMissingGroup, + /// Point an ACL rule's `from` at a vpc that is not in the peering. + NameAStrangerInARule, + /// Ask for flow-scoped ACL rules, which need a peering that is stateful throughout. + DemandFlowScope, + /// Put port 0 in a port range, which is not a port. + UsePortZero, + /// Duplicate one static-NAT expose over a sibling in the same manifest. + /// + /// The two exposes then claim the same private prefix, which `validate_expose_collisions` refuses + /// for every pair of NAT modes except masquerade-with-port-forwarding. + /// + /// The reason to aim at static NAT specifically is what happens downstream when the validator does + /// not catch it. `NatRuleTable::insert` takes no `Result` and checks nothing: two entries for one + /// prefix and the second **silently replaces** the first. Compare the port-forwarding table, whose + /// `RangeSet::insert_range` says "overlap is forbidden" and returns an error, and whose lookup is + /// a longest-prefix match for distinct prefixes -- structurally incapable of this. So static NAT + /// is where an unrefused duplicate becomes a table that quietly holds one of the two rules the + /// configuration asked for. + /// + /// The recipient is a copy of the donor whose translation range is moved to the prefix next door, + /// and both halves of that matter. + /// + /// A *plain* copy is not enough: the two exposes would then be identical, the table entry would be + /// overwritten with the same value, and there would be nothing to pick between. Run against a + /// validator with the overlap check removed, the ambiguity property saw no difference across 43,269 + /// comparisons -- and was right not to. Ambiguity needs one prefix with **two different** + /// translations. + /// + /// Moving the range to a *sibling* prefix keeps every length equal, so static NAT's equal-totals + /// rule still holds and overlap stays the only rule broken; keeps it disjoint from the donor's, so + /// the public-prefix rule holds too; and needs no agreement between the two exposes' prefix + /// lengths, which an earlier version required and which made it apply to one draw in 250. + DuplicateAStaticExpose, + /// Make two peers of one vpc expose the same prefix to it. + /// + /// `VpcRouteTable` is built per vpc from what its *peers* expose to it, so this leaves that vpc + /// with one destination and two places to send it. The rule the generator's slot scheme exists to + /// satisfy, deliberately broken: it was reached for a long time only because the generator broke + /// it by accident, and once that was fixed nothing tested it. + /// + /// Sometimes legal, and correctly so -- `VpcRoute::can_overlap` permits overlap between + /// masqueraded or default routes, and then only within one gateway group. + /// + /// **Note what a break test on this shows.** Delete the `OverlappingPrefixes` check from + /// `VpcRouteTable::validate` and the "whatever validates, builds" property still passes: every + /// configuration the weakened validator accepts still builds and enacts. So this rule is not + /// about *feasibility* -- the dataplane will cheerfully install two routes to one destination and + /// pick one -- it is about *ambiguity*, which only the validator is in a position to refuse. That + /// makes it a rule the near-miss property structurally cannot police, and the gap is in the + /// property, not the validator. See the note in `validator_completeness`. + OverlapWithAnotherPeer, +} + +impl Mutation { + /// How many mutations there are, so a harness can keep a counter per mutation without a lock. + pub const COUNT: usize = 15; + + /// This mutation's position in [`Mutation::all`]. + #[must_use] + pub fn index(self) -> usize { + Self::all() + .iter() + .position(|other| *other == self) + .unwrap_or_else(|| unreachable!()) + } + + /// Every mutation, the control included. + #[must_use] + pub fn all() -> Vec { + vec![ + Self::None, + Self::MismatchPortForwardPrefixes, + Self::MismatchStaticNatPrefixes, + Self::ExcludeFromPortForwarding, + Self::MixAddressFamilies, + Self::UseReservedPrefix, + Self::EmptyPrivatePrefixes, + Self::DropTranslationRange, + Self::MakeBothSidesStateful, + Self::NameAMissingGroup, + Self::NameAStrangerInARule, + Self::DemandFlowScope, + Self::UsePortZero, + Self::DuplicateAStaticExpose, + Self::OverlapWithAnotherPeer, + ] + } +} + +/// Lengthen a CIDR's mask by `by`, staying within the family's limit. +fn lengthen(cidr: &str, by: u8) -> Option { + let (address, len) = cidr.split_once('/')?; + let len: u8 = len.parse().ok()?; + let max = if address.contains(':') { 128 } else { 32 }; + let longer = len.saturating_add(by).min(max); + if longer == len { + return None; + } + Some(format!("{address}/{longer}")) +} + +/// The prefix next door: same length, differing only in its last network bit. +/// +/// A prefix and its sibling are disjoint by construction, and the sibling sits inside the same parent, +/// so it stays in whatever block and slot the original came from and cannot stray into a reserved +/// range or another expose's territory. +fn sibling(cidr: &str) -> Option { + let (address, len) = cidr.split_once('/')?; + let len: u8 = len.parse().ok()?; + if address.contains(':') { + let bits = address.parse::().ok()?.to_bits(); + let flipped = bits ^ (1u128 << (128u8.checked_sub(len)?)); + Some(format!("{}/{len}", std::net::Ipv6Addr::from(flipped))) + } else { + let bits = address.parse::().ok()?.to_bits(); + let flipped = bits ^ (1u32 << (32u8.checked_sub(len)?)); + Some(format!("{}/{len}", std::net::Ipv4Addr::from(flipped))) + } +} + +/// Every expose of every manifest of every peering, in a stable order. +fn exposes_mut(agent: &mut GatewayAgent) -> Vec<&mut GatewayAgentPeeringsPeeringExpose> { + agent + .spec + .peerings + .iter_mut() + .flatten() + .flat_map(|(_, peerings)| peerings.peering.iter_mut().flatten()) + .flat_map(|(_, manifest)| manifest.expose.iter_mut().flatten()) + .collect() +} + +fn is_port_forwarding(expose: &GatewayAgentPeeringsPeeringExpose) -> bool { + expose + .nat + .as_ref() + .is_some_and(|nat| nat.port_forward.is_some()) +} + +fn is_static(expose: &GatewayAgentPeeringsPeeringExpose) -> bool { + expose + .nat + .as_ref() + .is_some_and(|nat| nat.r#static.is_some()) +} + +/// Apply `mutation` to `agent`, reporting whether it found anything to change. +/// +/// A mutation that finds no target leaves the configuration legal, which is harmless -- the +/// properties are all conditional on what the validator says -- but it does mean the case tested +/// nothing new, so the caller counts them. +#[allow(clippy::too_many_lines)] +/// Whether every expose of a manifest translates statefully. +/// +/// `scope: flow` needs one side of a peering to be stateful throughout, since a flow-scoped rule has +/// nothing to attach to for connections that never reach the flow table. Mirrors +/// `Acl::validate_scope`, and the two being separate statements of the same rule is the point: a +/// mutation that asserts the validator refuses something has to know when it is entitled to. +fn stateful_throughout(manifest: &GatewayAgentPeeringsPeering) -> bool { + let exposes = manifest.expose.as_deref().unwrap_or(&[]); + !exposes.is_empty() + && exposes.iter().all(|expose| { + expose + .nat + .as_ref() + .is_some_and(|nat| nat.masquerade.is_some() || nat.port_forward.is_some()) + }) +} + +/// Whether an expose advertises exactly its `ips`, with nothing subtracted and nothing translated. +/// +/// Route destinations come from `VpcExpose::public_ips`, which is the **translation range** for any +/// expose that translates and the private prefixes only for one that does not. So a prefix copied out +/// of a static-NAT expose's `ips` never becomes a route at all, and copying one is not the overlap it +/// looks like -- the assertion that whatever a mutation touches must be refused caught exactly that +/// mistake, on a static expose, within seconds. +/// +/// Exclusions are ruled out for the same reason: `public_ips` is the prefixes *minus* the `not`s, so +/// an exclusion could carve away the very prefix being copied. +/// +/// Masquerade would be wrong twice over: it translates, and `VpcRoute::can_overlap` lets two +/// masqueraded routes share a destination legitimately. +fn advertises_its_ips(expose: &GatewayAgentPeeringsPeeringExpose) -> bool { + expose.nat.is_none() + && expose.default != Some(true) + && expose.ips.iter().flatten().all(|ip| ip.not.is_none()) +} + +/// Make two peers of one vpc advertise the same destination to it. +/// +/// Split out of [`apply`] only for length. Two passes, so nothing has to be borrowed mutably while +/// the search is still reading -- a single pass with two live borrows would need `leak()` or a clone +/// of the whole spec, and this runs millions of times. +fn overlap_with_another_peer(agent: &mut GatewayAgent) -> bool { + // Two passes, so nothing has to be borrowed mutably while the search is still reading. + // A single pass with two live borrows would need `leak()` or a clone of the whole spec, + // and this runs millions of times. + // + // First: find a vpc that appears in two peerings, and a prefix one of its peers exposes. + let mut donor: Option<(String, String, String)> = None; + for (key, peering) in agent.spec.peerings.iter().flatten() { + let Some(manifests) = peering.peering.as_ref() else { + continue; + }; + for shared in manifests.keys() { + let elsewhere = agent + .spec + .peerings + .iter() + .flatten() + .any(|(other, peering)| { + other != key + && peering + .peering + .as_ref() + .is_some_and(|m| m.contains_key(shared)) + }); + if !elsewhere { + continue; + } + // Only from an expose that advertises its `ips` verbatim, so the prefix taken + // really is one of this peer's route destinations. + let prefix = manifests + .iter() + .filter(|(name, _)| *name != shared) + .flat_map(|(_, manifest)| manifest.expose.iter().flatten()) + .filter(|expose| advertises_its_ips(expose)) + .flat_map(|expose| expose.ips.iter().flatten()) + .find_map(|ip| ip.cidr.clone()); + if let Some(prefix) = prefix { + donor = Some((key.clone(), shared.clone(), prefix)); + break; + } + } + if donor.is_some() { + break; + } + } + + // Second: give that prefix to the shared vpc's *other* peer, in a different peering. + let mut done = false; + if let Some((donor_key, shared, prefix)) = donor { + for (key, peering) in agent.spec.peerings.iter_mut().flatten() { + if *key == donor_key { + continue; + } + let Some(manifests) = peering.peering.as_mut() else { + continue; + }; + if !manifests.contains_key(&shared) { + continue; + } + let names: Vec = manifests + .keys() + .filter(|name| **name != shared) + .cloned() + .collect(); + for name in names { + let Some(manifest) = manifests.get_mut(&name) else { + continue; + }; + if let Some(ip) = manifest + .expose + .iter_mut() + .flatten() + .filter(|expose| advertises_its_ips(expose)) + .flat_map(|expose| expose.ips.iter_mut().flatten()) + .find(|ip| ip.cidr.is_some()) + { + ip.cidr = Some(prefix.clone()); + done = true; + break; + } + } + if done { + break; + } + } + } + done +} + +/// The mutations that break a peering's ACL or the group it names, rather than an expose's prefixes. +/// +/// Split out of [`apply`] only for length; none of these needs the driver. +fn mutate_peering_metadata(agent: &mut GatewayAgent, mutation: Mutation) -> bool { + match mutation { + Mutation::MakeBothSidesStateful => { + let mut done = false; + for (_, peerings) in agent.spec.peerings.iter_mut().flatten() { + let manifests = peerings.peering.iter_mut().flatten(); + let mut touched = 0; + for (_, manifest) in manifests { + for expose in manifest.expose.iter_mut().flatten() { + let nat = expose.nat.get_or_insert( + crate::gateway_agent_crd::GatewayAgentPeeringsPeeringExposeNat { + masquerade: None, + port_forward: None, + r#static: None, + }, + ); + nat.port_forward = None; + nat.r#static = None; + nat.masquerade = Some(GatewayAgentPeeringsPeeringExposeNatMasquerade { + idle_timeout: None, + }); + // masquerade needs somewhere to translate to + if expose.r#as.is_none() { + expose.r#as = Some(vec![GatewayAgentPeeringsPeeringExposeAs { + cidr: Some("172.31.0.0/16".to_string()), + not: None, + }]); + } + } + touched += 1; + } + if touched == 2 { + done = true; + break; + } + } + done + } + + Mutation::NameAMissingGroup => { + if let Some((_, peerings)) = agent.spec.peerings.iter_mut().flatten().next() { + peerings.gateway_group = Some("no-such-group".to_string()); + true + } else { + false + } + } + + Mutation::NameAStrangerInARule => { + let mut done = false; + for (_, peerings) in agent.spec.peerings.iter_mut().flatten() { + let Some(acl) = peerings.acl.as_mut() else { + continue; + }; + if let Some(rule) = acl.rules.iter_mut().flatten().next() { + rule.from = Some("not-in-this-peering".to_string()); + done = true; + break; + } + } + done + } + + Mutation::DemandFlowScope => { + let mut done = false; + for (_, peerings) in agent.spec.peerings.iter_mut().flatten() { + // Flow scope is legal where a side is stateful throughout, so asking for it there + // breaks nothing. Skip those peerings rather than produce a case whose legality is + // arguable: the assertion this feeds is "whatever was touched must be refused". + let allowed = peerings + .peering + .iter() + .flatten() + .any(|(_, manifest)| stateful_throughout(manifest)); + if allowed { + continue; + } + let Some(acl) = peerings.acl.as_mut() else { + continue; + }; + for rule in acl.rules.iter_mut().flatten() { + rule.scope = Some(GatewayAgentPeeringsAclRulesScope::Flow); + done = true; + } + if done { + break; + } + } + done + } + _ => unreachable!("mutate_peering_metadata called for {mutation:?}"), + } +} + +/// The mutations that break the *shape* of an expose's prefix lists -- lengths, families, exclusions. +/// +/// Split out of [`apply`] only for length; none of these needs the driver. +fn mutate_expose_shape(agent: &mut GatewayAgent, mutation: Mutation) -> bool { + match mutation { + Mutation::MismatchPortForwardPrefixes | Mutation::MismatchStaticNatPrefixes => { + let wanted: fn(&GatewayAgentPeeringsPeeringExpose) -> bool = + if mutation == Mutation::MismatchPortForwardPrefixes { + is_port_forwarding + } else { + is_static + }; + let mut done = false; + for expose in exposes_mut(agent) { + if !wanted(expose) { + continue; + } + // lengthening one side's mask leaves the two unequal, and for port forwarding + // leaves the address-port totals unequal too unless the ports compensate + if let Some(entry) = expose.r#as.iter_mut().flatten().next() + && let Some(cidr) = entry.cidr.as_ref() + && let Some(longer) = lengthen(cidr, 2) + { + entry.cidr = Some(longer); + done = true; + break; + } + } + done + } + + Mutation::ExcludeFromPortForwarding => { + let mut done = false; + for expose in exposes_mut(agent) { + if !is_port_forwarding(expose) { + continue; + } + let Some(inside) = expose + .ips + .iter() + .flatten() + .find_map(|ip| ip.cidr.as_ref()) + .and_then(|cidr| lengthen(cidr, 1)) + else { + continue; + }; + expose.ips.get_or_insert_with(Vec::new).push( + GatewayAgentPeeringsPeeringExposeIps { + cidr: None, + not: Some(inside), + vpc_subnet: None, + }, + ); + done = true; + break; + } + done + } + + Mutation::MixAddressFamilies => { + let mut done = false; + for expose in exposes_mut(agent) { + let Some(entry) = expose.ips.iter_mut().flatten().next() else { + continue; + }; + let Some(cidr) = entry.cidr.as_ref() else { + continue; + }; + // whichever family it is not + let other = if cidr.contains(':') { + "10.99.0.0/16" + } else { + "2001:db8:9999::/48" + }; + expose.ips.get_or_insert_with(Vec::new).push( + GatewayAgentPeeringsPeeringExposeIps { + cidr: Some(other.to_string()), + not: None, + vpc_subnet: None, + }, + ); + done = true; + break; + } + done + } + _ => unreachable!("mutate_expose_shape called for {mutation:?}"), + } +} + +/// Copy one static-NAT expose over a sibling in the same manifest, so both claim one private prefix, +/// then move the copy's translation range next door so the two claims disagree. +/// +/// Split out of [`apply`] only for length. +fn duplicate_a_static_expose(agent: &mut GatewayAgent) -> bool { + for (_, peering) in agent.spec.peerings.iter_mut().flatten() { + for manifest in peering.peering.iter_mut().flatten().map(|(_, m)| m) { + let Some(exposes) = manifest.expose.as_mut() else { + continue; + }; + // Both must use static NAT: it is the mode whose table silently replaces a duplicate + // rather than refusing it, and a pair of modes the validator lets overlap would make a + // case that is legitimately legal. + let statics: Vec = exposes + .iter() + .enumerate() + .filter(|(_, expose)| { + expose + .nat + .as_ref() + .is_some_and(|nat| nat.r#static.is_some()) + }) + .map(|(index, _)| index) + .collect(); + + let (Some(donor), Some(recipient)) = (statics.first(), statics.get(1)) else { + continue; + }; + let mut copy = exposes[*donor].clone(); + let moved = copy + .r#as + .as_mut() + .and_then(|ranges| ranges.first_mut()) + .and_then(|range| { + let next_door = sibling(range.cidr.as_ref()?)?; + range.cidr = Some(next_door); + Some(()) + }); + if moved.is_some() { + exposes[*recipient] = copy; + return true; + } + } + } + false +} + +/// Apply `mutation`, returning whether it found something to break. +/// +/// **A `true` return means the result is certainly illegal**, not merely that something changed. Every +/// mutation is written to decline rather than to produce a case whose legality is arguable -- see +/// `DemandFlowScope` and `OverlapWithAnotherPeer`, both of which break rules that have legitimate +/// exceptions and so check the exception does not apply before touching anything. That is what lets a +/// property assert the validator refuses whatever this touched, which is the only guard against the +/// validator growing more permissive about a rule nothing downstream enforces. +pub fn apply(d: &mut D, agent: &mut GatewayAgent, mutation: Mutation) -> Option { + let bit = match mutation { + Mutation::None => false, + + Mutation::MismatchPortForwardPrefixes + | Mutation::MismatchStaticNatPrefixes + | Mutation::ExcludeFromPortForwarding + | Mutation::MixAddressFamilies => mutate_expose_shape(agent, mutation), + Mutation::UseReservedPrefix => { + let reserved = ["127.0.0.0/8", "224.0.0.0/4", "0.0.0.0/8", "ff00::/8"]; + let choice = + reserved[d.gen_usize(Bound::Included(&0), Bound::Excluded(&reserved.len()))?]; + let mut done = false; + for expose in exposes_mut(agent) { + if let Some(entry) = expose.ips.iter_mut().flatten().next() + && entry.cidr.is_some() + { + entry.cidr = Some(choice.to_string()); + done = true; + break; + } + } + done + } + + Mutation::EmptyPrivatePrefixes => { + let mut done = false; + for expose in exposes_mut(agent) { + if expose.ips.is_some() { + expose.ips = None; + done = true; + break; + } + } + done + } + + Mutation::DropTranslationRange => { + let mut done = false; + for expose in exposes_mut(agent) { + if expose.nat.is_some() && expose.r#as.is_some() { + expose.r#as = None; + done = true; + break; + } + } + done + } + + Mutation::MakeBothSidesStateful + | Mutation::NameAMissingGroup + | Mutation::NameAStrangerInARule + | Mutation::DemandFlowScope => mutate_peering_metadata(agent, mutation), + Mutation::UsePortZero => { + let mut done = false; + for expose in exposes_mut(agent) { + let Some(nat) = expose.nat.as_mut() else { + continue; + }; + let Some(pf) = nat.port_forward.as_mut() else { + continue; + }; + if let Some(ports) = pf.ports.iter_mut().flatten().next() { + ports.port = Some("0-100".to_string()); + ports.r#as = Some("0-100".to_string()); + done = true; + break; + } + } + done + } + + Mutation::DuplicateAStaticExpose => duplicate_a_static_expose(agent), + Mutation::OverlapWithAnotherPeer => overlap_with_another_peer(agent), + }; + Some(bit) +} + +/// A legal configuration with one rule deliberately broken. +/// +/// Yields the mutation that was chosen and whether it found anything to change, alongside the +/// configuration, so a property can say what it was testing and a harness can tell whether the +/// generator is doing any work. +#[derive(Debug, Clone, Default)] +pub struct MutatedAgents(GatewayAgents); + +impl MutatedAgents { + #[must_use] + pub fn new(agents: GatewayAgents) -> Self { + Self(agents) + } +} + +impl ValueGenerator for MutatedAgents { + type Output = (Mutation, bool, GatewayAgent); + + fn generate(&self, d: &mut D) -> Option { + let mut agent = self.0.generate(d)?; + let all = Mutation::all(); + let mutation = all[d.gen_usize(Bound::Included(&0), Bound::Excluded(&all.len()))?]; + let bit = apply(d, &mut agent, mutation)?; + Some((mutation, bit, agent)) + } +} diff --git a/k8s-intf/src/bolero/peering.rs b/k8s-intf/src/bolero/peering.rs index cf7ad87af1..65bd48de7d 100644 --- a/k8s-intf/src/bolero/peering.rs +++ b/k8s-intf/src/bolero/peering.rs @@ -6,8 +6,10 @@ use std::ops::Bound; use bolero::{Driver, ValueGenerator}; -use crate::bolero::expose::LegalValueExposeGenerator; -use crate::bolero::{SubnetMap, VpcSubnetMap}; +use crate::bolero::acl::{AclGenerator, SideFacts}; +use crate::bolero::expose::{ExposeGenerator, Which}; +use crate::bolero::support::blocks; +use crate::bolero::{AddressFamily, NatFlavour, SubnetMap, VpcSubnetMap}; use crate::gateway_agent_crd::{GatewayAgentPeerings, GatewayAgentPeeringsPeering}; /// Generate legal values for `GatewayAgentPeeringsPeering` @@ -16,12 +18,29 @@ use crate::gateway_agent_crd::{GatewayAgentPeerings, GatewayAgentPeeringsPeering /// In particular, subnet names are restricted. Lengths of various lists is also limited to 16 pub struct LegalValuePeeringsPeeringGenerator<'a> { subnets: &'a SubnetMap, + flavours: &'a [NatFlavour], + family: AddressFamily, + max_exposes: u8, + /// The first block slot this manifest's vpc owns; see [`crate::bolero::support::blocks`]. + slot_base: u8, } impl<'a> LegalValuePeeringsPeeringGenerator<'a> { #[must_use] - pub fn new(subnets: &'a SubnetMap) -> Self { - Self { subnets } + pub fn new( + subnets: &'a SubnetMap, + flavours: &'a [NatFlavour], + family: AddressFamily, + max_exposes: u8, + vpc: u8, + ) -> Self { + Self { + subnets, + flavours, + family, + max_exposes, + slot_base: blocks::expose_slot(vpc, max_exposes, 0), + } } } @@ -29,11 +48,18 @@ impl ValueGenerator for LegalValuePeeringsPeeringGenerator<'_> { type Output = GatewayAgentPeeringsPeering; fn generate(&self, d: &mut D) -> Option { - let num_expose = d.gen_usize(Bound::Included(&1), Bound::Included(&16))?; - let expose_gen = LegalValueExposeGenerator::new(self.subnets); - let expose = (0..num_expose) - .map(|_| expose_gen.generate(d)) - .collect::>>()?; + let num_expose = d.gen_u8(Bound::Included(&1), Bound::Included(&self.max_exposes))?; + let mut expose = Vec::with_capacity(usize::from(num_expose)); + for index in 0..num_expose { + // The flavour has to be settled before the prefixes are drawn, since it constrains the + // shape and cannot be imposed afterwards. The family is settled for the whole peering, + // one level up: the two manifests must agree on it. + let flavour = self.flavours + [d.gen_usize(Bound::Included(&0), Bound::Excluded(&self.flavours.len()))?]; + let which = Which::nth(self.slot_base.saturating_add(index), index, num_expose); + expose + .push(ExposeGenerator::new(flavour, self.family, which, self.subnets).generate(d)?); + } Some(GatewayAgentPeeringsPeering { expose: Some(expose).filter(|e| !e.is_empty()), @@ -47,6 +73,10 @@ impl ValueGenerator for LegalValuePeeringsPeeringGenerator<'_> { pub struct LegalValuePeeringsGenerator<'a> { vpc_subnets: &'a VpcSubnetMap, vpc_names: Vec<&'a String>, + flavours: &'a [NatFlavour], + families: &'a [AddressFamily], + max_exposes: u8, + groups: &'a [String], } impl<'a> LegalValuePeeringsGenerator<'a> { @@ -55,46 +85,155 @@ impl<'a> LegalValuePeeringsGenerator<'a> { /// # Errors /// /// Returns an error if there are less than two VPCs in the subnet map. - pub fn new(vpc_subnets: &'a VpcSubnetMap) -> Result { + pub fn new( + vpc_subnets: &'a VpcSubnetMap, + flavours: &'a [NatFlavour], + families: &'a [AddressFamily], + max_exposes: u8, + groups: &'a [String], + ) -> Result { if vpc_subnets.len() < 2 { return Err("At least two VPCs are required to generate peerings".to_string()); } + if groups.is_empty() { + return Err("At least one gateway group is required".to_string()); + } let vpc_names = vpc_subnets.keys().collect(); Ok(Self { vpc_subnets, vpc_names, + flavours, + families, + max_exposes, + groups, }) } + + /// The flavours the *other* manifest of a peering may use, given that one side is stateful. + /// + /// A peering is refused if both manifests use a stateful flavour -- masquerade or port + /// forwarding -- so once one side may, the other is restricted to the stateless ones. If the + /// caller asked only for stateful flavours, the other side takes no NAT at all, which keeps the + /// peering legal while leaving the side the caller cares about alone. + fn stateless_of(&self) -> Vec { + let stateless: Vec = self + .flavours + .iter() + .copied() + .filter(|flavour| !flavour.is_stateful()) + .collect(); + if stateless.is_empty() { + vec![NatFlavour::None] + } else { + stateless + } + } } -fn pick2<'a, D: Driver, T>(d: &mut D, items: &[&'a T]) -> Option<[&'a T; 2]> { - assert!(items.len() >= 2); +/// Two distinct indices into a list of at least two items. +fn pick2(d: &mut D, len: usize) -> Option<[usize; 2]> { + assert!(len >= 2); - let index1 = d.gen_usize(Bound::Included(&0), Bound::Excluded(&items.len()))?; - let mut index2 = d.gen_usize(Bound::Included(&0), Bound::Excluded(&items.len()))?; + let index1 = d.gen_usize(Bound::Included(&0), Bound::Excluded(&len))?; + let mut index2 = d.gen_usize(Bound::Included(&0), Bound::Excluded(&len))?; if index1 == index2 { - index2 = (index2 + 1) % items.len(); + index2 = (index2 + 1) % len; } - Some([items[index1], items[index2]]) + Some([index1, index2]) } -impl ValueGenerator for LegalValuePeeringsGenerator<'_> { - type Output = GatewayAgentPeerings; +impl LegalValuePeeringsGenerator<'_> { + /// The unordered pairs of vpcs that could be peered, in a stable order. + /// + /// Validation refuses a configuration in which one pair of vpcs is peered twice + /// (`ConfigError::DuplicateVpcPeerings`), so a caller generating several peerings has to draw + /// *distinct* pairs. Drawing each one independently does not: with up to sixteen peerings over + /// as few as two vpcs a collision is close to certain, and a collision fails the whole + /// configuration. That is how the peering half of the model came to be generated in quantity and + /// never survive validation -- 28,000 peerings drawn, none validated. + #[must_use] + pub fn pairs(&self) -> Vec<[usize; 2]> { + let n = self.vpc_names.len(); + let mut out = Vec::with_capacity(n * n / 2); + for first in 0..n { + for second in (first + 1)..n { + out.push([first, second]); + } + } + out + } + + /// Generate a peering between the two named vpcs. + /// + /// The pair comes from the caller so that it can keep them distinct; see [`Self::pairs`]. + /// Indices rather than names, because a vpc's *position* is what decides which block slots its + /// prefixes come from -- and those have to differ between vpcs, not merely between the exposes of + /// one manifest. See [`blocks::expose_slot`]. + pub fn generate_for( + &self, + d: &mut D, + vpcs: [usize; 2], + ) -> Option { + let vpc_names = [*self.vpc_names.get(vpcs[0])?, *self.vpc_names.get(vpcs[1])?]; + // One address family for the whole peering: its two manifests must agree on it. + let family = self.families + [d.gen_usize(Bound::Included(&0), Bound::Excluded(&self.families.len()))?]; + + // At most one side may be stateful. Which one is drawn, so both orders are generated. + let stateful_side = d.gen_usize(Bound::Included(&0), Bound::Included(&1))?; + let stateless = self.stateless_of(); - fn generate(&self, d: &mut D) -> Option { - let vpc_names = pick2(d, &self.vpc_names)?; let empty_map = SubnetMap::new(); - let peerings_gens = vpc_names.map(|n| { - LegalValuePeeringsPeeringGenerator::new(self.vpc_subnets.get(n).unwrap_or(&empty_map)) - }); let peering = (0..=1) - .map(|i| Some((vpc_names[i].clone(), peerings_gens[i].generate(d)?))) + .map(|i| { + let flavours: &[NatFlavour] = if i == stateful_side { + self.flavours + } else { + &stateless + }; + let generator = LegalValuePeeringsPeeringGenerator::new( + self.vpc_subnets.get(vpc_names[i]).unwrap_or(&empty_map), + flavours, + family, + self.max_exposes, + u8::try_from(vpcs[i]).unwrap_or(u8::MAX), + ); + Some((vpc_names[i].clone(), generator.generate(d)?)) + }) .collect::>>()?; + // A gateway group that exists. The group table sits beside the peerings, so a name drawn + // freely is a name validation will not find. + let group = self.groups + [d.gen_usize(Bound::Included(&0), Bound::Excluded(&self.groups.len()))?] + .clone(); + + // The ACL is built from the manifests, not beside them: a rule's `match` is checked against + // what the two sides actually expose, and `scope: flow` against how they translate. + let acl = if d.produce::()? { + let facts: Vec = peering + .iter() + .map(|(vpc, manifest)| SideFacts::of(vpc, manifest)) + .collect(); + let [left, right] = <[SideFacts; 2]>::try_from(facts).ok()?; + Some(AclGenerator::new(left, right).generate(d)?) + } else { + None + }; + Some(GatewayAgentPeerings { - gateway_group: Some(d.produce::()?), + gateway_group: Some(group), peering: Some(peering), - acl: None, // FIXME: Add a proper implementation when used + acl, }) } } + +impl ValueGenerator for LegalValuePeeringsGenerator<'_> { + type Output = GatewayAgentPeerings; + + fn generate(&self, d: &mut D) -> Option { + let pair = pick2(d, self.vpc_names.len())?; + self.generate_for(d, pair) + } +} diff --git a/k8s-intf/src/bolero/permute.rs b/k8s-intf/src/bolero/permute.rs new file mode 100644 index 0000000000..fb293ed39f --- /dev/null +++ b/k8s-intf/src/bolero/permute.rs @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +//! A configuration, and the same configuration with its set-valued lists in a different order. +//! +//! Paired with `mutate`, which asks whether the validator is too permissive about configurations that +//! cannot be *enacted*. This asks the other question: whether an accepted configuration has only one +//! *meaning*. +//! +//! The two failures are nothing alike from where the user stands. An unenactable configuration at +//! least fails to build, and something somewhere gets an error. An **ambiguous** one builds +//! perfectly: two readings, both valid outputs of the code as written, and the chain silently takes +//! whichever its containers hand it first. Nothing reports that, and nothing can, because there is no +//! error to report -- only traffic going somewhere nobody chose. +//! +//! Permutation is the oracle, and its virtue is that it restates no rule. A CRD's `expose` list, and +//! the `ips` and `as` lists inside it, are *sets*: their order is not part of what the configuration +//! means. Nor is which name a peering happens to carry, since peering names reach no artifact -- the +//! names in the rendered FRR config come from vpcs. So reordering all of those must leave every +//! artifact the dataplane installs identical. Where it does not, the chain resolved a conflict by +//! position, and the configuration had two meanings. +//! +//! What is deliberately *not* permuted: an ACL's `rules`. Those are ordered by definition -- first +//! match wins -- so reordering them changes the configuration's meaning legitimately, and a property +//! that permuted them would be asserting something false. + +use std::collections::BTreeMap; +use std::ops::Bound; + +use bolero::{Driver, ValueGenerator}; + +use crate::bolero::mutate::{MutatedAgents, Mutation}; +use crate::gateway_agent_crd::{ + GatewayAgent, GatewayAgentPeerings, GatewayAgentPeeringsPeeringExpose, +}; + +/// Reorder `items`, if there is more than one way to. +/// +/// A rotation and a swap rather than a full shuffle: enough to move every element for the short lists +/// a configuration holds, and it costs two draws instead of one per element. +fn reorder(d: &mut D, items: &mut [T]) -> Option<()> { + if items.len() < 2 { + return Some(()); + } + let by = d.gen_usize(Bound::Included(&0), Bound::Excluded(&items.len()))?; + items.rotate_left(by); + let first = d.gen_usize(Bound::Included(&0), Bound::Excluded(&items.len()))?; + let second = d.gen_usize(Bound::Included(&0), Bound::Excluded(&items.len()))?; + items.swap(first, second); + Some(()) +} + +/// Reorder the lists inside one expose. +fn reorder_expose( + d: &mut D, + expose: &mut GatewayAgentPeeringsPeeringExpose, +) -> Option<()> { + if let Some(ips) = expose.ips.as_mut() { + reorder(d, ips)?; + } + if let Some(translations) = expose.r#as.as_mut() { + reorder(d, translations)?; + } + // Each port-forwarding `ports` block becomes an expose of its own during conversion, so the + // blocks are a set of rules and their order should not matter either. + if let Some(ports) = expose + .nat + .as_mut() + .and_then(|nat| nat.port_forward.as_mut()) + .and_then(|forward| forward.ports.as_mut()) + { + reorder(d, ports)?; + } + Some(()) +} + +/// Reorder everything inside one peering, but not its ACL's rules. +fn reorder_peering(d: &mut D, peering: &mut GatewayAgentPeerings) -> Option<()> { + for manifest in peering.peering.iter_mut().flatten().map(|(_, m)| m) { + if let Some(exposes) = manifest.expose.as_mut() { + reorder(d, exposes)?; + for expose in exposes.iter_mut() { + reorder_expose(d, expose)?; + } + } + } + + // A rule's `match` names sets of prefixes; the rules themselves are ordered and are left alone. + for rule in peering + .acl + .iter_mut() + .flat_map(|acl| acl.rules.iter_mut()) + .flatten() + { + let Some(pattern) = rule.r#match.as_mut() else { + continue; + }; + if let Some(src) = pattern.src.as_mut() { + reorder(d, src)?; + } + if let Some(dst) = pattern.dst.as_mut() { + reorder(d, dst)?; + } + } + Some(()) +} + +/// Reorder a whole configuration's set-valued lists. +/// +/// Returns whether anything actually moved, so a property can tell a real agreement from a +/// permutation that happened to be the identity on a configuration with nothing to reorder. +fn reorder_agent(d: &mut D, agent: &mut GatewayAgent) -> Option { + let before = format!("{:?}", agent.spec.peerings); + + // The peerings themselves: rotate which name holds which peering. Peering names reach no + // artifact -- the names in the rendered config are vpcs' -- so this is a permutation of the + // configuration and not a change to it, and it is the only one that reorders the *peering* walk + // that `VpcRouteTable::build` does. + if let Some(peerings) = agent.spec.peerings.as_mut() { + let names: Vec = peerings.keys().cloned().collect(); + let mut bodies: Vec = peerings.values().cloned().collect(); + reorder(d, &mut bodies)?; + *peerings = names.into_iter().zip(bodies).collect::>(); + + for peering in peerings.values_mut() { + reorder_peering(d, peering)?; + } + } + + Some(before != format!("{:?}", agent.spec.peerings)) +} + +/// Draws a configuration and a reordering of it. +/// +/// The reordering is a *different value* rather than an in-place edit, so the property receives both +/// and holds no ordering logic of its own. +/// +/// Built on [`MutatedAgents`] rather than on legal-by-construction configurations alone, and that +/// choice is the whole point. The generator is now careful to keep every vpc's prefixes disjoint, so +/// it *cannot* produce the overlapping-route ambiguity by itself -- a permutation property driven by +/// legal input only would pass without ever meeting the case it exists for. Feeding it near-misses +/// puts the question where it belongs: when the validator lets a rule slide, is the result still +/// unambiguous? A property that only sees configurations the generator was careful to make clean is +/// measuring its own generator. +#[derive(Debug, Default, Clone)] +pub struct PermutedAgents(MutatedAgents); + +impl PermutedAgents { + #[must_use] + pub fn new(agents: MutatedAgents) -> Self { + Self(agents) + } +} + +impl ValueGenerator for PermutedAgents { + /// The mutation applied, the configuration, its reordering, and whether the reordering moved + /// anything. + type Output = (Mutation, GatewayAgent, GatewayAgent, bool); + + fn generate(&self, d: &mut D) -> Option { + let (mutation, _applied, agent) = self.0.generate(d)?; + let mut permuted = agent.clone(); + let moved = reorder_agent(d, &mut permuted)?; + Some((mutation, agent, permuted, moved)) + } +} diff --git a/k8s-intf/src/bolero/reduce.rs b/k8s-intf/src/bolero/reduce.rs new file mode 100644 index 0000000000..acc72e2669 --- /dev/null +++ b/k8s-intf/src/bolero/reduce.rs @@ -0,0 +1,152 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +//! A configuration, and the same configuration with one expose taken out of it. +//! +//! The third of the near-miss family, after `mutate` (is an accepted configuration *enactable*?) and +//! `permute` (does it have only *one* meaning?). This one asks whether the dataplane is doing +//! everything the configuration asked for: **remove an expose and something the dataplane installs +//! must change.** +//! +//! The failure it hunts is a builder that silently ignores part of its input -- a shape it does not +//! handle, a `continue` on a branch nobody expected to be reachable. That is a class nothing else here +//! covers, and unlike overlap it needs no mutation to reach: the bug would be in the builder, not +//! behind a validator rule, so it shows up on configurations that are entirely legal. +//! +//! Whether an expose is ever *legitimately* a no-op is a domain question, and the answer taken here is +//! no: every expose contributes prefixes to the peer's import and advertise lists at the very least. +//! If that turns out to be wrong for some shape, this generator is where the exemption belongs. + +use std::ops::Bound; + +use bolero::{Driver, ValueGenerator}; + +use crate::bolero::mutate::{MutatedAgents, Mutation}; +use crate::gateway_agent_crd::GatewayAgent; + +/// Which expose was removed, for a failure message: peering, vpc, and its place in the manifest. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Dropped { + pub peering: String, + pub vpc: String, + pub index: usize, + /// The NAT mode it used, as the CRD spells it, so a property can say which artifact it expected to + /// change. `None` means the expose translated nothing. + pub nat: Option<&'static str>, +} + +impl std::fmt::Display for Dropped { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!( + f, + "{}/{} expose {} ({})", + self.peering, + self.vpc, + self.index, + self.nat.unwrap_or("no nat") + ) + } +} + +/// Every expose that could be removed, as (peering, vpc, index). +/// +/// Enumerated up front so the choice is uniform over exposes rather than biased by where in the walk it +/// happens to fall. Only manifests holding more than one expose are candidates: emptying a manifest +/// makes the configuration illegal outright (`NoExposes`), so drawing one would only produce a case +/// thrown away downstream. +fn candidates(agent: &GatewayAgent) -> Vec<(String, String, usize)> { + let mut out = Vec::new(); + for (peering_name, peering) in agent.spec.peerings.iter().flatten() { + for (vpc, manifest) in peering.peering.iter().flatten() { + let count = manifest.expose.as_ref().map_or(0, Vec::len); + if count < 2 { + continue; + } + for index in 0..count { + out.push((peering_name.clone(), vpc.clone(), index)); + } + } + } + out +} + +/// Take one of `choices` out of `agent`, chosen by the driver, and say which it was. +/// +/// A `None` return means the driver ran out of input, never that there was nothing to remove -- the +/// caller has already established there is. Keeping those two apart is why `choices` comes in as an +/// argument rather than being computed here. +fn drop_an_expose( + d: &mut D, + agent: &mut GatewayAgent, + mut choices: Vec<(String, String, usize)>, +) -> Option { + let choice = d.gen_usize(Bound::Included(&0), Bound::Excluded(&choices.len()))?; + let (peering_name, vpc, index) = choices.swap_remove(choice); + + let exposes = agent + .spec + .peerings + .as_mut()? + .get_mut(&peering_name)? + .peering + .as_mut()? + .get_mut(&vpc)? + .expose + .as_mut()?; + if index >= exposes.len() { + return None; + } + let removed = exposes.remove(index); + let nat = removed.nat.as_ref().and_then(|nat| { + if nat.r#static.is_some() { + Some("static") + } else if nat.masquerade.is_some() { + Some("masquerade") + } else if nat.port_forward.is_some() { + Some("port forwarding") + } else { + None + } + }); + + Some(Dropped { + peering: peering_name, + vpc, + index, + nat, + }) +} + +/// Draws a configuration and the same configuration with one expose removed. +/// +/// Built on [`MutatedAgents`] for the same reason [`crate::bolero::permute`] is: a property that only +/// ever sees configurations the generator took care to keep clean is measuring its own generator. Here +/// it also means the near-misses get asked the question, which is where a masked rule would show up. +#[derive(Debug, Default, Clone)] +pub struct ReducedAgents(MutatedAgents); + +impl ReducedAgents { + #[must_use] + pub fn new(agents: MutatedAgents) -> Self { + Self(agents) + } +} + +impl ValueGenerator for ReducedAgents { + /// The mutation, the configuration, the configuration less one expose, and which expose that was + /// (`None` when there was no manifest with two exposes to take one from). + type Output = (Mutation, GatewayAgent, GatewayAgent, Option); + + fn generate(&self, d: &mut D) -> Option { + let (mutation, _applied, agent) = self.0.generate(d)?; + let mut reduced = agent.clone(); + let choices = candidates(&reduced); + if choices.is_empty() { + // Nothing to remove, which is a case the property skips -- not the driver running out, so + // it must not be reported as `None` from here. + return Some((mutation, agent, reduced, None)); + } + let dropped = drop_an_expose(d, &mut reduced, choices)?; + Some((mutation, agent, reduced, Some(dropped))) + } +} diff --git a/k8s-intf/src/bolero/spec.rs b/k8s-intf/src/bolero/spec.rs index e89abbda46..1a097bdd3c 100644 --- a/k8s-intf/src/bolero/spec.rs +++ b/k8s-intf/src/bolero/spec.rs @@ -9,7 +9,7 @@ use bolero::{Driver, TypeGenerator, ValueGenerator}; use lpm::prefix::Prefix; use crate::bolero::peering::LegalValuePeeringsGenerator; -use crate::bolero::{LegalValue, SubnetMap, VpcSubnetMap}; +use crate::bolero::{AddressFamily, LegalValue, NatFlavour, SubnetMap, VpcSubnetMap}; use crate::gateway_agent_crd::{ GatewayAgentGateway, GatewayAgentGroups, GatewayAgentSpec, GatewayAgentVpcs, }; @@ -44,17 +44,127 @@ fn increment_string(s: &mut str) { } } +/// How much of a `GatewayAgentSpec` to generate, and of what. +/// +/// The knobs exist so that a property can aim: at one NAT flavour, at one address family, at a +/// single small vpc, at a fabric with a dozen. Without them every draw is a coin flip inside one +/// function and a property can only take what it is given. +/// +/// The defaults are deliberately **small**. The generator this replaced drew up to sixteen vpcs, +/// sixteen peerings per spec and sixteen exposes per peering, each with up to sixteen prefixes: a +/// single case ran to thousands of prefixes, which costs throughput and makes a counterexample +/// unreadable. Nothing here needs to be large to be interesting; a caller that wants large can ask. +#[derive(Debug, Clone)] +pub struct SpecBuilder { + max_vpcs: u8, + max_peerings: u8, + max_exposes: u8, + max_subnets: u8, + flavours: Vec, + families: Vec, +} + +impl Default for SpecBuilder { + fn default() -> Self { + Self { + max_vpcs: 4, + max_peerings: 3, + max_exposes: 2, + max_subnets: 3, + flavours: NatFlavour::all(), + families: AddressFamily::all(), + } + } +} + +impl SpecBuilder { + /// The most vpcs a generated spec will carry. At least two are needed for any peering. + #[must_use] + pub fn max_vpcs(mut self, max: u8) -> Self { + self.max_vpcs = max; + self + } + + /// The most peerings a generated spec will carry. + #[must_use] + pub fn max_peerings(mut self, max: u8) -> Self { + self.max_peerings = max; + self + } + + /// The most exposes one side of a peering will offer. + #[must_use] + pub fn max_exposes(mut self, max: u8) -> Self { + self.max_exposes = max; + self + } + + /// The most subnets a generated vpc will have. + #[must_use] + pub fn max_subnets(mut self, max: u8) -> Self { + self.max_subnets = max; + self + } + + /// Which NAT flavours the exposes may use. Empty is treated as all of them. + #[must_use] + pub fn flavours(mut self, flavours: Vec) -> Self { + if !flavours.is_empty() { + self.flavours = flavours; + } + self + } + + /// Which address families the exposes may use. Empty is treated as both. + #[must_use] + pub fn families(mut self, families: Vec) -> Self { + if !families.is_empty() { + self.families = families; + } + self + } + + /// The generator these knobs describe. + #[must_use] + pub fn build(self) -> GatewayAgentSpecs { + GatewayAgentSpecs(self) + } +} + +/// Draws `GatewayAgentSpec`s, as configured by a [`SpecBuilder`]. +#[derive(Debug, Clone)] +pub struct GatewayAgentSpecs(pub(crate) SpecBuilder); + +impl Default for GatewayAgentSpecs { + fn default() -> Self { + SpecBuilder::default().build() + } +} + /// Generate a random legal `GatewayAgentSpec` /// /// This does not cover all legal `GatewayAgentSpecs`, /// it is limited by the underlying generators and it generates /// vpcs and peerings with a fixed name pattern and not all /// vni combinations are generated. +/// +/// Delegates to [`GatewayAgentSpecs`] with its default knobs, so a caller who wants to aim can use +/// [`SpecBuilder`] instead. impl TypeGenerator for LegalValue { fn generate(d: &mut D) -> Option { - let num_vpcs = d.gen_usize(Bound::Included(&0), Bound::Included(&16))?; + Some(LegalValue(GatewayAgentSpecs::default().generate(d)?)) + } +} + +impl ValueGenerator for GatewayAgentSpecs { + type Output = GatewayAgentSpec; + + fn generate(&self, d: &mut D) -> Option { + let knobs = &self.0; + let num_vpcs = + usize::from(d.gen_u8(Bound::Included(&0), Bound::Included(&knobs.max_vpcs))?); let num_peerings = if num_vpcs > 1 { - d.gen_usize(Bound::Included(&0), Bound::Included(&16))? + usize::from(d.gen_u8(Bound::Included(&0), Bound::Included(&knobs.max_peerings))?) } else { 0 }; @@ -64,8 +174,12 @@ impl TypeGenerator for LegalValue { let mut vpc_internal_ids = HashSet::new(); for i in 0..num_vpcs { let vni_offset = u32::try_from(i).expect("too many vpcs"); - let lv_vpc = d.produce::>()?; - let mut vpc = lv_vpc.take(); + let mut vpc = crate::bolero::vpc::VpcGenerator::new( + u8::try_from(i).unwrap_or(u8::MAX), + knobs.max_subnets, + &knobs.families, + ) + .generate(d)?; let vpc_id = vpc.internal_id.as_mut().unwrap(); while !vpc_internal_ids.insert(vpc_id.clone()) { // We already have a VPC with this internal_id, "increment" the string to generate a @@ -78,19 +192,38 @@ impl TypeGenerator for LegalValue { let vpc_subnet_map = extract_subnets(&vpcs); - let mut peerings = BTreeMap::new(); - if num_peerings > 0 { - let peering_gen = LegalValuePeeringsGenerator::new(&vpc_subnet_map).unwrap(); - for i in 0..num_peerings { - peerings.insert(format!("peering{i}"), peering_gen.generate(d)?); - } - } - + // Gateway groups before peerings: a peering names one, and whole-config validation checks + // that the name exists. Generating the peerings first left them naming groups drawn freely, + // which is to say groups that do not exist. let num_groups = d.gen_usize(Bound::Included(&0), Bound::Included(&6))?; let mut groups = BTreeMap::new(); for i in 0..=num_groups { groups.insert(format!("gwgroup-{i}"), d.produce::()?); } + let group_names: Vec = groups.keys().cloned().collect(); + + let mut peerings = BTreeMap::new(); + if num_peerings > 0 { + let peering_gen = LegalValuePeeringsGenerator::new( + &vpc_subnet_map, + &knobs.flavours, + &knobs.families, + knobs.max_exposes, + &group_names, + ) + .unwrap(); + // Draw *distinct* vpc pairs. Validation refuses a configuration that peers one pair + // twice, so drawing each peering's pair independently -- as this used to -- makes almost + // every configuration with more than one peering invalid, and the whole peering half of + // the model never reaches anything downstream of validation. + let mut available = peering_gen.pairs(); + let wanted = num_peerings.min(available.len()); + for i in 0..wanted { + let choice = d.gen_usize(Bound::Included(&0), Bound::Excluded(&available.len()))?; + let pair = available.swap_remove(choice); + peerings.insert(format!("peering{i}"), peering_gen.generate_for(d, pair)?); + } + } let num_communities = d.gen_usize(Bound::Included(&0), Bound::Included(&9))?; let mut communities = BTreeMap::new(); @@ -99,7 +232,7 @@ impl TypeGenerator for LegalValue { communities.insert(i.to_string(), community); } - Some(LegalValue(GatewayAgentSpec { + Some(GatewayAgentSpec { agent_version: None, config: None, groups: Some(groups), @@ -107,6 +240,6 @@ impl TypeGenerator for LegalValue { gateway: Some(d.produce::>()?.take()), vpcs: Some(vpcs).filter(|v| !v.is_empty()), peerings: Some(peerings).filter(|p| !p.is_empty()), - })) + }) } } diff --git a/k8s-intf/src/bolero/support.rs b/k8s-intf/src/bolero/support.rs index b1880a381d..290daad323 100644 --- a/k8s-intf/src/bolero/support.rs +++ b/k8s-intf/src/bolero/support.rs @@ -254,15 +254,28 @@ pub fn choose(d: &mut D, choices: &[T]) -> Option { Some(choices[index].clone()) } +/// The shortest prefixes these generators will draw. +/// +/// A prefix shorter than this necessarily contains one of the special-use ranges that a `VpcExpose` +/// may not overlap -- a v4 `/0` covers loopback, a `/2` at 64 covers `127.0.0.0/8`, and so on -- so +/// drawing them only ever produces configurations that validation refuses. Longer prefixes can still +/// land inside a reserved range and be refused; they just are not guaranteed to. +const MIN_V4_MASK: u8 = 8; +const MIN_V6_MASK: u8 = 16; + pub fn generate_v4_prefixes(d: &mut D, count: u16) -> Option> { - let cidr4_gen = - UniqueV4CidrGenerator::new(count, d.gen_u8(Bound::Included(&0), Bound::Included(&32))?); + let cidr4_gen = UniqueV4CidrGenerator::new( + count, + d.gen_u8(Bound::Included(&MIN_V4_MASK), Bound::Included(&32))?, + ); cidr4_gen.generate(d) } pub fn generate_v6_prefixes(d: &mut D, count: u16) -> Option> { - let cidr6_gen = - UniqueV6CidrGenerator::new(count, d.gen_u8(Bound::Included(&0), Bound::Included(&128))?); + let cidr6_gen = UniqueV6CidrGenerator::new( + count, + d.gen_u8(Bound::Included(&MIN_V6_MASK), Bound::Included(&128))?, + ); cidr6_gen.generate(d) } @@ -416,3 +429,273 @@ mod test { } } } + +/// Prefixes drawn from blocks that are not special-use, with the two sides kept apart. +/// +/// An expose's prefixes are rejected if they overlap a special-use range, so drawing an address at +/// random is mostly a way of generating configurations that will be refused. These draw from blocks +/// this validator does not consider reserved -- `10.0.0.0/8` and `172.16.0.0/12` for v4, halves of +/// `2001:db8::/32` for v6 -- with the private side and the public side in different blocks so that +/// an expose's two sides can never be the same prefix. +/// +/// The same choice, and the same reasoning, as `config`'s own contract generators. +pub mod blocks { + use crate::bolero::AddressFamily; + use bolero::Driver; + use std::net::{Ipv4Addr, Ipv6Addr}; + + /// Each block is carved into fixed-size **slots**, and nothing is ever drawn that crosses one. + /// + /// Two prefixes taken from different slots therefore cannot overlap, whatever their lengths, so a + /// manifest that hands each of its exposes a slot of its own satisfies the validator's + /// no-overlap-within-a-manifest rule *by construction* rather than by luck. + /// + /// This is not a nicety. Before it, every expose in a manifest drew from one shared block, so + /// overlap was a matter of chance -- rare enough under uniform random input to look like noise. + /// A coverage-guided run found it and drove it hard: **89% of the configurations libfuzzer + /// produced that should have been legal were refused with `VPC prefixes overlap`**, against 6% + /// under random input. A fuzzer steers toward whatever reaches new code, and "the validator + /// rejects this" is new code. + /// + /// A slot length is also the shortest prefix its block can hold, since a shorter one would span + /// slots. It is long enough to sit inside the *narrower* of the two blocks (`172.16.0.0/12` for + /// v4, a half of `2001:db8::/32` for v6), so a length usable on one side is usable on the other: + /// static NAT and port forwarding both need the two sides to be the same length, and a length + /// either side cannot take is a length neither can. + pub const SLOT_V4_LEN: u8 = 20; + pub const SLOT_V6_LEN: u8 = 48; + pub const MIN_V4_LEN: u8 = SLOT_V4_LEN; + pub const MIN_V6_LEN: u8 = SLOT_V6_LEN; + + /// How many slots at the bottom of each *private* block are set aside for vpcs' own subnets: + /// one slot per vpc, so this is also the most vpcs the scheme separates. + /// + /// A subnet is subject to the same rules as a prefix written out in an expose, because an expose + /// may name one and a named subnet contributes its prefix. So a subnet has to be as carefully + /// placed as anything else: out of the slots exposes draw from, and out of the other vpcs'. + pub const SUBNET_SLOTS: u8 = 16; + + /// The slot an expose draws from, given which vpc owns it and which expose of that vpc it is. + /// + /// **A vpc's prefixes have to be disjoint from every other vpc's**, not merely from its own + /// siblings', because `VpcRouteTable` is built per vpc from the prefixes its *peers* expose to + /// it: if two peers of one vpc expose overlapping prefixes, that vpc has one destination and two + /// places to send it, and validation refuses the configuration. So the vpc is the outermost + /// level of the scheme and the slot index carries it. + /// + /// `slots_per_vpc` should be the most exposes any one manifest may hold. The arithmetic + /// saturates, so a caller asking for more vpcs or exposes than a `u8` of slots can separate gets + /// collisions rather than a panic -- see [`SUBNET_SLOTS`] and the 256 slots of `172.16.0.0/12` + /// for the budget. + #[must_use] + pub fn expose_slot(vpc: u8, slots_per_vpc: u8, expose: u8) -> u8 { + vpc.saturating_mul(slots_per_vpc).saturating_add(expose) + } + + /// Where one prefix is drawn from: a sub-slot of a slot of a block. + /// + /// Overlap is broken by *sharing an address range*, so the answer is to make sharing impossible: + /// every prefix is confined to a nested box, and two prefixes in different boxes cannot overlap + /// however long they are. Three levels, outermost first: + /// + /// * **block** -- private or public. Keeps an expose's two sides from being the same prefix. + /// * **slot** -- one per expose of a manifest. Keeps the exposes of a manifest from overlapping, + /// which is what `validate_expose_collisions` refuses. + /// * **sub-slot** -- one per prefix of an expose's own list. An expose's private list may hold + /// several prefixes, and those have to be disjoint from each other too. + #[derive(Debug, Clone, Copy)] + pub struct At { + /// Which expose of the manifest this is. + pub slot: u8, + /// Which prefix of this expose's list this is, and how many the list holds. + pub sub: u8, + pub subs: u8, + } + + impl At { + /// The whole of slot `slot`, for a caller drawing a single prefix. + #[must_use] + pub fn whole(slot: u8) -> Self { + Self { + slot, + sub: 0, + subs: 1, + } + } + + /// Prefix `sub` of `subs`, in slot `slot`. + #[must_use] + pub fn nth(slot: u8, sub: u8, subs: u8) -> Self { + Self { + slot, + sub, + subs: subs.max(sub.saturating_add(1)), + } + } + + /// How many bits of the slot the sub-slot index consumes. + fn sub_bits(self) -> u8 { + u8::try_from(self.subs.max(1).next_power_of_two().trailing_zeros()).unwrap_or(0) + } + + /// The prefix length at which this sub-slot begins: nothing shorter fits inside it. + fn level(self, family: AddressFamily) -> u8 { + min_len(family) + .saturating_add(self.sub_bits()) + .min(max_len(family)) + } + + /// The sub-slot's base address within `block_base`, and the length it begins at. + fn place(self, family: AddressFamily, block_base: u128, slot_index: u32) -> (u128, u8) { + let width = max_len(family); + let level = self.level(family); + let slot = u128::from(slot_index) << (width - min_len(family)); + // masked rather than trusted, so a caller passing `sub >= 1 << sub_bits` lands in a real + // sub-slot instead of bleeding into the next slot + let sub_mask = (1u128 << self.sub_bits()) - 1; + let sub = (u128::from(self.sub) & sub_mask) << (width - level); + (block_base | slot | sub, level) + } + } + + /// The shortest prefix length available at `at`. + /// + /// A sub-slot is shorter than a slot, so an expose drawing several prefixes has to draw longer + /// ones. A caller should draw the *count* first and take this as the floor for the length. + #[must_use] + pub fn min_len_at(family: AddressFamily, at: At) -> u8 { + at.level(family) + } + + fn v4(base: u32, block_len: u8, host: u32, len: u8) -> String { + let block_host_bits = 32 - block_len; + let within = if block_host_bits >= 32 { + host + } else { + host & ((1u32 << block_host_bits) - 1) + }; + let mask = u32::MAX.checked_shl(u32::from(32 - len)).unwrap_or(0); + let addr = (base | within) & mask; + format!("{}/{len}", Ipv4Addr::from(addr)) + } + + fn v6(base: u128, block_len: u8, host: u128, len: u8) -> String { + let block_host_bits = 128 - block_len; + let within = if block_host_bits >= 128 { + host + } else { + host & ((1u128 << block_host_bits) - 1) + }; + let mask = u128::MAX.checked_shl(u32::from(128 - len)).unwrap_or(0); + let addr = (base | within) & mask; + format!("{}/{len}", Ipv6Addr::from(addr)) + } + + /// A prefix of length `len` at `at` in the private block. + /// + /// The private block's slots start past the subnet reservation, since a vpc's subnets are private + /// addresses and an expose may name one. + pub fn private(d: &mut D, family: AddressFamily, at: At, len: u8) -> Option { + let slot = u32::from(SUBNET_SLOTS) + u32::from(at.slot); + Some(if family.is_v4() { + // 10.0.0.0/8 + let (base, level) = at.place(family, 0x0A00_0000, slot); + v4(u32::try_from(base).ok()?, level, d.produce::()?, len) + } else { + // the lower half of 2001:db8::/32, i.e. 2001:db8:0000::/33 + let (base, level) = at.place(family, 0x2001_0db8_0000_0000_0000_0000_0000_0000, slot); + v6(base, level, d.produce::()?, len) + }) + } + + /// A prefix of length `len` at `at` in the public block, disjoint from [`private`]. + /// + /// No reservation here: subnets are private, so the public block's slots start at zero. + pub fn public(d: &mut D, family: AddressFamily, at: At, len: u8) -> Option { + let slot = u32::from(at.slot); + Some(if family.is_v4() { + // 172.16.0.0/12, which holds 256 slots of SLOT_V4_LEN -- a u8 index cannot exceed it + let (base, level) = at.place(family, 0xAC10_0000, slot); + v4(u32::try_from(base).ok()?, level, d.produce::()?, len) + } else { + // the upper half of 2001:db8::/32, i.e. 2001:db8:8000::/33 + let (base, level) = at.place(family, 0x2001_0db8_8000_0000_0000_0000_0000_0000, slot); + v6(base, level, d.produce::()?, len) + }) + } + + /// The shortest prefix length one vpc's subnet slot can hold `count` distinct prefixes at. + /// + /// A caller draws the length *after* the count, since drawing it first can ask the slot for more + /// prefixes than it has at that length -- and [`private_run`] would then wrap and hand back + /// duplicates, which are overlapping subnets and refused. + #[must_use] + pub fn min_subnet_len(family: AddressFamily, count: u16) -> u8 { + let bits = u8::try_from(count.max(1).next_power_of_two().trailing_zeros()).unwrap_or(0); + min_len(family).saturating_add(bits).min(max_len(family)) + } + + /// `count` distinct prefixes of length `len` inside vpc `vpc`'s subnet slot. + /// + /// Consecutive rather than independently drawn, so they are distinct and non-overlapping without + /// a rejection loop. `len` should be at least [`min_subnet_len`] for this `count`, or the slot + /// runs out of room and the run wraps onto itself. + pub fn private_run( + d: &mut D, + family: AddressFamily, + vpc: u8, + len: u8, + count: u16, + ) -> Option> { + if count == 0 { + return Some(Vec::new()); + } + let slot_len = u32::from(min_len(family)); + let mut out = Vec::with_capacity(usize::from(count)); + if family.is_v4() { + // vpc `vpc`'s slot of 10.0.0.0/8, which holds 2^(len - SLOT_V4_LEN) prefixes of `len` + let base = 0x0A00_0000 | (u32::from(vpc) << (32 - slot_len)); + let slots = 1u32 + .checked_shl(u32::from(len) - slot_len) + .unwrap_or(u32::MAX); + let first = d.produce::()? % slots; + let shift = u32::from(32 - len); + for i in 0..u32::from(count) { + let slot = (first + i) % slots; + let addr = base | slot.checked_shl(shift).unwrap_or(0); + out.push(format!("{}/{len}", Ipv4Addr::from(addr))); + } + } else { + // likewise in the lower half of 2001:db8::/32 + let base = + 0x2001_0db8_0000_0000_0000_0000_0000_0000 | (u128::from(vpc) << (128 - slot_len)); + let slots = 1u128 + .checked_shl(u32::from(len) - slot_len) + .unwrap_or(u128::MAX); + let first = d.produce::()? % slots; + let shift = u32::from(128 - len); + for i in 0..u128::from(count) { + let slot = (first + i) % slots; + let addr = base | slot.checked_shl(shift).unwrap_or(0); + out.push(format!("{}/{len}", Ipv6Addr::from(addr))); + } + } + Some(out) + } + + /// The shortest prefix length either side of an expose may take. + #[must_use] + pub fn min_len(family: AddressFamily) -> u8 { + if family.is_v4() { + MIN_V4_LEN + } else { + MIN_V6_LEN + } + } + + /// The longest, which is a host route. + #[must_use] + pub fn max_len(family: AddressFamily) -> u8 { + if family.is_v4() { 32 } else { 128 } + } +} diff --git a/k8s-intf/src/bolero/vpc.rs b/k8s-intf/src/bolero/vpc.rs index 0eb84b6b92..35a047c633 100644 --- a/k8s-intf/src/bolero/vpc.rs +++ b/k8s-intf/src/bolero/vpc.rs @@ -7,8 +7,8 @@ use bolero::{Driver, TypeGenerator, ValueGenerator}; use net::vxlan::Vni; -use crate::bolero::LegalValue; -use crate::bolero::support::{UniqueV4CidrGenerator, UniqueV6CidrGenerator}; +use crate::bolero::support::blocks; +use crate::bolero::{AddressFamily, LegalValue}; use crate::gateway_agent_crd::{GatewayAgentVpcs, GatewayAgentVpcsSubnets}; fn generate_internal_id(d: &mut D) -> Option { @@ -21,20 +21,70 @@ fn generate_internal_id(d: &mut D) -> Option { Some(result) } -impl TypeGenerator for LegalValue { - fn generate(d: &mut D) -> Option { +/// Draws vpcs whose subnets are of the given families, and no more than `max_subnets` of each. +/// +/// A vpc's subnets can be named by an expose, and a named subnet contributes its own prefix -- so +/// they are subject to the same rules as an expose's own prefixes: a subnet that overlaps a +/// special-use range makes every expose naming it invalid. Hence the mask floors from +/// [`blocks`], which is what the previous generator, drawing masks from zero, could not respect. +#[derive(Debug, Clone)] +pub struct VpcGenerator<'a> { + /// Which vpc this is, which picks the block slot its subnets come from. A subnet may be named by + /// an expose, so it is a prefix like any other and must not collide with another vpc's. + vpc: u8, + max_subnets: u8, + families: &'a [AddressFamily], +} + +impl<'a> VpcGenerator<'a> { + #[must_use] + pub fn new(vpc: u8, max_subnets: u8, families: &'a [AddressFamily]) -> Self { + Self { + vpc, + max_subnets, + families, + } + } + + fn wants(&self, family: AddressFamily) -> bool { + self.families.contains(&family) + } +} + +impl ValueGenerator for VpcGenerator<'_> { + type Output = GatewayAgentVpcs; + + fn generate(&self, d: &mut D) -> Option { let internal_id = generate_internal_id(d)?; let vni = d.produce::()?; - let v4_masklen = d.gen_u8(Bound::Included(&0), Bound::Included(&32))?; - let num_v4_cidrs = d.gen_u16(Bound::Included(&0), Bound::Included(&16))?; + let num_v4_cidrs = if self.wants(AddressFamily::V4) { + u16::from(d.gen_u8(Bound::Included(&0), Bound::Included(&self.max_subnets))?) + } else { + 0 + }; + let num_v6_cidrs = if self.wants(AddressFamily::V6) { + u16::from(d.gen_u8(Bound::Included(&0), Bound::Included(&self.max_subnets))?) + } else { + 0 + }; - let v6_masklen = d.gen_u8(Bound::Included(&0), Bound::Included(&128))?; - let num_v6_cidrs = d.gen_u16(Bound::Included(&0), Bound::Included(&16))?; - let v4_gen = UniqueV4CidrGenerator::new(num_v4_cidrs, v4_masklen); - let v6_gen = UniqueV6CidrGenerator::new(num_v6_cidrs, v6_masklen); + // The count first, then a length the subnet region can hold that many distinct prefixes at. + // The other order lets the region run short, and `private_run` would wrap and hand back the + // same prefix twice -- two subnets that overlap, which the validator refuses. + let v4_masklen = d.gen_u8( + Bound::Included(&blocks::min_subnet_len(AddressFamily::V4, num_v4_cidrs)), + Bound::Included(&32), + )?; + let v6_masklen = d.gen_u8( + Bound::Included(&blocks::min_subnet_len(AddressFamily::V6, num_v6_cidrs)), + Bound::Included(&128), + )?; - let subnets_cidrs = vec![v4_gen.generate(d)?, v6_gen.generate(d)?]; + let subnets_cidrs = vec![ + blocks::private_run(d, AddressFamily::V4, self.vpc, v4_masklen, num_v4_cidrs)?, + blocks::private_run(d, AddressFamily::V6, self.vpc, v6_masklen, num_v6_cidrs)?, + ]; let subnets = subnets_cidrs .into_iter() .flatten() @@ -47,10 +97,20 @@ impl TypeGenerator for LegalValue { }) .collect::>(); - Some(LegalValue(GatewayAgentVpcs { + Some(GatewayAgentVpcs { internal_id: Some(internal_id), vni: Some(vni.into()), subnets: Some(subnets).filter(|s| !s.is_empty()), - })) + }) + } +} + +/// Delegates to [`VpcGenerator`] with default knobs, so existing users keep working. +/// +/// Vpc zero, since a caller drawing one vpc at a time has nothing to keep it disjoint from. +impl TypeGenerator for LegalValue { + fn generate(d: &mut D) -> Option { + let families = AddressFamily::all(); + Some(LegalValue(VpcGenerator::new(0, 3, &families).generate(d)?)) } } diff --git a/mgmt/Cargo.toml b/mgmt/Cargo.toml index 482e1a1396..11fd49675e 100644 --- a/mgmt/Cargo.toml +++ b/mgmt/Cargo.toml @@ -55,11 +55,22 @@ tokio = { workspace = true, features = ["macros", "rt", "sync", "time"] } tracing = { workspace = true, features = ["attributes"] } tracing-test = { workspace = true } +[lints.rust] +# `cargo bolero` sets `--cfg fuzzing` for every coverage-guided engine it drives. A property whose +# assertions describe the *distribution* of its inputs -- how often a generated case is refused, say +# -- only holds under the uniform random engine, since a coverage-guided one skews its inputs toward +# whatever reaches new code. Such a property reads the cfg to tell which engine it is under. +unexpected_cfgs = { level = "warn", check-cfg = ['cfg(fuzzing)'] } + [dev-dependencies] # internal dpdk = { workspace = true, features = ["test"] } # EAL for tests that build the rte_acl-backed ACL filter and flow-filter context fixin = { workspace = true } id = { workspace = true, features = ["bolero"] } +# for the generated exposes and `GatewayAgent` CRDs the config-chain properties feed through the +# builder +config = { workspace = true, features = ["bolero"] } +k8s-intf = { workspace = true, features = ["bolero"] } interface-manager = { workspace = true, features = ["bolero"] } lpm = { workspace = true, features = ["testing"] } n-vm = { workspace = true } diff --git a/mgmt/src/processor/confbuild/internal.rs b/mgmt/src/processor/confbuild/internal.rs index fc108e4871..d1ea848f5d 100644 --- a/mgmt/src/processor/confbuild/internal.rs +++ b/mgmt/src/processor/confbuild/internal.rs @@ -311,7 +311,7 @@ fn build_internal_overlay_config( Ok(()) } -const EVPN_RMAP_NO_ADV_COMM: &str = "EVPN-ROUTE-MAP-NO-ADV-COMM"; +pub(crate) const EVPN_RMAP_NO_ADV_COMM: &str = "EVPN-ROUTE-MAP-NO-ADV-COMM"; /// Create a route-map that adds community "no-advertise" to all routes fn route_map_add_noadv_comm() -> RouteMap { @@ -392,3 +392,245 @@ pub fn build_internal_config( debug!("Successfully built internal config for genid {genid}"); Ok(internal) } + +/// Properties over the whole configuration chain. +/// +/// A gateway configuration passes through four steps before the dataplane sees it: +/// +/// ```text +/// GatewayAgent (CRD) ──▶ ExternalConfig ──▶ validated ──▶ InternalConfig ──▶ FRR text +/// ``` +/// +/// The converters and the validator are well covered, and the renderers now are too -- but the +/// renderers were tested against an `InternalConfig` built by hand, and this builder was covered by +/// one sample configuration in a test that printed its output. So the third arrow, the one that +/// turns a *validated* configuration into the one the dataplane applies, has never seen a generated +/// input. +/// +/// That arrow is where the interesting failure lives, and there is precedent: a port-forwarding +/// expose that validated and could not be built (`fix(config): Refuse a port-forwarding expose the +/// dataplane cannot build`). It matters because `apply_gw_config` is a linear `?`-chain and there is +/// no transaction: by the time a late step fails, kernel interfaces, the flow filter, ACLs, static +/// NAT and the masquerade allocator have all been committed. +/// +/// So the claim is: **whatever validates, builds; and whatever builds, renders.** +#[cfg(test)] +mod chain_properties { + use super::*; + use config::{ExternalConfig, GenId}; + use k8s_intf::bolero::AddressFamily; + use k8s_intf::bolero::crd::{GatewayAgentBuilder, GatewayAgents}; + use k8s_intf::gateway_agent_crd::GatewayAgent; + use routing::Render; + use std::collections::BTreeSet; + + /// The configurations these properties draw from: legal, and **IPv4 only**. + /// + /// # Why IPv4 only + /// + /// Because IPv6 peering configuration cannot be rendered, and these properties say so loudly the + /// moment they are allowed to see it. `internal.rs` never uses `IpVer::V6`: + /// + /// * the advertise prefix list is built `IpVer::V4` over *unfiltered* prefixes, so a v6 prefix + /// reaches `PrefixList::add_entry` and returns `ConfigError::InternalFailure`; + /// * the import prefix list is `IpVer::V4` *and* filtered by `is_ipv4()`, so v6 prefixes are + /// dropped in silence -- no error, and no route either. + /// + /// Neither was reachable from a test until the generated gateway began joining its own gateway + /// groups, because `build_routing_config_peer` builds nothing for a peering whose group does not + /// list this gateway. See `.scratch/ipv6-peering-exec-summary.md`. + /// + /// **Widening this one function to `AddressFamily::all()` is the check for whether that is fixed**, + /// and it is deliberately the only place any of these properties names a family. + fn ipv4_agents() -> GatewayAgents { + GatewayAgentBuilder::new() + .families(vec![AddressFamily::V4]) + .build() + } + + /// Everything the chain produces for one generated CRD, or `None` if the configuration was + /// legal as a CRD but not a valid gateway configuration. + /// + /// Validation failing is not a defect: `LegalValue` generates values that are legal against the + /// *schema*, and plenty of those describe configurations that are semantically wrong -- two vpcs + /// claiming one vni, exposes that overlap. The claim starts after validation succeeds. + fn chain(agent: &GatewayAgent) -> Option<(GenId, InternalConfig)> { + let external = ExternalConfig::try_from(agent) + .unwrap_or_else(|e| panic!("a schema-legal CRD did not convert: {e}")); + let validated = external.validate().ok()?; + let genid = validated.genid(); + let internal = build_internal_config(&validated, None).unwrap_or_else(|e| { + panic!("a validated configuration would not build: {e}\n{validated:#?}") + }); + Some((genid, internal)) + } + + /// A configuration that validates can be built and rendered. + /// + /// The rendering half is not incidental: `Render` returns no `Result`, so the only way it can + /// fail is to panic, and the renderers had only ever been given hand-written input. + #[test] + fn whatever_validates_builds_and_renders() { + bolero::check!() + .with_generator(ipv4_agents()) + .for_each(|agent| { + let Some((genid, internal)) = chain(agent) else { + return; + }; + let text = internal.render(&genid).to_string(); + assert!( + text.contains(&format!("! config for gen {genid}")), + "the rendered config does not say which generation it is for" + ); + }); + } + + /// The built configuration carries a vrf for exactly the vnis the overlay's vpcs have. + /// + /// This is the correspondence the third arrow is supposed to establish. A vpc without a vrf is a + /// tenant with no forwarding table; a vrf without a vpc is one FRR will configure and nothing + /// will use. + #[test] + fn every_vpc_gets_a_vrf_and_no_more() { + bolero::check!() + .with_generator(ipv4_agents()) + .for_each(|agent| { + let external = ExternalConfig::try_from(agent) + .unwrap_or_else(|e| panic!("a schema-legal CRD did not convert: {e}")); + let Ok(validated) = external.validate() else { + return; + }; + let internal = build_internal_config(&validated, None) + .unwrap_or_else(|e| panic!("a validated configuration would not build: {e}")); + + let wanted: BTreeSet = validated + .external() + .overlay() + .vpc_table() + .values() + .map(|vpc| vpc.vni().as_u32()) + .collect(); + let built: BTreeSet = internal + .vrfs + .iter_by_name() + .filter_map(|vrf| vrf.vni.map(|vni| vni.as_u32())) + .collect(); + assert_eq!(built, wanted, "vrfs do not match the vpcs they come from"); + }); + } + + /// The properties above are not vacuous, and say how far they reach. + /// + /// They are all of the form "if it validates, then ..." -- so they are worth nothing if nothing + /// validates. It is worth measuring rather than assuming, and the measurement turned out to be + /// the most useful thing in this module. + /// + /// This started out measuring a failure. When first written, about a sixth of configurations + /// validated and **none of them had a peering** -- twenty-four thousand peerings drawn per four + /// thousand configurations, not one surviving. Peerings are where the exposes, the NAT and the + /// ACLs live, so that whole half of the model reached the builder never, while the CRD + /// generators sat at 94% coverage and every per-converter property passed, because those run + /// before validation. + /// + /// Seven causes, all in the generators, all now fixed: peering pairs drawn independently so a + /// duplicated pair was near-certain; exposes drawing a mix of address families when one expose + /// must be single-family; prefixes as short as `/0`, which always overlaps a reserved range; the + /// NAT flavour chosen *after* the shape it constrains; the two manifests of a peering drawing + /// families independently when they must agree; both manifests free to use a stateful flavour + /// when only one may; and a peering naming a gateway group drawn freely rather than one that + /// exists. + /// + /// So the assertions are now the ones worth making: most configurations validate, and they carry + /// peerings. + #[test] + fn the_properties_are_not_vacuous() { + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + static SEEN: AtomicUsize = AtomicUsize::new(0); + static VALIDATED: AtomicUsize = AtomicUsize::new(0); + static VPCS: AtomicUsize = AtomicUsize::new(0); + static PEERINGS: AtomicUsize = AtomicUsize::new(0); + static ACLS: AtomicUsize = AtomicUsize::new(0); + + bolero::check!() + .with_generator(ipv4_agents()) + .for_each(|agent| { + SEEN.fetch_add(1, Ordering::Relaxed); + let external = ExternalConfig::try_from(agent) + .unwrap_or_else(|e| panic!("a schema-legal CRD did not convert: {e}")); + if let Ok(validated) = external.validate() { + VALIDATED.fetch_add(1, Ordering::Relaxed); + let table = validated.external().overlay().vpc_table(); + VPCS.fetch_add(table.len(), Ordering::Relaxed); + PEERINGS.fetch_add( + table + .values() + .map(|vpc| vpc.peerings().len()) + .sum::(), + Ordering::Relaxed, + ); + ACLS.fetch_add( + table + .values() + .flat_map(|vpc| vpc.peerings()) + .filter(|peering| peering.acl().is_some()) + .count(), + Ordering::Relaxed, + ); + } + }); + + let seen = SEEN.load(Ordering::Relaxed); + let validated = VALIDATED.load(Ordering::Relaxed); + let vpcs = VPCS.load(Ordering::Relaxed); + let peerings = PEERINGS.load(Ordering::Relaxed); + let acls = ACLS.load(Ordering::Relaxed); + println!( + "{validated}/{seen} validated, carrying {vpcs} vpcs, {peerings} peerings, {acls} acls" + ); + assert!(seen > 0, "no configurations were generated"); + assert!( + validated * 2 >= seen, + "only {validated} of {seen} configurations validated: the properties above are \ + checking much less than they look like they are" + ); + assert!(vpcs > validated, "validated configurations carry no vpcs"); + assert!( + peerings > 0, + "no validated configuration carries a peering, so nothing downstream of validation \ + has seen the exposes or the NAT" + ); + // Not merely "more than none": a *share* of them. An ACL rule can be refused for reasons + // that have nothing to do with the rule -- `scope: flow` needs one side of the peering to be + // stateful throughout -- so a generator that gets those wrong still produces some valid + // ACLs, just far fewer. Asserting the share is what notices that. + assert!( + acls * 2 >= validated, + "only {acls} of {validated} validated configurations carry an ACL: most generated ACLs \ + are being refused for something other than what they say" + ); + } + + /// Building and rendering the same configuration twice gives the same text. + /// + /// Already checked over hand-built `InternalConfig`s; here the input comes from a generated CRD, + /// so the whole chain has to be deterministic, not just the last step of it. It matters because + /// `frr-reload.py` diffs the output against what FRR is running. + #[test] + fn the_chain_is_deterministic() { + bolero::check!() + .with_generator(ipv4_agents()) + .for_each(|agent| { + let Some((genid, once)) = chain(agent) else { + return; + }; + let (_, twice) = chain(agent).unwrap_or_else(|| { + panic!("the same CRD validated once and not the second time") + }); + assert_eq!( + once.render(&genid).to_string(), + twice.render(&genid).to_string(), + "the configuration chain is not deterministic" + ); + }); + } +} diff --git a/mgmt/src/tests/mgmt.rs b/mgmt/src/tests/mgmt.rs index 3aa26792a1..9880760933 100644 --- a/mgmt/src/tests/mgmt.rs +++ b/mgmt/src/tests/mgmt.rs @@ -171,7 +171,7 @@ pub mod test { } /* DEVICE configuration */ - fn sample_device_config() -> DeviceConfig { + pub(super) fn sample_device_config() -> DeviceConfig { DeviceConfig::new() } @@ -319,7 +319,7 @@ pub mod test { } /* build sample underlay config */ - fn sample_underlay_config() -> Underlay { + pub(super) fn sample_underlay_config() -> Underlay { /* main loopback for BGP and vtep */ let loopback = IpAddr::from_str("7.0.0.100").expect("Bad address"); let router_id = get_v4_addr(loopback); @@ -333,7 +333,7 @@ pub mod test { } #[rustfmt::skip] - fn sample_gw_groups() -> GwGroupTable { + pub(super) fn sample_gw_groups() -> GwGroupTable { let mut gwt = GwGroupTable::new(); let mut group = GwGroup::new("gw-group-1"); group.add_member(GwGroupMember::new("gw1", 1, IpAddr::from_str("172.128.0.1").unwrap())).unwrap(); @@ -348,7 +348,7 @@ pub mod test { gwt } - fn sample_community_table() -> PriorityCommunityTable { + pub(super) fn sample_community_table() -> PriorityCommunityTable { let mut comtable = PriorityCommunityTable::new(); comtable.insert(0, "65000:800").unwrap(); comtable.insert(1, "65000:801").unwrap(); @@ -508,3 +508,1062 @@ pub mod test { router.stop(); } } + +/// Properties over the third arrow of the configuration chain, with peerings. +/// +/// A gateway configuration passes through +/// +/// ```text +/// GatewayAgent (CRD) ─▶ ExternalConfig ─▶ validated ─▶ InternalConfig ─▶ FRR +/// ``` +/// +/// and `build_internal_config` -- the third arrow -- had only ever been given hand-built input: +/// `test::check_frr_config` above builds one sample configuration, renders it and prints the result. +/// +/// The properties in `processor::confbuild::internal` drive the whole chain from a generated +/// `GatewayAgent`, but measuring them showed that **no generated configuration with a peering ever +/// survives validation**, so the arrow has never been exercised for the half of the model that +/// carries the exposes, the NAT and the ACLs. Fixing that in the CRD generators is a separate piece +/// of work. +/// +/// This gets at the same question from the other side, and today rather than after that work: +/// `config`'s own contract generators already produce exposes that are valid *by construction* for +/// each of the three NAT flavours, so an overlay built around one of those, spliced into the sample +/// underlay, reaches the builder with a peering in it. +/// +/// The claim is the one that matters: **a configuration that validates can be built and rendered.** +/// A configuration that validates and then fails to build is a half-applied dataplane -- +/// `apply_gw_config` is a linear `?`-chain with no transaction, so by the time a late step fails the +/// kernel interfaces, the flow filter, the ACLs, static NAT and the masquerade allocator have all +/// been committed, and rolling the configuration back does not restore the masquerade flows already +/// torn down. +#[cfg(test)] +mod peering_chain { + use bolero::{Driver, ValueGenerator}; + use config::ExternalConfig; + use config::external::ExternalConfigBuilder; + use config::external::gwgroup::{GwGroup, GwGroupMember, GwGroupTable}; + use config::external::overlay::vpcpeering::VpcExpose; + use config::external::overlay::vpcpeering::contract::{ + LOCAL_VNI, MasqueradeExpose, PortForwardingExpose, REMOTE_VNI, StaticNatExpose, + overlay_with_exposes, + }; + use routing::Render; + use std::net::IpAddr; + use std::ops::Bound::Included; + use std::str::FromStr; + + use super::test::{ + sample_community_table, sample_device_config, sample_gw_groups, sample_underlay_config, + }; + use crate::processor::confbuild::internal::{EVPN_RMAP_NO_ADV_COMM, build_internal_config}; + + /// Which NAT flavour a generated expose uses. + #[derive(Debug, Clone, Copy, PartialEq, Eq)] + enum Flavour { + PortForwarding, + Masquerade, + Static, + } + + /// The most exposes one generated manifest offers. + /// + /// More than one because the interesting rejections, and the interesting things for the builder + /// to get wrong, are *between* exposes rather than within one. + const MAX_EXPOSES: u8 = 3; + + /// The autonomous system the sample underlay uses, so the rendered text can be checked for it. + const UNDERLAY_ASN: u32 = 65000; + + /// Draws exposes of each NAT flavour, each valid by construction, for one manifest. + #[derive(Debug, Clone, Copy, Default)] + struct AnyNatExposes; + + impl ValueGenerator for AnyNatExposes { + type Output = Vec<(Flavour, VpcExpose)>; + + fn generate(&self, driver: &mut D) -> Option { + let count = driver.gen_u8(Included(&1), Included(&MAX_EXPOSES))?; + let mut out = Vec::with_capacity(usize::from(count)); + for _ in 0..count { + out.push(match driver.gen_u8(Included(&0), Included(&2))? { + 0 => ( + Flavour::PortForwarding, + PortForwardingExpose.generate(driver)?, + ), + 1 => (Flavour::Masquerade, MasqueradeExpose.generate(driver)?), + _ => (Flavour::Static, StaticNatExpose.generate(driver)?), + }); + } + Some(out) + } + } + + /// The sample gateway groups, plus the `default` group the contract module's peering names. + /// + /// `VpcPeering::with_default_group` sets the group to `"default"`, and whole-config validation + /// checks that a peering's group exists. Validating the overlay on its own -- which is what + /// `overlay_offering` does, and what every existing user of these generators does -- cannot make + /// that check, because the group table sits beside the overlay rather than in it. So an overlay + /// those generators produce is not embeddable in a whole configuration without this. + fn gw_groups_with_default() -> GwGroupTable { + let mut groups = sample_gw_groups(); + let mut default = GwGroup::new("default"); + default + .add_member(GwGroupMember::new( + "gw-default", + 1, + IpAddr::from_str("172.128.0.9").unwrap_or_else(|_| unreachable!()), + )) + .unwrap_or_else(|e| unreachable!("{e}")); + groups + .add_group(default) + .unwrap_or_else(|e| unreachable!("{e}")); + groups + } + + /// A whole configuration: the sample underlay, and an overlay whose local vpc offers `exposes`. + fn external_offering(exposes: Vec) -> ExternalConfig { + let overlay = overlay_with_exposes(exposes).unwrap_or_else(|e| unreachable!("{e}")); + ExternalConfigBuilder::default() + .gwname("test-gw".to_string()) + .genid(1) + .device(sample_device_config()) + .underlay(sample_underlay_config()) + .overlay(overlay) + .gwgroups(gw_groups_with_default()) + .communities(sample_community_table()) + .build() + .unwrap_or_else(|e| unreachable!("{e}")) + } + + /// A configuration carrying a peering with NAT validates, builds, and renders. + /// + /// A single expose is always accepted -- the generators make it valid by construction -- but + /// several together may legitimately be refused, for overlapping prefixes or a port range + /// claimed twice. Those are skipped rather than treated as findings, and counted so that the + /// property cannot quietly become vacuous. + #[test] + fn a_config_with_a_nat_peering_builds_and_renders() { + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + static SEEN: AtomicUsize = AtomicUsize::new(0); + static BUILT: AtomicUsize = AtomicUsize::new(0); + static MULTI: AtomicUsize = AtomicUsize::new(0); + + bolero::check!() + .with_generator(AnyNatExposes) + .cloned() + .for_each(|offered: Vec<(Flavour, VpcExpose)>| { + SEEN.fetch_add(1, Ordering::Relaxed); + let flavours: Vec = offered.iter().map(|(f, _)| *f).collect(); + let exposes: Vec = offered.iter().map(|(_, e)| e.clone()).collect(); + let external = external_offering(exposes.clone()); + + let validated = match external.validate() { + Ok(validated) => validated, + Err(e) => { + // one expose on its own is valid by construction, so a rejection there is a + // finding; several may disagree with each other, which is not + assert!( + exposes.len() > 1, + "a single {:?} expose that is valid by construction was refused: {e}\n{exposes:#?}", + flavours[0] + ); + return; + } + }; + BUILT.fetch_add(1, Ordering::Relaxed); + if exposes.len() > 1 { + MULTI.fetch_add(1, Ordering::Relaxed); + } + + let internal = build_internal_config(&validated, None).unwrap_or_else(|e| { + panic!("a validated {flavours:?} configuration would not build: {e}\n{exposes:#?}") + }); + + // both vpcs of the peering reach the built configuration + let vnis: Vec = internal + .vrfs + .iter_by_name() + .filter_map(|vrf| vrf.vni.map(|vni| vni.as_u32())) + .collect(); + for vni in [LOCAL_VNI, REMOTE_VNI] { + assert!( + vnis.contains(&vni), + "vni {vni} of the peering is missing from the built config, got {vnis:?}" + ); + } + + // the underlay's own vrf reaches the built config too. Checking the overlay vnis + // alone would not notice it going missing, since the default vrf carries no vni. + assert!( + internal.vrfs.default_vrf_config().is_some(), + "the built config has no default vrf" + ); + + // the community table is carried across unchanged. It is copied rather than derived, + // so nothing else in this property would notice it being dropped -- and the + // communities are what the overlay's route maps tag routes with. + for order in 0..5 { + assert_eq!( + internal.commtable.get_community(order), + validated.external().communities().get_community(order), + "community {order} did not survive the build" + ); + } + + // the route-map that keeps learnt evpn routes from being re-advertised. Only + // `configure_bgp_peers` adds it, and it is what applies to the underlay neighbours' + // l2vpn-evpn address family -- so its absence means the neighbours were never + // configured, which the vni checks above would not notice. + assert!( + internal + .rmap_table + .values() + .any(|rmap| rmap.name == EVPN_RMAP_NO_ADV_COMM), + "the evpn no-advertise route-map is missing from the built config" + ); + + let text = internal.render(&validated.genid()).to_string(); + assert!( + text.contains("! config for gen 1"), + "the rendered config does not say which generation it is for" + ); + assert!( + text.contains(EVPN_RMAP_NO_ADV_COMM), + "the evpn no-advertise route-map is missing from the rendered config" + ); + // and the underlay's bgp instance reaches the rendered text, which is what ties the + // asn the overlay vrfs are built with back to the configuration it came from + assert!( + text.contains(&format!("router bgp {UNDERLAY_ASN}")), + "the underlay bgp instance is missing from the rendered config" + ); + for vni in [LOCAL_VNI, REMOTE_VNI] { + assert!( + text.contains(&format!(" vni {vni}")), + "vni {vni} is missing from the rendered config" + ); + } + }); + + let seen = SEEN.load(Ordering::Relaxed); + let built = BUILT.load(Ordering::Relaxed); + let multi = MULTI.load(Ordering::Relaxed); + println!("{built}/{seen} configurations built, {multi} of them with several exposes"); + assert!( + built * 2 >= seen, + "most configurations were skipped: {built}/{seen}" + ); + assert!( + multi > 0, + "no configuration with more than one expose was built" + ); + } +} + +/// Properties over the *rest* of applying a configuration: the dataplane tables. +/// +/// `build_internal_config` turns a validated configuration into the FRR half of it, and the chain +/// properties in `processor::confbuild::internal` cover that. The other half is the dataplane's own +/// tables, built from the same validated configuration by +/// +/// * `build_nat_configuration` -- static NAT, +/// * `MasqueradeConfig::new` and `update_nat_allocator` -- masquerade, +/// * `build_port_forwarding_configuration` and `PortFwTableWriter::update_table` -- port forwarding. +/// +/// All of them are fallible from a configuration that has already validated, and the last is where +/// the one confirmed bug of this class actually fired: `PortFwEntry::is_valid` refused a rule whose +/// two sides had matching address-port *totals* but mismatched prefix lengths, during apply, at the +/// last of the NAT stages -- by which point the kernel interfaces, the flow filter, the ACLs, the +/// static NAT tables and the masquerade allocator had all been committed. See +/// `fix(config): Refuse a port-forwarding expose the dataplane cannot build`. +/// +/// So the claim is the same one, carried one step further: **a configuration that validates builds +/// every table it implies.** +/// +/// One property per NAT flavour rather than one over all of them, using the generator knobs. A +/// single property over the default flavour mix reaches each flavour eventually; asking for one +/// reaches it in every case, and says in its name which one failed. +#[cfg(test)] +mod dataplane_tables { + use config::{ExternalConfig, ValidatedGwConfig}; + use flow_entry::flow_table::FlowTable; + use k8s_intf::bolero::NatFlavour; + use k8s_intf::bolero::crd::GatewayAgentBuilder; + use nat::masquerade::{MasqueradeConfig, NatAllocatorWriter}; + use nat::portfw::{PortFwTableWriter, build_port_forwarding_configuration}; + use nat::static_nat::NatTablesWriter; + use nat::static_nat::setup::build_nat_configuration; + + /// Build every dataplane table the configuration implies, in the order `apply_gw_config` does. + fn build_tables(validated: &ValidatedGwConfig, flavour: NatFlavour) { + let vpc_table = validated.external().overlay().vpc_table(); + + let nat_tables = build_nat_configuration(vpc_table).unwrap_or_else(|e| { + panic!("a validated {flavour:?} configuration would not build static NAT: {e}") + }); + let mut nattablesw = NatTablesWriter::new(); + nattablesw.update_nat_tables(nat_tables); + + let masquerade = MasqueradeConfig::new(vpc_table).set_randomize(false); + let mut natallocatorw = NatAllocatorWriter::new(); + let flow_table = FlowTable::new(16); + natallocatorw.update_nat_allocator(masquerade, validated.genid(), &flow_table); + + let ruleset = build_port_forwarding_configuration(vpc_table).unwrap_or_else(|e| { + panic!("a validated {flavour:?} configuration would not build port forwarding: {e}") + }); + let mut portfw_w = PortFwTableWriter::new(); + // The step that used to fail during apply: the ruleset builds and the table still refuses + // it, because a rule can be well-formed as configuration and unrepresentable as a rule. + portfw_w.update_table(&ruleset).unwrap_or_else(|e| { + panic!("a validated {flavour:?} port-forwarding ruleset was refused by the table: {e}") + }); + } + + /// Drive the whole chain for one NAT flavour, and report how much of it got through. + fn drive(flavour: NatFlavour) { + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + let seen = AtomicUsize::new(0); + let built = AtomicUsize::new(0); + + let generator = GatewayAgentBuilder::new().flavours(vec![flavour]).build(); + + bolero::check!() + .with_generator(generator) + .cloned() + .for_each(|agent| { + seen.fetch_add(1, Ordering::Relaxed); + let external = ExternalConfig::try_from(&agent) + .unwrap_or_else(|e| panic!("a schema-legal CRD did not convert: {e}")); + let Ok(validated) = external.validate() else { + return; + }; + built.fetch_add(1, Ordering::Relaxed); + build_tables(&validated, flavour); + }); + + let seen = seen.load(Ordering::Relaxed); + let built = built.load(Ordering::Relaxed); + println!("{flavour:?}: {built}/{seen} configurations validated and built their tables"); + assert!( + built * 2 >= seen, + "only {built} of {seen} {flavour:?} configurations validated, so this checked much \ + less than it looks like it did" + ); + } + + #[test] + fn a_static_nat_configuration_builds_its_tables() { + drive(NatFlavour::Static); + } + + #[test] + fn a_masquerade_configuration_builds_its_tables() { + drive(NatFlavour::Masquerade); + } + + /// The flavour the historical bug was in. + #[test] + fn a_port_forwarding_configuration_builds_its_tables() { + drive(NatFlavour::PortForward); + } + + #[test] + fn a_configuration_with_no_nat_builds_its_tables() { + drive(NatFlavour::None); + } +} + +/// The property the whole validation chain exists for. +/// +/// The validator ships as a wasm module in a process of its own. Its entire surface is +/// +/// ```text +/// ExternalConfig::try_from(&crd)?.validate()? +/// ``` +/// +/// -- convert, then validate, and nothing else. If it blesses a configuration, that process writes +/// the configuration to Kubernetes. The dataplane runs the same two steps later, but it has no way to +/// tell anyone that something is wrong: by then the configuration is already the desired state. +/// +/// So the requirement is not "the dataplane reports bad configurations well". It is: +/// +/// > **anything the validator accepts must be enactable.** +/// +/// A validator that is too strict is a nuisance -- the user sees an error and fixes their input. A +/// validator that is too permissive is unrecoverable, and every check that lives only in a +/// downstream builder is exactly that kind of gap. Nothing downstream of `validate` can help. +/// +/// A panic is the same failure in a different coat: in wasm it traps, so the calling process gets a +/// failure with no `ValidateError` in it and the user gets nothing to act on. +/// +/// The generator here is the near-miss one: a legal configuration with **one** rule deliberately +/// broken, which is the input most likely to slip past. The valid-by-construction properties +/// elsewhere test everything downstream of validation; these test validation itself. +#[cfg(test)] +/// The validator, and everything a blessed configuration makes the dataplane install. +/// +/// Shared by the three properties below rather than written out in each, since they differ in what they +/// *ask* of these artifacts and not in how the artifacts are obtained. +#[cfg(test)] +mod enacted { + use config::{ConfigError, ExternalConfig, ValidatedGwConfig}; + use flow_entry::flow_table::FlowTable; + use k8s_intf::gateway_agent_crd::GatewayAgent; + use nat::masquerade::{MasqueradeConfig, NatAllocatorWriter}; + use nat::portfw::{PortFwTableWriter, build_port_forwarding_configuration}; + use nat::static_nat::setup::build_nat_configuration; + use routing::Render; + + use crate::processor::confbuild::internal::build_internal_config; + + /// Exactly what the wasm validator does, and nothing more. + /// + /// The conversion's error type is not `ConfigError`, and a conversion failure is a rejection just as + /// much as a validation failure is, so it is folded in here. + pub(super) fn validator(crd: &GatewayAgent) -> Result { + let external = ExternalConfig::try_from(crd) + .map_err(|e| ConfigError::Invalid(format!("conversion: {e}")))?; + external.validate() + } + + /// What the dataplane installs, **kept apart by artifact**. + /// + /// Separate rather than merged, and that matters. Every expose contributes prefixes to the FRR + /// render whatever else it does, so a single merged blob would show a difference even where a NAT + /// builder had ignored the expose entirely -- which is the failure worth catching. Asking each + /// artifact its own question is what makes that visible. + pub(super) struct Artifacts { + pub frr: Vec, + pub static_nat: Vec, + pub port_forwarding: Vec, + pub masquerade: Vec, + } + + /// Text as a sorted multiset of its meaningful lines. + /// + /// Sorted because some of these tables are hash maps, whose iteration order is not part of a + /// configuration's meaning; comparing it would give false alarms. That costs nothing that matters, + /// since the artifacts whose order *is* semantic carry their sequence numbers in the text, so + /// reordering them changes the lines themselves. + fn lines(text: &str) -> Vec { + let mut out: Vec = text + .lines() + .map(|line| line.trim_end().to_string()) + .filter(|line| !line.is_empty()) + .collect(); + out.sort(); + out + } + + impl Artifacts { + /// Build every artifact, or `None` if any builder refuses. + /// + /// A refusal is not this module's business: `validator_completeness` is the property that says + /// a blessed configuration must build, and it says so with a panic. + pub(super) fn of(validated: &ValidatedGwConfig) -> Option { + let genid = validated.genid(); + let internal = build_internal_config(validated, None).ok()?; + let vpc_table = validated.external().overlay().vpc_table(); + + let nat_tables = build_nat_configuration(vpc_table).ok()?; + let ruleset = build_port_forwarding_configuration(vpc_table).ok()?; + let mut portfw = PortFwTableWriter::new(); + portfw.update_table(&ruleset).ok()?; + + let masquerade = MasqueradeConfig::new(vpc_table).set_randomize(false); + let mut writer = NatAllocatorWriter::new(); + writer.update_nat_allocator(masquerade, genid, &FlowTable::new(16)); + let allocator = writer.get_reader().get(); + + Some(Self { + frr: lines(&internal.render(&genid).to_string()), + static_nat: lines(&nat_tables.to_string()), + port_forwarding: lines( + &ruleset + .iter() + .map(ToString::to_string) + .collect::>() + .join("\n"), + ), + masquerade: allocator + .map(|allocator| lines(&allocator.to_string())) + .unwrap_or_default(), + }) + } + + /// Every artifact's lines together, for a caller that only wants to know whether *anything* + /// differs. + pub(super) fn all(&self) -> Vec { + let mut out = self.frr.clone(); + out.extend(self.static_nat.iter().cloned()); + out.extend(self.port_forwarding.iter().cloned()); + out.extend(self.masquerade.iter().cloned()); + out.sort(); + out + } + } +} + +mod validator_completeness { + use super::enacted::validator; + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + use config::{ConfigError, ValidatedGwConfig}; + use flow_entry::flow_table::FlowTable; + use k8s_intf::bolero::AddressFamily; + use k8s_intf::bolero::crd::GatewayAgentBuilder; + use k8s_intf::bolero::mutate::{MutatedAgents, Mutation}; + use k8s_intf::gateway_agent_crd::GatewayAgent; + use nat::masquerade::{MasqueradeConfig, NatAllocatorWriter}; + use nat::portfw::{PortFwTableWriter, build_port_forwarding_configuration}; + use nat::static_nat::NatTablesWriter; + use nat::static_nat::setup::build_nat_configuration; + use routing::Render; + + use crate::processor::confbuild::internal::build_internal_config; + + /// Everything the dataplane has to be able to do with a blessed configuration. + /// + /// Any error here is the failure this module exists to find: the validator said yes and the + /// dataplane cannot comply, with nowhere to report it. + fn enact(validated: &ValidatedGwConfig, mutation: Mutation) { + let genid = validated.genid(); + + let internal = build_internal_config(validated, None).unwrap_or_else(|e| { + panic!("{mutation:?}: validator accepted a config the builder rejects: {e}") + }); + let _ = internal.render(&genid).to_string(); + + let vpc_table = validated.external().overlay().vpc_table(); + + let nat_tables = build_nat_configuration(vpc_table).unwrap_or_else(|e| { + panic!("{mutation:?}: validator accepted a config static NAT rejects: {e}") + }); + NatTablesWriter::new().update_nat_tables(nat_tables); + + let masquerade = MasqueradeConfig::new(vpc_table).set_randomize(false); + NatAllocatorWriter::new().update_nat_allocator(masquerade, genid, &FlowTable::new(16)); + + let ruleset = build_port_forwarding_configuration(vpc_table).unwrap_or_else(|e| { + panic!("{mutation:?}: validator accepted a config port forwarding rejects: {e}") + }); + PortFwTableWriter::new() + .update_table(&ruleset) + .unwrap_or_else(|e| { + panic!("{mutation:?}: validator accepted a ruleset the table rejects: {e}") + }); + } + + /// Whatever the validator accepts, the dataplane can enact. + /// + /// **What this property cannot see.** It catches permissiveness that leads to a configuration the + /// dataplane cannot build. It does not catch permissiveness about rules with no downstream + /// enforcement -- `Mutation::OverlapWithAnotherPeer` is the worked example: deleting the + /// `OverlappingPrefixes` check from `VpcRouteTable::validate` leaves this property passing, + /// because two routes to one destination build perfectly well and the dataplane simply picks one. + /// Those rules are about *ambiguity* rather than *feasibility*, and policing them needs a + /// companion property -- "a mutation that breaks a rule must be refused" -- which in turn needs + /// each mutation to report whether the case it built is certainly illegal, since several of them + /// are legitimately legal some of the time. + /// + /// Also: it never panics, since reaching the assertions at all means it returned. In wasm a panic + /// is a trap, so it is a rejection with no reason attached -- worse for the user than any error. + #[test] + /// + /// # Why IPv4 only + /// + /// Over both families this fails in the first second, on a *legal* configuration, for a reason + /// that is a real defect and not this property's to fix: `internal.rs` renders no IPv6 peering + /// configuration at all. The advertise prefix list is built as `IpVer::V4` with unfiltered + /// prefixes, so a v6 prefix reaches `PrefixList::add_entry` and comes back as + /// `ConfigError::InternalFailure`; the import list is `IpVer::V4` *and* filtered by `is_ipv4()`, + /// so v6 prefixes are dropped without a word. See `.scratch/next-phase-assessment.md`. + /// + /// **Widening this back to `AddressFamily::all()` is how to check whether that is fixed**, and is + /// the only change needed. The restriction lives here rather than in a second, ignored test + /// because `cargo bolero` names a fuzz target after the function holding the `check!()`: two tests + /// sharing one body collapse to a single target, and then neither can be fuzzed. + fn whatever_the_validator_accepts_can_be_enacted_over_ipv4() { + let families = vec![AddressFamily::V4]; + + // What each mutation did, so the run can say whether the generator is doing any work: how + // often it was drawn, how often it found a target, and how often the result was refused. + // Counters per mutation rather than a map behind a lock, since a lock is not wanted here. + const N: usize = Mutation::COUNT; + #[allow(clippy::declare_interior_mutable_const)] + const ZERO: AtomicUsize = AtomicUsize::new(0); + static DRAWN: [AtomicUsize; N] = [ZERO; N]; + static APPLIED: [AtomicUsize; N] = [ZERO; N]; + static REFUSED: [AtomicUsize; N] = [ZERO; N]; + + bolero::check!() + .with_generator(MutatedAgents::new( + GatewayAgentBuilder::new().families(families).build(), + )) + .cloned() + .for_each(|(mutation, bit, agent): (Mutation, bool, GatewayAgent)| { + let outcome = validator(&agent); + let accepted = outcome.is_ok(); + + if let Ok(validated) = &outcome { + // A mutation reports `true` only when the result is *certainly* illegal, so the + // validator accepting it is a hole. This is the only guard against the validator + // growing more permissive about a rule nothing downstream enforces: for those, + // `enact` below succeeds happily and says nothing, which is exactly what the + // green break test on `OverlappingPrefixes` demonstrated. + assert!( + !bit, + "{mutation:?} broke a rule outright and the validator accepted the result, \ + so nothing downstream will ever report it" + ); + enact(validated, mutation); + } else if let Err(e) = &outcome { + // The control is legal by construction, so a rejection of it is a defect in the + // *generator*, and asserted rather than counted so bolero shrinks it. Counting it + // instead hid a real one for a long while: the tally said "6% of controls + // refused", which reads as tolerable noise, and finding the cause from a tally + // means reading configurations by eye. Asserted, the shrinker hands over a + // two-line counterexample. + assert!( + mutation != Mutation::None, + "an unmutated configuration was refused, so the generator is producing \ + illegal input and every mutated case is suspect: {e}" + ); + + // A rejection has to say something the user can act on. An internal failure says + // "this is our bug", which is not something anyone can fix from the outside. + assert!( + !matches!(e, ConfigError::InternalFailure(_)), + "{mutation:?}: rejected with an internal failure, which tells the user \ + nothing they can act on: {e}" + ); + } + + let slot = mutation.index(); + DRAWN[slot].fetch_add(1, Ordering::Relaxed); + if bit { + APPLIED[slot].fetch_add(1, Ordering::Relaxed); + } + if !accepted { + REFUSED[slot].fetch_add(1, Ordering::Relaxed); + } + }); + + // Everything from here on describes the *distribution* of the inputs, which is the random + // engine's contract and not a coverage-guided one's. libfuzzer keeps a corpus and steers + // toward inputs that reach new code, so it will happily spend a run replaying one mutation + // ten thousand times -- the right behaviour for finding a gap, and fatal to a check that + // every mutation gets drawn. The property above is what a fuzzing engine is here to break; + // these are here to catch the generator rotting, which only the random engine can see. + #[cfg(fuzzing)] + println!("under a coverage-guided engine: skipping the generator-health checks"); + + #[cfg(not(fuzzing))] + check_generator_health(&DRAWN, &APPLIED, &REFUSED); + } + + /// The generator-health checks, which only hold under the uniform random engine. + /// + /// Split out so the `cfg` above gates one call rather than half a function body. + #[cfg(not(fuzzing))] + fn check_generator_health( + drawn_at: &[AtomicUsize; Mutation::COUNT], + applied_at: &[AtomicUsize; Mutation::COUNT], + refused_at: &[AtomicUsize; Mutation::COUNT], + ) { + let mut total = 0; + for mutation in Mutation::all() { + let slot = mutation.index(); + let drawn = drawn_at[slot].load(Ordering::Relaxed); + let applied = applied_at[slot].load(Ordering::Relaxed); + let refused = refused_at[slot].load(Ordering::Relaxed); + println!("{mutation:<32?} {drawn:>7} drawn {applied:>7} applied {refused:>7} refused"); + total += drawn; + } + + // Nothing here about rejection rates. The control is asserted never to be refused and an + // applied mutation is asserted always to be, per case, which is both stronger and + // shrinkable; a ratio over the whole run could only restate them more weakly. + // + // What is left is coverage of the mutations themselves, and that needs a sample big enough to + // expect one. The default run is a second long, which is a few hundred cases across fourteen + // mutations -- too few to conclude anything from a mutation's absence, and the thresholds say + // so rather than letting a short run either fail spuriously or look like it checked. + let cases_for_drawn = 50 * Mutation::COUNT; + let cases_for_applied = 2_000 * Mutation::COUNT; + if total < cases_for_drawn { + println!( + "only {total} cases: too few to say anything about mutation coverage \ + (needs {cases_for_drawn} to check each was drawn, {cases_for_applied} to check each \ + found a target)" + ); + return; + } + + for mutation in Mutation::all() { + assert!( + drawn_at[mutation.index()].load(Ordering::Relaxed) > 0, + "{mutation:?} was never drawn in {total} cases" + ); + } + + if total < cases_for_applied { + println!( + "{total} cases: enough to check every mutation was drawn, too few to check each \ + found a target (needs {cases_for_applied})" + ); + return; + } + + // A mutation that has quietly stopped finding anything to break still shows up as drawn. The + // rarest is `OverlapWithAnotherPeer`, which needs two peerings sharing a vpc and an expose on + // each that advertises its `ips` verbatim; it applies to about one draw in forty. + for mutation in Mutation::all().into_iter().filter(|m| *m != Mutation::None) { + assert!( + applied_at[mutation.index()].load(Ordering::Relaxed) > 0, + "{mutation:?} never found anything to break in {total} cases, so it is testing \ + nothing" + ); + } + } +} + +/// A validated configuration must have exactly *one* meaning. +/// +/// [`super::validator_completeness`] asks whether an accepted configuration can be **enacted**. This +/// asks whether it can be enacted only one way, which is a different failure and a worse one. An +/// unenactable configuration fails to build, and something gets an error. An ambiguous one builds +/// perfectly: two readings, both valid outputs of the code as written, and the chain takes whichever +/// its containers hand it first. There is no error to report, so nothing reports it -- only traffic +/// going somewhere nobody chose, discovered much later. +/// +/// The oracle is permutation, and its virtue is that it restates no rule: it notices an ambiguity +/// whether or not anyone thought to forbid it. See `k8s_intf::bolero::permute`. +/// +/// # What this catches, and what it does not +/// +/// It catches ambiguity the chain resolves **at build time**: two readings of the input, one artifact, +/// and a container's iteration order deciding which. That is a real class and this is a real guard on +/// it, run over near-misses so the question is asked of configurations the validator was willing to +/// bless rather than only of ones the generator took care to keep clean. +/// +/// It does **not** catch ambiguity the chain resolves at *run* time, and the break test says so +/// plainly. Delete the `OverlappingPrefixes` check from `VpcRouteTable::validate`, so that two peers +/// of one vpc may advertise the same destination, and this property stays silent across 13,908 +/// comparisons. The reason is structural: an import prefix-list is rendered per peer, so both routes +/// are installed, in two lists, and the rendered configuration is **the same whichever order the +/// peerings are walked in**. Nothing was silently picked at build time. The picking happens later, in +/// the forwarding plane, on a packet. +/// +/// So the ambiguity concern has two halves and this is one of them: +/// +/// * **build-time** -- one artifact, two possible contents. Covered here. +/// * **run-time** -- one artifact, two rules inside it that match the same packet. *Not* covered, by +/// anything, and it is the half that misbehaves in production rather than in a build. +/// +/// The second needs a different oracle: a check over the *installed tables* that no two entries match +/// one packet with different actions. Worth knowing before writing it that for a rule with no +/// downstream consumer -- and the overlap rule is exactly that, as its green break test showed -- such +/// a check is necessarily a second statement of the requirement rather than an independent one. It +/// would still earn its place, since it would fail if the validator regressed, but it should be +/// written knowing what it is. +mod ambiguity { + use super::enacted::{Artifacts, validator}; + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + use k8s_intf::bolero::mutate::Mutation; + use k8s_intf::bolero::permute::PermutedAgents; + use k8s_intf::gateway_agent_crd::GatewayAgent; + + /// Reordering a configuration's sets does not change what it means. + /// + /// IPv4 only, for the same reason as `relevance`: this renders every artifact twice per case, and + /// `NatAllocator`'s `Display` never returns on an IPv6 masquerade pool. See that property's note. + #[test] + fn a_configuration_has_only_one_meaning() { + static MOVED: AtomicUsize = AtomicUsize::new(0); + static COMPARED: AtomicUsize = AtomicUsize::new(0); + + bolero::check!() + .with_generator(PermutedAgents::new( + k8s_intf::bolero::mutate::MutatedAgents::new( + k8s_intf::bolero::crd::GatewayAgentBuilder::new() + .families(vec![k8s_intf::bolero::AddressFamily::V4]) + .build(), + ), + )) + .cloned() + .for_each( + |(mutation, agent, permuted, moved): (Mutation, GatewayAgent, GatewayAgent, bool)| { + let Ok(first) = validator(&agent) else { + // Refused, which is the validator doing its job. Only what it *accepts* has to + // have one meaning. + return; + }; + + // A permutation cannot make a legal configuration illegal. If it does, the validator + // is reading order as meaning, which is its own kind of ambiguity. + let second = validator(&permuted).unwrap_or_else(|e| { + panic!( + "{mutation:?}: reordering a configuration's sets made the validator refuse \ + it, so it is treating list order as meaning: {e}" + ) + }); + + let (Some(before), Some(after)) = (Artifacts::of(&first), Artifacts::of(&second)) + else { + return; + }; + let (before, after) = (before.all(), after.all()); + + if moved { + MOVED.fetch_add(1, Ordering::Relaxed); + } + COMPARED.fetch_add(1, Ordering::Relaxed); + + if before != after { + let mut differences: Vec = Vec::new(); + for line in &before { + if !after.contains(line) { + differences.push(format!(" only before: {line}")); + } + } + for line in &after { + if !before.contains(line) { + differences.push(format!(" only after: {line}")); + } + } + differences.truncate(20); + panic!( + "{mutation:?}: reordering a configuration's sets changed what the dataplane \ + installs, so the configuration had more than one meaning and the chain \ + picked one:\n{}", + differences.join("\n") + ); + } + }, + ); + + // Without this the property passes trivially on configurations with nothing to reorder -- + // one peering with one expose holding one prefix has no second order to be in. A tenth is + // well above what a dead permutation would give and well below what is observed (a fifth, + // driven by near-misses, since several mutations shorten the very lists this reorders). + let compared = COMPARED.load(Ordering::Relaxed); + let moved = MOVED.load(Ordering::Relaxed); + println!("{moved} of {compared} comparisons were of a genuinely reordered configuration"); + // Distribution, which is the random engine's contract and not a coverage-guided one's; see + // the same reasoning in `validator_completeness`. + #[cfg(not(fuzzing))] + assert!( + compared > 0 && moved * 10 >= compared, + "only {moved} of {compared} comparisons actually reordered anything: the permutation is \ + not doing any work" + ); + } +} + +/// Every expose earns its place: remove one and the dataplane must install something different. +/// +/// The third question about a blessed configuration, after "can it be enacted" and "does it have one +/// meaning": **is the dataplane doing all of it?** +/// +/// The failure this hunts is a builder that silently ignores part of its input -- a shape it does not +/// handle, a `continue` on a branch nobody expected to be reachable. Nothing else here covers that, and +/// unlike the ambiguity work it needs no mutation to reach it: such a bug would live in the builder +/// rather than behind a validator rule, so it shows up on configurations that are entirely legal. +/// +/// The artifacts are asked **one at a time**, and that is the whole design. Every expose contributes +/// prefixes to the FRR render whatever else it does, so a merged comparison would report a difference +/// even where a NAT builder had ignored the expose completely -- which is precisely the case worth +/// catching. What each artifact is entitled to expect comes from the removed expose's own NAT mode, a +/// fact read straight off the CRD: no model of `collapse_prefixes` or of the PAT splitting is needed, +/// and none is wanted, since a wrong model would make this property lie rather than fail. +/// +/// It rests on there being no *legitimately* no-op expose. That is a domain claim, not a code one; it is +/// asserted here on the understanding that it holds, and `k8s_intf::bolero::reduce` is where an +/// exemption would belong if some shape turns out to be exempt. +mod relevance { + use super::enacted::{Artifacts, validator}; + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + use k8s_intf::bolero::mutate::Mutation; + use k8s_intf::bolero::reduce::{Dropped, ReducedAgents}; + use k8s_intf::gateway_agent_crd::GatewayAgent; + + /// Whether this gateway is a member of the group the named peering points at. + /// + /// The peering only reaches the routing configuration if it is. + fn handled_here(agent: &GatewayAgent, peering: &str) -> bool { + let Some(name) = agent.metadata.name.as_deref() else { + return false; + }; + let Some(group) = agent + .spec + .peerings + .as_ref() + .and_then(|peerings| peerings.get(peering)) + .and_then(|peering| peering.gateway_group.as_deref()) + else { + return false; + }; + agent + .spec + .groups + .as_ref() + .and_then(|groups| groups.get(group)) + .and_then(|group| group.members.as_ref()) + .is_some_and(|members| members.iter().any(|m| m.name == name)) + } + + /// Whether `left` and `right` differ, and by what, for a failure message. + fn difference(left: &[String], right: &[String]) -> Option { + if left == right { + return None; + } + let mut out: Vec = Vec::new(); + for line in left { + if !right.contains(line) { + out.push(format!(" only with it: {line}")); + } + } + for line in right { + if !left.contains(line) { + out.push(format!(" only without it: {line}")); + } + } + out.truncate(10); + Some(out.join("\n")) + } + + /// # Why IPv4 only + /// + /// `NatAllocator`'s `Display` never returns on an IPv6 masquerade pool. `ips_in_bitmap` walks + /// **every set bit** of the pool's bitmap (`for offset in &self.bitmap.0`), which is a few thousand + /// iterations for a v4 `/20` and unbounded for a v6 pool. Measured, not surmised: over IPv4 it + /// completes 380 times in a one-second run with a worst case of 6ms; over both families it does not + /// complete once in 200 seconds. + /// + /// This property is the one that trips over it because it renders every artifact, the allocator + /// included, twice per case. See `.scratch/next-phase-assessment.md` -- the defect is not confined + /// to tests, since that `Display` is a `CliSource` and holds a read lock across the whole print. + /// + /// # Yield + /// + /// About one case in fourteen reaches the comparison; the rest are configurations with no manifest + /// holding two exposes, overwhelmingly because they have no peerings at all. Hence `sizes` below + /// and the loose bound in the health check, which is set from that measurement rather than hope. + #[test] + fn every_expose_leaves_a_trace() { + static CHECKED: AtomicUsize = AtomicUsize::new(0); + static NOTHING_TO_DROP: AtomicUsize = AtomicUsize::new(0); + static REFUSED_WITHOUT: AtomicUsize = AtomicUsize::new(0); + static NOT_OURS: AtomicUsize = AtomicUsize::new(0); + static TRANSLATING: AtomicUsize = AtomicUsize::new(0); + + bolero::check!() + .with_generator(ReducedAgents::new( + k8s_intf::bolero::mutate::MutatedAgents::new( + k8s_intf::bolero::crd::GatewayAgentBuilder::new() + .families(vec![k8s_intf::bolero::AddressFamily::V4]) + // Manifests of three or four exposes, where the default is at most two: this + // property has to *remove* one and still leave a legal manifest, so a + // one-expose manifest is a case it can only throw away. At the default that + // was 84% of every case drawn. + .sizes(4, 3, 4, 3) + .build(), + ), + )) + .cloned() + .for_each( + |(mutation, agent, reduced, dropped): ( + Mutation, + GatewayAgent, + GatewayAgent, + Option, + )| { + let Some(dropped) = dropped else { + NOTHING_TO_DROP.fetch_add(1, Ordering::Relaxed); + return; + }; + let Ok(whole) = validator(&agent) else { + // Refused, which is `validator_completeness`'s business and not this one's. + return; + }; + // Removing an expose can make a configuration illegal for reasons of its own: a + // manifest left empty, or an ACL rule whose `match` named the departed prefix and + // now matches nothing. Those are honest refusals and the case is dropped, but + // counted, so a property that had quietly become all-skips would say so. + let Ok(less) = validator(&reduced) else { + REFUSED_WITHOUT.fetch_add(1, Ordering::Relaxed); + return; + }; + let (Some(with), Some(without)) = (Artifacts::of(&whole), Artifacts::of(&less)) + else { + return; + }; + + // A peering whose gateway group does not list *this* gateway is not this + // gateway's to route: `build_routing_config_peer` never runs for it, so nothing + // it contains reaches any artifact. That is correct behaviour and the one + // legitimate way an expose leaves no trace -- a fact of *context* rather than of + // the expose's shape, which is why reading the expose alone could never have + // predicted it. This property found it by failing. + if !handled_here(&agent, &dropped.peering) { + NOT_OURS.fetch_add(1, Ordering::Relaxed); + return; + } + + CHECKED.fetch_add(1, Ordering::Relaxed); + + // Every expose reaches the routing configuration, whatever else it does: its + // prefixes go into the lists its peer imports and advertises. + assert!( + difference(&with.frr, &without.frr).is_some(), + "{mutation:?}: removing {dropped} changed nothing in the routing \ + configuration, so the dataplane was never routing it" + ); + + // And an expose that translates must reach the table for the mode it asked for. A + // builder that ignores a shape it does not recognise fails here and nowhere else. + let (table, name) = match dropped.nat { + None => return, + Some("static") => (&with.static_nat, &without.static_nat), + Some("port forwarding") => { + (&with.port_forwarding, &without.port_forwarding) + } + Some("masquerade") => (&with.masquerade, &without.masquerade), + Some(other) => unreachable!("unknown nat mode {other}"), + }; + TRANSLATING.fetch_add(1, Ordering::Relaxed); + assert!( + difference(table, name).is_some(), + "{mutation:?}: removing {dropped} changed nothing in the table for its own \ + NAT mode, so that translation was never installed" + ); + }, + ); + + let checked = CHECKED.load(Ordering::Relaxed); + let nothing = NOTHING_TO_DROP.load(Ordering::Relaxed); + let refused = REFUSED_WITHOUT.load(Ordering::Relaxed); + let not_ours = NOT_OURS.load(Ordering::Relaxed); + let translating = TRANSLATING.load(Ordering::Relaxed); + println!( + "{checked} exposes checked ({translating} of them translating); {nothing} \ + configurations had nothing to drop, {refused} became illegal without it, {not_ours} \ + sat in a peering this gateway does not handle" + ); + + // Skipping is expected here in a way it is not in the other properties, so the guard is that + // skipping did not become the whole story. + let seen = checked + nothing + refused + not_ours; + #[cfg(not(fuzzing))] + if seen > 200 { + assert!( + checked * 40 > seen, + "only {checked} of {seen} cases got as far as comparing artifacts: this property has \ + become mostly skips" + ); + } + } +} diff --git a/miri.just b/miri.just index 091da10355..db51d93875 100644 --- a/miri.just +++ b/miri.just @@ -25,7 +25,20 @@ export target := cpu + "-unknown-linux-gnu" export provenance := "permissive" export schedule_seed := choose('5', "0123456789") export seeds := "1" -export stacked_borrow_check := "disabled" +# Overridable from the environment because `just` will not override a *module's* variables from the +# command line: `just miri stacked_borrow_check=enabled test` is parsed as a recipe name, and +# `just --set stacked_borrow_check enabled miri test` is refused outright ("not present in justfile"). +# Without `env()` these knobs can only be changed by editing this file, which is how +# `stacked_borrow_check` came to be effectively unreachable. +export stacked_borrow_check := env("STACKED_BORROW_CHECK", "disabled") + +# How long each bolero property gets, in milliseconds. +# +# Needed because this recipe launches its own `nix-shell`, which does not inherit the caller's +# environment: exporting `BOLERO_RANDOM_TEST_TIME_MS` before `just miri test` has no effect, and every +# property silently stops at bolero's one-second default. Under miri that is about 25 cases -- enough to +# prove the harness runs and nothing else. +export bolero_test_time_ms := env("BOLERO_TEST_TIME_MS", "30000") export preemption_rate := "0.10" export weak_failure_rate := "0.05" export randomize_struct_layout := "enabled" @@ -63,6 +76,7 @@ test *args="": # Umbrella cfg shared with the qemu-user path in nix/profiles.nix. RUSTFLAGS+="--cfg=emulated" declare -rx RUSTFLAGS + declare -rx BOLERO_RANDOM_TEST_TIME_MS="${bolero_test_time_ms}" declare -a cmd=("nice" "-n" "19" "cargo" "miri" "nextest" "run" "--profile=miri" "--target=${target}") if [ "${cores}" != "0" ]; then # nextest defaults --test-threads to the core count; the miri profile diff --git a/nat/Cargo.toml b/nat/Cargo.toml index e46d2740a2..b2e69b14e7 100644 --- a/nat/Cargo.toml +++ b/nat/Cargo.toml @@ -36,7 +36,7 @@ shuttle = { workspace = true, optional = true } [dev-dependencies] # internal -config = { workspace = true } +config = { workspace = true, features = ["bolero"] } fixin = { workspace = true } test-utils = { workspace = true } lpm = { workspace = true, features = ["testing"] } diff --git a/nat/src/portfw/portfwtable/setup.rs b/nat/src/portfw/portfwtable/setup.rs index 812e14b329..3194ec163e 100644 --- a/nat/src/portfw/portfwtable/setup.rs +++ b/nat/src/portfw/portfwtable/setup.rs @@ -111,3 +111,64 @@ pub fn build_port_forwarding_configuration( } Ok(ruleset) } + +#[cfg(test)] +mod tests { + use super::*; + use config::external::overlay::vpcpeering::VpcExpose; + use config::external::overlay::vpcpeering::contract::{ + LOCAL_VNI, PortForwardingExpose, REMOTE_VNI, overlay_offering, + }; + + /// A port-forwarding expose becomes exactly the rules it describes. + /// + /// The interesting part is not that rules come out, but that the two sides do not get crossed: + /// the expose's `as_range` is what traffic arrives on and its `ips` is where traffic goes, and + /// a rule that has them the other way round forwards to the wrong place while satisfying every + /// check the rule itself makes. + #[test] + fn an_expose_becomes_the_rules_it_describes() { + bolero::check!() + .with_generator(PortForwardingExpose) + .cloned() + .for_each(|expose: VpcExpose| { + let nat = expose.nat.as_ref().expect("port forwarding sets nat"); + let proto = nat.proto; + let internal = *expose.ips.first().expect("one prefix"); + let external = *nat.as_range.first().expect("one prefix"); + + let overlay = overlay_offering(expose.clone()).expect("overlay"); + let rules = build_port_forwarding_configuration(overlay.vpc_table()) + .expect("a validated port-forwarding expose should build"); + + // `Any` is served by one rule per protocol; anything else by one. + let expected = if proto == L4Protocol::Any { 2 } else { 1 }; + assert_eq!(rules.len(), expected, "for {expose}"); + + for rule in &rules { + assert_eq!(rule.ext_prefix, external.prefix(), "external prefix"); + assert_eq!(rule.int_prefix, internal.prefix(), "internal prefix"); + assert_eq!( + rule.ext_ports.first().get(), + external.ports().expect("ports").start(), + "external ports" + ); + assert_eq!( + rule.int_ports.first().get(), + internal.ports().expect("ports").start(), + "internal ports" + ); + // The rule forwards into the local VPC, and admits traffic from the peer. + assert_eq!(rule.dst_vpcd, VpcDiscriminant::from_vni(vni(LOCAL_VNI))); + assert_eq!( + rule.key.src_vpcd(), + VpcDiscriminant::from_vni(vni(REMOTE_VNI)) + ); + } + }); + } + + fn vni(raw: u32) -> net::vxlan::Vni { + net::vxlan::Vni::new_checked(raw).unwrap_or_else(|_| unreachable!()) + } +} diff --git a/nat/src/static_nat/setup/mod.rs b/nat/src/static_nat/setup/mod.rs index 484f3f8d4e..c8beb2c1bf 100644 --- a/nat/src/static_nat/setup/mod.rs +++ b/nat/src/static_nat/setup/mod.rs @@ -206,3 +206,103 @@ mod tests { .expect("Failed to build NAT tables"); } } + +#[cfg(test)] +mod config_driven { + use super::*; + use config::external::overlay::vpcpeering::VpcExpose; + use config::external::overlay::vpcpeering::contract::{ + LOCAL_VNI, REMOTE_VNI, StaticNatExpose, overlay_offering, + }; + use lpm::prefix::PrefixWithOptionalPorts; + use std::collections::BTreeSet; + use std::net::IpAddr; + + fn vni(raw: u32) -> Vni { + Vni::new_checked(raw).unwrap_or_else(|_| unreachable!()) + } + + // Every address a set of prefixes covers. Sizes are kept small by the generator so that this + // is a handful of addresses rather than a sweep. + fn addresses(prefixes: &BTreeSet) -> Vec { + let mut out = Vec::new(); + for prefix in prefixes { + let prefix = prefix.prefix(); + let (start, end) = (prefix.as_address(), prefix.last_address()); + let (mut bits, last) = match (start, end) { + (IpAddr::V4(a), IpAddr::V4(b)) => { + (u128::from(a.to_bits()), u128::from(b.to_bits())) + } + (IpAddr::V6(a), IpAddr::V6(b)) => (a.to_bits(), b.to_bits()), + _ => unreachable!("a prefix does not change address family"), + }; + while bits <= last { + out.push(match start { + IpAddr::V4(_) => IpAddr::V4( + u32::try_from(bits) + .unwrap_or_else(|_| unreachable!()) + .into(), + ), + IpAddr::V6(_) => IpAddr::V6(bits.into()), + }); + bits += 1; + } + } + out + } + + /// Static NAT maps the two sides of an expose one to one. + /// + /// This is the contract `RangeBuilder` exists to keep, and the reason it is not trivial: the + /// two sides hold the same number of addresses while being free to be cut up differently, so + /// the mapping runs across prefix boundaries that do not line up. Every private address must + /// come out somewhere in the public set, no two may land on the same place, and between them + /// they must cover it -- a bijection, checked by enumeration rather than by sampling. + #[test] + fn the_two_sides_of_an_expose_map_one_to_one() { + bolero::check!() + .with_generator(StaticNatExpose) + .cloned() + .for_each(|expose: VpcExpose| { + let private = addresses(&expose.ips); + let public: BTreeSet = addresses( + &expose + .nat + .as_ref() + .expect("static nat sets nat") + .as_range + .clone(), + ) + .into_iter() + .collect(); + + let overlay = overlay_offering(expose.clone()).expect("overlay"); + let tables = build_nat_configuration(overlay.vpc_table()) + .expect("a validated expose builds"); + let table = tables + .get_table(vni(LOCAL_VNI)) + .expect("the offering vpc has a table"); + + let mut seen = BTreeSet::new(); + for source in &private { + let (mapped, _) = table + .find_src_mapping(source, None, vni(REMOTE_VNI)) + .unwrap_or_else(|| panic!("{source} has no mapping in {expose}")); + let mapped = mapped.inner(); + assert!( + public.contains(&mapped), + "{source} mapped to {mapped}, which the expose does not offer" + ); + assert!( + seen.insert(mapped), + "{source} mapped to {mapped}, which another address already took" + ); + } + assert_eq!( + seen.len(), + public.len(), + "the mapping left part of the public side unused, for {expose}" + ); + }); + } +} diff --git a/net/src/headers/embedded.rs b/net/src/headers/embedded.rs index 8017c2231f..c98128d78b 100644 --- a/net/src/headers/embedded.rs +++ b/net/src/headers/embedded.rs @@ -22,6 +22,7 @@ use crate::udp::{TruncatedUdp, UdpChecksum, UdpPort}; use arrayvec::ArrayVec; use core::fmt::Debug; use derive_builder::Builder; +use std::net::IpAddr; use std::num::NonZero; #[cfg(any(test, feature = "bolero"))] @@ -609,42 +610,87 @@ impl EmbeddedTransport { } } + /// Fold the change of a 16-bit field into the header's checksum. + /// + /// `increment_update_checksum` computes the new checksum and hands it back; it does not store + /// it, despite taking `&mut self`. Setting it is the point of this method, and it is what makes + /// a translated quote's checksum agree with the bytes around it. + /// + /// Setting fails only on a header too truncated to hold a checksum, which is a header the + /// caller could not have read one out of either, so there is nothing to do about it here. pub fn update_checksum(&mut self, current_checksum: u16, old_value: u16, new_value: u16) { match self { EmbeddedTransport::Tcp(tcp) => { - // Silently ignore errors if transport header is truncated - let _ = tcp.increment_update_checksum( + let updated = tcp.increment_update_checksum( TcpChecksum::new(current_checksum), old_value, new_value, ); + let _ = tcp.set_checksum(updated); } EmbeddedTransport::Udp(udp) => { - // Silently ignore errors if transport header is truncated - let _ = udp.increment_update_checksum( + let updated = udp.increment_update_checksum( UdpChecksum::new(current_checksum), old_value, new_value, ); + let _ = udp.set_checksum(updated); } EmbeddedTransport::Icmp4(icmp) => { - // Silently ignore errors if transport header is truncated - let _ = icmp.increment_update_checksum( + let updated = icmp.increment_update_checksum( Icmp4Checksum::new(current_checksum), old_value, new_value, ); + let _ = icmp.set_checksum(updated); } EmbeddedTransport::Icmp6(icmp) => { - // Silently ignore errors if transport header is truncated - let _ = icmp.increment_update_checksum( + let updated = icmp.increment_update_checksum( Icmp6Checksum::new(current_checksum), old_value, new_value, ); + let _ = icmp.set_checksum(updated); } } } + + /// Fold a change to one of the quoted packet's IP addresses into this header's checksum. + /// + /// TCP, UDP and `ICMPv6` are checksummed over a pseudo-header built from the source and + /// destination addresses, so rewriting either address of a quoted packet leaves the quoted + /// transport checksum describing an address that is no longer there. `ICMPv4` has no + /// pseudo-header and is left alone. + /// + /// Incremental, because a quote is usually truncated: there is no payload to compute a + /// checksum over from scratch, only a delta to apply to the one already there. + pub fn update_checksum_for_address(&mut self, old: IpAddr, new: IpAddr) { + if matches!(self, EmbeddedTransport::Icmp4(_)) { + return; + } + // Translating between address families would change the shape of the pseudo-header rather + // than a value inside it, so there would be no delta to fold. + if old.is_ipv4() != new.is_ipv4() { + return; + } + for (old_word, new_word) in address_words(old).into_iter().zip(address_words(new)) { + let Some(current) = self.checksum() else { + return; + }; + self.update_checksum(current, old_word, new_word); + } + } +} + +// The 16-bit words of an address, in the order a checksum sums them. +fn address_words(addr: IpAddr) -> ArrayVec { + match addr { + IpAddr::V4(addr) => { + let [a, b, c, d] = addr.octets(); + ArrayVec::from_iter([u16::from_be_bytes([a, b]), u16::from_be_bytes([c, d])]) + } + IpAddr::V6(addr) => ArrayVec::from(addr.segments()), + } } impl DeParse for EmbeddedTransport { @@ -1378,6 +1424,115 @@ mod tests { // Before calling check_full_payload, should be false assert!(!headers.is_full_payload()); } + + // Checksum folding for a rewritten address in a quoted packet. + // + // A quote is usually truncated, so folding a delta into the checksum already there is the only + // option: there is no payload to compute one over. These build the one case where both routes + // are open -- a full header over a known payload -- and hold the fold against a computation + // from scratch. + // + // The checksum starts out correct on purpose. An RFC 1624 update is exact given a correct + // starting value and says nothing at all given a wrong one, so only a correct start tests + // anything. + mod address_folding { + use super::*; + use crate::icmp4::{Icmp4, Icmp4EchoRequest, Icmp4Type}; + use crate::ip::NextHeader; + use crate::ipv4::UnicastIpv4Addr; + use crate::ipv6::UnicastIpv6Addr; + use crate::tcp::{Tcp, TcpChecksumPayload}; + use crate::udp::{UdpChecksumPayload, UdpPort}; + use std::net::{Ipv4Addr, Ipv6Addr}; + + const PAYLOAD: [u8; 4] = [0xde, 0xad, 0xbe, 0xef]; + + fn v4_net(source: Ipv4Addr, next_header: NextHeader) -> Net { + let mut ip = Ipv4::default(); + ip.set_source(UnicastIpv4Addr::new(source).unwrap_or_else(|_| unreachable!())) + .set_destination(Ipv4Addr::new(192, 168, 1, 2)) + .set_next_header(next_header); + Net::Ipv4(ip) + } + + fn v6_net(source: Ipv6Addr, next_header: NextHeader) -> Net { + let mut ip = Ipv6::default(); + ip.set_source(UnicastIpv6Addr::new(source).unwrap_or_else(|_| unreachable!())) + .set_destination(Ipv6Addr::new(0xfe80, 0, 0, 0, 0, 0, 0, 2)) + .set_next_header(next_header); + Net::Ipv6(ip) + } + + /// Two words to fold, over a TCP pseudo-header. + #[test] + fn a_v4_address_change_matches_a_fresh_tcp_checksum() { + let (old, new) = (Ipv4Addr::new(192, 168, 1, 1), Ipv4Addr::new(10, 11, 12, 13)); + + let mut tcp = Tcp::new(123.try_into().unwrap(), 456.try_into().unwrap()); + tcp.update_checksum(&TcpChecksumPayload::new( + &v4_net(old, NextHeader::TCP), + &PAYLOAD, + )) + .unwrap_or_else(|()| unreachable!()); + + let mut quoted = EmbeddedTransport::Tcp(TruncatedTcp::FullHeader(tcp.clone())); + quoted.update_checksum_for_address(IpAddr::V4(old), IpAddr::V4(new)); + + let expected = tcp + .compute_checksum(&TcpChecksumPayload::new( + &v4_net(new, NextHeader::TCP), + &PAYLOAD, + )) + .unwrap_or_else(|()| unreachable!()); + assert_eq!(quoted.checksum(), Some(u16::from(expected))); + } + + /// Eight words to fold, over a UDP pseudo-header. + #[test] + fn a_v6_address_change_matches_a_fresh_udp_checksum() { + let old = Ipv6Addr::new(0xfe80, 0, 0, 0, 0, 0, 0, 1); + let new = Ipv6Addr::new(0x2001, 0xdb8, 0xdead, 0xbeef, 1, 2, 3, 4); + + let mut udp = Udp::new( + UdpPort::new_checked(123).unwrap(), + UdpPort::new_checked(456).unwrap(), + ); + udp.update_checksum(&UdpChecksumPayload::new( + &v6_net(old, NextHeader::UDP), + &PAYLOAD, + )) + .unwrap_or_else(|()| unreachable!()); + + let mut quoted = EmbeddedTransport::Udp(TruncatedUdp::FullHeader(udp.clone())); + quoted.update_checksum_for_address(IpAddr::V6(old), IpAddr::V6(new)); + + let expected = udp + .compute_checksum(&UdpChecksumPayload::new( + &v6_net(new, NextHeader::UDP), + &PAYLOAD, + )) + .unwrap_or_else(|()| unreachable!()); + assert_eq!(quoted.checksum(), Some(u16::from(expected))); + } + + /// `ICMPv4` is checksummed over itself alone, so the addresses around it are none of its + /// business and folding one must not disturb it. + #[test] + fn a_v4_address_change_leaves_an_icmpv4_quote_alone() { + let mut icmp = + Icmp4::with_type(Icmp4Type::EchoRequest(Icmp4EchoRequest { id: 18, seq: 2 })); + icmp.update_checksum(&PAYLOAD) + .unwrap_or_else(|()| unreachable!()); + + let mut quoted = EmbeddedTransport::Icmp4(TruncatedIcmp4::FullHeader(icmp)); + let before = quoted.checksum(); + quoted.update_checksum_for_address( + IpAddr::V4(Ipv4Addr::new(192, 168, 1, 1)), + IpAddr::V4(Ipv4Addr::new(10, 11, 12, 13)), + ); + assert_eq!(quoted.checksum(), before); + } + } } #[cfg(any(test, feature = "bolero"))] diff --git a/net/src/headers/embedded_view.rs b/net/src/headers/embedded_view.rs index fe6d5eba0e..f5aeee0ad6 100644 --- a/net/src/headers/embedded_view.rs +++ b/net/src/headers/embedded_view.rs @@ -1285,6 +1285,52 @@ mod tests { assert!(matches::<(&Ipv4, &TruncatedTcp)>(e)); } + /// An ICMP error quoting an ICMP packet, which the builder had never been asked to assemble. + /// + /// `EmbeddedAssembler::icmp4` and `::icmp6` were the two inner-transport methods no test called, + /// and with them the `EmbeddedTransport::Icmp4 => NextHeader::ICMP` arm of the fixup that sets + /// the quoted packet's protocol field. A ping that draws a destination-unreachable is the + /// ordinary way to produce one, so the gap was in the tests rather than in the scenario. + /// + /// The inner protocol number is checked directly: `fixup_embedded` is what writes it, the shape + /// match below would pass without it, and a quoted packet whose IP header disagrees with the + /// transport it carries is exactly what a peer cannot parse. + #[test] + fn icmp4_quoted_inside_an_icmp4_error() { + use crate::headers::builder::Blank; + use crate::icmp4::Icmp4; + use crate::ip::NextHeader; + let h = icmp4_with_embedded(|a| a.ipv4(|_| {}).icmp4(Icmp4::blank())); + let e = h.embedded_ip().expect("embedded must be present"); + assert!(matches::<(&Ipv4, &TruncatedIcmp4)>(e)); + let Some(Net::Ipv4(ip)) = e.net() else { + unreachable!("the quoted packet carries an IPv4 header") + }; + assert_eq!( + ip.next_header(), + NextHeader::ICMP, + "the quoted IP header does not name the transport it carries" + ); + } + + #[test] + fn icmp6_quoted_inside_an_icmp6_error() { + use crate::headers::builder::Blank; + use crate::icmp6::Icmp6; + use crate::ip::NextHeader; + let h = icmp6_with_embedded(|a| a.ipv6(|_| {}).icmp6(Icmp6::blank())); + let e = h.embedded_ip().expect("embedded must be present"); + assert!(matches::<(&Ipv6, &TruncatedIcmp6)>(e)); + let Some(Net::Ipv6(ip)) = e.net() else { + unreachable!("the quoted packet carries an IPv6 header") + }; + assert_eq!( + ip.next_header(), + NextHeader::ICMP6, + "the quoted IP header does not name the transport it carries" + ); + } + #[test] fn ipv6_truncated_udp_matches_full_inner_packet() { use crate::udp::UdpPort; @@ -1790,3 +1836,655 @@ mod tests { } } } + +// =========================================================================== +// Differential properties +// =========================================================================== + +#[cfg(test)] +#[allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] // fine to unwrap in tests +mod embedded_view_properties { + use super::*; + use crate::eth::Eth; + use crate::headers::{Headers, ShapedIcmpError, ShapedQuote}; + use crate::icmp4::Icmp4; + use crate::icmp6::Icmp6; + use crate::vlan::Vlan; + + /// Both structural walks over a quoted packet, across the range of outer and inner shapes. + /// + /// The pairing this checks is the one soundness rests on, and it is not the obvious one. + /// `as_embedded`/`as_embedded_mut` *decide* with [`embedded_sealed::Sealed::matches`], which + /// walks [`EmbeddedHeaders`] through [`EmbeddedStep`]. [`EmbeddedLookMut::look_mut`] then + /// *delivers* through [`EmbeddedMatcherMut`], a completely separate implementation with its own + /// pre-split [`EmbeddedFields`](super::super::pat::EmbeddedFields) and its own gap check, and it + /// unwraps that chain with `unreachable_unchecked` on the strength of the first one's answer. + /// Two implementations, one of them licensing the other to skip its own `None` branch. If they + /// ever disagree the result is undefined behaviour, not a wrong answer. + /// + /// So the oracle here is `pat()`/`pat_mut()`'s `.embedded()` chain, which is the same machinery + /// `look_mut` uses -- making this differential test exactly the invariant `look_mut` assumes. + /// + /// Arity matters twice over, because `matches`, `look` and `look_mut` are generated per *inner* + /// arity while `as_embedded`/`as_embedded_mut` are generated per *outer* arity, each threading + /// the extension cursor by hand. Before this the suite reached one outer arity and one inner + /// arity, of eight and three. + macro_rules! embedded_agrees { + ( + $read:ident, $mutable:ident, + ($outer:ty, $($ol:ident),+), + ($inner:ty, $($il:ident),+) + ) => { + #[test] + fn $read() { + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + static SEEN: AtomicUsize = AtomicUsize::new(0); + static HIT: AtomicUsize = AtomicUsize::new(0); + bolero::check!() + .with_generator(ShapedIcmpError) + .for_each(|h: &Headers| { + SEEN.fetch_add(1, Ordering::Relaxed); + let licensed = h + .as_view::<$outer>() + .is_some_and(|w| w.as_embedded::<$inner>().is_some()); + if licensed { + HIT.fetch_add(1, Ordering::Relaxed); + } + let deliverable = + h.pat()$(.$ol())+.embedded()$(.$il())+.done().is_some(); + assert_eq!( + licensed, deliverable, + concat!( + "`matches` and the read walk disagree for ", + stringify!($inner), + " inside ", + stringify!($outer), + ", so `look` would reach `unreachable_unchecked`: {:?}" + ), + h + ); + }); + agreement_is_not_vacuous( + concat!(stringify!($inner), " in ", stringify!($outer)), + SEEN.load(Ordering::Relaxed), + HIT.load(Ordering::Relaxed), + ); + } + + #[test] + fn $mutable() { + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + static SEEN: AtomicUsize = AtomicUsize::new(0); + static HIT: AtomicUsize = AtomicUsize::new(0); + bolero::check!() + .with_generator(ShapedIcmpError) + .for_each(|h: &Headers| { + let mut owned = h.clone(); + SEEN.fetch_add(1, Ordering::Relaxed); + // `as_embedded_mut`'s reference cannot outlive the `and_then` closure it + // would be produced in, so this asks the question without keeping it. + let licensed = match owned.as_view_mut::<$outer>() { + Some(w) => w.as_embedded_mut::<$inner>().is_some(), + None => false, + }; + if licensed { + HIT.fetch_add(1, Ordering::Relaxed); + } + let deliverable = + owned.pat_mut()$(.$ol())+.embedded()$(.$il())+.done().is_some(); + assert_eq!( + licensed, deliverable, + concat!( + "`matches` and the mutable walk disagree for ", + stringify!($inner), + " inside ", + stringify!($outer), + ", so `look_mut` would reach `unreachable_unchecked`: {:?}" + ), + h + ); + }); + agreement_is_not_vacuous( + concat!(stringify!($inner), " in ", stringify!($outer)), + SEEN.load(Ordering::Relaxed), + HIT.load(Ordering::Relaxed), + ); + } + }; + } + + /// A shape the generator never produces makes its property pass for the wrong reason. + /// + /// Two shapes compound here -- an outer arity and an inner one -- so a property can starve much + /// more quietly than in [`view`](super::super::view): demanding four VLAN tags *and* a + /// particular extension inside the quoted packet multiplies two small probabilities. Each pair + /// reports its own hit rate, and the pairs are chosen to vary one dimension at a time for + /// exactly that reason. + fn agreement_is_not_vacuous(shape: &str, seen: usize, hit: usize) { + println!("{shape}: matched {hit} of {seen}"); + // Twenty thousand rather than [`view`](super::super::view)'s five hundred, because + // compounding costs an order of magnitude: the widest pair here matches about one packet in + // twenty, and `(&Ipv6, &HopByHop, &TruncatedTcp)` inside an ICMPv6 error matches about one + // in a thousand. Five hundred cases at that rate would fail this check outright half the + // time it ran. A deliberately short run should say nothing rather than say something false. + // + // Ten thousand, not the twenty-five thousand a default run manages here. The thinnest shape + // in this module draws about eight hits per ten thousand, and coverage instrumentation costs + // about a fifth of the throughput -- measured at 17 hits in 20,206 cases against 21 in + // 25,857 without. A threshold set at the observed count would therefore switch itself off on + // any busier runner, silently. At ten thousand draws a shape that really is drawable misses + // entirely about three times in ten thousand runs, which is the trade this wants. + if seen > 10_000 { + assert!( + hit > 0, + "{shape} never matched in {seen} packets: the two walks agree only because the \ + generator cannot produce this shape" + ); + } + } + + type O4V4 = (&'static Eth, &'static Ipv4, &'static Icmp4); + type O4V6 = (&'static Eth, &'static Ipv6, &'static Icmp6); + + // ---- Inner shapes, at the one outer arity the builders could already express ---------- + // + // Varying the inner shape alone: network layer, both enum forms, every truncated transport, + // and the extension region. + embedded_agrees!( + read_inner_v4_tcp, + mutable_inner_v4_tcp, + (O4V4, eth, ipv4, icmp4), + ((&Ipv4, &TruncatedTcp), ipv4, tcp) + ); + embedded_agrees!( + read_inner_v6_udp, + mutable_inner_v6_udp, + (O4V6, eth, ipv6, icmp6), + ((&Ipv6, &TruncatedUdp), ipv6, udp) + ); + embedded_agrees!( + read_inner_net_only, + mutable_inner_net_only, + (O4V4, eth, ipv4, icmp4), + ((&Net,), net) + ); + embedded_agrees!( + read_inner_v6_only, + mutable_inner_v6_only, + (O4V6, eth, ipv6, icmp6), + ((&Ipv6,), ipv6) + ); + // An ICMP error quoting an ICMP packet -- a `TruncatedIcmp4` inside an `Icmp4`. Legitimate: + // an unreachable in response to a ping quotes the echo request. + embedded_agrees!( + read_inner_icmp_in_icmp4, + mutable_inner_icmp_in_icmp4, + (O4V4, eth, ipv4, icmp4), + ((&Ipv4, &TruncatedIcmp4), ipv4, icmp4) + ); + embedded_agrees!( + read_inner_icmp_in_icmp6, + mutable_inner_icmp_in_icmp6, + (O4V6, eth, ipv6, icmp6), + ((&Ipv6, &TruncatedIcmp6), ipv6, icmp6) + ); + + // ---- The embedded extension region --------------------------------------------------- + // + // `ext_gap_ok_embedded` versus `ext_gap_ok_mut_embedded`: naming an extension switches the + // quoted packet from skip-extensions-silently to consume-them-all, so these match only a quote + // carrying that extension and no other. The read pair shares `ext_gap_ok_embedded` between the + // two walks and so tests the by-hand `ec` threading; the mutable pair runs two genuinely + // different implementations against each other, one over `EmbeddedHeaders` and one over a + // pre-split `EmbeddedFields`. + embedded_agrees!( + read_inner_ext_v6, + mutable_inner_ext_v6, + (O4V6, eth, ipv6, icmp6), + ((&Ipv6, &HopByHop, &TruncatedTcp), ipv6, hop_by_hop, tcp) + ); + embedded_agrees!( + read_inner_ext_v4_auth, + mutable_inner_ext_v4_auth, + (O4V4, eth, ipv4, icmp4), + ((&Ipv4, &Ipv4Auth, &TruncatedTcp), ipv4, ipv4_auth, tcp) + ); + // Entering the region and stopping there: the gap check runs at the transport step, so with no + // transport named the remaining extensions are simply left unvisited. Telling this apart from + // the two above is the whole reason the check sits where it does. + embedded_agrees!( + read_inner_ext_no_transport, + mutable_inner_ext_no_transport, + (O4V6, eth, ipv6, icmp6), + ((&Ipv6, &HopByHop), ipv6, hop_by_hop) + ); + + // ---- Outer arities ------------------------------------------------------------------- + // + // `as_embedded` and `as_embedded_mut` are written out once per outer arity, eight times. Arities + // 1 and 2 cannot be instantiated: both require `EmbeddedHeaders: Within`, which holds + // only for `Icmp4` and `Icmp6`, and neither can appear that early. `Eth` is the only layer with + // `Within<()>`, so an arity-1 shape must be `(&Eth,)`; an arity-2 shape must then end in + // something `Within`, and no ICMP layer is. Asking for either is a compile error -- + // `Icmp4: Within<()> is not satisfied` and `Icmp4: Within is not satisfied` -- so those + // two macro arms are 44 lines that can never run and could be deleted. + // + // 3 through 8 is therefore all of them. The inner shape is held cheap on purpose here, so that a + // thin outer shape is the only improbable thing each property asks for. + embedded_agrees!( + read_outer_4, + mutable_outer_4, + ((&Eth, &Vlan, &Ipv4, &Icmp4), eth, vlan, ipv4, icmp4), + ((&Ipv4, &TruncatedTcp), ipv4, tcp) + ); + embedded_agrees!( + read_outer_5, + mutable_outer_5, + ( + (&Eth, &Vlan, &Vlan, &Ipv6, &Icmp6), + eth, + vlan, + vlan, + ipv6, + icmp6 + ), + ((&Ipv6, &TruncatedUdp), ipv6, udp) + ); + embedded_agrees!( + read_outer_6, + mutable_outer_6, + ( + (&Eth, &Vlan, &Vlan, &Vlan, &Ipv4, &Icmp4), + eth, + vlan, + vlan, + vlan, + ipv4, + icmp4 + ), + // An inner shape that can miss on a quote that *is* present, so the shape check's own + // rejection is reached rather than only the absent-quote early return above it. + ((&Ipv4, &TruncatedTcp), ipv4, tcp) + ); + embedded_agrees!( + read_outer_7, + mutable_outer_7, + ( + (&Eth, &Vlan, &Vlan, &Vlan, &Vlan, &Ipv4, &Icmp4), + eth, + vlan, + vlan, + vlan, + vlan, + ipv4, + icmp4 + ), + ((&Ipv4, &TruncatedTcp), ipv4, tcp) + ); + // Arity 8, and the only outer shape that also enters the *outer* extension region: `Icmp6` + // directly after a `HopByHop` makes the outer gap check strict too. + embedded_agrees!( + read_outer_8, + mutable_outer_8, + ( + (&Eth, &Vlan, &Vlan, &Vlan, &Vlan, &Ipv6, &HopByHop, &Icmp6), + eth, + vlan, + vlan, + vlan, + vlan, + ipv6, + hop_by_hop, + icmp6 + ), + // `Ipv6` rather than `Net`: a quote of the other family misses, which is the only way to + // reach this arity's shape-check rejection. + ((&Ipv6,), ipv6) + ); + + // The `EmbeddedTransport` enum has no read-side property above, and that is an asymmetry in + // `pat.rs` rather than a choice here: `EmbeddedMatcherMut::transport` exists but + // `EmbeddedMatcher::transport` does not, so the immutable oracle cannot name the enum even + // though every concrete variant and every other layer is paired between the two. The mutable + // side is checked below; adding the missing method would let the read side follow in one line. + #[test] + fn mutable_inner_transport_enum() { + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + static SEEN: AtomicUsize = AtomicUsize::new(0); + static HIT: AtomicUsize = AtomicUsize::new(0); + bolero::check!() + .with_generator(ShapedIcmpError) + .for_each(|h: &Headers| { + let mut owned = h.clone(); + SEEN.fetch_add(1, Ordering::Relaxed); + let licensed = match owned.as_view_mut::() { + Some(w) => w.as_embedded_mut::<(&Net, &EmbeddedTransport)>().is_some(), + None => false, + }; + if licensed { + HIT.fetch_add(1, Ordering::Relaxed); + } + let deliverable = owned + .pat_mut() + .eth() + .ipv4() + .icmp4() + .embedded() + .net() + .transport() + .done() + .is_some(); + assert_eq!( + licensed, deliverable, + "`matches` and the mutable walk disagree for (&Net, &EmbeddedTransport): {h:?}" + ); + }); + agreement_is_not_vacuous( + "(&Net, &EmbeddedTransport)", + SEEN.load(Ordering::Relaxed), + HIT.load(Ordering::Relaxed), + ); + } + + // ---- Delivering, not merely deciding ------------------------------------------------- + + /// Write through every reference `look_mut` hands out, in the extension-region shape. + /// + /// The properties above compare *decisions*. This one takes the decision up on its offer: it + /// calls `look_mut`, which runs the `unreachable_unchecked` the decision licenses, and then + /// writes through all three references. The assertions are close to beside the point -- what + /// matters is that three `&mut` into one `EmbeddedHeaders` are created and written through, so + /// that miri with stacked borrows enabled can judge whether + /// [`EmbeddedFields`](super::super::pat::EmbeddedFields) handed out two paths to the same + /// layer. The arity-3 shape is the one that borrows from all three of its fields at once, + /// including a slice element, which is the hardest of them to split soundly. + fn exercise_the_embedded_split() { + bolero::check!() + .with_generator(ShapedIcmpError) + .for_each(|h: &Headers| { + let mut owned = h.clone(); + let Some(outer) = owned.as_view_mut::() else { + return; + }; + let Some(ew) = outer.as_embedded_mut::<(&Ipv6, &HopByHop, &TruncatedTcp)>() else { + return; + }; + let (ip, ext, tcp) = ew.look_mut(); + + let want_hops = ip.hop_limit().wrapping_add(1); + ip.set_hop_limit(want_hops); + let seen_ext = ext.next_header(); + let seen_tcp = matches!(tcp, TruncatedTcp::FullHeader(_)); + + assert_eq!( + ip.hop_limit(), + want_hops, + "the write through ipv6 did not stick" + ); + assert_eq!( + ext.next_header(), + seen_ext, + "the extension header changed under a write to ipv6" + ); + assert_eq!( + matches!(tcp, TruncatedTcp::FullHeader(_)), + seen_tcp, + "the quoted transport changed under a write to ipv6" + ); + }); + } + + /// Shards of [`exercise_the_embedded_split`], so the machine can be used. + /// + /// Same reasoning as the shards in [`view`](super::super::view): under miri this is the + /// expensive property and the one that matters most, and its cost is wall-clock rather than + /// cores. Each shard seeds itself from the OS, so `N` shards explore `N` independent streams + /// and nextest runs them concurrently. Eight rather than sixteen -- `view`'s split is the wider + /// one and should keep the larger share of the cores. + macro_rules! split_shards { + ($($name:ident),* $(,)?) => { + $( + #[test] + fn $name() { + exercise_the_embedded_split(); + } + )* + }; + } + + split_shards!( + the_embedded_split_hands_out_distinct_layers, + embedded_split_shard_2, + embedded_split_shard_3, + embedded_split_shard_4, + embedded_split_shard_5, + embedded_split_shard_6, + embedded_split_shard_7, + embedded_split_shard_8, + ); + + /// `look` and `look_mut` pick the same slots, one inner shape per instantiation. + /// + /// Agreeing that a match exists is weaker than agreeing on *which* layers were selected, and + /// this is the only thing in the suite that asks the stronger question across the family. + /// + /// It is also the only thing that *calls* `look` and `look_mut` at more than a couple of shapes, + /// which matters more than it looks: the differential properties above go through + /// [`embedded_sealed::Sealed::matches`] and so exercise [`EmbeddedStep`], while `look_mut` is the + /// sole caller of [`EmbeddedStepMut`] -- a second, separate set of per-layer impls that delegate + /// into [`EmbeddedMatcherMut`]. Deciding a shape matches never runs a line of it. So there is one + /// instantiation per layer that has an `EmbeddedStepMut` impl, and the extension ones are the + /// point: one implementation indexes `net_ext` by a cursor, the other consumes a slice, and + /// nothing before this compared what they returned. + macro_rules! same_layers { + ($name:ident, $gen:expr, $outer:ty, $shape:ty, $($binding:ident),+) => { + #[test] + fn $name() { + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + static SEEN: AtomicUsize = AtomicUsize::new(0); + static HIT: AtomicUsize = AtomicUsize::new(0); + bolero::check!() + .with_generator($gen) + .for_each(|h: &Headers| { + let mut owned = h.clone(); + SEEN.fetch_add(1, Ordering::Relaxed); + // `h` and `&mut h` cannot be held at once, so the packet is pinned across + // both calls and the raw addresses compared afterwards. + let immutable = { + let Some(outer) = owned.as_view::<$outer>() else { + return; + }; + let Some(ew) = outer.as_embedded::<$shape>() else { + return; + }; + let ($($binding,)+) = ew.look(); + ($(std::ptr::from_ref($binding),)+) + }; + HIT.fetch_add(1, Ordering::Relaxed); + let mutable = { + let outer = owned.as_view_mut::<$outer>().unwrap_or_else(|| { + unreachable!("the same packet matched a moment ago") + }); + let ew = outer.as_embedded_mut::<$shape>().unwrap_or_else(|| { + unreachable!("the same shape matched a moment ago") + }); + let ($($binding,)+) = ew.look_mut(); + ($(std::ptr::from_ref(&*$binding),)+) + }; + assert_eq!( + immutable, mutable, + concat!( + "`look` and `look_mut` selected different layers of the quoted \ + packet for ", + stringify!($shape) + ) + ); + }); + agreement_is_not_vacuous( + concat!("look/look_mut ", stringify!($shape)), + SEEN.load(Ordering::Relaxed), + HIT.load(Ordering::Relaxed), + ); + } + }; + } + + /// The strict gap check applies to the transport *enum* too, not just concrete variants. + /// + /// Stated as a direct assertion rather than a differential, because the immutable oracle cannot + /// express it -- see the note above `mutable_inner_transport_enum`. That leaves the enum's own + /// gap check with nothing else reaching it, and it is worth reaching: naming an extension makes + /// the check strict, so a second, unnamed extension must turn a hit into a miss even though the + /// transport pattern is the permissive one. + /// + /// The second extension is a clone of the first, which keeps this free of any assumption about + /// what other extension types exist while still being two headers where the shape names one. + #[test] + fn the_transport_enum_still_refuses_an_unconsumed_extension() { + bolero::check!() + .with_generator(ShapedQuote { ext: 0, v4: false }) + .for_each(|h: &Headers| { + let mut owned = h.clone(); + let quoted = owned + .embedded_ip_mut() + .unwrap_or_else(|| unreachable!("ShapedQuote always attaches a quote")); + let first = quoted + .net_ext + .first() + .unwrap_or_else(|| unreachable!("ShapedQuote always places one extension")) + .clone(); + quoted.net_ext.push(first); + + let outer = owned + .as_view::() + .unwrap_or_else(|| unreachable!("ShapedQuote draws an ICMPv6 error")); + assert!( + outer + .as_embedded::<(&Ipv6, &HopByHop, &EmbeddedTransport)>() + .is_none(), + "naming one extension of two matched anyway, so the gap check did not run for \ + the transport enum" + ); + // Not naming an extension skips the region silently, which is the other half of the + // same rule and shows the miss above is the gap check rather than a broken shape. + assert!( + outer.as_embedded::<(&Ipv6, &EmbeddedTransport)>().is_some(), + "a shape that never entered the extension region was refused" + ); + }); + } + + // Network layer, both concrete and as the enum, at inner arity 1. These shapes are common + // enough in the broad generator's output -- around one packet in twenty -- to use it. + same_layers!(same_layers_v4_only, ShapedIcmpError, O4V4, (&Ipv4,), ip); + same_layers!(same_layers_v6_only, ShapedIcmpError, O4V6, (&Ipv6,), ip); + same_layers!(same_layers_net_only, ShapedIcmpError, O4V4, (&Net,), net); + + // Every truncated transport, and the transport enum. + same_layers!( + same_layers_v4_tcp, + ShapedIcmpError, + O4V4, + (&Ipv4, &TruncatedTcp), + ip, + tcp + ); + same_layers!( + same_layers_v6_udp, + ShapedIcmpError, + O4V6, + (&Ipv6, &TruncatedUdp), + ip, + udp + ); + same_layers!( + same_layers_icmp_in_icmp4, + ShapedIcmpError, + O4V4, + (&Ipv4, &TruncatedIcmp4), + ip, + icmp + ); + same_layers!( + same_layers_icmp_in_icmp6, + ShapedIcmpError, + O4V6, + (&Ipv6, &TruncatedIcmp6), + ip, + icmp + ); + same_layers!( + same_layers_transport_enum, + ShapedIcmpError, + O4V4, + (&Net, &EmbeddedTransport), + net, + transport + ); + + // Every extension header, which is where the two walks diverge most: one indexes `net_ext` by a + // cursor, the other consumes a slice. These use the narrow generator, because on the broad one + // they matched six to twenty-three packets in twenty-four thousand -- see [`ShapedQuote`]. + // + // Each therefore sees a quoted packet with exactly one extension, and that is not a limitation + // of the generator but of what is expressible: embedded shapes stop at arity 3, so a shape can + // name at most one extension, and naming one makes the gap check strict -- a second, unconsumed + // extension is a miss. One is the only number of extensions a matching quote can carry. Cursor + // arithmetic past position zero is consequently unreachable from here, and is covered by the + // differential properties above, which do draw multi-extension quotes and compare the two walks + // on the misses. + same_layers!( + same_layers_ext_hop_by_hop, + ShapedQuote { ext: 0, v4: false }, + O4V6, + (&Ipv6, &HopByHop, &TruncatedTcp), + ip, + ext, + tcp + ); + same_layers!( + same_layers_ext_dest_opts, + ShapedQuote { ext: 1, v4: false }, + O4V6, + (&Ipv6, &DestOpts, &TruncatedTcp), + ip, + ext, + tcp + ); + same_layers!( + same_layers_ext_routing, + ShapedQuote { ext: 2, v4: false }, + O4V6, + (&Ipv6, &Routing, &TruncatedTcp), + ip, + ext, + tcp + ); + same_layers!( + same_layers_ext_fragment, + ShapedQuote { ext: 3, v4: false }, + O4V6, + (&Ipv6, &Fragment, &TruncatedTcp), + ip, + ext, + tcp + ); + same_layers!( + same_layers_ext_v6_auth, + ShapedQuote { ext: 4, v4: false }, + O4V6, + (&Ipv6, &Ipv6Auth, &TruncatedTcp), + ip, + ext, + tcp + ); + same_layers!( + same_layers_ext_v4_auth, + ShapedQuote { ext: 0, v4: true }, + O4V4, + (&Ipv4, &Ipv4Auth, &TruncatedTcp), + ip, + ext, + tcp + ); +} diff --git a/net/src/headers/mod.rs b/net/src/headers/mod.rs index 22fb5237ca..6514cfec17 100644 --- a/net/src/headers/mod.rs +++ b/net/src/headers/mod.rs @@ -1194,7 +1194,10 @@ where mod contract { use crate::eth::ethtype::CommonEthType; use crate::eth::{Eth, GenWithEthType}; - use crate::headers::{Headers, Net, Transport}; + use crate::headers::{ + EmbeddedHeaders, EmbeddedTransport, Headers, MAX_NET_EXTENSIONS, MAX_VLANS, Net, NetExt, + Transport, + }; use crate::icmp4::Icmp4; use crate::icmp6::Icmp6; use crate::ipv4; @@ -1205,6 +1208,7 @@ mod contract { use crate::vxlan::Vxlan; use arrayvec::ArrayVec; use bolero::{Driver, TypeGenerator, ValueGenerator}; + use std::ops::Bound; impl TypeGenerator for Headers { /// Generate a completely arbitrary value of [`Headers`]. @@ -1250,6 +1254,393 @@ mod contract { } } + /// Draws [`Headers`] whose **layer structure** varies: VLAN tags, IPv6 extension headers, and + /// every combination of the two. + /// + /// [`CommonHeaders`] deliberately does not. All six of its construction sites set + /// `vlan: ArrayVec::default()` and `net_ext: ArrayVec::default()`, so it never produces a VLAN tag + /// or an extension header at all -- reasonable for the "sunny-day" packet processing it was written + /// for, and useless for the code whose whole subject is those two things: + /// + /// * [`HeadersView`](crate::headers::view::HeadersView) exists to decide whether a packet's + /// structure matches a type-level shape, and its contract is stated in terms of VLAN tags that + /// are not mentioned in the shape and extension regions the shape may or may not enter; + /// * [`Matcher`](crate::headers::pat::Matcher) threads the same `ExtGapCheck`. + /// + /// Neither semantics could be reached by any existing generator, which is a better explanation of + /// `view.rs` sitting at 42% line coverage than "nobody wrote tests" -- the tests are there. + /// + /// This is structural on purpose. `ViewStep::step` walks the in-memory layers with VLAN and + /// extension cursors, so what matters here is which slots are populated and how many, not whether + /// `next_header` agrees with the layer that follows it. A generator that also kept the wire fields + /// consistent would be the right tool for a parse round-trip property, and the wrong one for this: + /// it would couple a structural test to a byte-level invariant and shrink the space it explores. + #[allow(dead_code)] // constructed through `.with_generator()` + #[repr(transparent)] + pub struct ShapedHeaders; + + impl ValueGenerator for ShapedHeaders { + type Output = Headers; + + fn generate(&self, driver: &mut D) -> Option { + // Start from a common stack so the base layers are realistic, then vary the structure the + // view and matcher semantics actually turn on. + let mut headers = CommonHeaders.generate(driver)?; + + let vlans = driver.gen_usize(Bound::Included(&0), Bound::Included(&MAX_VLANS))?; + for _ in 0..vlans { + headers.vlan.push(driver.produce()?); + } + + // Extension headers hang off the net layer, so there is nothing to attach them to without + // one. + if headers.net.is_some() { + headers.net_ext = ext_run(driver, matches!(headers.net, Some(Net::Ipv4(_))))?; + } + + Some(headers) + } + } + + /// Draw a run of 0..=[`MAX_NET_EXTENSIONS`] extension headers for the given family. + /// + /// The IPv4 authentication header is the only extension that belongs on a v4 packet, so `v4` + /// selects between one choice and five rather than merely reweighting them. + /// + /// A quarter of the v6 runs follow RFC 8200's recommended order instead of drawing each slot + /// independently. Without that bias a shape naming three specific extensions in sequence is + /// drawn about once in fifteen thousand packets, which is enough to keep a property honest and + /// nowhere near enough for it to find anything: measured at two hits in 36,000 cases. The + /// ordered runs are also the ones a real peer sends, so this makes the generator both more + /// useful and more realistic. + fn ext_run( + driver: &mut D, + v4: bool, + ) -> Option> { + let mut out = ArrayVec::default(); + let count = driver.gen_usize(Bound::Included(&0), Bound::Included(&MAX_NET_EXTENSIONS))?; + let ordered = driver.gen_u8(Bound::Included(&0), Bound::Included(&3))? == 0; + for slot in 0..count { + let pick = if ordered { + u8::try_from(slot).unwrap_or(u8::MAX) + } else { + driver.gen_u8(Bound::Included(&0), Bound::Included(&4))? + }; + out.push(one_ext(driver, v4, pick)?); + } + Some(out) + } + + /// Draws a packet with layers deliberately missing, which is what the optional matchers are for. + /// + /// [`CommonHeaders`] sets every layer it knows about on every path, so `net` and `transport` are + /// always `Some` in anything it or [`ShapedHeaders`] produces. That makes the absent-layer arm of + /// every `opt_*` method in [`pat`](crate::headers::pat) -- `None => Some(a.append(None))`, the + /// arm that distinguishes "the layer is not there, which is fine" from "the layer is there and is + /// the wrong one, which is a miss" -- unreachable by any generator in the crate. The whole point + /// of an optional matcher is the packet that stops early, and nothing could draw one. + /// + /// Truncation here is by *suffix*, because that is the only shape a real short packet takes: a + /// layer sits inside the one below it, so a packet cannot carry a transport header without a + /// network header to carry it. Dropping a suffix also keeps `net_ext` consistent, since + /// extensions hang off the network layer and have to go when it does. + #[allow(dead_code)] // constructed through `.with_generator()` + #[repr(transparent)] + pub struct ThinHeaders; + + impl ValueGenerator for ThinHeaders { + type Output = Headers; + + fn generate(&self, driver: &mut D) -> Option { + let mut headers = ShapedHeaders.generate(driver)?; + // Keep the full stack a fifth of the time, so a property reading this generator still + // sees the ordinary case alongside the truncated ones. + match driver.gen_u8(Bound::Included(&0), Bound::Included(&4))? { + 0 => {} + 1 => headers.udp_encap = None, + 2 => { + headers.udp_encap = None; + headers.transport = None; + } + 3 => { + headers.udp_encap = None; + headers.transport = None; + headers.net_ext.clear(); + headers.net = None; + } + _ => { + headers.udp_encap = None; + headers.transport = None; + headers.net_ext.clear(); + headers.net = None; + headers.vlan.clear(); + headers.eth = None; + } + } + Some(headers) + } + } + + /// [`ShapedHeaders`] with the Ethernet header taken away one packet in eight. + /// + /// Every `Shape` starts at `Eth`, and the branch where that first step refuses is generated + /// separately at each of the eight arities -- eight copies of `return false`, none of which any + /// generator could reach, because `CommonHeaders` sets `eth` on all six of its paths. + /// + /// [`ThinHeaders`] reaches them, but it truncates four packets in five, and the deep shapes pay + /// for that: arity seven already needs exactly four VLAN tags, so another factor of five would + /// leave it matching too rarely to find anything. One in eight covers the branch at every arity + /// while leaving seven eighths of the draw doing the work it was doing before. + #[allow(dead_code)] // constructed through `.with_generator()` + #[repr(transparent)] + pub struct SometimesHeadless; + + impl ValueGenerator for SometimesHeadless { + type Output = Headers; + + fn generate(&self, driver: &mut D) -> Option { + let mut headers = ShapedHeaders.generate(driver)?; + if driver.gen_u8(Bound::Included(&0), Bound::Included(&7))? == 0 { + // The tags go with it. A VLAN tag is carried by the Ethernet header, so a packet + // holding one without the other is not a short packet, it is an impossible one, and + // the properties reading this generator are about short packets. + headers.vlan.clear(); + headers.eth = None; + } + Some(headers) + } + } + + /// Draw one extension header, with fuzzed contents. + /// + /// `pick` indexes the IPv6 extension order RFC 8200 recommends -- 0 hop-by-hop, 1 destination + /// options, 2 routing, 3 fragment, anything else authentication -- and is ignored for `v4`, + /// where the IPv4 authentication header is the only extension there is. + fn one_ext(driver: &mut D, v4: bool, pick: u8) -> Option { + if v4 { + return Some(NetExt::Ipv4Auth(driver.produce()?)); + } + Some(match pick { + 0 => NetExt::HopByHop(driver.produce()?), + 1 => NetExt::DestOpts(driver.produce()?), + 2 => NetExt::Routing(driver.produce()?), + 3 => NetExt::Fragment(driver.produce()?), + _ => NetExt::Ipv6Auth(driver.produce()?), + }) + } + + /// Draws an ICMP error whose quoted packet carries exactly one chosen extension header, and + /// nothing else in the way. + /// + /// Companion to [`ShapedIcmpError`], and the division between them is deliberate. A property + /// comparing *hit against miss* needs both, so it wants the broad generator. A property that + /// only inspects matches -- does `look` pick the same layer `look_mut` does -- gets nothing from + /// a packet it skips, so for that one a narrow generator is not a weaker test but a stronger + /// one. + /// + /// The difference is not marginal. Naming a specific extension inside a quoted packet compounds + /// six independent conditions, one of which is `P(no VLAN tags) = 1/5`, since a tag the outer + /// shape does not name is a miss. Measured on [`ShapedIcmpError`], + /// `(&Ipv6, &DestOpts, &TruncatedTcp)` matched 6 packets in 23,910: a property doing real work + /// six times a second, and a hit rate low enough that asserting it is ever non-zero is itself a + /// coin flip. This generator produces that shape every time, and fuzzes the contents instead. + #[allow(dead_code)] // constructed through `.with_generator()` + pub struct ShapedQuote { + /// Which extension header to place, as the `pick` index [`one_ext`] uses. + pub ext: u8, + /// Family of both the quoting message and the packet it quotes. + pub v4: bool, + } + + impl ValueGenerator for ShapedQuote { + type Output = Headers; + + fn generate(&self, driver: &mut D) -> Option { + let eth_type = if self.v4 { + CommonEthType::Ipv4 + } else { + CommonEthType::Ipv6 + }; + let eth = GenWithEthType(eth_type.into()).generate(driver)?; + + let mut quoted_ext = ArrayVec::default(); + quoted_ext.push(one_ext(driver, self.v4, self.ext)?); + let quoted_transport = EmbeddedTransport::Tcp(driver.produce()?); + + let (net, transport, quoted_net) = if self.v4 { + ( + Net::Ipv4( + ipv4::GenWithNextHeader(ipv4::CommonNextHeader::Icmp4.into()) + .generate(driver)?, + ), + Transport::Icmp4(driver.produce()?), + Net::Ipv4( + ipv4::GenWithNextHeader(ipv4::CommonNextHeader::Tcp.into()) + .generate(driver)?, + ), + ) + } else { + ( + Net::Ipv6( + ipv6::GenWithNextHeader(ipv6::CommonNextHeader::Icmp6.into()) + .generate(driver)?, + ), + Transport::Icmp6(driver.produce()?), + Net::Ipv6( + ipv6::GenWithNextHeader(ipv6::CommonNextHeader::Tcp.into()) + .generate(driver)?, + ), + ) + }; + + Some(Headers { + eth: Some(eth), + // No VLAN tags and no outer extensions: a tag the outer shape does not name is a + // miss, and this generator exists to stop producing misses. + vlan: ArrayVec::default(), + net: Some(net), + net_ext: ArrayVec::default(), + transport: Some(transport), + udp_encap: None, + embedded_ip: Some(EmbeddedHeaders::new( + Some(quoted_net), + Some(quoted_transport), + quoted_ext, + None, + )), + }) + } + } + + /// Draws an ICMP error message quoting an inner packet, with the layer structure of *both* the + /// outer message and the quoted packet varying. + /// + /// [`CommonHeaders`] cannot produce one at all: all six of its construction sites set + /// `embedded_ip: None`, so [`ShapedHeaders`], which builds on it, cannot either. That is a + /// better explanation of [`embedded_view`](crate::headers::embedded_view) sitting at 34% line + /// coverage than any claim about missing tests -- its hand-written tests are thorough, but every + /// one of them builds its packet through `HeaderStack`/`header_chain`, which pin the shape at + /// `(Eth, Ipv4, Icmp4)` outside and `(Ipv4, Tcp)` inside and offer no way to attach an extension + /// header to either side. Six of the eight `as_embedded` arities and the whole embedded + /// extension region were unreachable by construction. + /// + /// What varies here is what `EmbeddedShape`, `EmbeddedStep` and + /// [`ExtGapCheck`](crate::headers::pat::ExtGapCheck) actually turn on: + /// + /// * outer VLAN tags and outer extension headers, so the outer arity spans the range of + /// `as_embedded` impls instead of only the one a builder can express; + /// * whether the embedded section is present at all -- an ICMP message that is not an error + /// quotes nothing, and every embedded shape must miss on it; + /// * the inner network layer, its extension headers, and its truncated transport, including a + /// quote that stops before the transport header and the ICMP-inside-ICMP case. + /// + /// The inner family follows the outer family most of the time, because that is what a real ICMP + /// error looks like: an `ICMPv4` message quotes an IPv4 packet. It deliberately does not always. + /// A family mismatch is where two independent structural walks are most likely to disagree, so + /// it wants drawing rather than assuming away. + /// + /// Structural on purpose, for the same reason as [`ShapedHeaders`]: `next_header` is set to + /// match the transport where one was drawn, but nothing keeps the extension chain's own + /// `next_header` fields consistent, because no code under test reads them. + #[allow(dead_code)] // constructed through `.with_generator()` + #[repr(transparent)] + pub struct ShapedIcmpError; + + impl ValueGenerator for ShapedIcmpError { + type Output = Headers; + + fn generate(&self, driver: &mut D) -> Option { + let outer_v4 = driver.produce::()?; + let eth_type = if outer_v4 { + CommonEthType::Ipv4 + } else { + CommonEthType::Ipv6 + }; + let eth = GenWithEthType(eth_type.into()).generate(driver)?; + + let mut vlan = ArrayVec::default(); + let vlans = driver.gen_usize(Bound::Included(&0), Bound::Included(&MAX_VLANS))?; + for _ in 0..vlans { + vlan.push(driver.produce()?); + } + + let (net, transport) = if outer_v4 { + let ip = ipv4::GenWithNextHeader(ipv4::CommonNextHeader::Icmp4.into()) + .generate(driver)?; + (Net::Ipv4(ip), Transport::Icmp4(driver.produce()?)) + } else { + let ip = ipv6::GenWithNextHeader(ipv6::CommonNextHeader::Icmp6.into()) + .generate(driver)?; + (Net::Ipv6(ip), Transport::Icmp6(driver.produce()?)) + }; + + // Echo requests and replies are not errors and quote nothing, so absence is a shape the + // API has to handle, not an edge case to skip. + let embedded_ip = if driver.produce::()? { + Some(quoted_packet(driver, outer_v4)?) + } else { + None + }; + + Some(Headers { + eth: Some(eth), + vlan, + net: Some(net), + net_ext: ext_run(driver, outer_v4)?, + transport: Some(transport), + udp_encap: None, + embedded_ip, + }) + } + } + + /// Draw the packet quoted inside an ICMP error. `outer_v4` is the family of the quoting + /// message; see [`ShapedIcmpError`] for why the quoted packet usually but not always shares it. + fn quoted_packet(driver: &mut D, outer_v4: bool) -> Option { + // A quoting host copies as much of the offending packet as it can, and RFC 792 asked for + // only the header plus eight bytes. A quote can therefore be too short to hold even the + // network header -- which is the one case where the optional embedded matchers' absent-layer + // arm is reachable, since a quote that *has* a network layer always has a version. + if driver.gen_u8(Bound::Included(&0), Bound::Included(&7))? == 0 { + return Some(EmbeddedHeaders::new(None, None, ArrayVec::default(), None)); + } + let mismatch = driver.gen_u8(Bound::Included(&0), Bound::Included(&7))? == 0; + let v4 = outer_v4 != mismatch; + + // Pick the transport first so the network layer's `next_header` can name it. A quote that + // stops before the transport header is the truncation the `Truncated*` types exist for. + let transport = match driver.gen_u8(Bound::Included(&0), Bound::Included(&3))? { + 0 => Some(EmbeddedTransport::Tcp(driver.produce()?)), + 1 => Some(EmbeddedTransport::Udp(driver.produce()?)), + 2 if v4 => Some(EmbeddedTransport::Icmp4(driver.produce()?)), + 2 => Some(EmbeddedTransport::Icmp6(driver.produce()?)), + _ => None, + }; + + let net = if v4 { + let next = match transport { + Some(EmbeddedTransport::Udp(_)) => ipv4::CommonNextHeader::Udp, + Some(EmbeddedTransport::Icmp4(_)) => ipv4::CommonNextHeader::Icmp4, + _ => ipv4::CommonNextHeader::Tcp, + }; + Net::Ipv4(ipv4::GenWithNextHeader(next.into()).generate(driver)?) + } else { + let next = match transport { + Some(EmbeddedTransport::Udp(_)) => ipv6::CommonNextHeader::Udp, + Some(EmbeddedTransport::Icmp6(_)) => ipv6::CommonNextHeader::Icmp6, + _ => ipv6::CommonNextHeader::Tcp, + }; + Net::Ipv6(ipv6::GenWithNextHeader(next.into()).generate(driver)?) + }; + + Some(EmbeddedHeaders::new( + Some(net), + transport, + ext_run(driver, v4)?, + None, + )) + } + #[allow(dead_code)] // rustc not able to infer we construct this through .with_generator() #[repr(transparent)] pub struct CommonHeaders; @@ -1390,7 +1781,7 @@ mod test { use crate::tcp::{TcpChecksum, TcpChecksumPayload, TcpPort}; use crate::udp::{UdpChecksum, UdpChecksumPayload, UdpPort}; - fn parse_back_test(headers: &Headers) { + pub(crate) fn parse_back_test(headers: &Headers) { let mut buffer = [0_u8; 1024]; let bytes_written = match headers.deparse(&mut buffer[..headers.size().into_non_zero_usize().get()]) { diff --git a/net/src/headers/pat.rs b/net/src/headers/pat.rs index a5f640119d..d38a1f0436 100644 --- a/net/src/headers/pat.rs +++ b/net/src/headers/pat.rs @@ -2955,3 +2955,742 @@ mod tests { }; } } + +// =========================================================================== +// Optional-layer and combinator properties +// =========================================================================== + +#[cfg(test)] +#[allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] // fine to unwrap in tests +mod opt_properties { + use super::*; + use crate::headers::{Headers, ShapedIcmpError, ThinHeaders}; + use std::cell::Cell; + + /// `opt_X` is never stricter than `X`, for every layer and on both the read and mutable paths. + /// + /// One invariant, universally true, across three families whose internals differ sharply. + /// `opt_eth` cannot miss at all. `opt_vlan` and the optional extension methods cannot miss + /// either, but advance their cursor only when they matched, so they skip rather than refuse. + /// `opt_net`, the optional transport methods and `opt_vxlan` are three-way: present and right is + /// a hit, absent is a hit carrying `None`, present and wrong is a miss. Whatever the family, + /// weakening a requirement cannot turn a match into a miss -- and the direction is the thing a + /// mis-wiring would invert, since `and_then` and `map` differ by exactly that. + /// + /// Every one of these methods was uncovered before this: 250 of `pat.rs`'s 296 unreached lines + /// were the `opt_*` family and the combinators below. + macro_rules! opt_is_weaker { + ($read:ident, $mutable:ident, [$($pre:ident),*], $strict:ident, $opt:ident) => { + #[test] + fn $read() { + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + static STRICT: AtomicUsize = AtomicUsize::new(0); + static OPT_ONLY: AtomicUsize = AtomicUsize::new(0); + bolero::check!() + .with_generator(ThinHeaders) + .for_each(|h: &Headers| { + let strict = h.pat()$(.$pre())*.$strict().done().is_some(); + let opt = h.pat()$(.$pre())*.$opt().done().is_some(); + assert!( + !strict || opt, + concat!( + "`", stringify!($strict), "` matched where `", stringify!($opt), + "` did not, so the optional form is the stricter one: {:?}" + ), + h + ); + if strict { + STRICT.fetch_add(1, Ordering::Relaxed); + } else if opt { + OPT_ONLY.fetch_add(1, Ordering::Relaxed); + } + }); + both_outcomes_seen( + concat!(stringify!($opt), " (read)"), + STRICT.load(Ordering::Relaxed), + OPT_ONLY.load(Ordering::Relaxed), + ); + } + + #[test] + fn $mutable() { + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + static STRICT: AtomicUsize = AtomicUsize::new(0); + static OPT_ONLY: AtomicUsize = AtomicUsize::new(0); + bolero::check!() + .with_generator(ThinHeaders) + .for_each(|h: &Headers| { + let mut owned = h.clone(); + let strict = owned.pat_mut()$(.$pre())*.$strict().done().is_some(); + let opt = owned.pat_mut()$(.$pre())*.$opt().done().is_some(); + assert!( + !strict || opt, + concat!( + "`", stringify!($strict), "` matched where `", stringify!($opt), + "` did not on the mutable path: {:?}" + ), + h + ); + if strict { + STRICT.fetch_add(1, Ordering::Relaxed); + } else if opt { + OPT_ONLY.fetch_add(1, Ordering::Relaxed); + } + }); + both_outcomes_seen( + concat!(stringify!($opt), " (mut)"), + STRICT.load(Ordering::Relaxed), + OPT_ONLY.load(Ordering::Relaxed), + ); + } + }; + } + + /// An implication is satisfied for free when its antecedent never holds, and again when the two + /// sides never differ. + /// + /// `strict implies opt` would pass on a generator that produced nothing matching -- and it would + /// pass just as quietly on one where `opt` never accepted anything `strict` refused, which is the + /// more likely failure and the one that would make the optional method's whole reason for + /// existing untested. So both counts have to be non-zero: the requirement is sometimes met, and + /// relaxing it sometimes matters. + fn both_outcomes_seen(what: &str, strict: usize, opt_only: usize) { + println!("{what}: {strict} strict matches, {opt_only} matched only optionally"); + assert!( + strict > 0, + "{what}: the strict form never matched, so the implication held vacuously" + ); + assert!( + opt_only > 0, + "{what}: the optional form never accepted anything the strict form refused, so being \ + optional was never tested" + ); + } + + opt_is_weaker!(read_opt_eth, mut_opt_eth, [], eth, opt_eth); + opt_is_weaker!(read_opt_vlan, mut_opt_vlan, [eth], vlan, opt_vlan); + opt_is_weaker!(read_opt_net, mut_opt_net, [eth], net, opt_net); + opt_is_weaker!(read_opt_ipv4, mut_opt_ipv4, [eth], ipv4, opt_ipv4); + opt_is_weaker!(read_opt_ipv6, mut_opt_ipv6, [eth], ipv6, opt_ipv6); + opt_is_weaker!( + read_opt_hop_by_hop, + mut_opt_hop_by_hop, + [eth, ipv6], + hop_by_hop, + opt_hop_by_hop + ); + opt_is_weaker!( + read_opt_dest_opts, + mut_opt_dest_opts, + [eth, ipv6], + dest_opts, + opt_dest_opts + ); + opt_is_weaker!( + read_opt_routing, + mut_opt_routing, + [eth, ipv6], + routing, + opt_routing + ); + opt_is_weaker!( + read_opt_fragment, + mut_opt_fragment, + [eth, ipv6], + fragment, + opt_fragment + ); + opt_is_weaker!( + read_opt_ipv6_auth, + mut_opt_ipv6_auth, + [eth, ipv6], + ipv6_auth, + opt_ipv6_auth + ); + opt_is_weaker!( + read_opt_ipv4_auth, + mut_opt_ipv4_auth, + [eth, ipv4], + ipv4_auth, + opt_ipv4_auth + ); + opt_is_weaker!(read_opt_tcp, mut_opt_tcp, [eth, net], tcp, opt_tcp); + opt_is_weaker!(read_opt_udp, mut_opt_udp, [eth, net], udp, opt_udp); + opt_is_weaker!(read_opt_icmp4, mut_opt_icmp4, [eth, ipv4], icmp4, opt_icmp4); + opt_is_weaker!(read_opt_icmp6, mut_opt_icmp6, [eth, ipv6], icmp6, opt_icmp6); + opt_is_weaker!( + read_opt_transport, + mut_opt_transport, + [eth, net], + transport, + opt_transport + ); + opt_is_weaker!( + read_opt_vxlan, + mut_opt_vxlan, + [eth, net, udp], + vxlan, + opt_vxlan + ); + + /// The same invariant for the matchers over a quoted ICMP-error payload. + /// + /// Separate macro only because the chain has to pass through `.embedded()` partway, which is not + /// a layer name. The embedded families mirror the outer ones and are generated by their own set + /// of macros, so every arm needs reaching on its own -- covering `opt_hop_by_hop` says nothing + /// about the five sibling copies `embedded_ext!` emits. + macro_rules! embedded_opt_is_weaker { + ( + $read:ident, $mutable:ident, + [$($o:ident),*], [$($i:ident),*], + $strict:ident, $opt:ident + ) => { + #[test] + fn $read() { + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + static STRICT: AtomicUsize = AtomicUsize::new(0); + static OPT_ONLY: AtomicUsize = AtomicUsize::new(0); + bolero::check!() + .with_generator(ShapedIcmpError) + .for_each(|h: &Headers| { + let strict = h.pat()$(.$o())*.embedded()$(.$i())*.$strict() + .done().is_some(); + let opt = h.pat()$(.$o())*.embedded()$(.$i())*.$opt() + .done().is_some(); + assert!( + !strict || opt, + concat!( + "quoted `", stringify!($strict), "` matched where `", + stringify!($opt), "` did not: {:?}" + ), + h + ); + if strict { + STRICT.fetch_add(1, Ordering::Relaxed); + } else if opt { + OPT_ONLY.fetch_add(1, Ordering::Relaxed); + } + }); + both_outcomes_seen( + concat!("quoted ", stringify!($opt), " (read)"), + STRICT.load(Ordering::Relaxed), + OPT_ONLY.load(Ordering::Relaxed), + ); + } + + #[test] + fn $mutable() { + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + static STRICT: AtomicUsize = AtomicUsize::new(0); + static OPT_ONLY: AtomicUsize = AtomicUsize::new(0); + bolero::check!() + .with_generator(ShapedIcmpError) + .for_each(|h: &Headers| { + let mut owned = h.clone(); + let strict = owned.pat_mut()$(.$o())*.embedded()$(.$i())*.$strict() + .done().is_some(); + let opt = owned.pat_mut()$(.$o())*.embedded()$(.$i())*.$opt() + .done().is_some(); + assert!( + !strict || opt, + concat!( + "quoted `", stringify!($strict), "` matched where `", + stringify!($opt), "` did not on the mutable path: {:?}" + ), + h + ); + if strict { + STRICT.fetch_add(1, Ordering::Relaxed); + } else if opt { + OPT_ONLY.fetch_add(1, Ordering::Relaxed); + } + }); + both_outcomes_seen( + concat!("quoted ", stringify!($opt), " (mut)"), + STRICT.load(Ordering::Relaxed), + OPT_ONLY.load(Ordering::Relaxed), + ); + } + }; + } + + embedded_opt_is_weaker!( + read_quoted_opt_ipv4, + mut_quoted_opt_ipv4, + [eth, ipv4, icmp4], + [], + ipv4, + opt_ipv4 + ); + embedded_opt_is_weaker!( + read_quoted_opt_ipv6, + mut_quoted_opt_ipv6, + [eth, ipv6, icmp6], + [], + ipv6, + opt_ipv6 + ); + // No `opt_net` or `opt_transport` pair here, and no `transport` on the read side, because the + // embedded matchers do not have them. The enum-level vocabulary is complete on the outer + // matchers and mostly absent on the embedded ones: + // + // | | `net` | `opt_net` | `transport` | `opt_transport` | + // |----------------------|-------|-----------|-------------|-----------------| + // | `Matcher` | yes | yes | yes | yes | + // | `MatcherMut` | yes | yes | yes | yes | + // | `EmbeddedMatcher` | yes | no | no | no | + // | `EmbeddedMatcherMut` | yes | no | yes | no | + // + // Five methods missing, all hand-written rather than macro-generated, which is the likely reason + // -- the per-variant methods come from `embedded_net!` and `embedded_transport!` and are all + // present. The consequence is not cosmetic: a shape naming `Net` or `EmbeddedTransport` inside a + // quoted packet cannot be written as a matcher chain, so `embedded_view`'s differential + // properties have no read-side oracle for the enum forms either. + embedded_opt_is_weaker!( + read_quoted_opt_hop_by_hop, + mut_quoted_opt_hop_by_hop, + [eth, ipv6, icmp6], + [ipv6], + hop_by_hop, + opt_hop_by_hop + ); + embedded_opt_is_weaker!( + read_quoted_opt_dest_opts, + mut_quoted_opt_dest_opts, + [eth, ipv6, icmp6], + [ipv6], + dest_opts, + opt_dest_opts + ); + embedded_opt_is_weaker!( + read_quoted_opt_routing, + mut_quoted_opt_routing, + [eth, ipv6, icmp6], + [ipv6], + routing, + opt_routing + ); + embedded_opt_is_weaker!( + read_quoted_opt_fragment, + mut_quoted_opt_fragment, + [eth, ipv6, icmp6], + [ipv6], + fragment, + opt_fragment + ); + embedded_opt_is_weaker!( + read_quoted_opt_ipv6_auth, + mut_quoted_opt_ipv6_auth, + [eth, ipv6, icmp6], + [ipv6], + ipv6_auth, + opt_ipv6_auth + ); + embedded_opt_is_weaker!( + read_quoted_opt_ipv4_auth, + mut_quoted_opt_ipv4_auth, + [eth, ipv4, icmp4], + [ipv4], + ipv4_auth, + opt_ipv4_auth + ); + embedded_opt_is_weaker!( + read_quoted_opt_tcp, + mut_quoted_opt_tcp, + [eth, ipv4, icmp4], + [ipv4], + tcp, + opt_tcp + ); + embedded_opt_is_weaker!( + read_quoted_opt_udp, + mut_quoted_opt_udp, + [eth, ipv6, icmp6], + [ipv6], + udp, + opt_udp + ); + embedded_opt_is_weaker!( + read_quoted_opt_icmp4, + mut_quoted_opt_icmp4, + [eth, ipv4, icmp4], + [ipv4], + icmp4, + opt_icmp4 + ); + embedded_opt_is_weaker!( + read_quoted_opt_icmp6, + mut_quoted_opt_icmp6, + [eth, ipv6, icmp6], + [ipv6], + icmp6, + opt_icmp6 + ); + + // ---- Exact semantics, per family ------------------------------------------------------ + + /// The three-way families accept a matching layer or an absent one, and refuse a wrong one. + /// + /// `opt_is_weaker` above only pins the direction. This states the whole rule, and the arm it + /// exists for is the middle one: a packet that stops before the layer is a *hit* carrying `None`, + /// while a packet carrying the wrong layer is a miss. Conflating those two is the mistake this + /// family invites, and no generator could produce the first case until `ThinHeaders`. + #[test] + fn the_three_way_families_separate_an_absent_layer_from_a_wrong_one() { + bolero::check!() + .with_generator(ThinHeaders) + .for_each(|h: &Headers| { + // Reachability, spelled out once: a strict `.eth()` needs an Ethernet header, and + // the network step's gap check needs every VLAN tag consumed -- none were named. + let reached_net = h.eth().is_some() && h.vlan().is_empty(); + + assert_eq!( + h.pat().eth().opt_net().done().is_some(), + reached_net, + "the network enum is never the wrong variant, so absent or present must both \ + match: {h:?}" + ); + assert_eq!( + h.pat().eth().opt_ipv4().done().is_some(), + reached_net && !matches!(h.net(), Some(Net::Ipv6(_))), + "opt_ipv4 must accept IPv4 and absence, and refuse IPv6: {h:?}" + ); + assert_eq!( + h.pat().eth().opt_ipv6().done().is_some(), + reached_net && !matches!(h.net(), Some(Net::Ipv4(_))), + "opt_ipv6 must accept IPv6 and absence, and refuse IPv4: {h:?}" + ); + + let reached_transport = reached_net && h.net().is_some(); + assert_eq!( + h.pat().eth().net().opt_transport().done().is_some(), + reached_transport, + "the transport enum is never the wrong variant either: {h:?}" + ); + assert_eq!( + h.pat().eth().net().opt_tcp().done().is_some(), + reached_transport && matches!(h.transport(), None | Some(Transport::Tcp(_))), + "opt_tcp must accept TCP and absence, and refuse every other transport: {h:?}" + ); + + // `opt_vxlan` sits behind a concrete `.udp()`, so its own absent arm is the packet + // that carries UDP and no encapsulation -- much the commoner case in real traffic. + let reached_vxlan = + reached_transport && matches!(h.transport(), Some(Transport::Udp(_))); + assert_eq!( + h.pat().eth().net().udp().opt_vxlan().done().is_some(), + reached_vxlan, + "a UDP packet with no encapsulation must match opt_vxlan: {h:?}" + ); + }); + } + + /// The cursor families skip rather than refuse, and advance only on a hit. + /// + /// This is the sharpest observable difference between the two designs, and it is entirely about + /// the cursor. `opt_vlan` cannot miss, so a chain of them followed by a *strict* network step + /// succeeds exactly when the packet has no more tags than the chain has optional slots -- the + /// strict step's gap check is what makes the cursor's behaviour visible from outside. + #[test] + fn optional_vlans_absorb_one_tag_each_and_only_when_they_match() { + bolero::check!() + .with_generator(ThinHeaders) + .for_each(|h: &Headers| { + let reachable = h.eth().is_some() && h.net().is_some(); + let tags = h.vlan().len(); + assert_eq!( + h.pat().eth().opt_vlan().net().done().is_some(), + reachable && tags <= 1, + "one optional tag absorbed the wrong number of tags: {h:?}" + ); + assert_eq!( + h.pat().eth().opt_vlan().opt_vlan().net().done().is_some(), + reachable && tags <= 2, + "two optional tags absorbed the wrong number of tags: {h:?}" + ); + // `MAX_VLANS` is four, so four optional slots absorb any packet there can be. If + // `opt_vlan` advanced its cursor on a miss this would fail on the untagged packets. + assert_eq!( + h.pat() + .eth() + .opt_vlan() + .opt_vlan() + .opt_vlan() + .opt_vlan() + .net() + .done() + .is_some(), + reachable, + "four optional tags failed to absorb a packet with at most four: {h:?}" + ); + }); + } + + /// Naming an extension optionally still enters the region, which makes the gap check strict. + /// + /// The subtlety worth a test: `opt_hop_by_hop` cannot itself fail, so it looks harmless, but it + /// moves `Pos` to `HopByHop` and that is what `ExtGapCheck` dispatches on. The transport step + /// afterwards therefore demands every extension be consumed -- so an *optional* extension can + /// turn a later, unrelated step into a miss. + #[test] + fn an_optional_extension_still_makes_the_transport_gap_check_strict() { + bolero::check!() + .with_generator(ThinHeaders) + .for_each(|h: &Headers| { + let consumed = + usize::from(matches!(h.net_ext().first(), Some(NetExt::HopByHop(_)))); + let want = h.eth().is_some() + && h.vlan().is_empty() + && matches!(h.net(), Some(Net::Ipv6(_))) + && h.transport().is_some() + && h.net_ext().len() == consumed; + assert_eq!( + h.pat() + .eth() + .ipv6() + .opt_hop_by_hop() + .transport() + .done() + .is_some(), + want, + "an optional extension left the transport gap check lenient: {h:?}" + ); + }); + } + + // ---- Combinators ---------------------------------------------------------------------- + + /// `when`, `inspect` and `otherwise` on all four matchers. + /// + /// Three small methods repeated four times, and all twelve copies were unreached. Two of them + /// are side-effecting, which makes "did it run" the entire contract rather than a detail: + /// `inspect` must run exactly on a match and `otherwise` exactly on a miss, and neither may + /// change the outcome. `when` must be able to destroy a match and must never manufacture one -- + /// on a chain that already failed, a `true` predicate has nothing to revive. + macro_rules! combinators_fire_exactly_once_and_only_when_due { + ($name:ident, $gen:expr, $subject:expr, $fires:expr, $($chain:tt)*) => { + // The `mut` binding is what `pat_mut()` needs and what `pat()` does not, and the same + // macro serves both, so half the instantiations declare a `mut` they never use. + #[allow(unused_mut)] + #[test] + fn $name() { + bolero::check!() + .with_generator($gen) + .for_each(|h: &Headers| { + // The chain is a token sequence rather than a closure because a closure + // returning a matcher borrowed from its own argument needs a higher-ranked + // lifetime, which closure inference will not produce. + let mut owned = h.clone(); + let base = owned $($chain)* .done().is_some(); + // What the combinators actually track, which is not always `base`: see + // `the_embedded_combinators_track_the_inner_match_only` below. + let fires: bool = $fires(h); + + let mut owned = h.clone(); + assert!( + owned $($chain)* .when(|_| false).done().is_none(), + concat!($subject, ": a false predicate left the match standing: {:?}"), + h + ); + let mut owned = h.clone(); + assert_eq!( + owned $($chain)* .when(|_| true).done().is_some(), + base, + concat!($subject, ": a true predicate was not a no-op: {:?}"), + h + ); + + let ran = Cell::new(false); + let mut owned = h.clone(); + let after = owned $($chain)* + .inspect(|_| ran.set(true)) + .done() + .is_some(); + assert_eq!( + ran.get(), fires, + concat!($subject, ": inspect ran on a miss or skipped a match: {:?}"), + h + ); + assert_eq!( + after, base, + concat!($subject, ": inspect changed the result: {:?}"), + h + ); + + let ran = Cell::new(false); + let mut owned = h.clone(); + let after = owned $($chain)* + .otherwise(|| ran.set(true)) + .done() + .is_some(); + assert_eq!( + ran.get(), !fires, + concat!($subject, ": otherwise ran on a match or skipped a miss: {:?}"), + h + ); + assert_eq!( + after, base, + concat!($subject, ": otherwise changed the result: {:?}"), + h + ); + }); + } + }; + } + + /// The outer matchers carry one accumulator, so their combinators fire exactly on the result. + fn whole_chain(h: &Headers) -> bool { + h.pat().eth().net().done().is_some() + } + + /// The embedded matchers carry two, and their combinators watch only the inner one. + /// + /// `.embedded().ipv4()` leaves the inner accumulator populated exactly when a quote is present + /// and its network layer is IPv4 -- a condition that says nothing about whether the outer chain + /// that led there succeeded. + fn quoted_ipv4_matched(h: &Headers) -> bool { + h.embedded_ip() + .is_some_and(|e| matches!(e.net(), Some(Net::Ipv4(_)))) + } + + combinators_fire_exactly_once_and_only_when_due!( + matcher_combinators, ThinHeaders, "Matcher", whole_chain, + .pat().eth().net() + ); + combinators_fire_exactly_once_and_only_when_due!( + matcher_mut_combinators, ThinHeaders, "MatcherMut", whole_chain, + .pat_mut().eth().net() + ); + combinators_fire_exactly_once_and_only_when_due!( + embedded_matcher_combinators, ShapedIcmpError, "EmbeddedMatcher", quoted_ipv4_matched, + .pat().eth().ipv4().icmp4().embedded().ipv4() + ); + combinators_fire_exactly_once_and_only_when_due!( + embedded_matcher_mut_combinators, ShapedIcmpError, "EmbeddedMatcherMut", + quoted_ipv4_matched, + .pat_mut().eth().ipv4().icmp4().embedded().ipv4() + ); + + /// `otherwise` does not run for every chain that returns `None`, and `inspect` runs for some. + /// + /// Found by the property above, which originally assumed the combinators tracked `.done()`. + /// They do not, and the difference is only observable on the embedded matchers, which carry two + /// accumulators: `done()` requires *both* to be populated, while `when`, `inspect` and + /// `otherwise` all read the inner one alone. + /// + /// So a packet whose outer chain fails -- an unconsumed VLAN tag will do it -- but whose quoted + /// packet matches will run `inspect`, skip `otherwise`, and then return `None`. The doc comments + /// are accurate ("apply a predicate to the inner accumulator", "run a closure if the inner match + /// has already failed"), so this is documented rather than broken. It still seems worth a + /// decision: `otherwise` is the error-handling hook, and there is a whole class of failure it + /// stays silent for. + /// + /// This test pins the behaviour as it stands. If the combinators are ever changed to track + /// `done()` it will fail, which is the point -- that should be a decision rather than a + /// discovery. + #[test] + fn the_embedded_combinators_track_the_inner_match_only() { + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + static DIVERGED: AtomicUsize = AtomicUsize::new(0); + bolero::check!() + .with_generator(ShapedIcmpError) + .for_each(|h: &Headers| { + let whole = h + .pat() + .eth() + .ipv4() + .icmp4() + .embedded() + .ipv4() + .done() + .is_some(); + let inner = quoted_ipv4_matched(h); + if whole == inner { + return; + } + DIVERGED.fetch_add(1, Ordering::Relaxed); + // Divergence is one-directional: the inner match can succeed where the whole chain + // fails, never the reverse, since `done()` needs the inner accumulator too. + assert!( + inner && !whole, + "the whole chain matched while the inner one did not, which `done` forbids: \ + {h:?}" + ); + let ran = Cell::new(false); + let _ = h + .pat() + .eth() + .ipv4() + .icmp4() + .embedded() + .ipv4() + .otherwise(|| ran.set(true)) + .done(); + assert!( + !ran.get(), + "`otherwise` ran on a chain whose inner match succeeded; the divergence \ + documented here has been fixed, so this test should be deleted: {h:?}" + ); + }); + let diverged = DIVERGED.load(Ordering::Relaxed); + println!("outer failed while the quote matched: {diverged} packets"); + assert!( + diverged > 0, + "the two never diverged, so this test proved nothing about which one the combinators \ + follow" + ); + } + + /// The quoted transport enum refuses an unconsumed extension on the mutable path too. + /// + /// `EmbeddedMatcherMut::transport` is the one enum-level embedded method that exists, and its + /// gap-check rejection had nothing reaching it. Stated directly rather than differentially, + /// because the read-side counterpart it would be compared against is one of the five missing + /// methods listed above. + #[test] + fn the_quoted_transport_enum_refuses_an_unconsumed_extension() { + bolero::check!() + .with_generator(crate::headers::ShapedQuote { ext: 0, v4: false }) + .for_each(|h: &Headers| { + let mut owned = h.clone(); + let quoted = owned + .embedded_ip_mut() + .unwrap_or_else(|| unreachable!("ShapedQuote always attaches a quote")); + let first = quoted + .net_ext + .first() + .unwrap_or_else(|| unreachable!("ShapedQuote always places one extension")) + .clone(); + quoted.net_ext.push(first); + + let mut two = owned.clone(); + assert!( + two.pat_mut() + .eth() + .ipv6() + .icmp6() + .embedded() + .ipv6() + .hop_by_hop() + .transport() + .done() + .is_none(), + "one extension named of two, yet the transport enum matched: {h:?}" + ); + let mut one = h.clone(); + assert!( + one.pat_mut() + .eth() + .ipv6() + .icmp6() + .embedded() + .ipv6() + .hop_by_hop() + .transport() + .done() + .is_some(), + "the sole extension was named and consumed, yet the chain missed: {h:?}" + ); + }); + } +} diff --git a/net/src/headers/view.rs b/net/src/headers/view.rs index 516f60ff28..9f14ee8135 100644 --- a/net/src/headers/view.rs +++ b/net/src/headers/view.rs @@ -2575,3 +2575,682 @@ mod tests { assert_eq!(v.vid(), vid_updated); } } + +/// The `unsafe` boundary, checked against the safe implementation of the same semantics. +/// +/// [`HeadersView`] earns its zero-cost extraction with `unwrap_unchecked`: [`Look::look`] repeats the +/// [`ViewStep::step`] chain that [`sealed::Sealed::matches`] already ran and tells the compiler the +/// `None` arms cannot happen. **The soundness of that rests entirely on the two chains agreeing**, and +/// they are written out separately for every arity by the macro above -- eight-odd hand-written pairs, +/// each threading the VLAN and extension cursors through by hand. A transposed cursor in one of them +/// is not a wrong answer, it is undefined behaviour. +/// +/// That is the same shape as every defect this campaign found in `routing`: an invariant enforced at a +/// distance by a different function from the one relying on it. The difference is the consequence. +/// +/// The oracle is [`Matcher`](super::pat::Matcher), which decides the same question safely and returns +/// an `Option`. Comparing the two is a differential test between two existing implementations rather +/// than against a third transcription of the rules -- if they disagree, either `matches` admits a +/// packet `look` cannot extract from (undefined behaviour) or `Matcher` mis-matches in the datapath. +/// Both are worth knowing. +/// +/// References are compared by **address**, not by value: two VLAN tags with identical contents are a +/// pass under `assert_eq!` and a bug if the two implementations picked different ones. +/// +/// # Two lines of defence, and where each one fires +/// +/// Both `matches` and `look` call the same [`ViewStep::step`], so a bug *inside* `step` is invisible +/// here -- it moves both sides together. What is checked is the hand-written *chaining* around it, +/// which is where the duplication is. +/// +/// Against a divergence in that chaining there are two independent guards, and both were demonstrated +/// by breaking the arity-3 arm on purpose: +/// +/// 1. **This differential, which fires first.** The `Matcher` comparison happens *before* `look` is +/// called, so a divergence in either direction fails with a shrunk counterexample rather than by +/// invoking undefined behaviour. Making `matches` stricter, or making it accept everything, both +/// fail here in under a second. +/// 2. **The standard library's own check, as a backstop.** `unwrap_unchecked` bottoms out in +/// `hint::unreachable_unchecked`, whose `assert_unsafe_precondition!` is gated on `ub_checks` -- +/// which follows `-Cdebug-assertions`, and `profile.fuzz` sets that **on**. So if a divergence ever +/// slipped past guard 1 -- if `Matcher` carried the same bug, say -- reaching `look` aborts: +/// +/// ```text +/// unsafe precondition(s) violated: hint::unreachable_unchecked must never be reached +/// thread caused non-unwinding panic. aborting. +/// ``` +/// +/// Verified by calling `look` on a deliberately over-permissive `matches` with the differential +/// removed: `SIGABRT`, in the fuzz profile, no miri required. Note it is a *non-unwinding* panic, so +/// bolero cannot catch it and the process dies -- which under libfuzzer is exactly right, a saved +/// `crash-*` artifact rather than a silent pass. +/// +/// The practical consequence is that **`just fuzz` on the fuzz profile already detects the unsound +/// direction**, and that is where the assurance mostly comes from. Miri remains useful for what +/// `ub_checks` does not model -- aliasing and provenance across the `as_ref_unchecked` boundary -- but +/// it is not the only thing standing between this and undefined behaviour. +/// +/// These do run clean under `just miri test -p dataplane-net view_properties`. Worth knowing what that +/// is worth, though: bolero manages 5 cases a second under miri against roughly 35,000 native, and the +/// miri recipe spawns its own `nix-shell`, so `BOLERO_RANDOM_TEST_TIME_MS` from the caller's +/// environment never arrives and the run stops at **25 cases per property**. A smoke test, not a proof. +/// Raising it means setting the budget inside `miri.just`. +#[cfg(test)] +mod view_properties { + use crate::eth::Eth; + use crate::headers::view::Look; + use crate::headers::{Headers, Net, ShapedHeaders, Transport}; + use crate::vlan::Vlan; + + /// `as_view` and `Matcher` must agree on whether the packet has the shape, and on which layers. + #[test] + fn eth_net_transport_agrees_with_the_matcher() { + bolero::check!() + .with_generator(ShapedHeaders) + .for_each(|h: &Headers| { + let matched = h.pat().eth().net().transport().done(); + match h.as_view::<(&Eth, &Net, &Transport)>() { + None => assert!( + matched.is_none(), + "the matcher accepted a shape as_view refused: {h:?}" + ), + Some(view) => { + let Some((m_eth, m_net, m_transport)) = matched else { + panic!("as_view accepted a shape the matcher refused: {h:?}"); + }; + let (v_eth, v_net, v_transport) = view.look(); + assert!(std::ptr::eq(v_eth, m_eth), "eth differs: {h:?}"); + assert!(std::ptr::eq(v_net, m_net), "net differs: {h:?}"); + assert!( + std::ptr::eq(v_transport, m_transport), + "transport differs: {h:?}" + ); + } + } + }); + } + + /// The same, for a shape that names a VLAN tag. + /// + /// This is where the interesting half of the contract lives: a tag the shape does not mention is a + /// miss, so the two implementations have to agree about *how many* tags were consumed, not merely + /// that some were. + #[test] + fn eth_vlan_net_transport_agrees_with_the_matcher() { + bolero::check!() + .with_generator(ShapedHeaders) + .for_each(|h: &Headers| { + let matched = h.pat().eth().vlan().net().transport().done(); + match h.as_view::<(&Eth, &Vlan, &Net, &Transport)>() { + None => assert!( + matched.is_none(), + "the matcher accepted a vlan shape as_view refused: {h:?}" + ), + Some(view) => { + let Some((m_eth, m_vlan, m_net, m_transport)) = matched else { + panic!("as_view accepted a vlan shape the matcher refused: {h:?}"); + }; + let (v_eth, v_vlan, v_net, v_transport) = view.look(); + assert!(std::ptr::eq(v_eth, m_eth), "eth differs: {h:?}"); + assert!( + std::ptr::eq(v_vlan, m_vlan), + "the two implementations consumed different vlan tags: {h:?}" + ); + assert!(std::ptr::eq(v_net, m_net), "net differs: {h:?}"); + assert!( + std::ptr::eq(v_transport, m_transport), + "transport differs: {h:?}" + ); + } + } + }); + } + + // A shape starting at `Net` rather than `Eth` was tried here and does not compile: `Net` does not + // satisfy `Within<()>`, so the adjacency graph forbids a shape that begins mid-stack. That half of + // the contract is enforced at compile time and needs no property. +} + +/// The **mutable** half of the unsafe boundary, which is the more delicate one. +/// +/// [`Look::look`] and [`sealed::Sealed::matches`] at least walk the stack the same way: both chain +/// [`ViewStep::step`]. [`LookMut::look_mut`] does not. It builds a [`MatcherMut`](super::pat::MatcherMut) +/// from [`Headers::pat_mut`] and chains [`ViewStepMut::chain`] over it, then calls +/// `unreachable_unchecked` if that comes back `None`. +/// +/// So the invariant is established by one traversal and consumed by a **different** one. Nothing makes +/// `ViewStep::step` and `ViewStepMut::chain` agree except that they were written to; where the read path +/// risks a mis-threaded cursor between two copies of the same walk, this risks two walks disagreeing +/// outright. +/// +/// Note what is *not* worth testing here: `look_mut` against `MatcherMut` directly. `look_mut` **is** +/// `MatcherMut` plus an `unreachable_unchecked`, so that comparison is the implementation against +/// itself. The question worth asking is whether `matches` -- the `ViewStep` walk that licensed the +/// unchecked call -- agrees with the `ViewStepMut` walk that has to deliver on it. +/// +/// # Aliasing, and why `ub_checks` cannot help +/// +/// `look_mut` hands back several `&mut` into one [`Headers`], pre-split through +/// [`Fields`](super::pat::Fields). If that split ever aliased, two of those references would point at +/// the same layer -- undefined behaviour of a kind the `ub_checks` backstop does **not** model. It +/// checks the `unreachable_unchecked` precondition, nothing about aliasing. +/// +/// Only miri sees that, and only with stacked borrows *on*, which this repo turns off by default: +/// +/// ```text +/// STACKED_BORROW_CHECK=enabled just miri test -p dataplane-net view_mut_properties +/// ``` +/// +/// Through the environment, not `just`'s command line: a *module's* variables cannot be overridden +/// there. `just miri stacked_borrow_check=enabled test` parses as a recipe name and +/// `just --set stacked_borrow_check enabled miri test` is refused, which is why `miri.just` reads both +/// knobs via `env()`. +#[cfg(test)] +mod view_mut_properties { + use crate::eth::Eth; + use crate::headers::view::{Look, LookMut}; + use crate::headers::{Headers, Net, ShapedHeaders, SometimesHeadless, Transport}; + + /// The type a `Matcher` method name selects. + /// + /// A shape and the chain that matches it are the same statement written twice -- + /// `(&Eth, &Ipv6, &HopByHop, &Transport)` and `.eth().ipv6().hop_by_hop().transport()` -- and + /// writing both by hand at every instantiation is a standing invitation to write two different + /// statements. The pair that disagrees still compiles and still passes; it just quietly tests + /// something other than what it says. So the chain is the input and the shape is derived. + /// + /// The table is the only place the correspondence lives, and it is the whole of it: every entry + /// below names a method the `matcher_net!`, `matcher_ext!` and `matcher_transport!` invocations + /// in [`pat`](super::pat) generate, plus the four written out by hand. + macro_rules! layer_ty { + (eth) => { + crate::eth::Eth + }; + (vlan) => { + crate::vlan::Vlan + }; + (net) => { + crate::headers::Net + }; + (ipv4) => { + crate::ipv4::Ipv4 + }; + (ipv6) => { + crate::ipv6::Ipv6 + }; + (hop_by_hop) => { + crate::ipv6::HopByHop + }; + (dest_opts) => { + crate::ipv6::DestOpts + }; + (routing) => { + crate::ipv6::Routing + }; + (fragment) => { + crate::ipv6::Fragment + }; + (ipv4_auth) => { + crate::ip_auth::Ipv4Auth + }; + (ipv6_auth) => { + crate::ip_auth::Ipv6Auth + }; + (transport) => { + crate::headers::Transport + }; + (tcp) => { + crate::tcp::Tcp + }; + (udp) => { + crate::udp::Udp + }; + (icmp4) => { + crate::icmp4::Icmp4 + }; + (icmp6) => { + crate::icmp6::Icmp6 + }; + (vxlan) => { + crate::vxlan::Vxlan + }; + } + + /// The `Shape` a chain of matcher method names denotes. + macro_rules! shape_of { + ($($layer:ident),+ $(,)?) => { ($(&'static layer_ty!($layer),)+) }; + } + + /// The address of each layer in a tuple of references, without naming the arity. + /// + /// Agreeing that a shape matches is the soundness question; handing back the *same* layers is the + /// correctness one, and it was previously checked only at arities three and four, because that is + /// where a tuple can be destructured by hand into a fixed number of bindings. Reducing the tuple + /// to its addresses removes the need to name the arity at all, so the check applies wherever the + /// agreement check does. + /// + /// Addresses rather than values: two layers can hold equal bytes without being the same layer, and + /// picking the wrong VLAN tag out of four identical ones is exactly the cursor bug this is looking + /// for. + trait Addrs { + /// One address per element. + type Out: PartialEq + core::fmt::Debug; + /// Where each element of this tuple lives. + fn addrs(&self) -> Self::Out; + } + + macro_rules! impl_addrs { + ($n:literal; $($T:ident $idx:tt),+) => { + impl<'a, $($T),+> Addrs for ($(&'a $T,)+) { + type Out = [usize; $n]; + fn addrs(&self) -> [usize; $n] { + [$(core::ptr::from_ref::<$T>(self.$idx) as usize),+] + } + } + + impl<'a, $($T),+> Addrs for ($(&'a mut $T,)+) { + type Out = [usize; $n]; + fn addrs(&self) -> [usize; $n] { + [$(core::ptr::from_ref::<$T>(&*self.$idx) as usize),+] + } + } + }; + } + + impl_addrs!(1; A 0); + impl_addrs!(2; A 0, B 1); + impl_addrs!(3; A 0, B 1, C 2); + impl_addrs!(4; A 0, B 1, C 2, D 3); + impl_addrs!(5; A 0, B 1, C 2, D 3, E 4); + impl_addrs!(6; A 0, B 1, C 2, D 3, E 4, F 5); + impl_addrs!(7; A 0, B 1, C 2, D 3, E 4, F 5, G 6); + impl_addrs!(8; A 0, B 1, C 2, D 3, E 4, F 5, G 6, H 7); + + /// Both walks, at every arity the oracle can express. + /// + /// The decision is what soundness turns on: if `matches` says yes where the walk that must deliver + /// says no, `look`/`look_mut` reach `unreachable_unchecked`. So this compares decisions across the + /// whole family, for the read path and the mutable path both. + /// + /// Arity matters because `matches`, `look` and `look_mut` are generated separately for each one, by + /// a macro, with the VLAN and extension cursors threaded through by hand every time. Testing two + /// arities tested two of eight copies. + /// + /// Each property asks two things of a shape, and the second is the reason `look` and `look_mut` + /// appear here rather than only in the split test: + /// + /// * **the decision**, `matches` against the matcher chain. This is what soundness turns on: if + /// `matches` says yes where the walk that must deliver says no, `look`/`look_mut` reach + /// `unreachable_unchecked`. Disagreement here is undefined behaviour, not a wrong answer. + /// * **the delivery**, `look`/`look_mut` against the same chain's tuple, compared by address. + /// Agreeing that a shape is present is not the same as picking the same layers out of it, and + /// the cursor arithmetic that decides *which* VLAN tag or *which* extension header comes back + /// is written out by hand once per arity. A shape naming four tags has four chances to be off + /// by one and no way for the decision check to notice. + /// + /// Delivery was previously checked at arities three and four only, because a tuple has to be + /// destructured into a fixed number of bindings to be compared -- which is why [`Addrs`] exists. + macro_rules! arity_agrees { + ($read:ident, $mutable:ident, $gen:expr, $($layer:ident),+) => { + #[test] + fn $read() { + type Shape = shape_of!($($layer),+); + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + static SEEN: AtomicUsize = AtomicUsize::new(0); + static HIT: AtomicUsize = AtomicUsize::new(0); + bolero::check!() + .with_generator($gen) + .for_each(|h: &Headers| { + SEEN.fetch_add(1, Ordering::Relaxed); + let licensed = h.as_view::().is_some(); + if licensed { + HIT.fetch_add(1, Ordering::Relaxed); + } + let chain = h.pat()$(.$layer())+.done(); + assert_eq!( + licensed, chain.is_some(), + concat!( + "`matches` and the read walk disagree for ", + stringify!(($($layer),+)), + ", so `look` would reach `unreachable_unchecked`: {:?}" + ), + h + ); + if let (Some(view), Some(chain)) = (h.as_view::(), chain) { + assert_eq!( + view.look().addrs(), chain.addrs(), + concat!( + "`look` and the read walk agreed that ", + stringify!(($($layer),+)), + " is present and then handed back different layers: {:?}" + ), + h + ); + } + }); + agreement_is_not_vacuous( + stringify!(($($layer),+)), + SEEN.load(Ordering::Relaxed), + HIT.load(Ordering::Relaxed), + ); + } + + #[test] + fn $mutable() { + type Shape = shape_of!($($layer),+); + use concurrency::sync::atomic::{AtomicUsize, Ordering}; + static SEEN: AtomicUsize = AtomicUsize::new(0); + static HIT: AtomicUsize = AtomicUsize::new(0); + bolero::check!() + .with_generator($gen) + .for_each(|h: &Headers| { + let mut owned = h.clone(); + SEEN.fetch_add(1, Ordering::Relaxed); + let licensed = owned.as_view_mut::().is_some(); + if licensed { + HIT.fetch_add(1, Ordering::Relaxed); + } + // The two walks each want the whole of `owned` mutably, so they take turns + // and hand back addresses rather than references. Nothing is written + // between the two, so the addresses stay comparable. + let chain = owned.pat_mut()$(.$layer())+.done().map(|t| t.addrs()); + assert_eq!( + licensed, chain.is_some(), + concat!( + "`matches` and the mutable walk disagree for ", + stringify!(($($layer),+)), + ", so `look_mut` would reach `unreachable_unchecked`: {:?}" + ), + h + ); + if let Some(view) = owned.as_view_mut::() { + assert_eq!( + Some(view.look_mut().addrs()), chain, + concat!( + "`look_mut` and the mutable walk agreed that ", + stringify!(($($layer),+)), + " is present and then handed back different layers: {:?}" + ), + h + ); + } + }); + agreement_is_not_vacuous( + stringify!(($($layer),+)), + SEEN.load(Ordering::Relaxed), + HIT.load(Ordering::Relaxed), + ); + } + }; + } + + /// A shape the generator never produces makes its property pass for the wrong reason. + /// + /// Two implementations agreeing that nothing matches is not agreement worth having, and the higher + /// arities are where it would happen quietly: arity 7 needs a packet carrying exactly four VLAN + /// tags, since a tag the shape does not name is a miss. So every arity reports its own hit rate and + /// fails if it never matched at all. + fn agreement_is_not_vacuous(shape: &str, seen: usize, hit: usize) { + println!("{shape}: matched {hit} of {seen}"); + // The default run is a second long, which is plenty here -- these properties manage tens of + // thousands of cases a second -- but a short sample should say so rather than fail. + if seen > 500 { + assert!( + hit > 0, + "{shape} never matched in {seen} packets: the two walks agree only because the \ + generator cannot produce this shape" + ); + } + } + + // Every arity from one to eight, which is all of them. + // + // `Matcher` names every layer the `Within` graph does -- `matcher_net!`, `matcher_ext!` and + // `matcher_transport!` give it `.ipv4()`, `.ipv6()`, `.hop_by_hop()`, `.dest_opts()`, + // `.routing()`, `.fragment()`, `.ipv4_auth()`, `.ipv6_auth()`, `.tcp()`, `.udp()`, `.icmp4()` and + // `.icmp6()` alongside the generic `.eth()` / `.vlan()` / `.net()` / `.transport()`. So the + // oracle reaches as far as the code does: `Eth` + four VLAN tags (`MAX_VLANS`) + `Ipv6` + + // `HopByHop` + `Transport` is eight, and the extension region is expressible. + // + // `SometimesHeadless` rather than `ShapedHeaders`: the first step of every one of these is `Eth`, + // and the branch where that step refuses is generated once per arity. Nothing that always sets + // `eth` can reach any of them. + arity_agrees!(read_1, mutable_1, SometimesHeadless, eth); + arity_agrees!(read_2, mutable_2, SometimesHeadless, eth, net); + arity_agrees!(read_3, mutable_3, SometimesHeadless, eth, net, transport); + arity_agrees!( + read_4, + mutable_4, + SometimesHeadless, + eth, + vlan, + net, + transport + ); + arity_agrees!( + read_5, + mutable_5, + SometimesHeadless, + eth, + vlan, + vlan, + net, + transport + ); + arity_agrees!( + read_6, + mutable_6, + SometimesHeadless, + eth, + vlan, + vlan, + vlan, + net, + transport + ); + arity_agrees!( + read_7, + mutable_7, + SometimesHeadless, + eth, + vlan, + vlan, + vlan, + vlan, + net, + transport + ); + arity_agrees!( + read_8, + mutable_8, + SometimesHeadless, + eth, + vlan, + vlan, + vlan, + vlan, + ipv6, + hop_by_hop, + transport + ); + + // Shapes that enter the IPv6 extension region, where `ExtGapCheck` switches from + // skip-extensions-silently to consume-them-all. + // + // This is the subtlest part of the contract and the part the module documentation spends most of + // its words on. It is also where the two walks are least alike: on the read path both call + // `ExtGapCheck::ext_gap_ok`, so what is under test is the by-hand threading of the `ec` cursor + // through each separately generated arity -- which is the plausible bug. On the mutable path + // they call genuinely different implementations, `ext_gap_ok` against a `Headers` versus + // `ext_gap_ok_mut` against a pre-split `Fields`, so those two are checked against each other + // as well. + // + // Each shape below is deliberately exact: naming `HopByHop` and then `Transport` matches only a + // packet carrying that extension and no other, since an unconsumed extension is a miss once the + // chain has entered the region. + arity_agrees!( + read_ext_v6_one, + mutable_ext_v6_one, + SometimesHeadless, + eth, + ipv6, + hop_by_hop, + transport + ); + arity_agrees!( + read_ext_v6_two, + mutable_ext_v6_two, + SometimesHeadless, + eth, + ipv6, + hop_by_hop, + dest_opts, + transport + ); + arity_agrees!( + read_ext_v6_three, + mutable_ext_v6_three, + SometimesHeadless, + eth, + ipv6, + hop_by_hop, + dest_opts, + routing, + transport + ); + // The IPv4 authentication header is the only extension that belongs on a v4 packet, and it is + // the only way to reach the strict branch without IPv6. + arity_agrees!( + read_ext_v4_auth, + mutable_ext_v4_auth, + SometimesHeadless, + eth, + ipv4, + ipv4_auth, + transport + ); + // Entering the region and then *not* naming a transport: the gap check never runs, so every + // extension after the named one is simply left unvisited. Distinguishing this from the strict + // case above is the whole point of running the check at the transport step rather than the + // extension step. + arity_agrees!( + read_ext_v6_no_transport, + mutable_ext_v6_no_transport, + SometimesHeadless, + eth, + ipv6, + hop_by_hop + ); + + // The UDP encapsulation layer, which no shape named until now. + // + // `Vxlan` is the one `ViewStep` that runs no gap check at all -- it reads `udp_encap` and ignores + // both cursors -- and the one layer that is not part of the linear header stack, so nothing about + // it follows from the arities above. `CommonHeaders` has been drawing VXLAN packets the whole + // time; the gap was that no property ever asked for one. + // + // `ShapedHeaders` rather than `SometimesHeadless` here, because this shape is already narrow: + // `Udp` is one of three next headers, the encapsulation is a coin flip on top of that, and the + // net step demands no VLAN tags. Removing the Ethernet header as well would spend the hit rate + // on a branch the eight arities above already cover. + arity_agrees!( + read_vxlan_v4, + mutable_vxlan_v4, + ShapedHeaders, + eth, + ipv4, + udp, + vxlan + ); + arity_agrees!( + read_vxlan_net, + mutable_vxlan_net, + ShapedHeaders, + eth, + net, + udp, + vxlan + ); + + /// Exercise the multi-`&mut` split itself: write through every reference and read the writes back. + /// + /// The assertions are almost beside the point. What matters is that the references are *created and + /// written through*, so that miri with stacked borrows enabled can judge whether + /// [`Fields`](super::pat::Fields) handed out two paths to the same layer. Nothing else in the suite + /// does that. + fn exercise_the_mutable_split() { + bolero::check!() + .with_generator(ShapedHeaders) + .for_each(|h: &Headers| { + let mut owned = h.clone(); + let Some(view) = owned.as_view_mut::<(&Eth, &Net, &Transport)>() else { + return; + }; + let (eth, net, transport) = view.look_mut(); + + // A write through each reference, then a read back through the same one. If two of + // these aliased, the writes would interfere and miri would object to the borrow stack + // long before the values did. + let want_src = + crate::eth::mac::SourceMac::try_from(crate::eth::mac::Mac([2, 0, 0, 0, 0, 1])) + .unwrap_or_else(|_| { + unreachable!("a locally-administered unicast mac is a valid source") + }); + eth.set_source(want_src); + let seen_net = net.dst_addr(); + let seen_transport = transport.dst_port(); + + assert_eq!( + eth.source(), + want_src, + "the write through eth did not stick" + ); + assert_eq!(net.dst_addr(), seen_net, "net changed under a write to eth"); + assert_eq!( + transport.dst_port(), + seen_transport, + "transport changed under a write to eth" + ); + }); + } + + /// Shards of [`exercise_the_mutable_split`], so the machine can be used. + /// + /// Under miri this property is the expensive one and the one that matters most, and its cost is + /// wall-clock: bolero manages about five cases a second. Raising the time budget buys cases + /// linearly, but only on one core. + /// + /// Each shard seeds itself from the OS, so `N` shards explore `N` independent streams and nextest + /// runs them concurrently -- turning a machine with cores to spare into more cases for the same + /// wall time, which is the only lever that does not cost patience. Sixteen rather than sixty: miri + /// carries a large memory footprint per process, and the other properties want cores too. + macro_rules! split_shards { + ($($name:ident),* $(,)?) => { + $( + #[test] + fn $name() { + exercise_the_mutable_split(); + } + )* + }; + } + + split_shards!( + the_mutable_split_hands_out_distinct_layers, + split_shard_02, + split_shard_03, + split_shard_04, + split_shard_05, + split_shard_06, + split_shard_07, + split_shard_08, + split_shard_09, + split_shard_10, + split_shard_11, + split_shard_12, + split_shard_13, + split_shard_14, + split_shard_15, + split_shard_16, + ); +} diff --git a/net/src/headers/within.rs b/net/src/headers/within.rs index 2fc8646360..c2e2240537 100644 --- a/net/src/headers/within.rs +++ b/net/src/headers/within.rs @@ -554,3 +554,357 @@ impl Within for Tcp { impl Within for Udp { fn conform(_parent: &mut Net) {} } + +/// What [`Within::conform`] is for, checked against the parser. +/// +/// `conform` writes the parent's protocol field so it names the child: `EthType::IPV6` on an +/// Ethernet header carrying IPv6, `NextHeader::ROUTING` on an IPv6 header carrying a routing +/// extension. It is the only reason this trait has a method at all -- the ordering half of the +/// contract is enforced by the compiler, by the presence or absence of an impl, and needs no test. +/// +/// Thirty-two of these bodies had never run. Every one of them was an IPv6 extension header +/// transition or `Vlan` inside `Vlan`, which is to say: the whole extension region, plus the second +/// tag of a double-tagged frame. The reason is that `conform` is reached only through +/// [`HeaderStack::stack`](crate::headers::builder::HeaderStack::stack), and while the builder has had +/// `.hop_by_hop()`, `.dest_opts()`, `.routing()`, `.fragment()`, `.ipv4_auth()` and `.ipv6_auth()` +/// all along, no test ever called one. The generators reach the extension region constantly, but +/// they assemble [`Headers`](crate::headers::Headers) field by field and never go through the +/// builder, so they never conform anything. +/// +/// # The oracle +/// +/// A test that builds a packet and then reads back the field `conform` just wrote would be checking +/// the implementation against itself. So the check is deparse-then-parse: the parser decides what +/// follows an IPv6 header by reading its next-header field, which is the field `conform` sets, so a +/// `conform` that names the wrong protocol produces bytes the parser reads as a different packet -- +/// or as no valid packet at all. +/// +/// # Scrambling +/// +/// Each layer's next-header field is set to a fuzzed byte *before* the next layer is stacked, so +/// `conform` is always overwriting a wrong value rather than filling in a blank one. +/// +/// That is not a precaution, it is load-bearing, and the case that proves it is `Ipv4` inside `Eth`. +/// [`Blank`] for `Eth` produces `EthType::IPV4`, so on a blank Ethernet header the conform that sets +/// `EthType::IPV4` has nothing to do: delete its body and every packet still round-trips. Scrambled, +/// the same deletion fails two chains. `Vlan` has the same blank and the same exposure. +/// +/// # What is left, and why it stays uncovered +/// +/// Nineteen no-op `conform` bodies remain unrun: the enum-level impls, the `EmbeddedStart` impls and +/// everything `impl_truncated_within!` generates. They are not merely untested, they are unreachable +/// through the builder, and the compiler says so twice -- writing +/// `HeaderStack::new().eth(..).stack::(..)` fails with both `Net: Blank is not satisfied` and +/// `Headers: Install is not satisfied`. Either bound alone would be enough. +/// +/// They exist to give the pattern matcher its `Within` edges, which need the trait but not the +/// method. `conform` is a public trait method, so they are callable in principle by anyone holding +/// the parent; nothing in the tree does. Whether nineteen uncallable bodies are worth keeping is a +/// question for whoever owns the trait, not something a test can settle. +/// +/// [`Blank`]: crate::headers::builder::Blank +#[cfg(test)] +mod conform_properties { + use crate::eth::Eth; + use crate::eth::ethtype::EthType; + use crate::headers::builder::HeaderStack; + use crate::headers::test::parse_back_test; + use crate::headers::{Headers, Transport}; + use crate::icmp4::{ + Icmp4, Icmp4DestUnreachable, Icmp4EchoReply, Icmp4EchoRequest, Icmp4ParamProblem, + Icmp4Redirect, Icmp4TimeExceeded, Icmp4Type, + }; + use crate::icmp6::{ + Icmp6, Icmp6DestUnreachable, Icmp6EchoReply, Icmp6EchoRequest, Icmp6PacketTooBig, + Icmp6ParamProblem, Icmp6TimeExceeded, Icmp6Type, + }; + use crate::ip::NextHeader; + use crate::ip_auth::{Ipv4Auth, Ipv6Auth}; + use crate::ipv4::Ipv4; + use crate::ipv6::{DestOpts, Fragment, HopByHop, Ipv6, Routing}; + use crate::tcp::Tcp; + use crate::udp::Udp; + use crate::vlan::Vlan; + + /// Put a wrong protocol number in the field `conform` is responsible for. + /// + /// Implemented for every layer the builder can stack. The transport types are the leaves of + /// every chain -- nothing is ever stacked on top of one, so nothing ever conforms one -- and + /// their impls are deliberately empty rather than absent, so that the chain macro does not have + /// to know which layers are interior. + trait Scramble { + /// Overwrite the protocol field with something derived from `seed`. + fn scramble(&mut self, seed: u8); + } + + macro_rules! scramble_next_header { + ($($T:ty),+ $(,)?) => {$( + impl Scramble for $T { + fn scramble(&mut self, seed: u8) { + self.set_next_header(NextHeader::new(seed)); + } + } + )+}; + } + + macro_rules! scramble_leaf { + ($($T:ty),+ $(,)?) => {$( + impl Scramble for $T { + fn scramble(&mut self, _seed: u8) {} + } + )+}; + } + + scramble_next_header!( + Ipv4, Ipv6, HopByHop, DestOpts, Routing, Fragment, Ipv4Auth, Ipv6Auth + ); + scramble_leaf!(Tcp, Udp); + // The ICMP subtypes are the only layers that can sit on top of an ICMP header, and nothing sits + // on top of them. + scramble_leaf!( + Icmp4DestUnreachable, + Icmp4Redirect, + Icmp4TimeExceeded, + Icmp4ParamProblem, + Icmp4EchoRequest, + Icmp4EchoReply, + Icmp6DestUnreachable, + Icmp6PacketTooBig, + Icmp6TimeExceeded, + Icmp6ParamProblem, + Icmp6EchoRequest, + Icmp6EchoReply, + ); + + // ICMP is the one place where `conform` writes a message type rather than a protocol number, so + // `Unknown` is the scramble: it is the one variant matching no subtype, which makes it wrong for + // every chain below rather than accidentally right for one of them. + // + // The type byte is fixed rather than fuzzed, and has to be. `Unknown` stores the raw byte, so + // `Unknown { type_u8: 3 }` deparses to the same three bytes a destination-unreachable message + // does and parses back as one -- a header the round-trip is right to reject, and nothing to do + // with `conform`. 253 for v4 and 200 for v6 are reserved for experimentation and belong to no + // variant, so they survive the round trip as themselves. The rest of the message stays fuzzed. + impl Scramble for Icmp4 { + fn scramble(&mut self, seed: u8) { + self.set_type(crate::icmp4::Icmp4Type::Unknown { + type_u8: 253, + code_u8: seed, + bytes5to8: [seed; 4], + }); + } + } + + impl Scramble for Icmp6 { + fn scramble(&mut self, seed: u8) { + self.set_type(crate::icmp6::Icmp6Type::Unknown { + type_u8: 200, + code_u8: seed, + bytes5to8: [seed; 4], + }); + } + } + + impl Scramble for Eth { + fn scramble(&mut self, seed: u8) { + self.set_ether_type(EthType::new(u16::from(seed))); + } + } + + impl Scramble for Vlan { + fn scramble(&mut self, seed: u8) { + self.set_inner_ethtype(EthType::new(u16::from(seed))); + } + } + + /// Build the named chain through the builder, scrambling as it goes, and round-trip it. + /// + /// The chain is a list of `HeaderStack` method names, and the closures are written here rather + /// than at the call site: a closure written at the call site could not name the `seed` this + /// macro binds, macro hygiene being what it is, and every call site would then have to repeat + /// the same closure once per layer. + macro_rules! conform_chain { + ($name:ident, $($layer:ident),+ $(,)?) => { + conform_chain!(@build $name, |_| {}, $($layer),+); + }; + (specialized $name:ident, $($layer:ident),+ $(,)?) => { + conform_chain!(@build $name, icmp_type_was_specialized, $($layer),+); + }; + (@build $name:ident, $check:expr, $($layer:ident),+) => { + #[test] + fn $name() { + bolero::check!().with_type().for_each(|seed: &u8| { + let seed = *seed; + let built = HeaderStack::new() + $(.$layer(|l| l.scramble(seed)))+ + .build_headers(); + let headers = built.unwrap_or_else(|e| { + unreachable!("a blank {} chain does not overflow: {e:?}", stringify!($name)) + }); + parse_back_test(&headers); + $check(&headers); + }); + } + }; + } + + /// The scrambled ICMP type did not survive into the built packet. + /// + /// Only for chains that end in a subtype layer. A chain ending at a bare `.icmp4()` has nothing + /// above it to conform it, so the scramble is *supposed* to survive there, and round-tripping is + /// the only thing to check. + /// + /// Worth stating why this is separate from the round trip rather than folded into it: the + /// scrambled type is a well-formed ICMP message, so a packet still carrying it deparses and + /// parses back perfectly. The round trip cannot tell that the specialization never happened. + fn icmp_type_was_specialized(headers: &Headers) { + match headers.transport() { + Some(Transport::Icmp4(icmp)) => assert!( + !matches!(icmp.icmp_type(), Icmp4Type::Unknown { type_u8: 253, .. }), + "the scrambled ICMPv4 type survived the build, so nothing specialized it" + ), + Some(Transport::Icmp6(icmp)) => assert!( + !matches!(icmp.icmp_type(), Icmp6Type::Unknown { type_u8: 200, .. }), + "the scrambled ICMPv6 type survived the build, so nothing specialized it" + ), + other => unreachable!("a subtype chain builds an ICMP transport, got {other:?}"), + } + } + + // Seventeen chains, chosen to cover all thirty-two unreached transitions between them. Three + // extension headers is the ceiling -- `MAX_NET_EXTENSIONS` -- so the deeper regions of the graph + // have to be reached by several chains rather than one long one. + conform_chain!( + double_tag_then_three_extensions, + eth, + vlan, + vlan, + ipv6, + dest_opts, + routing, + fragment, + tcp + ); + conform_chain!( + hop_by_hop_routing_dest_opts, + eth, + ipv6, + hop_by_hop, + routing, + dest_opts, + udp + ); + conform_chain!( + routing_fragment_auth, + eth, + ipv6, + routing, + fragment, + ipv6_auth, + tcp + ); + conform_chain!( + fragment_then_dest_opts, + eth, + ipv6, + fragment, + dest_opts, + icmp6 + ); + conform_chain!( + hop_by_hop_then_fragment, + eth, + ipv6, + hop_by_hop, + fragment, + udp + ); + conform_chain!( + dest_opts_then_fragment, + eth, + ipv6, + dest_opts, + fragment, + icmp6 + ); + conform_chain!(auth_then_dest_opts, eth, ipv6, ipv6_auth, dest_opts, udp); + conform_chain!( + hop_by_hop_then_auth, + eth, + ipv6, + hop_by_hop, + ipv6_auth, + icmp6 + ); + conform_chain!(dest_opts_then_auth, eth, ipv6, dest_opts, ipv6_auth, tcp); + conform_chain!(routing_then_auth, eth, ipv6, routing, ipv6_auth, udp); + conform_chain!(hop_by_hop_then_udp, eth, ipv6, hop_by_hop, udp); + conform_chain!(routing_then_udp, eth, ipv6, routing, udp); + conform_chain!( + hop_by_hop_then_routing_then_tcp, + eth, + ipv6, + hop_by_hop, + routing, + tcp + ); + conform_chain!(hop_by_hop_then_icmp6, eth, ipv6, hop_by_hop, icmp6); + conform_chain!(routing_then_icmp6, eth, ipv6, routing, icmp6); + // The IPv4 authentication header is the only extension that belongs on a v4 packet. + conform_chain!(v4_auth_then_udp, eth, ipv4, ipv4_auth, udp); + conform_chain!(v4_auth_then_icmp4, eth, ipv4, ipv4_auth, icmp4); + + // The ICMP message subtypes, which the builder can specialize into and which no test had ever + // asked for. Ten of the twelve `Blank` impls behind them had never been called either -- the two + // that had were what made the shared macro bodies look covered, since a `macro_rules!` line + // counts as run once any one of its expansions runs. Worth knowing generally: a table of twelve + // generated impls reports as covered when one of the twelve is exercised, and only the + // hand-written part of each -- here `blank()` -- shows the difference. + conform_chain!(specialized icmp4_dest_unreachable, eth, ipv4, icmp4, dest_unreachable); + conform_chain!(specialized icmp4_redirect, eth, ipv4, icmp4, redirect); + conform_chain!(specialized icmp4_time_exceeded, eth, ipv4, icmp4, time_exceeded); + conform_chain!(specialized icmp4_param_problem, eth, ipv4, icmp4, param_problem); + conform_chain!(specialized icmp4_echo_request, eth, ipv4, icmp4, echo_request); + conform_chain!(specialized icmp4_echo_reply, eth, ipv4, icmp4, echo_reply); + conform_chain!(specialized icmp6_dest_unreachable, eth, ipv6, icmp6, dest_unreachable6); + conform_chain!(specialized icmp6_packet_too_big, eth, ipv6, icmp6, packet_too_big6); + conform_chain!(specialized icmp6_time_exceeded, eth, ipv6, icmp6, time_exceeded6); + conform_chain!(specialized icmp6_param_problem, eth, ipv6, icmp6, param_problem6); + conform_chain!(specialized icmp6_echo_request, eth, ipv6, icmp6, echo_request6); + conform_chain!(specialized icmp6_echo_reply, eth, ipv6, icmp6, echo_reply6); + + /// A subtype the caller customized, which is what separates the two writers of the ICMP type. + /// + /// The chains above cannot tell `Within::conform` from `Install` for these layers, and no test + /// could, because both write the same value: `conform` sets `DestUnreachable(blank())` and + /// `Install` sets `DestUnreachable(value)`, and `value` is `blank()` whenever the caller does not + /// change it. Empty either one alone and the other still produces the right packet. + /// + /// Choosing a code other than the blank one separates them, and the answer is that `conform` is + /// the redundant half. `Install` runs unconditionally from `build_headers`, after `conform`, and + /// overwrites whatever `conform` wrote -- so all twelve of these `conform` bodies could be empty + /// with no observable change. Nothing can be stacked on a subtype, so there is no arrangement in + /// which `conform` gets the last word. + /// + /// Pinned here rather than acted on: emptying them is a call for whoever owns the builder. + #[test] + fn a_customized_subtype_survives_the_build() { + let headers = HeaderStack::new() + .eth(|l| l.scramble(0)) + .ipv4(|l| l.scramble(0)) + .icmp4(|l| l.scramble(0)) + .dest_unreachable(|d| *d = Icmp4DestUnreachable::Port) + .build_headers() + .unwrap_or_else(|e| unreachable!("a blank chain does not overflow: {e:?}")); + + let Some(Transport::Icmp4(icmp)) = headers.transport() else { + unreachable!("the chain builds an ICMPv4 transport") + }; + assert_eq!( + icmp.icmp_type(), + Icmp4Type::DestUnreachable(Icmp4DestUnreachable::Port), + "the code the caller chose did not reach the built packet" + ); + parse_back_test(&headers); + } +} diff --git a/nix/overlays/dataplane-dev.nix b/nix/overlays/dataplane-dev.nix index c8829018f2..e318617b71 100644 --- a/nix/overlays/dataplane-dev.nix +++ b/nix/overlays/dataplane-dev.nix @@ -12,6 +12,12 @@ let }; in { + # callPackage rather than a bare import: duvet links against OpenSSL and so needs pkg-config and + # the library from the package set, while still taking our overridden rust toolchain. + duvet = final.callPackage ../pkgs/duvet { + src = sources.duvet; + inherit (override-packages) rustPlatform; + }; kopium = import ../pkgs/kopium ( override-packages // { @@ -21,9 +27,48 @@ in opengrep = final.callPackage ../pkgs/opengrep { src = sources.opengrep; }; + # cargoDeps must be fetched from the overridden source too. + bugstalker = prev.bugstalker.overrideAttrs (orig: { + version = final.lib.removePrefix "v" sources.bugstalker.version; + src = sources.bugstalker; + cargoDeps = prev.rustPlatform.fetchCargoVendor { + src = sources.bugstalker; + hash = "sha256-GGi5hnrK5WpvnXHNckpsBch/SJ4lDvH7peSlrCdk218="; + }; + }); + # lurk disables ASLR in the tracee before exec, and treats failure as fatal. + # Docker's default seccomp profile answers personality(ADDR_NO_RANDOMIZE) with + # EPERM, so under a plain `docker run` the traced program never starts -- and + # lurk still exits 0 after emitting a well-formed JSON trace of its own child + # failing, which the `jq -R 'fromjson? // empty'` filter we document accepts + # without complaint. + # + # Nothing in the tracer image symbolizes an address, so a fixed layout buys us + # nothing. Make it advisory rather than telling users to pass + # `--security-opt seccomp=unconfined`, which drops confinement on a container + # whose whole job is ptracing a process. + # + # Two single-line substitutions rather than one spanning both: nix strips the + # common indentation from an indented string, so a multi-line search pattern + # would not match the source's own indentation. + lurk = prev.lurk.overrideAttrs (orig: { + postPatch = (orig.postPatch or "") + '' + substituteInPlace src/lib.rs \ + --replace-fail \ + 'personality::set(Persona::ADDR_NO_RANDOMIZE)' \ + 'let _ = personality::set(Persona::ADDR_NO_RANDOMIZE);' \ + --replace-fail \ + '.map_err(|_| anyhow!("Unable to set ADDR_NO_RANDOMIZE"))?;' \ + "" + ''; + }); + cargo-bolero = prev.cargo-bolero.override { inherit (override-packages) rustPlatform; }; cargo-deny = prev.cargo-deny.override { inherit (override-packages) rustPlatform; }; cargo-edit = prev.cargo-edit.override { inherit (override-packages) rustPlatform; }; + cargo-expand = prev.cargo-expand.override { inherit (override-packages) rustPlatform; }; + cargo-show-asm = prev.cargo-show-asm.override { inherit (override-packages) rustPlatform; }; + cargo-mutants = prev.cargo-mutants.override { inherit (override-packages) rustPlatform; }; cargo-llvm-cov = (prev.cargo-llvm-cov.override override-packages).overrideAttrs (orig: { # the test suite is very impractical in our CI (fails on nightly for spurious reasons), and has nothing to do with # our project. @@ -43,15 +88,113 @@ in destination = "/src/fabric/${p}"; }; - gdb' = prev.gdb.overrideAttrs (orig: { - CFLAGS = "-Os -flto"; - CXXFLAGS = "-Os -flto"; - LDFLAGS = "-flto -Wl,--as-needed,--gc-sections -static-libstdc++ -static-libgcc"; - buildInputs = (orig.buildInputs or [ ]); - configureFlags = (orig.configureFlags or [ ]) ++ [ - "--enable-static" - "--disable-inprocess-agent" - "--disable-source-highlight" # breaks static compile - ]; - }); + # A gdb that can be copied into a VM or an image that has no nix store, and + # still run: musl, no interpreter, no shared libraries at all. + # + # Configure flags cannot get you here. `--enable-static` and + # `--disable-shared` -- which nixpkgs already passes -- only decide whether + # the libbfd, libopcodes, and libctf that this tree builds are archives; they + # say nothing about how the gdb executable links against readline, ncurses, + # expat, or python, and those have no static outputs in the default package + # set. Hence pkgsStatic, which rebuilds the dependencies rather than the + # link line. + # + # pkgsStatic gates python support on host == build, so this gdb configures + # `--without-python` and cannot load the Rust pretty printers. It is a + # bare-metal debugger, not a replacement for the ordinary `gdb` that + # `containers.dataplane-core-viewer` ships with `rust-gdb-printers`. + gdb' = final.pkgsStatic.gdb.override { + # dejagnu is a buildInput only so that gdb's own test suite can run, and we + # never run it. It also cannot be built here: expect resolves `tclStubsPtr` + # from tcl's stub library, which exists to be filled in by a dynamic loader, + # so a static link leaves it undefined. + dejagnu = final.pkgsStatic.emptyDirectory; + }; + + # A perf that can be copied into a VM or an image with no nix store, for the + # same reason as `gdb'`. + # + # Unlike gdb this cannot come from pkgsStatic: elfutils carries + # `badPlatforms = isStatic` because its Makefile builds libelf.so + # unconditionally, and a static toolchain cannot emit a shared object at all + # (`crtbeginT.o: relocation R_X86_64_32 against hidden symbol __TMC_END__`). + # perf without libelf/libdw is not worth shipping, so build against ordinary + # glibc packages -- whose elfutils already installs libelf.a and libdw.a -- + # and make only the final link static. + perf' = + let + # Every dependency below is either unusable in a static binary or not + # worth its transitive static closure. Dropping them at the argument + # layer keeps them out of the build; the NO_* flags tell perf's own + # configure-equivalent the same thing, so the two cannot disagree. + none = final.emptyDirectory; + in + (final.perf.override { + stdenv = final.stdenvAdapters.makeStaticBinaries final.stdenv; + withPython = false; + withLibcap = false; + newt = none; + slang = none; + babeltrace = none; + libunwind = none; + libpfm = none; + numactl = none; + openssl = none; + libopcodes = none; + libtraceevent = none; + systemtap-unwrapped = none; + }).overrideAttrs + (orig: { + # dlfilters are dlopen-ed plugins, which a static perf could not load + # even if the toolchain could build them. + postPatch = orig.postPatch + '' + substituteInPlace Makefile.perf \ + --replace-fail \ + 'DLFILTERS := dlfilter-test-api-v0.so dlfilter-test-api-v2.so dlfilter-show-cycles.so' \ + 'DLFILTERS :=' \ + --replace-fail '$(INSTALL) $(DLFILTERS) ' 'true ' + ''; + # perf keys off -static in LDFLAGS to add `-lelf -lz -llzma -lbz2 -ldl` + # to the libdw link. Without it the libdw probe fails and perf builds + # with DWARF support silently off -- it still links, so this is only + # visible in `perf version --build-options`. Pass it in the + # environment, not via makeFlags, so Makefile.config's own `LDFLAGS +=` + # still appends. + env = orig.env // { + LDFLAGS = "-static"; + }; + # A static link does not follow a library's own dependencies, so + # libelf.a and libdw.a's compression backends have to be named here, + # as archives rather than the shared objects the normal outputs carry. + buildInputs = orig.buildInputs ++ [ + final.zlib.static + (final.zstd.override { static = true; }) + (final.xz.override { enableStatic = true; }) + (final.bzip2.override { enableStatic = true; }) + ]; + makeFlags = orig.makeFlags ++ [ + "NO_LIBPYTHON=1" + "NO_LIBPERL=1" + "NO_SLANG=1" + "NO_NEWT=1" + "NO_LIBBABELTRACE=1" + "NO_LIBNUMA=1" + "NO_LIBAUDIT=1" + "NO_LIBBPF=1" + "NO_LIBPFM4=1" + "NO_LIBCRYPTO=1" + "NO_JVMTI=1" + "NO_LIBUNWIND=1" + "NO_LIBDEBUGINFOD=1" + "NO_LIBTRACEEVENT=1" + "NO_SDT=1" + # Only C++ demangling. perf's Rust v0 demangler is built in and + # unaffected, which is what matters for a Rust dataplane. + "NO_DEMANGLE=1" + ]; + # wrapProgram would replace the binary with a shell script, defeating + # the point of a static build. It only put objdump on PATH for + # `perf annotate`, which needs a toolchain on the target regardless. + preFixup = ""; + }); } diff --git a/nix/overlays/llvm.nix b/nix/overlays/llvm.nix index 334be96d21..9b7c46d39c 100644 --- a/nix/overlays/llvm.nix +++ b/nix/overlays/llvm.nix @@ -36,7 +36,7 @@ let "clippy" "llvm-tools" "rust-analyzer" - "rust-docs" + # Avoid retaining 680M of unused prebuilt documentation. "rust-src" "rust-std" "rustc" diff --git a/nix/pkgs/duvet/default.nix b/nix/pkgs/duvet/default.nix new file mode 100644 index 0000000000..14377edd96 --- /dev/null +++ b/nix/pkgs/duvet/default.nix @@ -0,0 +1,36 @@ +# duvet: specification compliance coverage. +# +# Parses a specification into its individual requirements and matches them against citations left +# in the source -- `//= ` followed by the requirement's text, quoted -- so that a requirement +# with no implementation, or an implementation with no test, becomes visible. +# +# It answers the question neither bolero nor cargo-mutants can. Bolero says a property holds; +# cargo-mutants says enough properties exist that nothing goes unasserted; neither can say the +# properties are the ones the specification asked for. See development/code/mutation-testing.md. +# +# Taken from the published crate rather than the git tag, deliberately. duvet embeds +# `www/public/script.js` with `include_str!`, and that file is a JavaScript build product: it is +# absent from the git tree and present in the crate. Building from the tag would mean carrying a +# node toolchain and a second lockfile to produce a file the crate already ships. The crate also +# carries the `Cargo.lock` the git tree omits, so the dependency set is pinned upstream instead of +# by us. +# +# The cost of that choice is that the pin is a plain URL, so npins cannot discover new versions: +# bumping duvet means editing the version in npins/sources.json by hand. +{ + src, + rustPlatform, + pkg-config, + openssl, + ... +}: +rustPlatform.buildRustPackage (final: { + pname = "duvet"; + version = "0.4.2"; + src = src.outPath; + cargoLock.lockFile = "${final.src}/Cargo.lock"; + nativeBuildInputs = [ pkg-config ]; + buildInputs = [ openssl ]; + # Upstream's tests reach the network to fetch the specifications they exercise. + doCheck = false; +}) diff --git a/npins/sources.json b/npins/sources.json index 369e701a56..722332c937 100644 --- a/npins/sources.json +++ b/npins/sources.json @@ -16,6 +16,22 @@ "url": "https://api.github.com/repos/KaTeX/KaTeX/tarball/refs/tags/v0.18.3", "hash": "sha256-FZpiUhKFI2GAZmr667gA5yHFezAoRcAADoS0+NrNsJQ=" }, + "bugstalker": { + "type": "GitRelease", + "repository": { + "type": "GitHub", + "owner": "godzie44", + "repo": "BugStalker" + }, + "pre_releases": false, + "version_upper_bound": null, + "release_prefix": null, + "submodules": false, + "version": "v0.4.7", + "revision": "9c18e546eca6a1d68ef69da340a6fb0f2bc1bab7", + "url": "https://api.github.com/repos/godzie44/BugStalker/tarball/refs/tags/v0.4.7", + "hash": "sha256-AAeSvy/rvyylPH2jTVBGN95QIc6gumREQYuruhRo2ZI=" + }, "crane": { "type": "GitRelease", "repository": { @@ -71,6 +87,12 @@ "url": "https://github.com/githedgehog/dplane-rpc/archive/e8fc33db10e1d00785f2a2b90cbadcad7900f200.tar.gz", "hash": "sha256-tjN4qSbKrWfosOV3wt2AnQxmVL0BPZYBjAHG3X00+aM=" }, + "duvet": { + "type": "Url", + "url": "https://static.crates.io/crates/duvet/duvet-0.4.2.crate", + "unpack": true, + "hash": "sha256-ey5eGuJv65ARtv+q+LDYMRunfOliNGHCTJ24tMgXbjs=" + }, "fabric": { "type": "GitRelease", "repository": { diff --git a/routing/src/atable/atablerw.rs b/routing/src/atable/atablerw.rs index 3b4ae9eb75..c173b63caf 100644 --- a/routing/src/atable/atablerw.rs +++ b/routing/src/atable/atablerw.rs @@ -82,3 +82,298 @@ impl AtableReaderFactory { AtableReader(self.0.handle()) } } + +/// Model-based properties over the adjacency table and its left-right wrapper. +/// +/// The table is a map. What is not map-like is the *publishing*: `add_adjacency`, `del_adjacency` +/// and `clear` all take a `publish` flag, and `AtResolver::refresh_atable_from_proc` relies on it -- +/// it clears the table with `publish: false`, adds every entry it found with `publish: false`, and +/// publishes once at the end. So a reader must never observe the cleared-but-not-yet-repopulated +/// state. If it could, the egress stage would find an empty adjacency table on every refresh and +/// have no destination mac for anything. +#[cfg(test)] +mod atable_properties { + use super::*; + use bolero::{Driver, ValueGenerator}; + use net::eth::mac::Mac; + use std::collections::BTreeMap; + use std::net::IpAddr; + use std::ops::Bound::Included; + use std::str::FromStr; + + const NUM_IFACES: u8 = 2; + const NUM_ADDRESSES: u8 = 3; + const NUM_MACS: u8 = 2; + const MAX_CHANGES: u8 = 12; + + fn ifindexes() -> Vec { + (1..=u32::from(NUM_IFACES)) + .map(|i| InterfaceIndex::try_new(i).unwrap_or_else(|_| unreachable!())) + .collect() + } + + fn addresses() -> Vec { + ["10.0.0.1", "10.0.0.2", "10.0.0.3"] + .iter() + .map(|a| IpAddr::from_str(a).unwrap_or_else(|_| unreachable!())) + .collect() + } + + fn macs() -> Vec { + vec![ + Mac::from([0x00, 0xaa, 0x00, 0x00, 0x00, 0x01]), + Mac::from([0x00, 0xbb, 0x00, 0x00, 0x00, 0x02]), + ] + } + + /// One change, over indices into the pools above. Every mutation carries the writer's `publish` + /// flag, since whether a change is visible yet is the point. + #[derive(Debug, Clone)] + enum Change { + Add { + iface: usize, + address: usize, + mac: usize, + publish: bool, + }, + Del { + iface: usize, + address: usize, + publish: bool, + }, + Clear { + publish: bool, + }, + Publish, + } + + /// Draws sequences of [`Change`]s. + #[derive(Debug, Clone, Copy, Default)] + struct ChangeSequences; + + fn index(driver: &mut D, count: u8) -> Option { + driver + .gen_u8(Included(&0), Included(&(count - 1))) + .map(usize::from) + } + + impl ValueGenerator for ChangeSequences { + type Output = Vec; + + fn generate(&self, driver: &mut D) -> Option> { + let len = driver.gen_u8(Included(&0), Included(&MAX_CHANGES))?; + let mut out = Vec::with_capacity(usize::from(len)); + for _ in 0..len { + let change = match driver.gen_u8(Included(&0), Included(&3))? { + 0 => Change::Add { + iface: index(driver, NUM_IFACES)?, + address: index(driver, NUM_ADDRESSES)?, + mac: index(driver, NUM_MACS)?, + publish: driver.produce::()?, + }, + 1 => Change::Del { + iface: index(driver, NUM_IFACES)?, + address: index(driver, NUM_ADDRESSES)?, + publish: driver.produce::()?, + }, + 2 => Change::Clear { + publish: driver.produce::()?, + }, + _ => Change::Publish, + }; + out.push(change); + } + Some(out) + } + } + + /// The adjacencies, by `(interface, address)` index pair, to mac index. + type Entries = BTreeMap<(usize, usize), usize>; + + /// Two states, which is the whole point: what the writer has appended, and what a reader is + /// entitled to see. + #[derive(Debug, Clone, Default)] + struct Model { + appended: Entries, + published: Entries, + } + + impl Model { + fn publish(&mut self) { + self.published = self.appended.clone(); + } + + fn apply(&mut self, change: &Change) { + match change { + Change::Add { + iface, + address, + mac, + publish, + } => { + // an adjacency is keyed by the interface and address it carries, so re-learning + // one with a different mac replaces it + self.appended.insert((*iface, *address), *mac); + if *publish { + self.publish(); + } + } + Change::Del { + iface, + address, + publish, + } => { + self.appended.remove(&(*iface, *address)); + if *publish { + self.publish(); + } + } + Change::Clear { publish } => { + self.appended.clear(); + if *publish { + self.publish(); + } + } + Change::Publish => self.publish(), + } + } + } + + fn apply_to_table(writer: &mut AtableWriter, change: &Change) { + let ifaces = ifindexes(); + let addrs = addresses(); + match change { + Change::Add { + iface, + address, + mac, + publish, + } => writer.add_adjacency( + Adjacency::new(addrs[*address], ifaces[*iface], macs()[*mac]), + *publish, + ), + Change::Del { + iface, + address, + publish, + } => writer.del_adjacency(addrs[*address], ifaces[*iface], *publish), + Change::Clear { publish } => writer.clear(*publish), + Change::Publish => writer.publish(), + } + } + + /// Everything a reader can see of the table, as index pairs. + fn seen(table: &AdjacencyTable) -> Entries { + let ifaces = ifindexes(); + let addrs = addresses(); + let all = macs(); + let mut out = Entries::new(); + for (iface, ifindex) in ifaces.iter().enumerate() { + for (address, addr) in addrs.iter().enumerate() { + if let Some(adjacency) = table.get_adjacency(*addr, *ifindex) { + let mac = all + .iter() + .position(|m| *m == adjacency.get_mac()) + .unwrap_or_else(|| unreachable!()); + // and the adjacency agrees with the key it was found under + assert_eq!(adjacency.get_ifindex(), *ifindex, "adjacency ifindex"); + assert_eq!(adjacency.get_ip(), *addr, "adjacency address"); + out.insert((iface, address), mac); + } + } + } + assert_eq!( + out.len(), + table.len(), + "the table holds entries outside the pools" + ); + out + } + + /// The pools and the constants that index them agree. + #[test] + fn the_pools_are_the_size_the_generator_thinks() { + assert_eq!(ifindexes().len(), usize::from(NUM_IFACES)); + assert_eq!(addresses().len(), usize::from(NUM_ADDRESSES)); + assert_eq!(macs().len(), usize::from(NUM_MACS)); + // the macs must be distinguishable, or a replacement would not be visible + assert_ne!(macs()[0], macs()[1]); + } + + /// A reader sees the table as of the last publish, and never an intermediate state. + /// + /// The unpublished half is what `AtResolver::refresh_atable_from_proc` depends on: it clears and + /// repopulates the whole table without publishing, so between the two a reader must still see + /// the previous contents rather than nothing. + #[test] + fn a_reader_sees_the_table_as_of_the_last_publish() { + bolero::check!() + .with_generator(ChangeSequences) + .cloned() + .for_each(|changes: Vec| { + let (mut writer, reader) = AtableWriter::new(); + let mut model = Model::default(); + + for (step, change) in changes.iter().enumerate() { + apply_to_table(&mut writer, change); + model.apply(change); + let at = format!("at step {step} of {changes:?}"); + + let view = reader.enter().unwrap_or_else(|| unreachable!()); + assert_eq!(seen(&view), model.published, "reader {at}"); + } + + // and once everything is published, the reader sees everything appended + writer.publish(); + model.publish(); + let view = reader.enter().unwrap_or_else(|| unreachable!()); + assert_eq!(seen(&view), model.appended, "reader after a final publish"); + }); + } + + /// A refresh that clears and repopulates without publishing is invisible until it does. + /// + /// The same claim as above, spelled out in the shape the resolver actually uses, because that is + /// the sequence whose failure would empty the adjacency table under the egress stage. + #[test] + fn a_clear_and_repopulate_is_invisible_until_published() { + let (mut writer, reader) = AtableWriter::new(); + let ifindex = ifindexes()[0]; + let (old, new) = (addresses()[0], addresses()[1]); + + writer.add_adjacency(Adjacency::new(old, ifindex, macs()[0]), true); + assert!( + reader + .enter() + .unwrap_or_else(|| unreachable!()) + .get_adjacency(old, ifindex) + .is_some() + ); + + // a refresh: clear, repopulate, and only then publish + writer.clear(false); + writer.add_adjacency(Adjacency::new(new, ifindex, macs()[1]), false); + + let view = reader.enter().unwrap_or_else(|| unreachable!()); + assert!( + view.get_adjacency(old, ifindex).is_some(), + "the table emptied under a reader mid-refresh" + ); + assert!( + view.get_adjacency(new, ifindex).is_none(), + "an unpublished addition was visible" + ); + drop(view); + + writer.publish(); + let view = reader.enter().unwrap_or_else(|| unreachable!()); + assert!( + view.get_adjacency(old, ifindex).is_none(), + "the clear was lost" + ); + assert!( + view.get_adjacency(new, ifindex).is_some(), + "the addition was lost" + ); + } +} diff --git a/routing/src/atable/resolver.rs b/routing/src/atable/resolver.rs index c86f342e99..c97cfc1991 100644 --- a/routing/src/atable/resolver.rs +++ b/routing/src/atable/resolver.rs @@ -148,6 +148,80 @@ impl AtResolver { } } +/// Properties over the one part of the resolver that does not need `/proc`. +/// +/// `refresh_atable_from_proc` reads the kernel's ARP table and its interface list, so it cannot be +/// driven from a test. What can is the step between them: resolving the device name an ARP entry +/// carries to an interface index. Every ARP entry the kernel reports goes through it, and an entry +/// it cannot resolve is dropped -- so getting it wrong loses adjacencies silently. +#[cfg(test)] +mod resolver_properties { + use super::*; + use netdev::Interface; + + fn interface(index: u32, name: &str) -> Interface { + Interface { + index, + name: name.to_string(), + ..Interface::dummy() + } + } + + /// A device name resolves to the index of the interface that has it, and to nothing otherwise. + #[test] + fn a_device_name_resolves_to_its_own_interface() { + let interfaces = [interface(2, "eth0"), interface(3, "eth1")]; + + for (index, name) in [(2, "eth0"), (3, "eth1")] { + let found = get_interface_ifindex(&interfaces, name) + .unwrap_or_else(|e| unreachable!("{e}")) + .unwrap_or_else(|| panic!("{name} did not resolve")); + assert_eq!(found.to_u32(), index, "{name} resolved to the wrong index"); + } + + assert_eq!( + get_interface_ifindex(&interfaces, "eth2").unwrap_or_else(|e| unreachable!("{e}")), + None, + "an unknown device must resolve to nothing, not to something else" + ); + assert_eq!( + get_interface_ifindex(&[], "eth0").unwrap_or_else(|e| unreachable!("{e}")), + None, + "no interfaces, nothing to resolve to" + ); + } + + /// An interface index of zero is refused rather than turned into an adjacency. + /// + /// `InterfaceIndex` is non-zero, and the kernel should never report a zero index -- but the + /// resolver takes the number from outside, so the distinction between "no such device" and "a + /// device whose index we cannot represent" has to survive: the first is `Ok(None)` and drops the + /// entry quietly, the second is an error and says why. + #[test] + fn an_interface_index_of_zero_is_an_error_not_a_miss() { + let interfaces = [interface(0, "eth0")]; + assert!( + get_interface_ifindex(&interfaces, "eth0").is_err(), + "index zero must be an error" + ); + assert_eq!( + get_interface_ifindex(&interfaces, "eth1").unwrap_or_else(|e| unreachable!("{e}")), + None, + "a different name is still just a miss" + ); + } + + /// The first interface with a name wins, and a later one with the same name does not shadow it. + #[test] + fn a_repeated_device_name_resolves_to_the_first() { + let interfaces = [interface(2, "eth0"), interface(9, "eth0")]; + let found = get_interface_ifindex(&interfaces, "eth0") + .unwrap_or_else(|e| unreachable!("{e}")) + .unwrap_or_else(|| unreachable!()); + assert_eq!(found.to_u32(), 2); + } +} + #[cfg(test)] pub mod tests { use super::*; diff --git a/routing/src/cli/display.rs b/routing/src/cli/display.rs index 9c40338de0..e400c9793e 100644 --- a/routing/src/cli/display.rs +++ b/routing/src/cli/display.rs @@ -21,7 +21,7 @@ use crate::router::cpi::{CpiStats, CpiStatus, StatsRow}; use crate::rib::VrfTable; use crate::rib::encapsulation::{Encapsulation, VxlanEncapsulation}; -use crate::rib::nexthop::{FwAction, Nhop, NhopKey, NhopStore}; +use crate::rib::nexthop::{FwAction, Nhop, NhopKey, NhopStore, Visited}; use crate::rib::vrf::{Route, RouteFlags, RouteOrigin, ShimNhop, Vrf, VrfStatus}; use crate::interfaces::iftable::IfTable; @@ -40,7 +40,7 @@ use net::vxlan::Vni; use std::fmt::Display; use std::fmt::Write; use std::os::unix::net::SocketAddr; -use std::rc::Rc; +use std::rc::{Rc, Weak}; use std::time::Duration; use std::time::Instant; @@ -126,27 +126,40 @@ impl Display for Nhop { if self.is_unresolved() { write!(f, " (unresolved)")?; } - fmt_nhop_resolvers(f, self, 2) + fmt_nhop_resolvers(f, self, 2, &mut vec![self.id()]) } } -fn fmt_nhop_resolvers(f: &mut std::fmt::Formatter<'_>, rc: &Nhop, depth: u8) -> std::fmt::Result { +/// Print a next-hop's resolvers, and theirs, and so on down. +/// +/// `path` holds the next-hops between the one being displayed and this one. A resolver already on +/// that path closes a resolution loop: we name it and stop, since the recursion has nothing new to +/// show and would otherwise run until `depth` overflowed -- which it did, panicking from inside a +/// `Display` impl that both the CLI and the warning about resolution loops go through. +fn fmt_nhop_resolvers( + f: &mut std::fmt::Formatter<'_>, + rc: &Nhop, + depth: u8, + path: &mut Visited, +) -> std::fmt::Result { let Ok(resolvers) = rc.resolvers.try_borrow() else { warn!("Try-borrow on nhop resolvers failed!"); return Ok(()); }; let tab = 5 * depth as usize; let indent = " ".repeat(tab); - if !resolvers.is_empty() { - for r in resolvers.iter() { - if let Some(r) = r.upgrade().as_ref() { - write!(f, "\n{indent} {}", r.key)?; - if r.is_unresolved() { - write!(f, " (UNRESOLVED)")?; - } - fmt_nhop_resolvers(f, r, depth + 1)?; - } + for r in resolvers.iter().filter_map(Weak::upgrade) { + write!(f, "\n{indent} {}", r.key)?; + if r.is_unresolved() { + write!(f, " (UNRESOLVED)")?; + } + if path.contains(&r.id()) { + write!(f, " (LOOP)")?; + continue; } + path.push(r.id()); + fmt_nhop_resolvers(f, &r, depth.saturating_add(1), path)?; + path.pop(); } Ok(()) } @@ -168,7 +181,14 @@ fn fmt_nhop_instruction(f: &mut std::fmt::Formatter<'_>, rc: &Nhop) -> std::fmt: // formats nhop using the display of the key, recoursing over resolvers // Does not use Nhop::fmt(). -fn fmt_nhop_rec(f: &mut std::fmt::Formatter<'_>, rc: &Rc, depth: u8) -> std::fmt::Result { +// +// `path` guards against a resolution loop, as in `fmt_nhop_resolvers` above. +fn fmt_nhop_rec( + f: &mut std::fmt::Formatter<'_>, + rc: &Rc, + depth: u8, + path: &mut Visited, +) -> std::fmt::Result { let tab = 8 * depth as usize; let indent = " ".repeat(tab); @@ -184,6 +204,9 @@ fn fmt_nhop_rec(f: &mut std::fmt::Formatter<'_>, rc: &Rc, depth: u8) -> st if rc.is_unresolved() { write!(f, " (UNRESOLVED)")?; } + if path.contains(&rc.id()) { + return writeln!(f, " (LOOP)"); + } writeln!(f)?; // fmt_nhop_instruction(f, rc)?; @@ -191,11 +214,11 @@ fn fmt_nhop_rec(f: &mut std::fmt::Formatter<'_>, rc: &Rc, depth: u8) -> st error!("Try-borrow on next-hop resolvers failed!"); return Ok(()); }; - for r in resolvers.iter() { - if let Some(r) = r.upgrade().as_ref() { - fmt_nhop_rec(f, r, depth + 1)?; - } + path.push(rc.id()); + for r in resolvers.iter().filter_map(Weak::upgrade) { + fmt_nhop_rec(f, &r, depth.saturating_add(1), path)?; } + path.pop(); // if let Ok(fg) = rc.as_ref().fibgroup.read() { // writeln!(f, "FibG {}", fg)?; // } @@ -206,7 +229,7 @@ impl Display for NhopStore { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { Heading(format!("Next-hop Store ({})", self.len())).fmt(f)?; for nhop in self.iter() { - fmt_nhop_rec(f, nhop, 0)?; + fmt_nhop_rec(f, nhop, 0, &mut Visited::new())?; fmt_nhop_instruction(f, nhop)?; } line(f) @@ -407,7 +430,7 @@ impl Display for VrfV4Nexthops<'_> { .filter(|nh| nh.key.address.is_none_or(|a| a.is_ipv4())); for nhop in iter { - fmt_nhop_rec(f, nhop, 0)?; + fmt_nhop_rec(f, nhop, 0, &mut Visited::new())?; } line(f) } @@ -425,7 +448,7 @@ impl Display for VrfV6Nexthops<'_> { .filter(|nh| nh.key.address.is_none_or(|a| a.is_ipv6())); for nhop in iter { - fmt_nhop_rec(f, nhop, 0)?; + fmt_nhop_rec(f, nhop, 0, &mut Visited::new())?; } line(f) } diff --git a/routing/src/fib/fibobjects.rs b/routing/src/fib/fibobjects.rs index 70c7b25151..4830ebd96f 100644 --- a/routing/src/fib/fibobjects.rs +++ b/routing/src/fib/fibobjects.rs @@ -262,3 +262,197 @@ pub enum PktInstruction { Encap(Encapsulation), /* encapsulate the packet */ Egress(EgressObject), /* send the packet over interface to some ip */ } + +#[cfg(test)] +mod squash_properties { + use super::*; + use crate::rib::encapsulation::VxlanEncapsulation; + use bolero::{Driver, ValueGenerator}; + use std::net::Ipv4Addr; + use std::num::NonZero; + use std::ops::Bound::Included; + + // Values come from a small alphabet, so that several egress objects disagreeing about the same + // field is the common case rather than a rarity. Deciding between them is the whole of what + // `squash` does; independently drawn values would almost never collide. + const ADDRESSES: [IpAddr; 3] = [ + IpAddr::V4(Ipv4Addr::new(10, 0, 0, 1)), + IpAddr::V4(Ipv4Addr::new(10, 0, 0, 2)), + IpAddr::V4(Ipv4Addr::new(10, 0, 0, 3)), + ]; + const IFNAMES: [&str; 2] = ["eth0", "eth1"]; + + fn index(raw: u8) -> InterfaceIndex { + InterfaceIndex::new(NonZero::new(u32::from(raw)).unwrap_or_else(|| unreachable!())) + } + + // Zero selects `None`, which is one of the choices for every field: an unresolved ifindex or a + // missing address is exactly what the merge rules are about. The draw is separate from the + // choice so that a driver running out of input stays distinct from a generated `None`. + fn choose(pick: u8, choices: &[T]) -> Option { + (pick > 0).then(|| choices[usize::from(pick - 1)].clone()) + } + + fn egress(driver: &mut D) -> Option { + let ifindex = driver.gen_u8(Included(&0), Included(&3))?; + let address = driver.gen_u8(Included(&0), Included(&3))?; + let ifname = driver.gen_u8(Included(&0), Included(&2))?; + Some(EgressObject::new( + choose(ifindex, &[index(1), index(2), index(3)]), + choose(address, &ADDRESSES), + choose(ifname, &IFNAMES).map(str::to_string), + )) + } + + fn instruction(driver: &mut D) -> Option { + Some(match driver.gen_u8(Included(&0), Included(&3))? { + // `Local` carries a distinguishable index so that order preservation can be checked + // rather than merely counted. + 0 => PktInstruction::Local(index(driver.gen_u8(Included(&1), Included(&3))?)), + 1 => PktInstruction::Drop, + 2 => PktInstruction::Encap(Encapsulation::Vxlan(VxlanEncapsulation::new( + Vni::new_checked(u32::from(driver.gen_u8(Included(&1), Included(&3))?)) + .unwrap_or_else(|_| unreachable!()), + ADDRESSES[0], + ))), + _ => PktInstruction::Egress(egress(driver)?), + }) + } + + #[derive(Debug, Clone, Copy, Default)] + struct Entry; + + impl ValueGenerator for Entry { + type Output = FibEntry; + + fn generate(&self, driver: &mut D) -> Option { + let count = driver.gen_u8(Included(&0), Included(&5))?; + let mut entry = FibEntry::new(); + for _ in 0..count { + entry.add(instruction(driver)?); + } + Some(entry) + } + } + + fn egresses(entry: &FibEntry) -> Vec<&EgressObject> { + entry + .iter() + .filter_map(|inst| match inst { + PktInstruction::Egress(e) => Some(e), + _ => None, + }) + .collect() + } + + fn others(entry: &FibEntry) -> Vec<&PktInstruction> { + entry + .iter() + .filter(|inst| !matches!(inst, PktInstruction::Egress(_))) + .collect() + } + + /// Squashing an entry leaves everything that is not an egress exactly as it was. + /// + /// The instructions before the egress are what gets executed on the way out -- encapsulation + /// above all -- so reordering or dropping one of them changes what goes on the wire. + #[test] + fn squash_preserves_the_other_instructions_in_order() { + bolero::check!() + .with_generator(Entry) + .cloned() + .for_each(|entry: FibEntry| { + let before: Vec = others(&entry).into_iter().cloned().collect(); + let mut squashed = entry.clone(); + squashed.squash(); + // A single instruction is returned untouched, egress or not. + if entry.len() == 1 { + assert_eq!(squashed, entry); + return; + } + let after: Vec = others(&squashed).into_iter().cloned().collect(); + assert_eq!(after, before, "for {entry:?}"); + }); + } + + /// What comes out has at most one egress, and it is last. + /// + /// Last because `FibEntry::is_valid` requires it: a multi-instruction entry is only usable if + /// its final instruction is an egress with a known interface. + #[test] + fn squash_leaves_at_most_one_egress_and_puts_it_last() { + bolero::check!() + .with_generator(Entry) + .cloned() + .for_each(|entry: FibEntry| { + if entry.len() == 1 { + return; + } + let mut squashed = entry.clone(); + squashed.squash(); + + assert!(egresses(&squashed).len() <= 1, "for {entry:?}"); + if let Some(position) = squashed + .iter() + .position(|inst| matches!(inst, PktInstruction::Egress(_))) + { + assert_eq!(position, squashed.len() - 1, "for {entry:?}"); + } + }); + } + + /// The merged egress is the first interface, the last address and the first name. + /// + /// The asymmetry is deliberate and load-bearing: `rib2fib` relies on it so that the address of + /// a recursive next-hop reaches the fib while the interface of the nearest resolver wins. An + /// oracle worked out from the input directly, rather than by folding with `merge` again. + #[test] + fn squash_merges_first_interface_last_address_first_name() { + bolero::check!() + .with_generator(Entry) + .cloned() + .for_each(|entry: FibEntry| { + if entry.len() == 1 { + return; + } + let inputs = egresses(&entry); + let ifindex = inputs.iter().find_map(|e| *e.ifindex()); + let address = inputs.iter().rev().find_map(|e| *e.address()); + let ifname = inputs.iter().find_map(|e| e.ifname().clone()); + + let mut squashed = entry.clone(); + squashed.squash(); + + match egresses(&squashed).first() { + Some(merged) => { + assert_eq!(*merged.ifindex(), ifindex, "interface, for {entry:?}"); + assert_eq!(*merged.address(), address, "address, for {entry:?}"); + assert_eq!(*merged.ifname(), ifname, "name, for {entry:?}"); + } + // An egress survives exactly when one of the inputs knew an interface. With + // none, there is nowhere to send the packet and the egress is dropped -- which + // is what leaves the entry to be refused by `is_valid`. + None => assert!(ifindex.is_none(), "for {entry:?}"), + } + }); + } + + /// Squashing twice is squashing once. + /// + /// Worth pinning because the entries are built by accumulating down a resolution chain and + /// squashed at the end of it; a squash that drifted on a second application would make the + /// result depend on how many times the chain was walked. + #[test] + fn squash_is_idempotent() { + bolero::check!() + .with_generator(Entry) + .cloned() + .for_each(|entry: FibEntry| { + let mut once = entry.clone(); + once.squash(); + let mut twice = once.clone(); + twice.squash(); + assert_eq!(twice, once, "for {entry:?}"); + }); + } +} diff --git a/routing/src/fib/fibtable.rs b/routing/src/fib/fibtable.rs index 5fbab2a74a..ef62d9c024 100644 --- a/routing/src/fib/fibtable.rs +++ b/routing/src/fib/fibtable.rs @@ -39,9 +39,14 @@ impl FibTable { self.entries.insert(id, entry); } /// Delete a `Fib`, by unregistering a `FibReaderFactory` for it + /// + /// Every key that reaches the fib goes, not just its own: a fib registered under a [`Vni`] is + /// reachable by that alias too, and an alias must not outlive the fib it names. Each entry + /// records the identity of the fib it points at, so the table can find its own aliases rather + /// than relying on the caller to remember which vni a fib was registered under. fn del_fib(&mut self, id: FibKey) { info!("Unregistering Fib with id {id} from the FibTable"); - self.entries.remove(&id); + self.entries.retain(|_, entry| entry.id != id); } /// Register an existing `Fib` with a given [`Vni`]. /// This allows looking up a Fib (`FibReaderFactory`) from a [`Vni`] @@ -144,12 +149,14 @@ impl FibTableWriter { self.0.append(FibTableChange::UnRegisterVni(vni)); self.0.publish(); } - pub fn del_fib(&mut self, vrfid: VrfId, vni: Option) { - let fibid = FibKey::from_vrfid(vrfid); - self.0.append(FibTableChange::Del(fibid)); - if let Some(vni) = vni { - self.0.append(FibTableChange::UnRegisterVni(vni)); - } + /// Remove the fib for `vrfid`, and with it every key that reached it. + /// + /// This used to take the fib's [`Vni`] so as to drop that alias as well, which made a leaked + /// alias a matter of the caller passing the right thing. [`FibTable::del_fib`] now finds the + /// aliases itself. + pub fn del_fib(&mut self, vrfid: VrfId) { + self.0 + .append(FibTableChange::Del(FibKey::from_vrfid(vrfid))); self.0.publish(); } } @@ -234,3 +241,215 @@ impl FibTableReader { Ok(FibReader::rc_from_rc_rhandle(rhandle)) } } + +/// Model-based properties over a [`FibTable`]. +/// +/// The table is a map, so most of it is uninteresting. The part that is not is the **vni alias**: a +/// fib is reachable both by its own [`FibKey::Id`] and, optionally, by a [`FibKey::Vni`] pointing +/// at the same entry. Nothing in the table ties the two together -- `del_fib` removes the alias +/// only because the caller passes the vni it was registered with -- so an alias outliving its fib +/// is the failure worth generating for. +#[cfg(test)] +mod fibtable_properties { + use super::*; + use crate::fib::fibtype::FibWriter; + use bolero::{Driver, ValueGenerator}; + use std::ops::Bound::Included; + + const NUM_VRFS: u8 = 3; + const NUM_VNIS: u8 = 2; + const MAX_CHANGES: u8 = 10; + + fn vrf_ids() -> Vec { + (0..u32::from(NUM_VRFS)).collect() + } + + fn vnis() -> Vec { + (1..=u32::from(NUM_VNIS)) + .map(|i| Vni::new_checked(100 * i).unwrap_or_else(|_| unreachable!())) + .collect() + } + + /// Every key a generated table could be looked up by. + fn keys() -> Vec { + vrf_ids() + .into_iter() + .map(FibKey::from_vrfid) + .chain(vnis().into_iter().map(FibKey::from_vni)) + .collect() + } + + /// One change, as [`FibTableWriter`] exposes them, over indices into the pools above. + #[derive(Debug, Clone)] + enum Change { + AddFib { vrf: usize, vni: Option }, + RegisterByVni { vrf: usize, vni: usize }, + UnregisterVni { vni: usize }, + DelFib { vrf: usize }, + } + + /// Draws sequences of [`Change`]s. + #[derive(Debug, Clone, Copy, Default)] + struct ChangeSequences; + + fn index(driver: &mut D, count: u8) -> Option { + driver + .gen_u8(Included(&0), Included(&(count - 1))) + .map(usize::from) + } + + impl ValueGenerator for ChangeSequences { + type Output = Vec; + + fn generate(&self, driver: &mut D) -> Option> { + let len = driver.gen_u8(Included(&0), Included(&MAX_CHANGES))?; + let mut out = Vec::with_capacity(usize::from(len)); + for _ in 0..len { + let change = match driver.gen_u8(Included(&0), Included(&3))? { + 0 => { + // A fib created with no vni and aliased later, or never, is the ordinary + // case, so it is worth drawing. Drawn as one index over `NUM_VNIS + 1` + // with the last meaning "none": a helper returning `Option>` + // cannot tell "no vni" from the driver running out of input. + let vrf = index(driver, NUM_VRFS)?; + let drawn = index(driver, NUM_VNIS + 1)?; + Change::AddFib { + vrf, + vni: (drawn < usize::from(NUM_VNIS)).then_some(drawn), + } + } + 1 => Change::RegisterByVni { + vrf: index(driver, NUM_VRFS)?, + vni: index(driver, NUM_VNIS)?, + }, + 2 => Change::UnregisterVni { + vni: index(driver, NUM_VNIS)?, + }, + _ => Change::DelFib { + vrf: index(driver, NUM_VRFS)?, + }, + }; + out.push(change); + } + Some(out) + } + } + + /// Which fib each key should reach, by vrf id. The table stripped of its left-right wrapping, + /// its `Arc` sharing and its reader factories. + type Model = BTreeMap; + + /// The writers behind a generated table, which the harness has to keep alive: a `FibReader` + /// whose `FibWriter` is gone cannot be entered, and that would look like an alias fault. + struct Fibs { + live: BTreeMap, + /// Writers displaced by a second `add_fib` for the same vrf. Nothing in the table points at + /// them any more, but they are not destroyed either, so they are parked rather than + /// dropped -- dropping one is not what production does on a replacement. + retired: Vec, + } + + fn apply(table: &mut FibTableWriter, fibs: &mut Fibs, model: &mut Model, change: &Change) { + let vrfs = vrf_ids(); + let all_vnis = vnis(); + match change { + Change::AddFib { vrf, vni } => { + let vrf = vrfs[*vrf]; + let vni = vni.map(|i| all_vnis[i]); + let writer = table.add_fib(vrf, vni); + if let Some(displaced) = fibs.live.insert(vrf, writer) { + fibs.retired.push(displaced); + } + model.insert(FibKey::from_vrfid(vrf), vrf); + if let Some(vni) = vni { + model.insert(FibKey::from_vni(vni), vrf); + } + } + Change::RegisterByVni { vrf, vni } => { + let vrf = vrfs[*vrf]; + let vni = all_vnis[*vni]; + table.register_fib_by_vni(vrf, vni); + // the table refuses to alias a fib it does not hold + if model.contains_key(&FibKey::from_vrfid(vrf)) { + model.insert(FibKey::from_vni(vni), vrf); + } + } + Change::UnregisterVni { vni } => { + let vni = all_vnis[*vni]; + table.unregister_vni(vni); + model.remove(&FibKey::from_vni(vni)); + } + Change::DelFib { vrf } => { + let vrf = vrfs[*vrf]; + table.del_fib(vrf); + // every key that reached this fib goes, alias included + model.retain(|_, named| *named != vrf); + // production destroys the fib once the table no longer names it, which is what + // makes a leaked alias observable: it would hand out a reader that cannot be + // entered + if let Some(writer) = fibs.live.remove(&vrf) { + writer.destroy(); + } + } + } + } + + /// The pools and the constants that index them agree. + #[test] + fn the_pools_are_the_size_the_generator_thinks() { + assert_eq!(vrf_ids().len(), usize::from(NUM_VRFS)); + assert_eq!(vnis().len(), usize::from(NUM_VNIS)); + assert_eq!(keys().len(), usize::from(NUM_VRFS + NUM_VNIS)); + } + + /// Every key a fib table holds reaches a live fib, and reaches it under its own identity. + /// + /// Two things at once, and the second is the point. A `FibKey::Vni` is an alias for a + /// `FibKey::Id`, and the thread-local read-handle cache keys on the identity the table reports + /// for a key, not on the key asked for -- so an alias reporting the wrong identity would have + /// two threads caching handles to different fibs under one name. Nothing else checks that the + /// alias and the entry it aliases stay in step. + #[test] + fn every_key_in_a_fib_table_reaches_the_fib_it_names() { + let keys = keys(); + + bolero::check!() + .with_generator(ChangeSequences) + .cloned() + .for_each(|changes: Vec| { + let (mut table, _reader) = FibTableWriter::new(); + let mut fibs = Fibs { + live: BTreeMap::new(), + retired: Vec::new(), + }; + let mut model = Model::new(); + + for (step, change) in changes.iter().enumerate() { + apply(&mut table, &mut fibs, &mut model, change); + + let at = || format!("at step {step} of {changes:?}"); + let held = table.enter().unwrap_or_else(|| unreachable!()); + + assert_eq!(held.len(), model.len(), "{}", at()); + + for key in &keys { + let Some(reader) = held.get_fib(*key) else { + assert!(!model.contains_key(key), "{key} missing {}", at()); + continue; + }; + let want = *model + .get(key) + .unwrap_or_else(|| panic!("{key} unexpected {}", at())); + + assert!(reader.is_valid(), "{key} reaches a dead fib {}", at()); + assert_eq!( + reader.get_id(), + Some(FibKey::from_vrfid(want)), + "{key} reaches the wrong fib {}", + at() + ); + } + } + }); + } +} diff --git a/routing/src/fib/fibtype.rs b/routing/src/fib/fibtype.rs index ea368bb36b..f2d597c40f 100644 --- a/routing/src/fib/fibtype.rs +++ b/routing/src/fib/fibtype.rs @@ -504,3 +504,412 @@ impl FibReaderFactory { FibReader(self.0.handle()) } } + +/// Model-based properties over a [`Fib`] driven through its writer. +/// +/// The generator here is the one the pipeline harness will want: it produces a *populated* fib, +/// reached the way production reaches one -- a sequence of `FibChange`s through a `FibWriter` -- +/// rather than by reaching into the tries. Everything the fib is asked afterwards is checked +/// against a model kept alongside it. +#[cfg(test)] +mod fib_properties { + use super::*; + use crate::fib::fibgroupstore::tests::{build_fib_entry_egress, build_fibgroup}; + use bolero::{Driver, ValueGenerator}; + use std::collections::{BTreeMap, BTreeSet}; + use std::ops::Bound::Included; + use std::str::FromStr; + + const NUM_NHOPS: u8 = 4; + const NUM_PREFIXES: u8 = 8; + const NUM_ENTRIES: u8 = 4; + const MAX_CHANGES: u8 = 12; + const MAX_KEYS_PER_ROUTE: u8 = 3; + const MAX_ENTRIES_PER_GROUP: u8 = 3; + + /// The drop next-hop, first in [`nhop_keys`]. The store creates its group at construction and + /// refuses to delete it. + const DROP_KEY: usize = 0; + /// `0.0.0.0/0` and `::/0`, at these indices in [`prefixes`]. A fib always carries a route for + /// both: `Fib::lpm` has no answer for an address nothing covers, and says so with an + /// `unreachable!()` on the forwarding path. + const ROOT_V4: usize = 0; + const ROOT_V6: usize = 5; + + /// The next-hop keys a generated fib may mention. + /// + /// A small pool on purpose. What is worth exercising is the collisions -- a route pinning a + /// group against deletion, a registration mutating a group two routes share -- and collisions + /// need a small pool to happen often. The drop key is in it because the rib does register + /// groups under it, and because the store treats it as permanent. + fn nhop_keys() -> Vec { + vec![ + NhopKey::with_drop(), + NhopKey::with_addr_ifindex("10.0.0.1", 1), + NhopKey::with_addr_ifindex("10.0.0.2", 2), + NhopKey::with_ifindex(3), + ] + } + + /// The prefixes a generated fib may carry routes for. Nested on purpose, so a longest match + /// has something to be longer than, and both roots so that deleting one is reachable. + fn prefixes() -> Vec { + [ + "0.0.0.0/0", + "10.0.0.0/8", + "10.1.0.0/16", + "10.1.2.0/24", + "10.1.2.3/32", + "::/0", + "2001:db8::/32", + "2001:db8:1::/48", + ] + .iter() + .map(|p| Prefix::from_str(p).unwrap_or_else(|_| unreachable!())) + .collect() + } + + /// Addresses to look up: one inside each level of the nesting, and one outside all of them in + /// each family, so a lookup that falls through to the root is exercised too. + fn probes() -> Vec { + [ + "9.9.9.9", + "10.9.9.9", + "10.1.9.9", + "10.1.2.9", + "10.1.2.3", + "2000::1", + "2001:db8::1", + "2001:db8:1::1", + ] + .iter() + .map(|a| IpAddr::from_str(a).unwrap_or_else(|_| unreachable!())) + .collect() + } + + /// The entries a generated group may be built from. + fn entry_pool() -> Vec { + (1..=u32::from(NUM_ENTRIES)) + .map(|i| build_fib_entry_egress(i, &format!("10.0.9.{i}"), &format!("eth{i}"))) + .collect() + } + + /// One change, as the writer API exposes them, over indices into the pools above. + #[derive(Debug, Clone)] + enum Change { + RegisterGroup { key: usize, entries: Vec }, + UnregisterGroup { key: usize }, + AddRoute { prefix: usize, keys: Vec }, + DelRoute { prefix: usize }, + } + + /// Draws sequences of [`Change`]s. + #[derive(Debug, Clone, Copy, Default)] + struct ChangeSequences; + + fn index(driver: &mut D, count: u8) -> Option { + driver + .gen_u8(Included(&0), Included(&(count - 1))) + .map(usize::from) + } + + fn indices(driver: &mut D, count: u8, most: u8) -> Option> { + // Deliberately able to draw none: an empty group and a route with no next-hops are both + // things the fib is supposed to refuse, and refusing is behaviour worth checking. + let len = driver.gen_u8(Included(&0), Included(&most))?; + let mut out = Vec::with_capacity(usize::from(len)); + for _ in 0..len { + out.push(index(driver, count)?); + } + Some(out) + } + + impl ValueGenerator for ChangeSequences { + type Output = Vec; + + fn generate(&self, driver: &mut D) -> Option> { + let len = driver.gen_u8(Included(&0), Included(&MAX_CHANGES))?; + let mut out = Vec::with_capacity(usize::from(len)); + for _ in 0..len { + let change = match driver.gen_u8(Included(&0), Included(&3))? { + 0 => Change::RegisterGroup { + key: index(driver, NUM_NHOPS)?, + entries: indices(driver, NUM_ENTRIES, MAX_ENTRIES_PER_GROUP)?, + }, + 1 => Change::UnregisterGroup { + key: index(driver, NUM_NHOPS)?, + }, + 2 => Change::AddRoute { + prefix: index(driver, NUM_PREFIXES)?, + keys: indices(driver, NUM_NHOPS, MAX_KEYS_PER_ROUTE)?, + }, + _ => Change::DelRoute { + prefix: index(driver, NUM_PREFIXES)?, + }, + }; + out.push(change); + } + Some(out) + } + } + + /// What the fib should hold, tracked beside it. + /// + /// Not a reimplementation: two maps, which is the fib with the tries, the group store's + /// reference counting and its `UnsafeCell` sharing all taken away. What the model does have to + /// get right is which changes the fib *refuses*, and that is the part worth writing down -- + /// three guards in three files decide it, and none of them says so where the next one can see. + #[derive(Debug, Clone)] + struct Model { + /// next-hop key index -> the entries of its group. + groups: BTreeMap>, + /// prefix index -> the next-hop keys of its route, in order. + routes: BTreeMap>, + } + + impl Model { + /// A fresh fib: a drop group, and a route to it for each root. + fn new() -> Self { + Self { + groups: BTreeMap::from([(DROP_KEY, vec![FibEntry::drop_fibentry()])]), + routes: BTreeMap::from([(ROOT_V4, vec![DROP_KEY]), (ROOT_V6, vec![DROP_KEY])]), + } + } + + fn referenced(&self, key: usize) -> bool { + self.routes.values().any(|keys| keys.contains(&key)) + } + + /// Drop every group no route points at. The store does this by reference count; here the + /// routes are the reference count. + fn purge(&mut self) { + let referenced: BTreeSet = self + .routes + .values() + .flatten() + .copied() + .collect::>(); + self.groups + .retain(|key, _| *key == DROP_KEY || referenced.contains(key)); + } + + fn apply(&mut self, change: &Change, pool: &[FibEntry]) { + match change { + Change::RegisterGroup { key, entries } => { + // a group with no entries is refused: a route reaching one would leave the + // forwarder with nothing to execute + if entries.is_empty() { + return; + } + let entries = entries.iter().map(|i| pool[*i].clone()).collect(); + self.groups.insert(*key, entries); + } + Change::UnregisterGroup { key } => { + // the drop group is permanent, and a group a route still names is pinned + if *key == DROP_KEY || self.referenced(*key) { + return; + } + self.groups.remove(key); + } + Change::AddRoute { prefix, keys } => { + // a route with no next-hops is refused, and so is one naming a group that was + // never registered -- whole, not in part + if keys.is_empty() || keys.iter().any(|k| !self.groups.contains_key(k)) { + return; + } + // note: no purge here. Replacing a route releases the old route's hold on its + // groups, but the fib leaves them in the store until something purges. + self.routes.insert(*prefix, keys.clone()); + } + Change::DelRoute { prefix } => { + // a root route is not deleted but reset to drop, so that a lookup always has + // an answer + let removed = if *prefix == ROOT_V4 || *prefix == ROOT_V6 { + self.routes.insert(*prefix, vec![DROP_KEY]) + } else { + self.routes.remove(prefix) + }; + if removed.is_some() { + self.purge(); + } + } + } + } + + /// The longest prefix carrying a route that covers `addr`. + fn lpm(&self, addr: &IpAddr, prefixes: &[Prefix]) -> Option { + self.routes + .keys() + .copied() + .filter(|i| prefixes[*i].covers_addr(addr)) + .max_by_key(|i| prefixes[*i].length()) + } + + /// The entries a route offers: its groups' entries, concatenated in next-hop key order. + fn entries_for(&self, prefix: usize) -> Vec { + self.routes[&prefix] + .iter() + .flat_map(|key| self.groups[key].iter().cloned()) + .collect() + } + } + + fn apply_to_fib(writer: &mut FibWriter, change: &Change, pool: &[FibEntry], keys: &[NhopKey]) { + let prefixes = prefixes(); + match change { + Change::RegisterGroup { key, entries } => { + let entries: Vec = entries.iter().map(|i| pool[*i].clone()).collect(); + writer.register_fibgroup(&keys[*key], &build_fibgroup(&entries), true); + } + Change::UnregisterGroup { key } => writer.unregister_fibgroup(&keys[*key], true), + Change::AddRoute { + prefix, + keys: route, + } => { + let route = route.iter().map(|k| keys[*k].clone()).collect(); + writer.add_fibroute(prefixes[*prefix], route, true); + } + Change::DelRoute { prefix } => writer.del_fibroute(prefixes[*prefix]), + } + } + + /// The pools and the constants that index them agree. + #[test] + fn the_pools_are_the_size_the_generator_thinks() { + assert_eq!(nhop_keys().len(), usize::from(NUM_NHOPS)); + assert_eq!(prefixes().len(), usize::from(NUM_PREFIXES)); + assert_eq!(entry_pool().len(), usize::from(NUM_ENTRIES)); + assert_eq!(nhop_keys()[DROP_KEY], NhopKey::with_drop()); + assert_eq!(prefixes()[ROOT_V4], Prefix::root_v4()); + assert_eq!(prefixes()[ROOT_V6], Prefix::root_v6()); + for probe in probes() { + assert!( + prefixes().iter().any(|p| p.covers_addr(&probe)), + "probe {probe} is covered by no prefix, not even a root" + ); + } + } + + /// After any sequence of changes, a fib answers every lookup the way the model says. + /// + /// This is the whole of the fib's read path against an independent account of its contents: + /// which prefix the lookup lands on, and which entries the route there offers. It covers the + /// group store's sharing too, since registering a group under a key two routes name has to + /// change what both of them offer. + #[test] + fn a_fib_answers_lookups_the_way_the_model_says() { + let keys = nhop_keys(); + let prefixes = prefixes(); + let probes = probes(); + let pool = entry_pool(); + + bolero::check!() + .with_generator(ChangeSequences) + .cloned() + .for_each(|changes: Vec| { + let (mut writer, _reader) = FibWriter::new(FibKey::from_vrfid(1)); + let mut model = Model::new(); + + for (step, change) in changes.iter().enumerate() { + apply_to_fib(&mut writer, change, &pool, &keys); + model.apply(change, &pool); + + let fib = writer.enter().unwrap_or_else(|| unreachable!()); + let at = || format!("at step {step} of {changes:?}"); + + assert_eq!(fib.len_groups(), model.groups.len(), "{}", at()); + + for probe in &probes { + let want = model + .lpm(probe, &prefixes) + .unwrap_or_else(|| panic!("model has no route for {probe} {}", at())); + + let (hit, route) = fib.lpm_with_prefix(probe); + assert_eq!(hit, prefixes[want], "for {probe} {}", at()); + + let got: Vec = route + .iter() + .flat_map(|group| group.entries().iter().cloned()) + .collect(); + assert_eq!(got, model.entries_for(want), "for {probe} {}", at()); + } + } + }); + } + + /// A lookup always lands on a route with at least one entry to execute. + /// + /// `Fib::lpm_entry_prefix` panics outright on a route with none -- "hit route without + /// fibgroups/entries. This is a bug." -- on the forwarding path, for every packet that reaches + /// it. The invariant that saves it is held jointly by three guards in three files: the store + /// refuses an empty group, the writer refuses a route with no next-hop keys, and + /// `FibRoute::from_nhopkeys` refuses a route naming a group that is not registered. Nothing + /// states the invariant they add up to, so state it here. + #[test] + fn every_route_a_lookup_reaches_has_an_entry_to_execute() { + let keys = nhop_keys(); + let probes = probes(); + let pool = entry_pool(); + + bolero::check!() + .with_generator(ChangeSequences) + .cloned() + .for_each(|changes: Vec| { + let (mut writer, _reader) = FibWriter::new(FibKey::from_vrfid(1)); + for change in &changes { + apply_to_fib(&mut writer, change, &pool, &keys); + } + + let fib = writer.enter().unwrap_or_else(|| unreachable!()); + for probe in &probes { + let (_, route) = fib.lpm_with_prefix(probe); + assert!(route.len() > 0, "no entry for {probe} after {changes:?}"); + // and the index arithmetic that picks among them is total over that range + for index in 0..route.len() { + let _ = route.get_fibentry(index); + } + } + }); + } + + /// A reader sees what the writer sees, once every change has been published. + #[test] + fn a_reader_and_a_writer_agree_after_publishing() { + let keys = nhop_keys(); + let probes = probes(); + let pool = entry_pool(); + + bolero::check!() + .with_generator(ChangeSequences) + .cloned() + .for_each(|changes: Vec| { + let (mut writer, reader) = FibWriter::new(FibKey::from_vrfid(1)); + for change in &changes { + apply_to_fib(&mut writer, change, &pool, &keys); + } + + for probe in &probes { + let (want_prefix, want_entries) = { + let fib = writer.enter().unwrap_or_else(|| unreachable!()); + let (prefix, route) = fib.lpm_with_prefix(probe); + let entries: Vec = route + .iter() + .flat_map(|group| group.entries().iter().cloned()) + .collect(); + (prefix, entries) + }; + + let (got_prefix, route) = reader + .lpm_route_with_prefix(*probe) + .unwrap_or_else(|| unreachable!()); + let got_entries: Vec = route + .iter() + .flat_map(|group| group.entries().iter().cloned()) + .collect(); + + assert_eq!(got_prefix, want_prefix, "for {probe} after {changes:?}"); + assert_eq!(got_entries, want_entries, "for {probe} after {changes:?}"); + } + }); + } +} diff --git a/routing/src/fib/test.rs b/routing/src/fib/test.rs index bac97ec8ca..e2e9ec8ca1 100644 --- a/routing/src/fib/test.rs +++ b/routing/src/fib/test.rs @@ -367,7 +367,7 @@ mod tests { } if updates.is_multiple_of(50) && fibw.is_some() { - fibtw.del_fib(vrfid, None); + fibtw.del_fib(vrfid); if let Some(fib) = fibw.take() { // fib is destroyed here fib.destroy(); @@ -519,7 +519,7 @@ mod concurrency_tests { loop { let fibw = fibtw.add_fib(vrfid, None); thread::sleep(Duration::from_millis(5)); - fibtw.del_fib(vrfid, None); + fibtw.del_fib(vrfid); fibw.destroy(); iterations += 1; if iterations == MAX_ITERATIONS { diff --git a/routing/src/frr/frrmi.rs b/routing/src/frr/frrmi.rs index 50f9ad802d..c8311e71e2 100644 --- a/routing/src/frr/frrmi.rs +++ b/routing/src/frr/frrmi.rs @@ -327,6 +327,11 @@ impl Frrmi { return Err(FrrErr::NotConnected); }; loop { + if let Some(announced) = self.readb.oversized() { + error!("Frr-agent announced a {announced}-octet message: refusing it"); + self.readb.clear(); + return Err(FrrErr::DecodeFailure); + } let pending = self.readb.next_read_len(); debug!("Recv data (read:{} pending:{pending})", self.readb.used); match Self::recv(sock, &mut self.readb, pending) { @@ -400,6 +405,14 @@ struct IoBuffer { used: usize, } impl IoBuffer { + /// Octets of `|length|genid|` ahead of every message body. + const HEADER_LEN: usize = 16; + + /// The largest message body we will accept off the wire. Responses from the frr-agent are a + /// status word or an error message, so this is generous by orders of magnitude; it is here to + /// bound the allocation, not to constrain the protocol. + const MAX_MSG_LEN: usize = 16 * 1024 * 1024; + #[must_use] #[allow(unused)] pub fn new() -> Self { @@ -412,6 +425,10 @@ impl IoBuffer { self.buffer.clear(); self.used = 0; } + /// The size of the buffer. Meaningful on the **write** side only, where `serialize` fills it: + /// there, `buffer.len()` is the size of the message and `used` is how much of it has gone out. + /// On the read side `recv` resizes the buffer to the size of the read it is about to attempt, so + /// `buffer.len()` says what we hoped for and only `used` says what arrived. #[must_use] fn len(&self) -> usize { self.buffer.len() @@ -426,37 +443,45 @@ impl IoBuffer { self.extend(msg); } - /// Tell the length that a message (encoded as |length|genid|data|) must have. - /// If less than 8 octets have been read it is not possible to know how big the message is yet. + /// Tell the length that a message (encoded as |length|genid|data|) must have, once the whole + /// header has been received. `None` until then. + /// + /// Off `used`, not off `buffer.len()`: see [`IoBuffer::len`]. Reading the length out of a + /// buffer sized for a read that has not happened yet takes whatever `resize` zero-filled for + /// the tail of the header, so a half-arrived header reads as a complete one announcing a + /// shorter message. #[must_use] fn msg_len(&self) -> Option { - if self.buffer.len() < 8 { - None - } else { - let len_buf = &self.buffer[0..8] - .try_into() - .unwrap_or_else(|_| unreachable!()); - - #[allow(clippy::cast_possible_truncation)] - let msg_len = u64::from_ne_bytes(*len_buf) as usize; - Some(msg_len) + if self.used < Self::HEADER_LEN { + return None; } + let len_buf: &[u8; 8] = &self.buffer[0..8] + .try_into() + .unwrap_or_else(|_| unreachable!()); + + #[allow(clippy::cast_possible_truncation)] + let msg_len = u64::from_ne_bytes(*len_buf) as usize; + Some(msg_len) + } + + /// The announced message length, if it is one we refuse to accept. + /// + /// The length prefix comes off the wire and `recv` resizes the read buffer to it, so without a + /// bound a confused or hostile frr-agent could ask us to allocate up to `u64::MAX`. + #[must_use] + fn oversized(&self) -> Option { + self.msg_len().filter(|len| *len > Self::MAX_MSG_LEN) } - /// Tell the number of octets that should be read next according to the contents of the read buffer - /// to get a message or be able to determine its length. - /// If less than 16 octets have been received, this returns the number needed to have exactly 16. - /// Else, we return the number of octets that are pending to have the complete message. + + /// Tell the number of octets that should be read next according to what has been received so + /// far, to get a message or be able to determine its length. + /// Until the whole header has arrived, this returns the number needed to complete it. + /// After that, the number of octets still pending to have the complete message. #[must_use] fn next_read_len(&self) -> usize { - if self.len() < 16 { - 16 - self.len() - } else { - let msg_len = self.msg_len().unwrap_or_else(|| unreachable!()); - if msg_len > (self.len() - 16) { - msg_len - (self.len() - 16) - } else { - 0 - } + match self.msg_len() { + None => Self::HEADER_LEN - self.used, + Some(msg_len) => msg_len.saturating_sub(self.used - Self::HEADER_LEN), } } @@ -464,7 +489,9 @@ impl IoBuffer { #[must_use] fn is_ready(&self) -> bool { match self.msg_len() { - Some(m) => self.len() == m + 16, + // subtracting rather than adding 16: the announced length is peer-supplied, and + // `m + 16` overflows for one close enough to `usize::MAX` + Some(msg_len) => self.used - Self::HEADER_LEN == msg_len, None => false, } } @@ -488,3 +515,195 @@ impl IoBuffer { Ok(FrrmiResponse { genid, data }) } } + +/// Properties over the frrmi wire framing. +/// +/// `IoBuffer` exists to cope with partial reads and writes on a stream socket -- so the property +/// worth having is that the message survives *any* division of its octets into reads. The oracle is +/// the message itself: serialize, deliver the bytes in generated chunks, and what comes back out +/// must be what went in. +#[cfg(test)] +mod framing_properties { + use super::*; + use bolero::{Driver, ValueGenerator}; + use std::ops::Bound::Included; + + const MAX_BODY: u8 = 20; + const MAX_CHUNK: u8 = 24; + const MAX_CHUNKS: u8 = 8; + const MAX_MESSAGES: u8 = 4; + + /// A message, and the way the peer's octets happen to arrive. + #[derive(Debug, Clone)] + struct Delivery { + genid: GenId, + /// The response body. Empty is worth generating: a zero-length body makes every octet of + /// the length prefix zero, which is the value a half-read header is indistinguishable from. + body: String, + /// Sizes of the successive writes the peer makes. Applied in order and clamped to what is + /// left; anything still unsent after the list runs out goes in one final write. + chunks: Vec, + } + + /// Draws [`Delivery`]s. + #[derive(Debug, Clone, Copy, Default)] + struct Deliveries; + + impl ValueGenerator for Deliveries { + type Output = Delivery; + + fn generate(&self, driver: &mut D) -> Option { + let genid = GenId::from(driver.gen_u8(Included(&0), Included(&3))?); + let body_len = usize::from(driver.gen_u8(Included(&0), Included(&MAX_BODY))?); + let body: String = (0..body_len) + .map(|i| char::from(b'a' + u8::try_from(i % 26).unwrap_or_else(|_| unreachable!()))) + .collect(); + + let count = driver.gen_u8(Included(&0), Included(&MAX_CHUNKS))?; + let mut chunks = Vec::with_capacity(usize::from(count)); + for _ in 0..count { + chunks.push(usize::from( + driver.gen_u8(Included(&1), Included(&MAX_CHUNK))?, + )); + } + Some(Delivery { + genid, + body, + chunks, + }) + } + } + + /// A `Frrmi` reading from one end of a socket pair, with the peer's end alongside it. + fn connected_pair() -> (UnixStream, Frrmi) { + let (peer, ours) = UnixStream::pair().unwrap_or_else(|e| unreachable!("{e}")); + let frrmi = Frrmi { + sock: Some(ours), + ..Frrmi::default() + }; + (peer, frrmi) + } + + /// Draws a sequence of [`Delivery`]s to send down one connection. + #[derive(Debug, Clone, Copy, Default)] + struct Streams; + + impl ValueGenerator for Streams { + type Output = Vec; + + fn generate(&self, driver: &mut D) -> Option> { + let count = driver.gen_u8(Included(&1), Included(&MAX_MESSAGES))?; + let mut out = Vec::with_capacity(usize::from(count)); + for _ in 0..count { + out.push(Deliveries.generate(driver)?); + } + Some(out) + } + } + + /// A message survives being delivered in any sequence of chunks. + #[test] + fn a_message_survives_any_division_into_reads() { + bolero::check!() + .with_generator(Deliveries) + .cloned() + .for_each(|delivery: Delivery| { + let mut wire = IoBuffer::new(); + wire.serialize(delivery.genid, delivery.body.as_bytes()); + let bytes = wire.buffer; + + let (mut peer, mut frrmi) = connected_pair(); + + let mut sent = 0; + let mut got = None; + let mut sizes = delivery.chunks.iter().copied(); + while sent < bytes.len() { + let take = sizes.next().unwrap_or(usize::MAX).min(bytes.len() - sent); + peer.write_all(&bytes[sent..sent + take]) + .unwrap_or_else(|e| unreachable!("{e}")); + sent += take; + + match frrmi.recv_msg() { + Ok(Some(response)) => { + assert!(got.is_none(), "two messages from one, for {delivery:?}"); + got = Some(response); + } + Ok(None) => (), + Err(e) => panic!("recv failed with {e} for {delivery:?}"), + } + } + + let response = got.unwrap_or_else(|| panic!("no message, for {delivery:?}")); + assert_eq!(response.genid, delivery.genid, "genid for {delivery:?}"); + assert_eq!(response.data, delivery.body, "body for {delivery:?}"); + }); + } + + /// A connection carries one message after another, however the octets are divided. + /// + /// The chunking runs over the whole stream rather than over each message, so a single write may + /// straddle a message boundary or carry several messages at once. `deserialize` clears the read + /// buffer, and this is what says nothing is left in it to confuse the message after. + #[test] + fn a_connection_carries_one_message_after_another() { + bolero::check!() + .with_generator(Streams) + .cloned() + .for_each(|deliveries: Vec| { + let mut bytes = Vec::new(); + for delivery in &deliveries { + let mut wire = IoBuffer::new(); + wire.serialize(delivery.genid, delivery.body.as_bytes()); + bytes.extend_from_slice(&wire.buffer); + } + + let (mut peer, mut frrmi) = connected_pair(); + + // one chunk list over the whole stream, taken from the deliveries in turn + let mut sizes = deliveries.iter().flat_map(|d| d.chunks.iter().copied()); + let mut got: Vec<(GenId, String)> = Vec::new(); + let mut sent = 0; + while sent < bytes.len() { + let take = sizes.next().unwrap_or(usize::MAX).min(bytes.len() - sent); + peer.write_all(&bytes[sent..sent + take]) + .unwrap_or_else(|e| unreachable!("{e}")); + sent += take; + + // drain: a single write may have completed more than one message + loop { + match frrmi.recv_msg() { + Ok(Some(response)) => got.push((response.genid, response.data)), + Ok(None) => break, + Err(e) => panic!("recv failed with {e} for {deliveries:?}"), + } + } + } + + let want: Vec<(GenId, String)> = deliveries + .iter() + .map(|d| (d.genid, d.body.clone())) + .collect(); + assert_eq!(got, want, "for {deliveries:?}"); + }); + } + + /// A message longer than we will accept is refused, not allocated for. + /// + /// The length prefix is whatever the peer put on the wire, and `recv` resizes the read buffer to + /// it. Announcing `u64::MAX` used to make that resize overflow its own length arithmetic. + #[test] + fn an_absurd_announced_length_is_refused() { + let mut header = Vec::new(); + header.extend_from_slice(&u64::MAX.to_ne_bytes()); + header.extend_from_slice(&0i64.to_ne_bytes()); + + let (mut peer, mut frrmi) = connected_pair(); + peer.write_all(&header) + .unwrap_or_else(|e| unreachable!("{e}")); + + assert!( + matches!(frrmi.recv_msg(), Err(FrrErr::DecodeFailure)), + "an absurd length must be refused" + ); + } +} diff --git a/routing/src/frr/renderer/mod.rs b/routing/src/frr/renderer/mod.rs index 64a7f89837..75de135fa6 100644 --- a/routing/src/frr/renderer/mod.rs +++ b/routing/src/frr/renderer/mod.rs @@ -66,3 +66,531 @@ impl Render for InternalConfig { cfg } } + +/// Properties over the FRR config renderers. +/// +/// A renderer's failure mode is not a crash: it is a BFD session, an OSPF area or a vni that never +/// gets configured because a line was not emitted, or that gets configured twice. Neither shows up +/// anywhere but in FRR's own state. Each renderer had one test that printed its output and asserted +/// nothing, so `ospf`, `bfd` and this module were at zero coverage. +/// +/// A round-trip oracle would need an FRR parser and is not worth building. What is worth checking is +/// weaker and still catches both failure modes: **every value the config holds appears in the +/// output, and no value it does not hold appears.** That is enough to catch an omitted field, a +/// field emitted when it should not be, and a top-level renderer that forgot to call a sub-renderer. +#[cfg(test)] +mod renderer_properties { + use super::*; + use bolero::{Driver, ValueGenerator}; + use config::external::overlay::vpc::VpcId; + use config::internal::device::DeviceConfig; + use config::internal::routing::bfd::{ + BFD_DETECT_MULTIPLIER, BFD_RECEIVE_INTERVAL_MS, BFD_TRANSMIT_INTERVAL_MS, BfdPeer, + }; + use config::internal::routing::ospf::{Ospf, OspfInterface, OspfNetwork}; + use config::internal::routing::vrf::{VrfConfig, VrfConfigTable}; + use net::route::RouteTableId; + use net::vxlan::Vni; + use std::net::{IpAddr, Ipv4Addr}; + use std::ops::Bound::Included; + use std::str::FromStr; + + const NUM_ADDRESSES: u8 = 3; + const NUM_NETWORKS: u8 = 4; + const MAX_PEERS: u8 = 3; + const MAX_VRFS: u8 = 3; + + fn addresses() -> Vec { + ["10.0.0.1", "10.0.0.2", "2001:db8::1"] + .iter() + .map(|a| IpAddr::from_str(a).unwrap_or_else(|_| unreachable!())) + .collect() + } + + fn networks() -> Vec { + vec![ + OspfNetwork::Broadcast, + OspfNetwork::NonBroadcast, + OspfNetwork::Point2Point, + OspfNetwork::Point2Multipoint, + ] + } + + /// The keyword FRR expects for each network type, written out here rather than taken from + /// `OspfNetwork::rendered`, so a wrong pairing is visible. + fn network_keyword(network: &OspfNetwork) -> &'static str { + match network { + OspfNetwork::Broadcast => "broadcast", + OspfNetwork::NonBroadcast => "non-broadcast", + OspfNetwork::Point2Point => "point-to-point", + OspfNetwork::Point2Multipoint => "point-to-multipoint", + } + } + + /// A BFD peer as the generator describes it. + #[derive(Debug, Clone, Copy)] + struct PeerSpec { + address: usize, + multihop: bool, + source: Option, + } + + /// A vrf as the generator describes it. + /// + /// Name, table id, vni and vpc id are all derived from its position rather than generated. + /// `VrfConfigTable` is a multi-index map with a *unique* index over each of them, and an + /// `Option` field's `None` counts as a value there -- so it can hold at most one vrf without a + /// vni, and at most one without a vpc id. Production satisfies that (the default vrf has + /// neither; every other vrf is a vpc vrf and has both), but the types do not say so, and + /// `add_vrf_config` calls a collision "a bug". So the harness gives every vrf its own, and the + /// "renders it only when it has one" cases are checked against `VrfConfig` directly below. + #[derive(Debug, Clone, Copy)] + struct VrfSpec { + ospf: Option, + } + + #[derive(Debug, Clone)] + struct Fabric { + genid: GenId, + peers: Vec, + vrfs: Vec, + } + + /// Draws [`Fabric`]s. + #[derive(Debug, Clone, Copy, Default)] + struct Fabrics; + + fn index(driver: &mut D, count: u8) -> Option { + driver + .gen_u8(Included(&0), Included(&(count - 1))) + .map(usize::from) + } + + /// An index into a pool of `count`, or `count` itself to mean "absent". + /// + /// One draw with a sentinel rather than an `Option>`, whose outer `None` -- the driver + /// running out of input -- cannot be told from the inner one. + fn maybe_index(driver: &mut D, count: u8) -> Option { + index(driver, count + 1) + } + + /// Read a [`maybe_index`] draw back as an option. + fn drawn(value: usize, count: u8) -> Option { + (value < usize::from(count)).then_some(value) + } + + impl ValueGenerator for Fabrics { + type Output = Fabric; + + fn generate(&self, driver: &mut D) -> Option { + let genid = GenId::from(driver.gen_u8(Included(&1), Included(&9))?); + + let count = driver.gen_u8(Included(&0), Included(&MAX_PEERS))?; + let mut peers = Vec::with_capacity(usize::from(count)); + for _ in 0..count { + peers.push(PeerSpec { + address: index(driver, NUM_ADDRESSES)?, + multihop: driver.produce::()?, + source: drawn(maybe_index(driver, NUM_ADDRESSES)?, NUM_ADDRESSES), + }); + } + + let count = driver.gen_u8(Included(&0), Included(&MAX_VRFS))?; + let mut vrfs = Vec::with_capacity(usize::from(count)); + for _ in 0..count { + vrfs.push(VrfSpec { + ospf: drawn(maybe_index(driver, NUM_ADDRESSES)?, NUM_ADDRESSES), + }); + } + + Some(Fabric { genid, peers, vrfs }) + } + } + + fn peer(spec: PeerSpec) -> BfdPeer { + BfdPeer::new(addresses()[spec.address]) + .set_multihop(spec.multihop) + .set_source(spec.source.map(|i| addresses()[i])) + } + + /// A router id for vrf `index`, distinct per vrf so it can be looked for in the output. + fn router_id(index: usize) -> Ipv4Addr { + Ipv4Addr::new( + 192, + 168, + 0, + u8::try_from(index + 1).unwrap_or_else(|_| unreachable!()), + ) + } + + fn vrf_name(index: usize) -> String { + format!("VPC-{index}") + } + + /// A vpc id for vrf `index`. + /// + /// Every non-default vrf needs one: `VrfConfigTable` holds a unique index over `vpc_id`, so two + /// vrfs without one collide. Real configs always have them -- non-default vrfs are vpc vrfs -- + /// but it is not obvious from the type, which takes an `Option`. + fn vpc_id(index: usize) -> VpcId { + VpcId::try_from(format!("vpc{index:02}").as_str()).unwrap_or_else(|_| unreachable!()) + } + + fn vni_for(index: usize) -> Vni { + Vni::new_checked(3000 + u32::try_from(index).unwrap_or_else(|_| unreachable!())) + .unwrap_or_else(|_| unreachable!()) + } + + fn internal_config(fabric: &Fabric) -> InternalConfig { + let mut config = InternalConfig::new("GW1", DeviceConfig::new()); + config.bfd_peers = fabric.peers.iter().copied().map(peer).collect(); + + let mut vrfs = VrfConfigTable::new(); + for (index, spec) in fabric.vrfs.iter().enumerate() { + let mut vrf = VrfConfig::new(&vrf_name(index), Some(vni_for(index)), false) + .set_table_id( + RouteTableId::try_from( + 100 + u32::try_from(index).unwrap_or_else(|_| unreachable!()), + ) + .unwrap_or_else(|_| unreachable!()), + ) + .set_vpc_id(vpc_id(index)); + if spec.ospf.is_some() { + let mut ospf = Ospf::new(router_id(index)); + ospf.set_vrf_name(vrf_name(index)); + vrf.set_ospf(ospf); + } + vrfs.add_vrf_config(vrf) + .unwrap_or_else(|e| unreachable!("{e}")); + } + config.vrfs = vrfs; + config + } + + /// How many times `needle` appears in `haystack`. + fn occurrences(haystack: &str, needle: &str) -> usize { + haystack.matches(needle).count() + } + + /// The pools and the constants that index them agree. + #[test] + fn the_pools_are_the_size_the_generator_thinks() { + assert_eq!(addresses().len(), usize::from(NUM_ADDRESSES)); + assert_eq!(networks().len(), usize::from(NUM_NETWORKS)); + // the derived names, table ids and vnis must be distinct, or `add_vrf_config` would refuse + // them and the harness would be testing its own collision handling + let count = usize::from(MAX_VRFS); + for derived in [ + (0..count).map(vrf_name).collect::>(), + (0..count).map(|i| format!("{:?}", vpc_id(i))).collect(), + (0..count).map(|i| vni_for(i).to_string()).collect(), + (0..count).map(|i| router_id(i).to_string()).collect(), + ] { + let mut sorted = derived.clone(); + sorted.sort(); + sorted.dedup(); + assert_eq!( + sorted.len(), + derived.len(), + "derived values must be distinct" + ); + } + } + + /// A BFD peer renders every field it carries, and none it does not. + /// + /// Note the one rule that is not "render what is set": a source address is emitted only for a + /// multihop peer. A single-hop peer with a source silently loses it, which is deliberate -- + /// FRR has nowhere to put it -- and worth having written down. + #[test] + fn a_bfd_peer_renders_the_fields_it_carries() { + bolero::check!() + .with_generator(Fabrics) + .cloned() + .for_each(|fabric: Fabric| { + for spec in &fabric.peers { + let peer = peer(*spec); + let text = peer.render(&()).to_string(); + + assert_eq!( + occurrences(&text, &format!(" peer {}", peer.address)), + 1, + "peer address once, for {spec:?} in {text}" + ); + assert_eq!( + occurrences(&text, " multihop"), + usize::from(spec.multihop), + "multihop iff set, for {spec:?} in {text}" + ); + + let source_shown = spec.multihop && spec.source.is_some(); + assert_eq!( + occurrences(&text, " source "), + usize::from(source_shown), + "a source is rendered only for a multihop peer, for {spec:?} in {text}" + ); + if source_shown { + let source = addresses()[spec.source.unwrap_or_else(|| unreachable!())]; + assert_eq!( + occurrences(&text, &format!(" source {source}")), + 1, + "the source that was set, for {spec:?} in {text}" + ); + } + + // and the timing parameters FRR needs to bring the session up at all + for line in [ + " no shutdown".to_string(), + format!(" detect-multiplier {BFD_DETECT_MULTIPLIER}"), + format!(" transmit-interval {BFD_TRANSMIT_INTERVAL_MS}"), + format!(" receive-interval {BFD_RECEIVE_INTERVAL_MS}"), + ] { + assert_eq!(occurrences(&text, &line), 1, "{line} once, in {text}"); + } + } + }); + } + + /// A BFD section appears only when there are peers, and holds each of them once. + #[test] + fn a_bfd_section_appears_only_for_peers_it_has() { + bolero::check!() + .with_generator(Fabrics) + .cloned() + .for_each(|fabric: Fabric| { + let peers: Vec = fabric.peers.iter().copied().map(peer).collect(); + let text = peers.render(&()).to_string(); + + if peers.is_empty() { + assert!( + !text.contains("bfd"), + "an empty peer list must render no bfd section, got {text}" + ); + return; + } + + assert_eq!( + occurrences(&text, "\nbfd\n"), + 1, + "one bfd section in {text}" + ); + assert_eq!(occurrences(&text, "\nexit\n"), 1, "one exit in {text}"); + for address in addresses() { + let wanted = peers.iter().filter(|p| p.address == address).count(); + assert_eq!( + occurrences(&text, &format!(" peer {address}")), + wanted, + "{address} appears once per peer that has it, in {text}" + ); + } + }); + } + + /// An OSPF instance renders its router id, and its vrf only when it has one. + #[test] + fn an_ospf_instance_renders_its_router_id_and_vrf() { + bolero::check!() + .with_generator(Fabrics) + .cloned() + .for_each(|fabric: Fabric| { + for (index, _) in fabric.vrfs.iter().enumerate() { + let id = router_id(index); + let name = vrf_name(index); + + let plain = Ospf::new(id).render(&()).to_string(); + assert_eq!( + occurrences(&plain, "router ospf\n"), + 1, + "an ospf instance with no vrf, in {plain}" + ); + assert_eq!( + occurrences(&plain, &format!(" ospf router-id {id}")), + 1, + "the router id, in {plain}" + ); + + let mut in_vrf = Ospf::new(id); + in_vrf.set_vrf_name(name.clone()); + let text = in_vrf.render(&()).to_string(); + assert_eq!( + occurrences(&text, &format!("router ospf vrf {name}")), + 1, + "an ospf instance in a vrf, in {text}" + ); + assert_eq!( + occurrences(&text, &format!(" ospf router-id {id}")), + 1, + "the router id, in {text}" + ); + } + }); + } + + /// An OSPF interface renders its area, and each option only when it is set. + #[test] + fn an_ospf_interface_renders_the_options_it_has() { + bolero::check!() + .with_generator(bolero::produce::<(u8, bool, Option, Option)>()) + .cloned() + .for_each( + |(area, passive, cost, network): (u8, bool, Option, Option)| { + let area = Ipv4Addr::new(0, 0, 0, area); + let network = + network.map(|n| networks()[usize::from(n) % networks().len()].clone()); + + let mut interface = OspfInterface::new(area).set_passive(passive); + if let Some(cost) = cost { + interface = interface.set_cost(cost); + } + if let Some(network) = network.clone() { + interface = interface.set_network(network); + } + let text = interface.render(&()).to_string(); + + assert_eq!( + occurrences(&text, &format!(" ip ospf area {area}")), + 1, + "the area, in {text}" + ); + assert_eq!( + occurrences(&text, " ip ospf passive"), + usize::from(passive), + "passive iff set, in {text}" + ); + assert_eq!( + occurrences(&text, " ip ospf cost "), + usize::from(cost.is_some()), + "cost iff set, in {text}" + ); + if let Some(cost) = cost { + assert_eq!(occurrences(&text, &format!(" ip ospf cost {cost}")), 1); + } + assert_eq!( + occurrences(&text, " ip ospf network "), + usize::from(network.is_some()), + "network iff set, in {text}" + ); + if let Some(network) = &network { + assert_eq!( + occurrences( + &text, + &format!(" ip ospf network {}", network_keyword(network)) + ), + 1, + "the network keyword FRR expects, in {text}" + ); + } + }, + ); + } + + /// Everything the config holds reaches the rendered output. + /// + /// This is the property the per-renderer ones cannot give: a sub-renderer that works perfectly is + /// no use if the top level never calls it, and an object silently missing from an FRR config is + /// a session or a vni that never comes up. + #[test] + fn everything_configured_reaches_the_output() { + bolero::check!() + .with_generator(Fabrics) + .cloned() + .for_each(|fabric: Fabric| { + let config = internal_config(&fabric); + let text = config.render(&fabric.genid).to_string(); + + assert_eq!( + occurrences(&text, &format!("! config for gen {}", fabric.genid)), + 1, + "the generation this config is for, in {text}" + ); + + for address in addresses() { + let wanted = fabric + .peers + .iter() + .filter(|p| addresses()[p.address] == address) + .count(); + assert_eq!( + occurrences(&text, &format!(" peer {address}")), + wanted, + "bfd peer {address}, in {text}" + ); + } + + for (index, spec) in fabric.vrfs.iter().enumerate() { + let name = vrf_name(index); + assert_eq!( + occurrences(&text, &format!("\nvrf {name}\n")), + 1, + "vrf {name} declared once, in {text}" + ); + assert_eq!( + occurrences(&text, &format!(" vni {}", vni_for(index))), + 1, + "vni of {name}, in {text}" + ); + assert_eq!( + occurrences(&text, &format!("router ospf vrf {name}")), + usize::from(spec.ospf.is_some()), + "ospf instance of {name} iff it has one, in {text}" + ); + assert_eq!( + occurrences(&text, &format!(" ospf router-id {}", router_id(index))), + usize::from(spec.ospf.is_some()), + "router id of {name} iff it has ospf, in {text}" + ); + } + }); + } + + /// A vrf renders its own name and vni only when it should. + /// + /// The default vrf is the interesting case: its configuration belongs at the top level of the + /// FRR config, so it must *not* be wrapped in `vrf ` / `exit-vrf`. Wrapping it would put + /// the underlay's static routes and vni into a vrf that FRR does not have. + #[test] + fn a_vrf_renders_its_wrapper_only_when_it_is_not_the_default() { + bolero::check!() + .with_generator(bolero::produce::<(bool, bool)>()) + .cloned() + .for_each(|(default, has_vni): (bool, bool)| { + let name = if default { "default" } else { "VPC-1" }; + let vni = has_vni.then(|| vni_for(0)); + let text = VrfConfig::new(name, vni, default).render(&()).to_string(); + + let wrapped = usize::from(!default); + assert_eq!( + occurrences(&text, &format!("\nvrf {name}\n")), + wrapped, + "a vrf declaration iff not the default, in {text}" + ); + assert_eq!( + occurrences(&text, "exit-vrf"), + wrapped, + "an exit-vrf iff not the default, in {text}" + ); + assert_eq!( + occurrences(&text, " vni "), + usize::from(has_vni), + "a vni iff it has one, in {text}" + ); + }); + } + + /// Rendering the same config twice gives the same text. + /// + /// Worth its own property because the output is handed to `frr-reload.py`, which diffs it against + /// what FRR is running. A rendering that varied -- an unordered table iterated, say -- would look + /// like a configuration change on every pass and reload FRR for nothing. + #[test] + fn rendering_is_deterministic() { + bolero::check!() + .with_generator(Fabrics) + .cloned() + .for_each(|fabric: Fabric| { + let once = internal_config(&fabric).render(&fabric.genid).to_string(); + let twice = internal_config(&fabric).render(&fabric.genid).to_string(); + assert_eq!(once, twice, "rendering is not deterministic"); + }); + } +} diff --git a/routing/src/interfaces/iftablerw.rs b/routing/src/interfaces/iftablerw.rs index 29f231098e..df2bf78ea0 100644 --- a/routing/src/interfaces/iftablerw.rs +++ b/routing/src/interfaces/iftablerw.rs @@ -80,10 +80,6 @@ impl IfTableWriter { (IfTableWriter(w), IfTableReader(r)) } #[must_use] - pub fn as_reader(&self) -> IfTableReader { - IfTableReader::new(self.0.clone()) - } - #[must_use] pub fn enter(&self) -> Option> { self.0.enter() } @@ -217,3 +213,465 @@ impl IfTableReaderFactory { #[allow(unsafe_code)] unsafe impl Send for IfTableWriter {} + +/// Model-based properties over the interface table. +/// +/// The table is a map, but two things about it are not: an interface's *attachment* names a vrf, +/// which lives in a different structure and can be removed underneath it; and a reconfiguration has +/// to leave the runtime state -- addresses, attachment, operational state -- alone, since none of it +/// comes from the configuration that is being replaced. +#[cfg(test)] +mod iftable_properties { + use super::*; + use crate::fib::fibtable::FibTableWriter; + use crate::interfaces::interface::{Attachment, IfType}; + use crate::rib::vrf::{RouterVrfConfig, Vrf}; + use bolero::{Driver, ValueGenerator}; + use net::interface::address::IfAddr; + use std::collections::{BTreeMap, BTreeSet}; + use std::net::IpAddr; + use std::ops::Bound::Included; + use std::str::FromStr; + + const NUM_IFACES: u8 = 3; + const NUM_VRFS: u8 = 2; + const NUM_ADDRESSES: u8 = 2; + const NUM_STATES: u8 = 3; + const MAX_CHANGES: u8 = 12; + + /// Interface indices. `InterfaceIndex` is non-zero, so these start at one. + fn ifindexes() -> Vec { + (1..=u32::from(NUM_IFACES)) + .map(|i| InterfaceIndex::try_new(i).unwrap_or_else(|_| unreachable!())) + .collect() + } + + /// Non-default vrf ids. The default vrf is 0 and `VrfTable::new` makes it; it is left out so + /// that removing a vrf is always allowed. + fn vrf_ids() -> Vec { + (1..=u32::from(NUM_VRFS)).collect() + } + + fn addresses() -> Vec { + ["10.0.0.1", "10.0.0.2"] + .iter() + .map(|a| { + IfAddr::new(IpAddr::from_str(a).unwrap_or_else(|_| unreachable!()), 24) + .unwrap_or_else(|_| unreachable!()) + }) + .collect() + } + + fn states() -> Vec { + vec![IfState::Unknown, IfState::Down, IfState::Up] + } + + /// One change, over indices into the pools above. + #[derive(Debug, Clone)] + enum Change { + /// Add an interface. `renamed` picks between two names so that a modification is visible. + AddInterface { + iface: usize, + renamed: bool, + }, + ModInterface { + iface: usize, + renamed: bool, + }, + DelInterface { + iface: usize, + }, + AddAddress { + iface: usize, + address: usize, + }, + DelAddress { + iface: usize, + address: usize, + }, + SetOperState { + iface: usize, + state: usize, + }, + SetAdminState { + iface: usize, + state: usize, + }, + AttachToVrf { + iface: usize, + vrf: usize, + }, + Detach { + iface: usize, + }, + DetachVrf { + vrf: usize, + }, + AddVrf { + vrf: usize, + }, + RemoveVrf { + vrf: usize, + }, + } + + /// Draws sequences of [`Change`]s. + #[derive(Debug, Clone, Copy, Default)] + struct ChangeSequences; + + fn index(driver: &mut D, count: u8) -> Option { + driver + .gen_u8(Included(&0), Included(&(count - 1))) + .map(usize::from) + } + + impl ValueGenerator for ChangeSequences { + type Output = Vec; + + fn generate(&self, driver: &mut D) -> Option> { + let len = driver.gen_u8(Included(&0), Included(&MAX_CHANGES))?; + let mut out = Vec::with_capacity(usize::from(len)); + for _ in 0..len { + let iface = index(driver, NUM_IFACES)?; + let change = match driver.gen_u8(Included(&0), Included(&11))? { + 0 => Change::AddInterface { + iface, + renamed: driver.produce::()?, + }, + 1 => Change::ModInterface { + iface, + renamed: driver.produce::()?, + }, + 2 => Change::DelInterface { iface }, + 3 => Change::AddAddress { + iface, + address: index(driver, NUM_ADDRESSES)?, + }, + 4 => Change::DelAddress { + iface, + address: index(driver, NUM_ADDRESSES)?, + }, + 5 => Change::SetOperState { + iface, + state: index(driver, NUM_STATES)?, + }, + 6 => Change::SetAdminState { + iface, + state: index(driver, NUM_STATES)?, + }, + 7 => Change::AttachToVrf { + iface, + vrf: index(driver, NUM_VRFS)?, + }, + 8 => Change::Detach { iface }, + 9 => Change::DetachVrf { + vrf: index(driver, NUM_VRFS)?, + }, + 10 => Change::AddVrf { + vrf: index(driver, NUM_VRFS)?, + }, + _ => Change::RemoveVrf { + vrf: index(driver, NUM_VRFS)?, + }, + }; + out.push(change); + } + Some(out) + } + } + + fn name_of(iface: usize, renamed: bool) -> String { + if renamed { + format!("eth{iface}-renamed") + } else { + format!("eth{iface}") + } + } + + fn config_for(iface: usize, renamed: bool) -> RouterInterfaceConfig { + let mut config = RouterInterfaceConfig::new(&name_of(iface, renamed), ifindexes()[iface]); + config.set_iftype(IfType::Unknown); + config + } + + /// What the model believes about one interface. + #[derive(Debug, Clone, PartialEq)] + struct IfaceState { + name: String, + admin: IfState, + oper: IfState, + /// The vrf it is attached to, by index into [`vrf_ids`]. + attached: Option, + addresses: BTreeSet, + } + + #[derive(Debug, Clone)] + struct Model { + interfaces: BTreeMap, + vrfs: BTreeSet, + } + + impl Model { + fn new() -> Self { + Self { + interfaces: BTreeMap::new(), + vrfs: BTreeSet::new(), + } + } + } + + /// The table, the vrfs it attaches to, and the reader that sees both. + struct World { + iftw: IfTableWriter, + iftr: IfTableReader, + vrftable: VrfTable, + } + + fn world() -> World { + let (fibtw, _fibtr) = FibTableWriter::new(); + let (iftw, iftr) = IfTableWriter::new(); + World { + iftw, + iftr, + vrftable: VrfTable::new(fibtw), + } + } + + /// Add an interface, which is refused if one with that index is already there. + fn apply_add(world: &mut World, model: &mut Model, iface: usize, renamed: bool) { + let result = world.iftw.add_interface(config_for(iface, renamed)); + if model.interfaces.contains_key(&iface) { + assert!(result.is_err(), "a duplicate interface was accepted"); + return; + } + assert!(result.is_ok(), "a new interface was refused: {result:?}"); + model.interfaces.insert( + iface, + IfaceState { + name: name_of(iface, renamed), + admin: IfState::Up, + oper: IfState::Unknown, + attached: None, + addresses: BTreeSet::new(), + }, + ); + } + + /// Reconfigure an interface. + /// + /// The configuration is replaced; the runtime state is not. Addresses, the vrf attachment and + /// the operational state are all learned out of band, and a reconfiguration knows nothing about + /// any of them. + fn apply_mod(world: &mut World, model: &mut Model, iface: usize, renamed: bool) { + let result = world.iftw.mod_interface(config_for(iface, renamed)); + let Some(state) = model.interfaces.get_mut(&iface) else { + assert!(result.is_err(), "an unknown interface was modified"); + return; + }; + assert!(result.is_ok(), "a known interface was refused: {result:?}"); + state.name = name_of(iface, renamed); + state.admin = IfState::Up; + } + + /// Attach an interface to a vrf. Both halves have to be there: the interface, and a vrf with a + /// fib whose id can be named. + fn apply_attach(world: &mut World, model: &mut Model, iface: usize, vrf: usize) { + let result = + world + .iftw + .attach_interface_to_vrf(ifindexes()[iface], vrf_ids()[vrf], &world.vrftable); + let attachable = model.interfaces.contains_key(&iface) && model.vrfs.contains(&vrf); + assert_eq!(result.is_ok(), attachable, "attaching {iface} to {vrf}"); + if attachable { + model + .interfaces + .get_mut(&iface) + .unwrap_or_else(|| unreachable!()) + .attached = Some(vrf); + } + } + + /// Detach every interface attached to `vrf`. + fn detach_all_from(model: &mut Model, vrf: usize) { + for state in model.interfaces.values_mut() { + if state.attached == Some(vrf) { + state.attached = None; + } + } + } + + /// Remove a vrf, which detaches the interfaces that were attached to it. + fn apply_remove_vrf(world: &mut World, model: &mut Model, vrf: usize) { + let result = world.vrftable.remove_vrf(vrf_ids()[vrf], &mut world.iftw); + assert_eq!( + result.is_ok(), + model.vrfs.contains(&vrf), + "removing vrf {vrf}" + ); + if model.vrfs.remove(&vrf) { + detach_all_from(model, vrf); + } + } + + fn apply(world: &mut World, model: &mut Model, change: &Change) { + let ifaces = ifindexes(); + let vrfs = vrf_ids(); + match change { + Change::AddInterface { iface, renamed } => apply_add(world, model, *iface, *renamed), + Change::ModInterface { iface, renamed } => apply_mod(world, model, *iface, *renamed), + Change::AttachToVrf { iface, vrf } => apply_attach(world, model, *iface, *vrf), + Change::RemoveVrf { vrf } => apply_remove_vrf(world, model, *vrf), + Change::DelInterface { iface } => { + world.iftw.del_interface(ifaces[*iface]); + model.interfaces.remove(iface); + } + Change::AddAddress { iface, address } => { + world + .iftw + .add_ip_address(ifaces[*iface], addresses()[*address]); + // an address for an interface we do not have is dropped, and the caller is not + // told: the error is logged inside `absorb_first` and goes no further + if let Some(state) = model.interfaces.get_mut(iface) { + state.addresses.insert(*address); + } + } + Change::DelAddress { iface, address } => { + world + .iftw + .del_ip_address(ifaces[*iface], addresses()[*address]); + if let Some(state) = model.interfaces.get_mut(iface) { + state.addresses.remove(address); + } + } + Change::SetOperState { iface, state } => { + world + .iftw + .set_iface_oper_state(ifaces[*iface], states()[*state]); + if let Some(entry) = model.interfaces.get_mut(iface) { + entry.oper = states()[*state]; + } + } + Change::SetAdminState { iface, state } => { + world + .iftw + .set_iface_admin_state(ifaces[*iface], states()[*state]); + if let Some(entry) = model.interfaces.get_mut(iface) { + entry.admin = states()[*state]; + } + } + Change::Detach { iface } => { + world.iftw.detach_interface(ifaces[*iface]); + if let Some(state) = model.interfaces.get_mut(iface) { + state.attached = None; + } + } + Change::DetachVrf { vrf } => { + world.iftw.detach_interfaces_from_vrf(vrfs[*vrf]); + detach_all_from(model, *vrf); + } + Change::AddVrf { vrf } => { + let config = RouterVrfConfig::new(vrfs[*vrf], &format!("vrf{vrf}")); + let result = world.vrftable.add_vrf(&config); + assert_eq!( + result.is_ok(), + !model.vrfs.contains(vrf), + "adding vrf {vrf}" + ); + model.vrfs.insert(*vrf); + } + } + } + + fn check(world: &World, model: &Model, at: &str) { + let ifaces = ifindexes(); + let vrfs = vrf_ids(); + let addrs = addresses(); + + for view in [ + world.iftw.enter().unwrap_or_else(|| unreachable!()), + world.iftr.enter().unwrap_or_else(|| unreachable!()), + ] { + assert_eq!(view.len(), model.interfaces.len(), "interface count {at}"); + + for (index, ifindex) in ifaces.iter().enumerate() { + let Some(iface) = view.get_interface(*ifindex) else { + assert!( + !model.interfaces.contains_key(&index), + "interface {index} missing {at}" + ); + continue; + }; + let want = model + .interfaces + .get(&index) + .unwrap_or_else(|| panic!("interface {index} unexpected {at}")); + + assert_eq!(iface.ifindex, *ifindex, "filed under the wrong key {at}"); + assert_eq!(iface.name, want.name, "name of {index} {at}"); + assert_eq!(iface.admin_state, want.admin, "admin state of {index} {at}"); + assert_eq!(iface.oper_state, want.oper, "oper state of {index} {at}"); + + let held: BTreeSet = (0..addrs.len()) + .filter(|i| iface.addresses.contains(&addrs[*i])) + .collect(); + assert_eq!(held, want.addresses, "addresses of {index} {at}"); + assert_eq!( + iface.addresses.len(), + want.addresses.len(), + "stray addresses on {index} {at}" + ); + + match (&iface.attachment, want.attached) { + (None, None) => (), + (Some(Attachment::Vrf(key)), Some(vrf)) => { + assert_eq!(*key, FibKey::Id(vrfs[vrf]), "attachment of {index} {at}"); + } + (got, want) => { + panic!("attachment of {index} is {got:?}, expected {want:?} {at}") + } + } + + // and the vrf it names still exists. Nothing in the types ties an attachment to the + // life of the vrf it points at; `VrfTable::remove_vrf` detaching them is the whole + // of it + if let Some(Attachment::Vrf(FibKey::Id(vrfid))) = &iface.attachment { + assert!( + world.vrftable.contains(*vrfid), + "interface {index} is attached to vrf {vrfid}, which is gone {at}" + ); + } + } + } + } + + /// The pools and the constants that index them agree. + #[test] + fn the_pools_are_the_size_the_generator_thinks() { + assert_eq!(ifindexes().len(), usize::from(NUM_IFACES)); + assert_eq!(vrf_ids().len(), usize::from(NUM_VRFS)); + assert_eq!(addresses().len(), usize::from(NUM_ADDRESSES)); + assert_eq!(states().len(), usize::from(NUM_STATES)); + // the default vrf is excluded, so every vrf in the pool can be removed + assert!(!vrf_ids().contains(&Vrf::DEFAULT_VRFID)); + assert_ne!(name_of(0, false), name_of(0, true)); + } + + /// After any sequence of changes, the interface table holds what the model says -- and no + /// interface is left attached to a vrf that has gone. + #[test] + fn an_interface_tables_state_and_attachments_stay_in_step() { + bolero::check!() + .with_generator(ChangeSequences) + .cloned() + .for_each(|changes: Vec| { + let mut world = world(); + let mut model = Model::new(); + + check(&world, &model, "on a fresh table"); + for (step, change) in changes.iter().enumerate() { + apply(&mut world, &mut model, change); + check(&world, &model, &format!("at step {step} of {changes:?}")); + } + }); + } +} diff --git a/routing/src/rib/nexthop.rs b/routing/src/rib/nexthop.rs index f4a52e0582..4196ffccc1 100644 --- a/routing/src/rib/nexthop.rs +++ b/routing/src/rib/nexthop.rs @@ -60,6 +60,12 @@ pub struct NhopKey { pub ifindex: Option, pub encap: Option, pub fwaction: FwAction, + /// The name of `ifindex`'s interface, for diagnostics only. + /// + /// Not populated when a next-hop is learned over the CPI, and it must not be: the name has to + /// be looked up against the interface table, which is populated out of band, so a key carrying + /// one would differ before and after we learn about the interface -- two next-hops where there + /// is one. See `RouteNhop::from_rpc_nhop`, and the same argument for a vxlan dmac above it. pub ifname: Option, } @@ -161,6 +167,20 @@ impl Hash for Nhop { } } +/// A next-hop's address in memory, used to tell next-hops apart in the visited sets that the +/// walks over the resolver graph keep. +/// +/// Address rather than key: keys are unique within one [`NhopStore`], but a next-hop may hold +/// resolvers that belong to another one, where an equal key would name a different object. A +/// resolution loop is a loop in the object graph, and next-hops live in `Rc`s, which do not move. +pub(crate) type NhopId = *const Nhop; + +/// The next-hops a walk over the resolver graph has already visited. +/// +/// A `Vec` rather than a set: resolver chains are a handful of next-hops long and fan out by two +/// or three, so a linear scan costs less than hashing. +pub(crate) type Visited = Vec; + impl Nhop { /// Create a new Nhop object from a key object fn from_key(key: &NhopKey) -> Self { @@ -184,23 +204,47 @@ impl Nhop { self } - /// Recursive method to check if a next-hop resolves via another, `checked`. - /// We use this method to avoid resolution loops that would happen in case of routing loops. - /// Resolution loops would cause us to stack overflow. This method is recursive, but - /// short-circuits in case of loop. The method takes the advantage that there cannot be two - /// next-hops with the same key. + /// This next-hop's identity for the visited sets of the walks over the resolver graph. + pub(crate) fn id(&self) -> NhopId { + std::ptr::from_ref(self) + } + + /// Tell if a next-hop resolves via another, `checked`: that is, whether `checked` is reachable + /// from `self` along resolver edges, `self` included. + /// + /// This is the guard against routing loops. `lazy_resolve` refuses an edge from `a` to `r` + /// exactly when `r.resolves_with(a)`, since such an edge would close a cycle, and a cycle in + /// the resolver graph would send the walks over it round for ever. + /// + /// The walk is total whether or not the graph already holds a cycle, because `visited` stops + /// it from entering any next-hop twice. That matters for two reasons: the guard should not + /// depend on the very invariant it exists to maintain, and a diamond-shaped graph is otherwise + /// re-walked once per path through it. fn resolves_with(&self, checked: &Nhop) -> bool { + self.resolves_with_rec(checked, &mut Visited::new()) + } + + fn resolves_with_rec(&self, checked: &Nhop, visited: &mut Visited) -> bool { // resolve to oneself is forbidden if self.key == checked.key { error!("Loop detected for next-hop {}!", self.key); return true; } + // a next-hop already visited leads nowhere new: either we are inside a cycle, or we + // reached it by another path and have already looked at everything beyond it. + // a next-hop already visited leads nowhere new: either we are inside a cycle, or we + // reached it by another path and have already looked at everything beyond it. + if visited.contains(&self.id()) { + return false; + } + visited.push(self.id()); + // resolvers should not refer back to the checked next-hop let resolvers = self.resolvers.borrow(); resolvers .iter() .filter_map(Weak::upgrade) - .any(|res| res.resolves_with(checked)) + .any(|res| res.resolves_with_rec(checked, visited)) } /// Tell if a next-hop requires resolution @@ -264,8 +308,15 @@ impl Nhop { } /// Auxiliary recursive method used by `Nhop::quick_resolve()`. + /// + /// `visited` guards against a resolution loop, as in `resolves_with` above. #[cfg(test)] - fn quick_resolve_rec(&self, result: &mut BTreeSet) { + fn quick_resolve_rec(&self, result: &mut BTreeSet, visited: &mut Visited) { + if visited.contains(&self.id()) { + return; + } + visited.push(self.id()); + let Ok(resolvers) = self.resolvers.try_borrow_mut() else { error!("Try-borrow-mut() failed on next-hop resolvers!"); return; @@ -297,7 +348,7 @@ impl Nhop { self.key.ifname.clone(), )); } else { - r.quick_resolve_rec(result); + r.quick_resolve_rec(result, visited); } } } @@ -311,7 +362,7 @@ impl Nhop { #[cfg(test)] pub fn quick_resolve(&self) -> BTreeSet { let mut out: BTreeSet = BTreeSet::new(); - self.quick_resolve_rec(&mut out); + self.quick_resolve_rec(&mut out, &mut Visited::new()); out } } @@ -1044,4 +1095,282 @@ mod tests { a.add_resolver(&checked); assert!(a.resolves_with(checked.as_ref())); } + + #[cfg_attr(not(emulated), traced_test)] + #[test] + /// Displaying a next-hop caught in a resolution loop terminates, and says which edge closes it. + /// + /// It used to walk the resolvers with an untracked `u8` depth, so a loop recursed until that + /// depth overflowed -- a panic raised from inside a `Display` impl that both the CLI and the + /// warning this module logs about resolution loops go through. + fn test_display_of_a_resolution_loop_terminates() { + let mut store = NhopStore::new(); + let a = store.add_nhop(&NhopKey::from_address("7.0.0.1")); + let b = store.add_nhop(&NhopKey::from_address("8.0.0.2")); + a.add_resolver(&b); + b.add_resolver(&a); + + let nhop = format!("{a}"); + assert!(nhop.contains("(LOOP)"), "loop not reported in {nhop}"); + + let whole_store = format!("{store}"); + assert!( + whole_store.contains("(LOOP)"), + "loop not reported in {whole_store}" + ); + } +} + +#[cfg(test)] +mod fibgroup_properties { + use super::*; + use crate::fib::fibobjects::FibEntry; + use bolero::{Driver, ValueGenerator}; + use std::ops::Bound::Included; + + const MAX_NODES: u8 = 6; + const MAX_RESOLVERS: u8 = 2; + + /// A next-hop graph, given as an adjacency list over node indices. + /// + /// **Cycles included**, self-loops among them. A routing loop is exactly a cycle here, and + /// both walks over the resolver graph have to survive one -- `resolves_with` because it is the + /// guard that keeps cycles out and so cannot presume its own success, and + /// `build_nhop_fibgroup_rec` because nothing in the types ties it to that guard. + #[derive(Debug, Clone)] + struct Graph { + /// `edges[i]` are the nodes that `i` resolves via, in the order they were wired. + edges: Vec>, + /// Whether node `i` knows an interface, and so needs no resolving. + grounded: Vec, + } + + impl Graph { + /// Which nodes are reachable from `start`, itself included. A plain closure over the + /// adjacency list, computed without asking any next-hop anything. + fn reachable_from(&self, start: usize) -> Vec { + let mut seen = vec![false; self.edges.len()]; + let mut stack = vec![start]; + while let Some(node) = stack.pop() { + if std::mem::replace(&mut seen[node], true) { + continue; + } + stack.extend_from_slice(&self.edges[node]); + } + seen + } + } + + /// Draws [`Graph`]s. + #[derive(Debug, Clone, Copy, Default)] + struct Graphs; + + impl ValueGenerator for Graphs { + type Output = Graph; + + fn generate(&self, driver: &mut D) -> Option { + let nodes = usize::from(driver.gen_u8(Included(&1), Included(&MAX_NODES))?); + let last = u8::try_from(nodes - 1).ok()?; + let mut edges = Vec::with_capacity(nodes); + let mut grounded = Vec::with_capacity(nodes); + for _ in 0..nodes { + let count = driver.gen_u8(Included(&0), Included(&MAX_RESOLVERS))?; + let mut resolvers = Vec::with_capacity(usize::from(count)); + for _ in 0..count { + resolvers.push(usize::from(driver.gen_u8(Included(&0), Included(&last))?)); + } + edges.push(resolvers); + grounded.push(driver.produce::()?); + } + Some(Graph { edges, grounded }) + } + } + + // Build the graph in an `NhopStore`, returning the next-hops by index. + fn realize(graph: &Graph) -> (NhopStore, Vec>) { + let mut store = NhopStore::new(); + let nodes: Vec> = (0..graph.edges.len()) + .map(|index| { + let raw = u8::try_from(index).unwrap_or_else(|_| unreachable!()); + let mut key = NhopKey::from_address(&format!("10.0.0.{}", raw + 1)); + if graph.grounded[index] { + key.ifindex = Some( + InterfaceIndex::try_new(u32::from(raw) + 1) + .unwrap_or_else(|_| unreachable!()), + ); + } + store.add_nhop(&key) + }) + .collect(); + + for (from, resolvers) in graph.edges.iter().enumerate() { + for to in resolvers { + nodes[from].add_resolver(&nodes[*to]); + } + } + (store, nodes) + } + + // The oracle: every *simple* root-to-leaf path, its next-hops' instructions concatenated, + // squashed, and kept if the forwarder could execute it. + // + // Enumerated over the generated adjacency list rather than by walking the recursion under + // test. "Simple" is the loop guard restated: a path that would revisit a node stops there and + // contributes nothing, because going round a loop is not forwarding. + fn expected( + graph: &Graph, + nodes: &[Rc], + from: usize, + path: &mut Vec, + prefix: &FibEntry, + out: &mut Vec, + ) { + if path.contains(&from) { + return; + } + path.push(from); + + let mut entry = prefix.clone(); + entry.extend_from_slice(&nodes[from].instructions.borrow()); + + if graph.edges[from].is_empty() { + // A next-hop with neither an interface nor a way to reach one contributes nothing. + if !nodes[from].must_be_resolved() { + entry.squash(); + if entry.is_valid() { + out.push(entry); + } + } + } else { + for to in &graph.edges[from] { + expected(graph, nodes, *to, path, &entry, out); + } + } + + path.pop(); + } + + /// A next-hop's fib group is one entry per usable resolution path, and never empty. + #[test] + fn a_fibgroup_is_the_usable_paths_through_the_graph() { + let rstore = RmacStore::new(); + bolero::check!() + .with_generator(Graphs) + .cloned() + .for_each(|graph: Graph| { + let (_store, nodes) = realize(&graph); + for node in &nodes { + node.build_nhop_instructions(&rstore); + } + + let mut want = Vec::new(); + expected( + &graph, + &nodes, + 0, + &mut Vec::new(), + &FibEntry::new(), + &mut want, + ); + if want.is_empty() { + // Nothing usable: the group carries a drop so packets are not misrouted. + want.push(FibEntry::drop_fibentry()); + } + + let got = nodes[0].build_nhop_fibgroup(); + assert_eq!(got.entries(), &want, "for {graph:?}"); + }); + } + + /// Every entry a fib group offers is one the forwarder can execute. + /// + /// `FibEntry::is_valid` is the written-down form of that, and `rib2fib` filters on it -- but + /// the drop injected when nothing is usable bypasses the filter, so it is worth asserting over + /// the group rather than trusting the one call site. + #[test] + fn every_entry_in_a_fibgroup_is_usable() { + let rstore = RmacStore::new(); + bolero::check!() + .with_generator(Graphs) + .cloned() + .for_each(|graph: Graph| { + let (_store, nodes) = realize(&graph); + for node in &nodes { + node.build_nhop_instructions(&rstore); + } + let group = nodes[0].build_nhop_fibgroup(); + assert!(!group.is_empty(), "for {graph:?}"); + for entry in group.iter() { + assert!(entry.is_valid(), "unusable entry {entry:?} for {graph:?}"); + } + }); + } + + /// A next-hop every one of whose paths loops back gets a drop, not an infinite walk. + /// + /// The general case falls out of the two properties above -- they only terminate because the + /// walk does -- but a routing loop is the failure this guard exists for, so it is worth one + /// case that says so in as many words. + #[test] + fn a_next_hop_in_a_resolution_loop_drops() { + let rstore = RmacStore::new(); + let mut store = NhopStore::new(); + + // 7.0.0.1 -> 8.0.0.2 -> 9.0.0.3 -> 7.0.0.1, and no way out to an interface. + let a = store.add_nhop(&NhopKey::from_address("7.0.0.1")); + let b = store.add_nhop(&NhopKey::from_address("8.0.0.2")); + let c = store.add_nhop(&NhopKey::from_address("9.0.0.3")); + a.add_resolver(&b); + b.add_resolver(&c); + c.add_resolver(&a); + store.rebuild_nhop_instructions(&rstore); + + let group = a.build_nhop_fibgroup(); + assert_eq!( + group.entries(), + &vec![FibEntry::drop_fibentry()], + "a packet caught in a routing loop must be dropped" + ); + } + + /// `resolves_with` answers reachability in the resolver graph. + /// + /// That is the whole of what the loop guard rests on: `lazy_resolve` refuses an edge from `a` + /// to `r` exactly when `r.resolves_with(a)`, which is to say when `a` is already reachable + /// from `r` and the edge would close a cycle. Checked against a closure computed over the + /// adjacency list, which asks no next-hop anything -- and, now that the graphs may contain + /// cycles, over graphs where `resolves_with` has to terminate on its own account. + #[test] + fn resolves_with_answers_reachability() { + bolero::check!() + .with_generator(Graphs) + .cloned() + .for_each(|graph: Graph| { + let (_store, nodes) = realize(&graph); + for (from, node) in nodes.iter().enumerate() { + let reachable = graph.reachable_from(from); + for (to, other) in nodes.iter().enumerate() { + assert_eq!( + node.resolves_with(other), + reachable[to], + "{from} -> {to}, for {graph:?}" + ); + } + } + }); + } + + /// A next-hop always resolves via itself, which is what makes the guard refuse a self-loop. + #[test] + fn a_next_hop_resolves_via_itself() { + bolero::check!() + .with_generator(Graphs) + .cloned() + .for_each(|graph: Graph| { + let (_store, nodes) = realize(&graph); + for node in &nodes { + assert!(node.resolves_with(node)); + } + }); + } } diff --git a/routing/src/rib/rib2fib.rs b/routing/src/rib/rib2fib.rs index 99ae73cd0f..738c921062 100644 --- a/routing/src/rib/rib2fib.rs +++ b/routing/src/rib/rib2fib.rs @@ -9,7 +9,7 @@ use tracing::{debug, trace, warn}; use crate::evpn::RmacStore; use crate::fib::fibobjects::{EgressObject, FibEntry, FibGroup, PktInstruction}; use crate::rib::encapsulation::{Encapsulation, VxlanEncapsulation}; -use crate::rib::nexthop::{FwAction, Nhop}; +use crate::rib::nexthop::{FwAction, Nhop, Visited}; use crate::rib::vrf::RouteOrigin; use std::rc::Weak; @@ -103,8 +103,44 @@ impl Nhop { ////////////////////////////////////////////////////////////////////// /// Recursive helper to build [`FibGroup`] for a next-hop. We accumulate /// a next-hop's packet instructions with those of its resolvers. + /// + /// `path` holds the next-hops between the root of the walk and this one. A next-hop that turns + /// up on its own resolution path closes a routing loop: following it would recurse until the + /// stack ran out, so we stop there and contribute nothing. If that leaves no usable path at + /// all, `build_nhop_fibgroup` injects a drop, which is what a packet caught in a routing loop + /// should meet anyway. + /// + /// This makes the walk safe on any graph rather than only on the acyclic ones that + /// `Nhop::resolves_with` lets `lazy_resolve` build. The two guards are deliberately + /// independent: nothing in the types ties this recursion to the one that used to be its only + /// protection, and a caller wiring resolvers by another route would lose it silently. ////////////////////////////////////////////////////////////////////// - fn build_nhop_fibgroup_rec(&self, fibgroup: &mut FibGroup, mut entry: FibEntry) { + fn build_nhop_fibgroup_rec( + &self, + fibgroup: &mut FibGroup, + entry: FibEntry, + path: &mut Visited, + ) { + if path.contains(&self.id()) { + warn!("Resolution loop at next-hop {self}: will not use this path"); + return; + } + path.push(self.id()); + self.build_nhop_fibgroup_visit(fibgroup, entry, path); + path.pop(); + } + + ////////////////////////////////////////////////////////////////////// + /// The body of [`Nhop::build_nhop_fibgroup_rec`], for a next-hop known not to be on its own + /// resolution path already. Split out so that the push and the pop of `path` sit next to each + /// other and no early return here can leave the path unbalanced. + ////////////////////////////////////////////////////////////////////// + fn build_nhop_fibgroup_visit( + &self, + fibgroup: &mut FibGroup, + mut entry: FibEntry, + path: &mut Visited, + ) { // add the instructions for a next-hop to the entry let instructions = self.instructions.borrow().clone(); entry.extend_from_slice(&instructions); @@ -136,7 +172,7 @@ impl Nhop { } } else { for resolver in resolvers.iter().filter_map(Weak::upgrade) { - resolver.build_nhop_fibgroup_rec(fibgroup, entry.clone()); + resolver.build_nhop_fibgroup_rec(fibgroup, entry.clone(), path); } } } @@ -149,7 +185,7 @@ impl Nhop { ////////////////////////////////////////////////////////////////////// pub(crate) fn build_nhop_fibgroup(&self) -> FibGroup { let mut fibgroup = FibGroup::new(); - self.build_nhop_fibgroup_rec(&mut fibgroup, FibEntry::new()); + self.build_nhop_fibgroup_rec(&mut fibgroup, FibEntry::new(), &mut Visited::new()); if fibgroup.is_empty() { warn!("Next-hop {self} has empty fibgroup: will add DROP FibEntry"); fibgroup.add(FibEntry::drop_fibentry()); diff --git a/routing/src/rib/vrf.rs b/routing/src/rib/vrf.rs index 6739a9d915..8abb46186d 100644 --- a/routing/src/rib/vrf.rs +++ b/routing/src/rib/vrf.rs @@ -4,10 +4,11 @@ //! VRF module to store Ipv4 and Ipv6 routing tables use bitflags::bitflags; +use std::borrow::Cow; use std::hash::Hash; use std::net::IpAddr; use std::rc::{Rc, Weak}; -use tracing::debug; +use tracing::{debug, warn}; #[cfg(test)] use common::cliprovider::Frame; @@ -113,7 +114,7 @@ impl ShimNhop { } } -#[derive(Copy, Clone, PartialEq)] +#[derive(Copy, Clone, Debug, PartialEq)] #[allow(unused)] pub enum VrfStatus { Active, @@ -392,6 +393,27 @@ impl Vrf { } } + ///////////////////////////////////////////////////////////////////////// + /// The next-hops to install for a route, substituting an explicit drop if there are none. + /// + /// A route with no next-hops cannot forward anything, and leaving it out of the table + /// altogether is worse than installing it as a drop: the rib would keep it while the fib + /// declined it -- `FibWriter::add_fibroute` refuses a route with no next-hop keys -- so a + /// packet matching it would fall through to a shorter prefix in the fib and be forwarded + /// somewhere else rather than dropped. Resolving via a default is how a routing loop starts. + /// + /// `add_route_rpc` used to do this, for the one path that could produce an empty list. Doing + /// it here means no caller can skip it. + ///////////////////////////////////////////////////////////////////////// + fn nhops_or_drop<'a>(prefix: &Prefix, nhops: &'a [RouteNhop]) -> Cow<'a, [RouteNhop]> { + if nhops.is_empty() { + warn!("Route to {prefix} has no next-hop: will install it with action drop"); + Cow::Owned(vec![RouteNhop::default()]) + } else { + Cow::Borrowed(nhops) + } + } + ///////////////////////////////////////////////////////////////////////// // Route Insertion ///////////////////////////////////////////////////////////////////////// @@ -403,7 +425,7 @@ impl Vrf { vrf0: Option<&Vrf>, ) { // register next-hops and let the route keep references to the shared nexthops created/found - route.s_nhops = self.register_shared_nhops(nhops); + route.s_nhops = self.register_shared_nhops(&Self::nhops_or_drop(prefix, nhops)); // resolve the new route next-hops. This is only for testing. In prod code, // this method is only used for drop routes which require no resolution. @@ -464,7 +486,7 @@ impl Vrf { rstore: &RmacStore, ) { // register next-hops and let the route keep references to the shared nexthops created/found - route.s_nhops = self.register_shared_nhops(nhops); + route.s_nhops = self.register_shared_nhops(&Self::nhops_or_drop(prefix, nhops)); let rvrf = vrf0.unwrap_or(self); @@ -1140,3 +1162,401 @@ pub mod tests { } } + +/// Model-based properties over a [`Vrf`]'s route table. +/// +/// Two structures move together on every route change: the route tries, and the [`NhopStore`] whose +/// entries the routes hold `Rc`s into. A route that comes or goes has to leave the store holding +/// exactly the next-hops the remaining routes name -- one too many is a leak that keeps a stale +/// fib group alive, one too few and a route names a next-hop nothing will resolve. A third, the +/// vrf's `Fib`, is written through on the same calls and has to end up describing the same +/// prefixes. +#[cfg(test)] +mod vrf_properties { + use super::*; + use crate::fib::fibtype::{FibKey, FibWriter}; + use crate::rib::nexthop::NhopKey; + use bolero::{Driver, ValueGenerator}; + use std::collections::{BTreeMap, BTreeSet}; + use std::ops::Bound::Included; + use std::str::FromStr; + + const NUM_PREFIXES: u8 = 8; + const NUM_NHOPS: u8 = 3; + const MAX_CHANGES: u8 = 10; + const MAX_NHOPS_PER_ROUTE: u8 = 3; + const NUM_STATUSES: u8 = 3; + + /// `0.0.0.0/0` and `::/0`, at these indices in [`prefixes`]. A vrf always carries a route for + /// both: `Vrf::lpm` has no answer otherwise and says so with an `unreachable!()`, and + /// `check_deletion` reaches for them by name. + const ROOT_V4: usize = 0; + const ROOT_V6: usize = 5; + + /// The prefixes a generated vrf may carry routes for. Nested, and both roots. + fn prefixes() -> Vec { + [ + "0.0.0.0/0", + "10.0.0.0/8", + "10.1.0.0/16", + "10.1.2.0/24", + "10.1.2.3/32", + "::/0", + "2001:db8::/32", + "2001:db8:1::/48", + ] + .iter() + .map(|p| Prefix::from_str(p).unwrap_or_else(|_| unreachable!())) + .collect() + } + + /// Addresses to look up: inside each level of the nesting, and outside all of them. + fn probes() -> Vec { + [ + "9.9.9.9", + "10.9.9.9", + "10.1.9.9", + "10.1.2.9", + "10.1.2.3", + "2000::1", + "2001:db8::1", + "2001:db8:1::1", + ] + .iter() + .map(|a| IpAddr::from_str(a).unwrap_or_else(|_| unreachable!())) + .collect() + } + + fn statuses() -> Vec { + vec![VrfStatus::Active, VrfStatus::Deleting, VrfStatus::Deleted] + } + + /// The next-hops a generated route may use. Small, so that routes share them: sharing is what + /// makes the store's reference counting do any work. + fn nhops() -> Vec { + vec![ + tests::build_test_nhop(Some("10.0.0.1"), Some(1), 0, None), + tests::build_test_nhop(Some("10.0.0.2"), None, 0, None), + tests::build_test_nhop(None, Some(3), 0, None), + ] + } + + /// One change, over indices into the pools above. + #[derive(Debug, Clone)] + enum Change { + /// A route. `nhops` may be empty: `add_route_complete` takes whatever the control plane + /// hands it, and a route with nowhere to go is the interesting end of that. + AddRoute { + prefix: usize, + nhops: Vec, + }, + DelRoute { + prefix: usize, + }, + SetStale { + value: bool, + }, + RemoveStale, + /// Move the vrf's status. Worth generating because `del_route` calls `check_deletion`, + /// whose whole body sits behind `status == Deleting` -- so without this, the transition + /// that `VrfTable::remove_deleting_vrfs` depends on never runs at all. + SetStatus { + status: usize, + }, + } + + /// Draws sequences of [`Change`]s. + #[derive(Debug, Clone, Copy, Default)] + struct ChangeSequences; + + fn index(driver: &mut D, count: u8) -> Option { + driver + .gen_u8(Included(&0), Included(&(count - 1))) + .map(usize::from) + } + + impl ValueGenerator for ChangeSequences { + type Output = Vec; + + fn generate(&self, driver: &mut D) -> Option> { + let len = driver.gen_u8(Included(&0), Included(&MAX_CHANGES))?; + let mut out = Vec::with_capacity(usize::from(len)); + for _ in 0..len { + let change = match driver.gen_u8(Included(&0), Included(&4))? { + 0 => { + let prefix = index(driver, NUM_PREFIXES)?; + let count = driver.gen_u8(Included(&0), Included(&MAX_NHOPS_PER_ROUTE))?; + let mut nhops = Vec::with_capacity(usize::from(count)); + for _ in 0..count { + nhops.push(index(driver, NUM_NHOPS)?); + } + Change::AddRoute { prefix, nhops } + } + 1 => Change::DelRoute { + prefix: index(driver, NUM_PREFIXES)?, + }, + 2 => Change::SetStale { + value: driver.produce::()?, + }, + 3 => Change::RemoveStale, + _ => Change::SetStatus { + status: index(driver, NUM_STATUSES)?, + }, + }; + out.push(change); + } + Some(out) + } + } + + /// Which prefixes carry routes, and for each the next-hops it names and whether it is stale. + /// + /// The two root routes are the preset drop routes a vrf is born with; they are never removed, + /// only reset, and `set_stale` skips them. + /// What the model believes about one route. + #[derive(Debug, Clone, PartialEq)] + struct RouteState { + /// The next-hop keys it names, in order. Keys rather than pool indices, because the preset + /// drop route names one the generator cannot ask for. + nhops: Vec, + stale: bool, + /// Whether this is still the drop route a vrf installs for each root on creation. + /// + /// Distinct from "names the drop next-hop": a generated route for a root prefix with no + /// next-hops ends up naming it too, but carries a real origin, distance and metric, so + /// `Route::is_preset_drop_route` says no -- and `check_deletion` asks that question. + preset: bool, + } + + #[derive(Debug, Clone)] + struct Model { + routes: BTreeMap, + status: VrfStatus, + } + + impl Model { + /// The route a vrf installs for each root on creation: drop, so that a lookup always has an + /// answer. + fn preset() -> RouteState { + RouteState { + nhops: vec![NhopKey::with_drop()], + stale: false, + preset: true, + } + } + + fn new() -> Self { + Self { + routes: BTreeMap::from([(ROOT_V4, Self::preset()), (ROOT_V6, Self::preset())]), + status: VrfStatus::Active, + } + } + + fn is_root(prefix: usize) -> bool { + prefix == ROOT_V4 || prefix == ROOT_V6 + } + + /// Every next-hop key the routes name -- the set the store should hold, no more and no less. + fn referenced(&self) -> BTreeSet { + self.routes + .values() + .flat_map(|route| route.nhops.iter().cloned()) + .collect() + } + + /// `Vrf::check_deletion`, which `del_route` calls: a vrf on its way out becomes deletable + /// once the only routes left are the two preset drop ones. + fn check_deletion(&mut self) { + let only_presets = self.routes.len() == 2 + && [ROOT_V4, ROOT_V6] + .iter() + .all(|root| self.routes.get(root).is_some_and(|route| route.preset)); + if self.status == VrfStatus::Deleting && only_presets { + self.status = VrfStatus::Deleted; + } + } + + /// The longest prefix carrying a route that covers `addr`. + fn lpm(&self, addr: &IpAddr, pool: &[Prefix]) -> Option { + self.routes + .keys() + .copied() + .filter(|i| pool[*i].covers_addr(addr)) + .max_by_key(|i| pool[*i].length()) + } + + fn apply(&mut self, change: &Change, pool: &[RouteNhop]) { + match change { + Change::AddRoute { prefix, nhops } => { + // a route with no next-hops is installed as a drop rather than left out + let nhops = if nhops.is_empty() { + vec![NhopKey::with_drop()] + } else { + nhops.iter().map(|i| pool[*i].key.clone()).collect() + }; + self.routes.insert( + *prefix, + RouteState { + nhops, + stale: false, + preset: false, + }, + ); + } + Change::DelRoute { prefix } => { + if Self::is_root(*prefix) { + // a root route is reset to the preset drop route, not removed + self.routes.insert(*prefix, Self::preset()); + } else { + self.routes.remove(prefix); + } + self.check_deletion(); + } + Change::SetStale { value } => { + // `Vrf::set_stale` skips a route whose prefix is a root, and separately one + // that is still a preset drop route. Generated routes are never the latter, so + // here the root test is the whole of it. + for (prefix, route) in &mut self.routes { + if !Self::is_root(*prefix) { + route.stale = *value; + } + } + } + Change::RemoveStale => { + let stale: Vec = self + .routes + .iter() + .filter_map(|(prefix, route)| route.stale.then_some(*prefix)) + .collect(); + for prefix in stale { + self.apply(&Change::DelRoute { prefix }, pool); + } + } + Change::SetStatus { status } => { + // the vrf under test is not the default one, so the move always takes + self.status = statuses()[*status]; + } + } + } + } + + fn apply_to_vrf(vrf: &mut Vrf, rstore: &RmacStore, change: &Change, pool: &[RouteNhop]) { + let prefixes = prefixes(); + match change { + Change::AddRoute { prefix, nhops } => { + let route = tests::build_test_route(RouteOrigin::Bgp, 20, 100); + let nhops: Vec = nhops.iter().map(|i| pool[*i].clone()).collect(); + vrf.add_route_complete(&prefixes[*prefix], route, &nhops, None, rstore); + } + Change::DelRoute { prefix } => vrf.del_route(prefixes[*prefix], None, rstore), + Change::SetStale { value } => vrf.set_stale(*value), + Change::RemoveStale => vrf.remove_stale_routes(None, rstore), + Change::SetStatus { status } => vrf.set_status(statuses()[*status]), + } + } + + /// Check the vrf, its next-hop store and its fib against the one model. + fn check(vrf: &Vrf, model: &Model, at: &str) { + let prefixes = prefixes(); + let probes = probes(); + + // 1. the route tries hold exactly the prefixes the model says + let held: BTreeSet = (0..prefixes.len()) + .filter(|i| vrf.get_route(prefixes[*i]).is_some()) + .collect(); + let want: BTreeSet = model.routes.keys().copied().collect(); + assert_eq!(held, want, "route set {at}"); + assert_eq!( + vrf.len_v4() + vrf.len_v6(), + model.routes.len(), + "route count {at}" + ); + + // 2. each route names the next-hops the model says, in order + for (prefix, want) in &model.routes { + let route = vrf + .get_route(prefixes[*prefix]) + .unwrap_or_else(|| panic!("no route for {prefix} {at}")); + let got: Vec = route.s_nhops.iter().map(|s| s.rc.key.clone()).collect(); + assert_eq!(got, want.nhops, "next-hops of {prefix} {at}"); + assert_eq!(route.is_stale(), want.stale, "stale flag of {prefix} {at}"); + assert_eq!( + route.is_preset_drop_route(), + want.preset, + "preset-drop-route of {prefix} {at}" + ); + } + + // the status, and with it `check_deletion`: a vrf on its way out becomes deletable exactly + // when the only routes left are the two preset drop ones + assert_eq!(vrf.status, model.status, "status {at}"); + + // 3. the next-hop store holds exactly the next-hops the routes name. One too many is a + // leak that keeps a stale fib group alive; one too few and a route names something + // nothing will resolve + let stored: BTreeSet = vrf.nhstore.iter().map(|rc| rc.key.clone()).collect(); + assert_eq!(stored, model.referenced(), "next-hop store {at}"); + + // 4. lpm resolves for every address, and lands where the model says + for probe in &probes { + let want = model + .lpm(probe, &prefixes) + .unwrap_or_else(|| panic!("model has no route for {probe} {at}")); + let (hit, _) = vrf.lpm(*probe); + assert_eq!(hit, prefixes[want], "lpm for {probe} {at}"); + } + + // 5. the fib describes the same prefixes as the vrf. Otherwise a packet matching a route + // the fib never heard about falls through to a shorter prefix and is forwarded + // somewhere else rather than dropped + let fibw = vrf.fibw.as_ref().unwrap_or_else(|| unreachable!()); + let fib = fibw.enter().unwrap_or_else(|| unreachable!()); + let mut want_v4 = BTreeSet::new(); + let mut want_v6 = BTreeSet::new(); + for prefix in model.routes.keys() { + match prefixes[*prefix] { + Prefix::IPV4(p) => want_v4.insert(p), + Prefix::IPV6(p) => want_v6.insert(p), + }; + } + let fib_v4: BTreeSet = fib.iter_v4().map(|(prefix, _)| prefix).collect(); + let fib_v6: BTreeSet = fib.iter_v6().map(|(prefix, _)| prefix).collect(); + assert_eq!(fib_v4, want_v4, "fib ipv4 prefixes {at}"); + assert_eq!(fib_v6, want_v6, "fib ipv6 prefixes {at}"); + } + + /// The pools and the constants that index them agree. + #[test] + fn the_pools_are_the_size_the_generator_thinks() { + assert_eq!(prefixes().len(), usize::from(NUM_PREFIXES)); + assert_eq!(nhops().len(), usize::from(NUM_NHOPS)); + assert_eq!(statuses().len(), usize::from(NUM_STATUSES)); + assert_eq!(prefixes()[ROOT_V4], Prefix::root_v4()); + assert_eq!(prefixes()[ROOT_V6], Prefix::root_v6()); + } + + /// After any sequence of route changes, the vrf, its next-hop store and its fib agree. + #[test] + fn a_vrfs_routes_and_next_hops_stay_in_step() { + let pool = nhops(); + let rstore = RmacStore::new(); + + bolero::check!() + .with_generator(ChangeSequences) + .cloned() + .for_each(|changes: Vec| { + let config = RouterVrfConfig::new(1, "test"); + let mut vrf = Vrf::new(&config); + let (fibw, _fibr) = FibWriter::new(FibKey::from_vrfid(1)); + vrf.set_fibw(fibw); + let mut model = Model::new(); + + check(&vrf, &model, "on a fresh vrf"); + for (step, change) in changes.iter().enumerate() { + apply_to_vrf(&mut vrf, &rstore, change, &pool); + model.apply(change, &pool); + check(&vrf, &model, &format!("at step {step} of {changes:?}")); + } + }); + } +} diff --git a/routing/src/rib/vrftable.rs b/routing/src/rib/vrftable.rs index d3035239ec..b1110fdecf 100644 --- a/routing/src/rib/vrftable.rs +++ b/routing/src/rib/vrftable.rs @@ -176,6 +176,17 @@ impl VrfTable { vrfid: VrfId, iftablew: &mut IfTableWriter, ) -> Result<(), RouterError> { + // The default vrf is not removable. `Vrf::set_status` says as much and keeps the default + // vrf `Active` so that the sweeps below never pick it up, but nothing stopped a caller + // naming it here -- and `get_default_vrf` treats the default vrf's existence as given, + // with an `unreachable!()` rather than an error. + if vrfid == Vrf::DEFAULT_VRFID { + error!("Refusing to remove the default vrf"); + return Err(RouterError::Internal( + "Bug: the default vrf cannot be removed", + )); + } + // remove the vrf from the vrf table debug!("Removing VRF {vrfid}..."); let Some(mut vrf) = self.by_id.remove(&vrfid) else { @@ -189,7 +200,7 @@ impl VrfTable { // delete the corresponding fib if let Some(fibw) = vrf.fibw.take() { debug!("Deleting Fib for vrf {vrfid} from the FibTable"); - self.fibtablew.del_fib(vrfid, vrf.vni); + self.fibtablew.del_fib(vrfid); fibw.destroy(); } @@ -937,3 +948,787 @@ mod tests { test_vrf_fibgroup(build_test_vrf_nhops_partially_resolved()); } } + +/// Model-based properties over a [`VrfTable`]. +/// +/// The vrf table is where four key spaces have to agree: `by_id`, `by_vni`, each [`Vrf`]'s own +/// `vni` field, and the fib table's two -- `FibKey::Id` and the `FibKey::Vni` alias. Nothing holds +/// them together but the table's own methods doing the right number of things in the right order, +/// so what is generated here is sequences of those methods, and what is checked is that all four +/// still describe the same set of vrfs afterwards. +#[cfg(test)] +mod vrftable_properties { + use super::*; + use crate::interfaces::iftablerw::IfTableWriter; + use crate::rib::vrf::VrfStatus; + use bolero::{Driver, ValueGenerator}; + use std::collections::BTreeMap; + use std::ops::Bound::Included; + + const NUM_VRFS: u8 = 3; + const NUM_VNIS: u8 = 2; + const NUM_STATUSES: u8 = 3; + const MAX_CHANGES: u8 = 12; + + /// Vrf ids a generated table may hold, the default among them: `remove_vrf` takes any id, and + /// whether it should take that one is exactly the question worth generating for. + fn vrf_ids() -> Vec { + (0..u32::from(NUM_VRFS)).collect() + } + + fn vnis() -> Vec { + (1..=u32::from(NUM_VNIS)) + .map(|i| Vni::new_checked(100 * i).unwrap_or_else(|_| unreachable!())) + .collect() + } + + fn statuses() -> Vec { + vec![VrfStatus::Active, VrfStatus::Deleting, VrfStatus::Deleted] + } + + /// One change, as [`VrfTable`] exposes them, over indices into the pools above. + #[derive(Debug, Clone)] + enum Change { + AddVrf { vrf: usize, vni: Option }, + SetVni { vrf: usize, vni: usize }, + UnsetVni { vrf: usize }, + RemoveVrf { vrf: usize }, + SetStatus { vrf: usize, status: usize }, + RemoveDeleted, + RemoveDeleting, + } + + /// Draws sequences of [`Change`]s. + #[derive(Debug, Clone, Copy, Default)] + struct ChangeSequences; + + fn index(driver: &mut D, count: u8) -> Option { + driver + .gen_u8(Included(&0), Included(&(count - 1))) + .map(usize::from) + } + + impl ValueGenerator for ChangeSequences { + type Output = Vec; + + fn generate(&self, driver: &mut D) -> Option> { + let len = driver.gen_u8(Included(&0), Included(&MAX_CHANGES))?; + let mut out = Vec::with_capacity(usize::from(len)); + for _ in 0..len { + let change = match driver.gen_u8(Included(&0), Included(&6))? { + 0 => { + // `NUM_VNIS` means "created without a vni", which is the ordinary case. + // One draw rather than an `Option>`, which cannot be told apart + // from the driver running out of input. + let vrf = index(driver, NUM_VRFS)?; + let drawn = index(driver, NUM_VNIS + 1)?; + Change::AddVrf { + vrf, + vni: (drawn < usize::from(NUM_VNIS)).then_some(drawn), + } + } + 1 => Change::SetVni { + vrf: index(driver, NUM_VRFS)?, + vni: index(driver, NUM_VNIS)?, + }, + 2 => Change::UnsetVni { + vrf: index(driver, NUM_VRFS)?, + }, + 3 => Change::RemoveVrf { + vrf: index(driver, NUM_VRFS)?, + }, + 4 => Change::SetStatus { + vrf: index(driver, NUM_VRFS)?, + status: index(driver, NUM_STATUSES)?, + }, + 5 => Change::RemoveDeleted, + _ => Change::RemoveDeleting, + }; + out.push(change); + } + Some(out) + } + } + + /// Which vrfs exist, and for each the vni it carries and the status it is in. + /// + /// One map, from which all four of the table's views are derivable -- which is the point. The + /// table keeps them as four separate structures updated by hand; if any method updates three + /// of them, this says which one it missed. + type Model = BTreeMap, VrfStatus)>; + + fn owner_of(model: &Model, vni: Vni) -> Option { + model + .iter() + .find_map(|(id, (carried, _))| (*carried == Some(vni)).then_some(*id)) + } + + fn fresh_model() -> Model { + Model::from([(Vrf::DEFAULT_VRFID, (None, VrfStatus::Active))]) + } + + fn apply(table: &mut VrfTable, iftw: &mut IfTableWriter, model: &mut Model, change: &Change) { + let ids = vrf_ids(); + let all_vnis = vnis(); + match change { + Change::AddVrf { vrf, vni } => { + let id = ids[*vrf]; + let vni = vni.map(|i| all_vnis[i]); + let config = RouterVrfConfig::new(id, &format!("vrf{id}")).set_vni(vni); + let _ = table.add_vrf(&config); + // refused if the id is taken, or if the vni is + if model.contains_key(&id) || vni.is_some_and(|v| owner_of(model, v).is_some()) { + return; + } + model.insert(id, (vni, VrfStatus::Active)); + } + Change::SetVni { vrf, vni } => { + let id = ids[*vrf]; + let vni = all_vnis[*vni]; + let _ = table.set_vni(id, vni); + match owner_of(model, vni) { + // another vrf holds it: refused. The same vrf already holds it: nothing to do + Some(_) => (), + // otherwise the vrf drops whatever vni it had and takes this one -- but only + // if it exists at all + None => { + if let Some(entry) = model.get_mut(&id) { + entry.0 = Some(vni); + } + } + } + } + Change::UnsetVni { vrf } => { + let id = ids[*vrf]; + let _ = table.unset_vni(id); + if let Some(entry) = model.get_mut(&id) { + entry.0 = None; + } + } + Change::RemoveVrf { vrf } => { + let id = ids[*vrf]; + let _ = table.remove_vrf(id, iftw); + // the default vrf is refused + if id != Vrf::DEFAULT_VRFID { + model.remove(&id); + } + } + Change::SetStatus { vrf, status } => { + let id = ids[*vrf]; + let status = statuses()[*status]; + if let Ok(vrf) = table.get_vrf_mut(id) { + vrf.set_status(status); + } + // the default vrf's status is fixed: it is what keeps the sweeps below from + // deleting it + if id != Vrf::DEFAULT_VRFID + && let Some(entry) = model.get_mut(&id) + { + entry.1 = status; + } + } + Change::RemoveDeleted => { + table.remove_deleted_vrfs(iftw); + model.retain(|_, (_, status)| *status != VrfStatus::Deleted); + } + Change::RemoveDeleting => { + table.remove_deleting_vrfs(iftw); + model.retain(|_, (_, status)| *status != VrfStatus::Deleting); + } + } + } + + /// Check every view of the table against the one model, and against each other. + fn check(table: &VrfTable, model: &Model, at: &str) { + let ids = vrf_ids(); + let all_vnis = vnis(); + + // 1. by_id holds exactly the vrfs the model says, each carrying what the model says + assert_eq!(table.len(), model.len(), "vrf count {at}"); + for id in &ids { + let Ok(vrf) = table.get_vrf(*id) else { + assert!(!model.contains_key(id), "vrf {id} missing {at}"); + continue; + }; + let (vni, status) = model + .get(id) + .unwrap_or_else(|| panic!("vrf {id} unexpected {at}")); + assert_eq!(vrf.vrfid, *id, "vrf {id} filed under the wrong key {at}"); + assert_eq!(vrf.vni, *vni, "vrf {id} vni {at}"); + assert_eq!(vrf.status, *status, "vrf {id} status {at}"); + } + + // 2. by_vni is exactly the inverse of the vnis the vrfs carry -- no stale entry left by a + // removal, and none missing after a vni was set + assert_eq!( + table.by_vni.len(), + model.values().filter(|(vni, _)| vni.is_some()).count(), + "vni index size {at}" + ); + for vni in &all_vnis { + assert_eq!( + table.get_vrfid_by_vni(*vni).ok(), + owner_of(model, *vni), + "vni {vni} index {at}" + ); + assert_eq!( + table.get_vrf_by_vni(*vni).map(|vrf| vrf.vrfid).ok(), + owner_of(model, *vni), + "vni {vni} lookup {at}" + ); + } + + // 3. the fib table holds a fib for every vrf, under its id and under its vni if it has + // one, and every one of those keys reaches a live fib with the right identity + let fibs = table.fibtablew.enter().unwrap_or_else(|| unreachable!()); + let expected_keys = model.len() + model.values().filter(|(v, _)| v.is_some()).count(); + assert_eq!(fibs.len(), expected_keys, "fib table size {at}"); + for id in &ids { + let key = FibKey::from_vrfid(*id); + let Some(fib) = fibs.get_fib(key) else { + assert!(!model.contains_key(id), "no fib for vrf {id} {at}"); + continue; + }; + assert!(fib.is_valid(), "fib for vrf {id} is dead {at}"); + assert_eq!( + fib.get_id(), + Some(key), + "fib for vrf {id} is not its own {at}" + ); + } + for vni in &all_vnis { + let Some(fib) = fibs.get_fib(FibKey::from_vni(*vni)) else { + assert!(owner_of(model, *vni).is_none(), "no fib for vni {vni} {at}"); + continue; + }; + let owner = owner_of(model, *vni) + .unwrap_or_else(|| panic!("fib aliased by vni {vni} with no owner {at}")); + assert!(fib.is_valid(), "fib aliased by vni {vni} is dead {at}"); + assert_eq!( + fib.get_id(), + Some(FibKey::from_vrfid(owner)), + "vni {vni} reaches the wrong fib {at}" + ); + } + drop(fibs); + + // 4. the default vrf is always there and always active. `get_default_vrf` treats that as + // given, with an `unreachable!()` rather than an error + assert!(table.contains(Vrf::DEFAULT_VRFID), "no default vrf {at}"); + assert_eq!( + table.get_default_vrf().status, + VrfStatus::Active, + "default vrf not active {at}" + ); + + // 5. the table's own consistency check agrees. `check_vni` is the in-tree half of this + // oracle; it should pass for a vrf with a vni and fail for one without + for id in &ids { + let Some((vni, _)) = model.get(id) else { + continue; + }; + assert_eq!( + table.check_vni(*id).is_ok(), + vni.is_some(), + "check_vni disagrees for vrf {id} {at}" + ); + } + } + + /// The pools and the constants that index them agree. + #[test] + fn the_pools_are_the_size_the_generator_thinks() { + assert_eq!(vrf_ids().len(), usize::from(NUM_VRFS)); + assert_eq!(vnis().len(), usize::from(NUM_VNIS)); + assert_eq!(statuses().len(), usize::from(NUM_STATUSES)); + assert_eq!(vrf_ids()[0], Vrf::DEFAULT_VRFID); + } + + /// After any sequence of changes, every view of the vrf table still describes the same vrfs. + #[test] + fn a_vrf_tables_four_views_stay_in_step() { + bolero::check!() + .with_generator(ChangeSequences) + .cloned() + .for_each(|changes: Vec| { + let (fibtw, _fibtr) = FibTableWriter::new(); + let (mut iftw, _iftr) = IfTableWriter::new(); + let mut table = VrfTable::new(fibtw); + let mut model = fresh_model(); + + check(&table, &model, "on a fresh table"); + for (step, change) in changes.iter().enumerate() { + apply(&mut table, &mut iftw, &mut model, change); + check(&table, &model, &format!("at step {step} of {changes:?}")); + } + }); + } +} + +/// Properties over resolution **across** vrfs. +/// +/// `VrfTable::refresh_non_default_fibs` and `refresh_fibs_by_vni` hand the default vrf to every +/// other vrf as its resolution vrf, which is how a next-hop in an overlay vrf reaches an interface +/// the underlay knows about. Nothing exercised that path: both functions, `set_stale`, +/// `remove_stale_routes` and `values_mut_except_default` were at zero coverage, and every generated +/// next-hop graph so far has lived inside a single vrf. +#[cfg(test)] +mod crossvrf_properties { + use super::*; + use crate::fib::fibobjects::{EgressObject, FibEntry, PktInstruction}; + use crate::rib::vrf::tests::{build_test_nhop, build_test_route, mk_addr}; + use crate::rib::vrf::{Route, RouteNhop, RouteOrigin}; + use bolero::{Driver, ValueGenerator}; + use lpm::prefix::Prefix; + use net::interface::InterfaceIndex; + use std::collections::BTreeMap; + use std::net::IpAddr; + use std::ops::Bound::Included; + use std::str::FromStr; + + const NUM_VRFS: u8 = 2; // non-default vrfs + const NUM_VNIS: u8 = 2; + const NUM_UNDERLAY: u8 = 2; + const NUM_OVERLAY: u8 = 2; + const NUM_IFINDEXES: u8 = 3; + const NUM_VIAS: u8 = 3; + const MAX_ROUTES: u8 = 3; + + /// Non-default vrf ids. The default vrf is 0 and `VrfTable::new` makes it. + fn vrf_ids() -> Vec { + (1..=u32::from(NUM_VRFS)).collect() + } + + fn vnis() -> Vec { + (1..=u32::from(NUM_VNIS)) + .map(|i| Vni::new_checked(100 * i).unwrap_or_else(|_| unreachable!())) + .collect() + } + + /// Prefixes the default vrf carries routes for. Nested, so a longest match has to be chosen. + fn underlay() -> Vec { + ["7.0.0.0/8", "7.1.0.0/16"] + .iter() + .map(|p| Prefix::from_str(p).unwrap_or_else(|_| unreachable!())) + .collect() + } + + /// Prefixes the other vrfs carry routes for. + fn overlay() -> Vec { + ["10.0.0.0/8", "10.1.0.0/16"] + .iter() + .map(|p| Prefix::from_str(p).unwrap_or_else(|_| unreachable!())) + .collect() + } + + fn ifindexes() -> Vec { + (1..=u32::from(NUM_IFINDEXES)).collect() + } + + /// On-link addresses a directly connected underlay next-hop may carry, one per interface. + /// + /// Whether the resolving next-hop has an address of its own is what decides which address ends + /// up in the fib, and `rib2fib` is explicit that the deeper one wins: the next-hop being + /// resolved supplies the layer-2 target "unless a next-hop deeper in the resolution chain + /// provides one of its own". Generating both cases is what lets this property tell + /// `EgressObject::merge`'s rule -- first ifindex, last address -- from its inverse. + fn onlink_addrs() -> Vec { + (1..=NUM_IFINDEXES) + .map(|i| mk_addr(&format!("7.200.0.{i}"))) + .collect() + } + + /// Addresses an overlay route may point via. `8.0.0.1` is covered by no underlay prefix, so it + /// reaches only the default vrf's root -- whose preset next-hop is a drop. + fn vias() -> Vec { + ["7.0.0.1", "7.1.0.1", "8.0.0.1"] + .iter() + .map(|a| mk_addr(a)) + .collect() + } + + /// A generated arrangement of vrfs and routes. + #[derive(Debug, Clone)] + struct Topology { + /// Whether each non-default vrf carries a vni. Vrf `i` gets vni `i`, so two vrfs can never + /// ask for the same one -- `add_vrf` rightly refuses that, and it is the `VrfTable` + /// property's business rather than this one's. + vrfs: Vec, + /// Routes in the default vrf, as (underlay prefix index, ifindex index, on-link). Directly + /// connected: the next-hop carries an interface, and an address of its own when `on-link`. + underlay: Vec<(usize, usize, bool)>, + /// Routes in the other vrfs, as (vrf index, overlay prefix index, via index). Recursive: + /// the next-hop carries an address and no interface, so it has to be resolved. + overlay: Vec<(usize, usize, usize)>, + /// One more default-vrf route, applied after the first refresh, so that a later refresh has + /// something to notice. + later: Option<(usize, usize, bool)>, + /// Vnis to pass to `refresh_fibs_by_vni`. + selected: Vec, + } + + /// Draws [`Topology`]s. + #[derive(Debug, Clone, Copy, Default)] + struct Topologies; + + fn index(driver: &mut D, count: u8) -> Option { + driver + .gen_u8(Included(&0), Included(&(count - 1))) + .map(usize::from) + } + + impl ValueGenerator for Topologies { + type Output = Topology; + + fn generate(&self, driver: &mut D) -> Option { + let mut vrfs = Vec::with_capacity(usize::from(NUM_VRFS)); + for _ in 0..NUM_VRFS { + vrfs.push(driver.produce::()?); + } + + let count = driver.gen_u8(Included(&0), Included(&MAX_ROUTES))?; + let mut underlay = Vec::with_capacity(usize::from(count)); + for _ in 0..count { + underlay.push(( + index(driver, NUM_UNDERLAY)?, + index(driver, NUM_IFINDEXES)?, + driver.produce::()?, + )); + } + + let count = driver.gen_u8(Included(&0), Included(&MAX_ROUTES))?; + let mut overlay = Vec::with_capacity(usize::from(count)); + for _ in 0..count { + overlay.push(( + index(driver, NUM_VRFS)?, + index(driver, NUM_OVERLAY)?, + index(driver, NUM_VIAS)?, + )); + } + + let later = if driver.produce::()? { + Some(( + index(driver, NUM_UNDERLAY)?, + index(driver, NUM_IFINDEXES)?, + driver.produce::()?, + )) + } else { + None + }; + + let count = driver.gen_u8(Included(&0), Included(&NUM_VNIS))?; + let mut selected = Vec::with_capacity(usize::from(count)); + for _ in 0..count { + selected.push(index(driver, NUM_VNIS)?); + } + + Some(Topology { + vrfs, + underlay, + overlay, + later, + selected, + }) + } + } + + /// The default vrf's routes, as the model sees them: prefix index -> (ifindex index, on-link). + /// A later route for a prefix replaces an earlier one, as the trie does. + type Underlay = BTreeMap; + + /// The other vrfs' routes: (vrf index, overlay prefix index) -> via index. A map for the same + /// reason: the generator may name one prefix twice, and only the last route survives. + type Overlay = BTreeMap<(usize, usize), usize>; + + /// The interface the default vrf leads to for `via`, or `None` if `via` reaches only the root -- + /// whose preset next-hop is a drop, so nothing is reachable through it. + /// + /// A longest-prefix match over the generated route list, asking no next-hop anything. + fn resolves_to(model: &Underlay, via: usize) -> Option<(u32, bool)> { + let address = vias()[via]; + let prefixes = underlay(); + model + .iter() + .filter(|(prefix, _)| prefixes[**prefix].covers_addr(&address)) + .max_by_key(|(prefix, _)| prefixes[**prefix].length()) + .map(|(_, (ifindex, onlink))| (ifindexes()[*ifindex], *onlink)) + } + + /// What the fib should hold for an overlay route pointing via `via`. + /// + /// One entry, and the shape of it is the whole point of resolving recursively: the interface + /// comes from the *resolver* in the default vrf, and the address stays that of the next-hop that + /// needed resolving. `EgressObject::merge` keeping the first ifindex and the last address is + /// what produces that, and `rib2fib` explains why it must -- otherwise the egress stage would + /// resolve the packet's own destination at layer 2, which is only right when it is on-link. + fn expected_entry(model: &Underlay, via: usize) -> FibEntry { + match resolves_to(model, via) { + // the interface comes from the resolver; the address is the resolver's own if it has + // one, and otherwise stays that of the next-hop that needed resolving + Some((ifindex, onlink)) => { + let index = usize::try_from(ifindex).unwrap_or_else(|_| unreachable!()) - 1; + let address = if onlink { + onlink_addrs()[index] + } else { + vias()[via] + }; + FibEntry::with_inst(PktInstruction::Egress(EgressObject::new( + InterfaceIndex::try_new(ifindex).ok(), + Some(address), + None, + ))) + } + None => FibEntry::drop_fibentry(), + } + } + + fn underlay_route(ifindex: usize, onlink: bool) -> (Route, Vec) { + let address = onlink.then(|| onlink_addrs()[ifindex].to_string()); + ( + build_test_route(RouteOrigin::Connected, 0, 0), + vec![build_test_nhop( + address.as_deref(), + Some(ifindexes()[ifindex]), + 0, + None, + )], + ) + } + + fn overlay_route(via: usize) -> (Route, Vec) { + ( + build_test_route(RouteOrigin::Bgp, 20, 100), + vec![build_test_nhop( + Some(&vias()[via].to_string()), + None, + 0, + None, + )], + ) + } + + /// Build the table described by `topology`, without refreshing anything yet. + fn realize(topology: &Topology, rstore: &RmacStore) -> (VrfTable, Underlay, Overlay) { + let (fibtw, _fibtr) = FibTableWriter::new(); + let mut table = VrfTable::new(fibtw); + let ids = vrf_ids(); + let all_vnis = vnis(); + + for (vrf, has_vni) in topology.vrfs.iter().enumerate() { + let config = RouterVrfConfig::new(ids[vrf], &format!("vrf{vrf}")) + .set_vni(has_vni.then(|| all_vnis[vrf])); + table + .add_vrf(&config) + .unwrap_or_else(|e| unreachable!("{e}")); + } + + let mut model = Underlay::new(); + for (prefix, ifindex, onlink) in &topology.underlay { + let (route, nhops) = underlay_route(*ifindex, *onlink); + let vrf0 = table + .get_vrf_mut(Vrf::DEFAULT_VRFID) + .unwrap_or_else(|e| unreachable!("{e}")); + vrf0.add_route_complete(&underlay()[*prefix], route, &nhops, None, rstore); + model.insert(*prefix, (*ifindex, *onlink)); + } + + let mut overlay_model = Overlay::new(); + for (vrf, prefix, via) in &topology.overlay { + let (route, nhops) = overlay_route(*via); + // deliberately inserted with no resolution vrf, so the route starts resolved against + // its own vrf -- where the address reaches nothing. Only the table-level refresh can + // put it right, which is what makes the refresh load-bearing here. + let target = table + .get_vrf_mut(ids[*vrf]) + .unwrap_or_else(|e| unreachable!("{e}")); + target.add_route_complete(&overlay()[*prefix], route, &nhops, None, rstore); + overlay_model.insert((*vrf, *prefix), *via); + } + + (table, model, overlay_model) + } + + /// The fib entries a vrf offers for one overlay prefix, or `None` if it has no such route. + fn fib_entries(table: &VrfTable, vrfid: VrfId, prefix: Prefix) -> Option> { + let vrf = table.get_vrf(vrfid).unwrap_or_else(|e| unreachable!("{e}")); + let fibw = vrf.fibw.as_ref().unwrap_or_else(|| unreachable!()); + let fib = fibw.enter().unwrap_or_else(|| unreachable!()); + let Prefix::IPV4(wanted) = prefix else { + unreachable!() + }; + fib.iter_v4().find(|(p, _)| *p == wanted).map(|(_, route)| { + route + .iter() + .flat_map(|group| group.entries().iter().cloned()) + .collect() + }) + } + + /// Every entry in every fib of every vrf is one the forwarder can execute. + /// + /// This is the rib-to-fib contract: `FibEntry::is_valid` is the written-down half of it, and + /// `rib2fib` filters on it -- but the drop injected for an empty group bypasses that filter, and + /// nothing checked the whole table at once. + fn every_entry_is_executable(table: &VrfTable, at: &str) { + for vrf in table.values() { + let fibw = vrf.fibw.as_ref().unwrap_or_else(|| unreachable!()); + let fib = fibw.enter().unwrap_or_else(|| unreachable!()); + for (prefix, route) in fib.iter_v4() { + for group in route.iter() { + assert!(!group.is_empty(), "empty group for {prefix} {at}"); + for entry in group.iter() { + assert!( + entry.is_valid(), + "vrf {} offers unusable {entry:?} for {prefix} {at}", + vrf.vrfid + ); + } + } + } + } + } + + /// The pools and the constants that index them agree. + #[test] + fn the_pools_are_the_size_the_generator_thinks() { + assert_eq!(vrf_ids().len(), usize::from(NUM_VRFS)); + assert_eq!(vnis().len(), usize::from(NUM_VNIS)); + assert_eq!(underlay().len(), usize::from(NUM_UNDERLAY)); + assert_eq!(overlay().len(), usize::from(NUM_OVERLAY)); + assert_eq!(ifindexes().len(), usize::from(NUM_IFINDEXES)); + assert_eq!(vias().len(), usize::from(NUM_VIAS)); + // vrf `i` takes vni `i`, so there must be at least as many vnis as vrfs + const { assert!(NUM_VNIS >= NUM_VRFS) }; + // the last via must be reachable through no underlay prefix, so that the unresolvable case + // is generated + assert_eq!(onlink_addrs().len(), usize::from(NUM_IFINDEXES)); + let all: Underlay = (0..usize::from(NUM_UNDERLAY)) + .map(|p| (p, (0, false))) + .collect(); + assert!(resolves_to(&all, usize::from(NUM_VIAS) - 1).is_none()); + } + + /// A refresh resolves every other vrf's next-hops through the default vrf. + /// + /// The interface has to come from the default vrf's route, and the address has to stay that of + /// the next-hop being resolved. + #[test] + fn a_refresh_resolves_other_vrfs_through_the_default_one() { + let rstore = RmacStore::new(); + bolero::check!() + .with_generator(Topologies) + .cloned() + .for_each(|topology: Topology| { + let (mut table, model, routes) = realize(&topology, &rstore); + table.refresh_non_default_fibs(&rstore); + every_entry_is_executable(&table, "after a refresh"); + + let ids = vrf_ids(); + for ((vrf, prefix), via) in &routes { + let got = fib_entries(&table, ids[*vrf], overlay()[*prefix]) + .unwrap_or_else(|| panic!("no fib route for {prefix} in vrf {vrf}")); + assert_eq!( + got, + vec![expected_entry(&model, *via)], + "vrf {vrf} prefix {prefix} via {via}, for {topology:?}" + ); + } + }); + } + + /// `refresh_fibs_by_vni` refreshes the vrfs whose vni is named, and leaves the rest alone. + #[test] + fn refreshing_by_vni_touches_only_those_vnis() { + let rstore = RmacStore::new(); + bolero::check!() + .with_generator(Topologies) + .cloned() + .for_each(|topology: Topology| { + let Some(later) = topology.later else { return }; + let (mut table, mut model, routes) = realize(&topology, &rstore); + table.refresh_non_default_fibs(&rstore); + + let ids = vrf_ids(); + let all_vnis = vnis(); + + // what each overlay route offers before anything changes + let before: BTreeMap<(usize, usize), Option>> = routes + .keys() + .map(|(vrf, prefix)| { + ( + (*vrf, *prefix), + fib_entries(&table, ids[*vrf], overlay()[*prefix]), + ) + }) + .collect(); + + // change the underlay, then refresh only the named vnis + let (prefix, ifindex, onlink) = later; + let (route, nhops) = underlay_route(ifindex, onlink); + let vrf0 = table + .get_vrf_mut(Vrf::DEFAULT_VRFID) + .unwrap_or_else(|e| unreachable!("{e}")); + vrf0.add_route_complete(&underlay()[prefix], route, &nhops, None, &rstore); + model.insert(prefix, (ifindex, onlink)); + + let selected: Vec = topology.selected.iter().map(|i| all_vnis[*i]).collect(); + table.refresh_fibs_by_vni(&selected, &rstore); + every_entry_is_executable(&table, "after refreshing by vni"); + + for ((vrf, prefix), via) in &routes { + let got = fib_entries(&table, ids[*vrf], overlay()[*prefix]); + let vni = table + .get_vrf(ids[*vrf]) + .unwrap_or_else(|e| unreachable!("{e}")) + .vni; + if vni.is_some_and(|vni| selected.contains(&vni)) { + assert_eq!( + got, + Some(vec![expected_entry(&model, *via)]), + "refreshed vrf {vrf} prefix {prefix}, for {topology:?}" + ); + } else { + assert_eq!( + got, + before[&(*vrf, *prefix)], + "untouched vrf {vrf} prefix {prefix}, for {topology:?}" + ); + } + } + }); + } + + /// Marking everything stale and sweeping leaves every vrf with only its preset drop routes. + /// + /// The default vrf is swept separately from the rest, since it is the resolution vrf for them. + #[test] + fn a_stale_sweep_empties_every_vrf() { + let rstore = RmacStore::new(); + bolero::check!() + .with_generator(Topologies) + .cloned() + .for_each(|topology: Topology| { + let (mut table, _model, _routes) = realize(&topology, &rstore); + table.refresh_non_default_fibs(&rstore); + + table.set_stale(true); + table.remove_stale_routes(&rstore); + + for vrf in table.values() { + assert_eq!(vrf.len_v4(), 1, "vrf {} kept ipv4 routes", vrf.vrfid); + assert_eq!(vrf.len_v6(), 1, "vrf {} kept ipv6 routes", vrf.vrfid); + for prefix in [Prefix::root_v4(), Prefix::root_v6()] { + let route = vrf + .get_route(prefix) + .unwrap_or_else(|| panic!("vrf {} lost {prefix}", vrf.vrfid)); + assert!( + route.is_preset_drop_route(), + "vrf {} left {prefix} as something other than the preset drop route", + vrf.vrfid + ); + } + } + every_entry_is_executable(&table, "after a stale sweep"); + }); + } +} diff --git a/routing/src/router/cpi.rs b/routing/src/router/cpi.rs index 68d9d0a220..ddec4dc920 100644 --- a/routing/src/router/cpi.rs +++ b/routing/src/router/cpi.rs @@ -209,14 +209,13 @@ impl RpcOperation for IpRoute { fn add(&self, db: &mut Self::ObjectStore) -> RpcResultCode { let rmac_store = &db.rmac_store; let vrftable = &mut db.vrftable; - let iftabler = &db.iftw.as_reader(); if self.vrfid == Vrf::DEFAULT_VRFID { let Ok(vrf0) = vrftable.get_vrf_mut(self.vrfid) else { error!("Unable to find default VRF!"); return RpcResultCode::Failure; }; - vrf0.add_route_rpc(self, None, rmac_store, iftabler); + vrf0.add_route_rpc(self, None, rmac_store); vrftable.refresh_non_default_fibs(rmac_store); } else { // this assumes that we always resolve non-default vrfs with the default vrf @@ -229,7 +228,7 @@ impl RpcOperation for IpRoute { error!("Unable to get vrf with id {}", self.vrfid); return RpcResultCode::Failure; }; - vrf.add_route_rpc(self, Some(vrf0), rmac_store, iftabler); + vrf.add_route_rpc(self, Some(vrf0), rmac_store); } RpcResultCode::Ok } @@ -498,3 +497,390 @@ pub fn process_cpi_data(rio: &mut Rio, peer: &SocketAddr, data: &mut Bytes, db: } } } + +/// Properties over the control-plane operations, driven through a whole [`RoutingDb`]. +/// +/// This is the layer above `rpc_adapt`: it picks the vrf, decides what a lookup failure means, and +/// chains an operation's effects into the rest of the database -- an rmac arriving refreshes the +/// fibs of the vrfs on its vni, and deleting a route can delete the vrf with it. Almost none of it +/// was covered. +#[cfg(test)] +mod cpi_properties { + use super::*; + use crate::atable::atablerw::AtableWriter; + use crate::config::RouterConfig; + use crate::evpn::RmacStore; + use crate::fib::fibobjects::{FibEntry, PktInstruction}; + use crate::fib::fibtable::FibTableWriter; + use crate::interfaces::iftablerw::IfTableWriter; + use crate::interfaces::tests::build_test_iftable; + use crate::rib::encapsulation::Encapsulation; + use crate::rib::vrf::tests::{build_test_nhop, build_test_route}; + use crate::rib::vrf::{RouteOrigin, RouterVrfConfig, VrfStatus}; + use bolero::{Driver, ValueGenerator}; + use dplane_rpc::msg::{ForwardAction, NextHop, VxlanEncap}; + use dplane_rpc::objects::MacAddress; + use lpm::prefix::Prefix; + use net::eth::mac::Mac; + use net::vxlan::Vni; + use std::net::IpAddr; + use std::ops::Bound::Included; + use std::str::FromStr; + + /// The vrf the overlay routes live in, and the vni it is reachable by. + const OVERLAY_VRF: VrfId = 7; + const OVERLAY_VNI: u32 = 3000; + /// The interface the underlay route egresses on. + const UNDERLAY_IFINDEX: u32 = 2; + + const NUM_VTEPS: u8 = 2; + const NUM_MACS: u8 = 2; + + fn addr(a: &str) -> IpAddr { + IpAddr::from_str(a).unwrap_or_else(|_| unreachable!()) + } + + /// Remote vtep addresses an overlay next-hop may point at. Both are covered by the underlay + /// route installed below, so reachability is never the reason a route fails to resolve. + fn vteps() -> Vec { + vec![addr("7.0.0.1"), addr("7.0.0.2")] + } + + fn macs() -> Vec<[u8; 6]> { + vec![ + [0x00, 0xaa, 0x00, 0x00, 0x00, 0x01], + [0x00, 0xbb, 0x00, 0x00, 0x00, 0x02], + ] + } + + /// A database with an interface table, an underlay route in the default vrf towards the vteps, + /// and one overlay vrf on [`OVERLAY_VNI`]. + fn fabric() -> RoutingDb { + let (fibtw, _fibtr) = FibTableWriter::new(); + let (iftw, _iftr) = IfTableWriter::new_with_data(build_test_iftable()); + let (_atw, atabler) = AtableWriter::new(); + let mut db = RoutingDb::new(fibtw, iftw, atabler); + + // the underlay: 7.0.0.0/8 out of an interface, so a vtep address resolves + let vrf0 = db + .vrftable + .get_vrf_mut(Vrf::DEFAULT_VRFID) + .unwrap_or_else(|e| unreachable!("{e}")); + vrf0.add_route_complete( + &Prefix::from_str("7.0.0.0/8").unwrap_or_else(|_| unreachable!()), + build_test_route(RouteOrigin::Connected, 0, 0), + &[build_test_nhop(None, Some(UNDERLAY_IFINDEX), 0, None)], + None, + &RmacStore::new(), + ); + + let vni = Vni::new_checked(OVERLAY_VNI).unwrap_or_else(|_| unreachable!()); + let config = RouterVrfConfig::new(OVERLAY_VRF, "overlay").set_vni(Some(vni)); + db.vrftable + .add_vrf(&config) + .unwrap_or_else(|e| unreachable!("{e}")); + db + } + + /// An overlay route: prefix reachable by vxlan to `vtep` on [`OVERLAY_VNI`]. + fn overlay_route(vrfid: VrfId, prefix: &str, vtep: IpAddr) -> IpRoute { + let (address, len) = prefix.split_once('/').unwrap_or_else(|| unreachable!()); + IpRoute { + prefix: addr(address), + prefix_len: len.parse().unwrap_or_else(|_| unreachable!()), + vrfid, + tableid: 254, + rtype: RouteType::Bgp, + distance: 20, + metric: 100, + nhops: vec![NextHop { + fwaction: ForwardAction::Forward, + address: Some(vtep), + ifindex: None, + vrfid, + encap: Some(NextHopEncap::VXLAN(VxlanEncap { vni: OVERLAY_VNI })), + }], + } + } + + fn rmac_msg(vtep: IpAddr, mac: [u8; 6]) -> Rmac { + Rmac { + address: vtep, + mac: MacAddress::new(mac), + vni: OVERLAY_VNI, + } + } + + /// The entries a vrf's fib offers for one prefix. + fn fib_entries(db: &RoutingDb, vrfid: VrfId, prefix: &str) -> Vec { + let prefix = Prefix::from_str(prefix).unwrap_or_else(|_| unreachable!()); + let Prefix::IPV4(wanted) = prefix else { + unreachable!() + }; + let vrf = db + .vrftable + .get_vrf(vrfid) + .unwrap_or_else(|e| unreachable!("{e}")); + let fibw = vrf.fibw.as_ref().unwrap_or_else(|| unreachable!()); + let fib = fibw.enter().unwrap_or_else(|| unreachable!()); + fib.iter_v4() + .find(|(p, _)| *p == wanted) + .map(|(_, route)| { + route + .iter() + .flat_map(|group| group.entries().iter().cloned()) + .collect() + }) + .unwrap_or_default() + } + + /// Every entry is one the forwarder can execute, and any `Drop` in it comes before anything + /// that would act on the packet. + /// + /// The second half is why an entry like `[Drop, Egress]` -- which resolution does produce, for + /// an encapsulation that could not be completed -- is safe: the forwarder stops at the first + /// instruction that finishes the packet, so a `Drop` reached first means nothing after it runs. + fn entries_are_well_formed(entries: &[FibEntry], at: &str) { + for entry in entries { + assert!(entry.is_valid(), "unusable {entry:?} {at}"); + let drop_at = entry + .iter() + .position(|inst| matches!(inst, PktInstruction::Drop)); + if let Some(index) = drop_at { + assert_eq!(index, 0, "a drop is not first in {entry:?} {at}"); + } + } + } + + /// Draws a `(vtep, mac)` pair. + #[derive(Debug, Clone, Copy, Default)] + struct Fabrics; + + impl ValueGenerator for Fabrics { + type Output = (usize, usize); + + fn generate(&self, driver: &mut D) -> Option<(usize, usize)> { + let vtep = driver.gen_u8(Included(&0), Included(&(NUM_VTEPS - 1)))?; + let mac = driver.gen_u8(Included(&0), Included(&(NUM_MACS - 1)))?; + Some((usize::from(vtep), usize::from(mac))) + } + } + + /// The pools and the constants that index them agree. + #[test] + fn the_pools_are_the_size_the_generator_thinks() { + assert_eq!(vteps().len(), usize::from(NUM_VTEPS)); + assert_eq!(macs().len(), usize::from(NUM_MACS)); + // every vtep must be reachable through the underlay route, so a failure to resolve is only + // ever about the router mac + let underlay = Prefix::from_str("7.0.0.0/8").unwrap_or_else(|_| unreachable!()); + for vtep in vteps() { + assert!(underlay.covers_addr(&vtep), "{vtep} is not in the underlay"); + } + } + + /// An overlay route drops until the router mac for its vtep arrives, and encapsulates after. + /// + /// This is the evpn data path end to end: the route resolves through the underlay either way, + /// but the encapsulation cannot be completed without a router mac, so until one arrives the + /// only safe thing is to drop. When `Rmac::add` stores one it refreshes the fibs on that vni, + /// which is what turns the entry into an encapsulate-then-egress. + #[test] + fn an_overlay_route_drops_until_its_router_mac_arrives() { + bolero::check!().with_generator(Fabrics).cloned().for_each( + |(vtep, mac): (usize, usize)| { + let mut db = fabric(); + let vtep = vteps()[vtep]; + let prefix = "10.0.0.0/24"; + + assert_eq!( + overlay_route(OVERLAY_VRF, prefix, vtep).add(&mut db), + RpcResultCode::Ok + ); + + // no router mac yet: the encapsulation cannot be completed + let before = fib_entries(&db, OVERLAY_VRF, prefix); + entries_are_well_formed(&before, "before the rmac"); + assert!( + before + .iter() + .all(|entry| matches!(entry.iter().next(), Some(PktInstruction::Drop))), + "an overlay route with no router mac must drop, got {before:?}" + ); + + assert_eq!(rmac_msg(vtep, macs()[mac]).add(&mut db), RpcResultCode::Ok); + + let after = fib_entries(&db, OVERLAY_VRF, prefix); + entries_are_well_formed(&after, "after the rmac"); + let expected_mac = Mac::from(macs()[mac]); + for entry in &after { + let mut instructions = entry.iter(); + match instructions.next() { + Some(PktInstruction::Encap(Encapsulation::Vxlan(vxlan))) => { + assert_eq!(vxlan.vni.as_u32(), OVERLAY_VNI, "vni in {entry:?}"); + assert_eq!(vxlan.remote, vtep, "remote in {entry:?}"); + assert_eq!(vxlan.dmac, Some(expected_mac), "dmac in {entry:?}"); + } + other => panic!("expected an encapsulation first, got {other:?}"), + } + match instructions.next() { + Some(PktInstruction::Egress(egress)) => { + assert_eq!( + egress.ifindex().map(InterfaceIndex::to_u32), + Some(UNDERLAY_IFINDEX), + "egress interface in {entry:?}" + ); + assert_eq!( + *egress.address(), + Some(vtep), + "egress address in {entry:?}" + ); + } + other => panic!("expected an egress second, got {other:?}"), + } + assert!(instructions.next().is_none(), "extra work in {entry:?}"); + } + }, + ); + } + + /// Withdrawing a router mac does not stop traffic. + /// + /// `Rmac::del` marks the entry stale rather than removing it, and `VxlanEncapsulation::resolve` + /// accepts a stale one -- "ok if we found a mac, even if the entry is stale". Forwarding to a + /// mac that may have moved beats blackholing the vni while the control plane catches up. + #[test] + fn withdrawing_a_router_mac_leaves_the_route_forwarding() { + bolero::check!().with_generator(Fabrics).cloned().for_each( + |(vtep, mac): (usize, usize)| { + let mut db = fabric(); + let vtep = vteps()[vtep]; + let prefix = "10.0.0.0/24"; + let rmac = rmac_msg(vtep, macs()[mac]); + + overlay_route(OVERLAY_VRF, prefix, vtep).add(&mut db); + rmac.add(&mut db); + let before = fib_entries(&db, OVERLAY_VRF, prefix); + + assert_eq!(rmac.del(&mut db), RpcResultCode::Ok); + // the fib is only rebuilt on the next refresh, so ask for one + db.vrftable.refresh_non_default_fibs(&db.rmac_store); + + let after = fib_entries(&db, OVERLAY_VRF, prefix); + entries_are_well_formed(&after, "after withdrawing the rmac"); + assert_eq!(after, before, "withdrawing a router mac changed the fib"); + }, + ); + } + + /// A route for a vrf we do not have fails on add, and succeeds on delete until we have a config. + /// + /// The asymmetry is deliberate: a delete for a vrf we never had is a route we do not have + /// either, so reporting failure would leave frr retrying something already true. Once a config + /// has been applied there is no such excuse, and the same lookup is a real failure. + #[test] + fn an_unknown_vrf_fails_on_add_and_forgives_on_delete() { + let missing = OVERLAY_VRF + 1; + let prefix = "10.9.0.0/24"; + let vtep = vteps()[0]; + + let mut db = fabric(); + assert_eq!( + overlay_route(missing, prefix, vtep).add(&mut db), + RpcResultCode::Failure + ); + assert_eq!( + overlay_route(missing, prefix, vtep).del(&mut db), + RpcResultCode::Ok, + "a delete for an unknown vrf is forgiven while we have no config" + ); + + db.set_config(RouterConfig::new(1)); + assert!(db.have_config()); + assert_eq!( + overlay_route(missing, prefix, vtep).del(&mut db), + RpcResultCode::Failure, + "once a config is applied the same lookup is a real failure" + ); + } + + /// Deleting the last route of a vrf on its way out takes the vrf with it. + #[test] + fn deleting_the_last_route_of_a_dying_vrf_removes_it() { + let mut db = fabric(); + let prefix = "10.0.0.0/24"; + let vtep = vteps()[0]; + let route = overlay_route(OVERLAY_VRF, prefix, vtep); + route.add(&mut db); + + // while the vrf is active, deleting its routes leaves it in place + assert_eq!(route.del(&mut db), RpcResultCode::Ok); + assert!(db.vrftable.contains(OVERLAY_VRF)); + + route.add(&mut db); + db.vrftable + .get_vrf_mut(OVERLAY_VRF) + .unwrap_or_else(|e| unreachable!("{e}")) + .set_status(VrfStatus::Deleting); + + // now the same delete empties it, which makes it deletable, which removes it + assert_eq!(route.del(&mut db), RpcResultCode::Ok); + assert!( + !db.vrftable.contains(OVERLAY_VRF), + "a vrf that became deletable was left behind" + ); + } + + /// An interface address is refused unless both the mask and the interface index are usable. + #[test] + fn an_interface_address_is_refused_unless_it_is_usable() { + let cases = [ + // (ifindex, mask, expected) + (UNDERLAY_IFINDEX, 24, RpcResultCode::Ok), + (UNDERLAY_IFINDEX, 0, RpcResultCode::InvalidRequest), + (UNDERLAY_IFINDEX, 33, RpcResultCode::InvalidRequest), + (0, 24, RpcResultCode::InvalidRequest), + ]; + for (ifindex, mask, want) in cases { + let mut db = fabric(); + let message = IfAddress { + ifname: "eth0".to_string(), + address: addr("10.0.0.1"), + mask_len: mask, + ifindex, + vrfid: Vrf::DEFAULT_VRFID, + }; + let present = |db: &RoutingDb| { + let iftable = db.iftw.enter().unwrap_or_else(|| unreachable!()); + let Ok(index) = InterfaceIndex::try_new(ifindex) else { + return false; + }; + iftable + .get_interface(index) + .is_some_and(|iface| !iface.addresses.is_empty()) + }; + + assert_eq!(message.add(&mut db), want, "adding {message}"); + assert_eq!( + present(&db), + want == RpcResultCode::Ok, + "after adding {message}" + ); + + assert_eq!(message.del(&mut db), want, "deleting {message}"); + assert!(!present(&db), "the address survived its own deletion"); + } + } + + /// `nonlocal_nhop` spots a route whose next-hops live in another vrf. + #[test] + fn a_next_hop_in_another_vrf_is_nonlocal() { + let vtep = vteps()[0]; + let mut route = overlay_route(OVERLAY_VRF, "10.0.0.0/24", vtep); + assert!(!nonlocal_nhop(&route), "its own vrf is not nonlocal"); + route.nhops[0].vrfid = Vrf::DEFAULT_VRFID; + assert!(nonlocal_nhop(&route), "another vrf is nonlocal"); + route.nhops.clear(); + assert!(!nonlocal_nhop(&route), "no next-hops, nothing nonlocal"); + } +} diff --git a/routing/src/router/rpc_adapt.rs b/routing/src/router/rpc_adapt.rs index ff2e39bbc9..b735b27522 100644 --- a/routing/src/router/rpc_adapt.rs +++ b/routing/src/router/rpc_adapt.rs @@ -11,7 +11,6 @@ use crate::errors::RouterError; use crate::evpn::{RmacEntry, RmacStore}; -use crate::interfaces::iftablerw::IfTableReader; use crate::rib::encapsulation::{Encapsulation, VxlanEncapsulation}; use crate::rib::nexthop::{FwAction, NhopKey}; use crate::rib::vrf::{Route, RouteFlags, RouteNhop, RouteOrigin, Vrf}; @@ -100,11 +99,7 @@ impl TryFrom<&Rmac> for RmacEntry { impl RouteNhop { #[tracing::instrument(level = "debug")] - fn from_rpc_nhop( - nh: &NextHop, - origin: RouteOrigin, - iftabler: &IfTableReader, - ) -> Result { + fn from_rpc_nhop(nh: &NextHop, origin: RouteOrigin) -> Result { let mut ifindex = nh .ifindex .map(|i| match InterfaceIndex::try_new(i) { @@ -129,22 +124,21 @@ impl RouteNhop { None => None, }; - // lookup interface name - let ifname = match ifindex { - None => None, - Some(k) => iftabler - .enter() - .and_then(|iftable| iftable.get_interface(k).map(|iface| iface.name.clone())), - }; - - // build key for this next hop + // build key for this next hop. + // + // No interface name: it would have to be looked up from `ifindex` against the interface + // table, which is populated out of band, so the same next-hop off the wire would key + // differently before and after we learn about the interface -- two `Nhop`s, two fib groups, + // for one next-hop. This is the same reasoning that keeps a vxlan dmac out of the key, + // written down above: a next-hop key has to be immutable for keying purposes, so nothing + // derived from mutable state outside it belongs in one. let key = NhopKey::new( origin, nh.address, ifindex, encap, FwAction::from(nh.fwaction), - ifname, + None, ); // validate next hop from its key @@ -180,13 +174,7 @@ impl Route { } impl Vrf { - pub fn add_route_rpc( - &mut self, - iproute: &IpRoute, - vrf0: Option<&Vrf>, - rstore: &RmacStore, - iftabler: &IfTableReader, - ) { + pub fn add_route_rpc(&mut self, iproute: &IpRoute, vrf0: Option<&Vrf>, rstore: &RmacStore) { let prefix = match Prefix::try_from((iproute.prefix, iproute.prefix_len)) { Ok(p) => p, Err(e) => { @@ -216,18 +204,18 @@ impl Vrf { let route = Route::from_iproute(&prefix, iproute); let mut nhops = Vec::with_capacity(iproute.nhops.len()); for nhop in &iproute.nhops { - match RouteNhop::from_rpc_nhop(nhop, route.origin, iftabler) { + match RouteNhop::from_rpc_nhop(nhop, route.origin) { Ok(nh) => nhops.push(nh), Err(e) => error!("Omitting next-hop {nhop} in route to {prefix}: {e}"), } } - // If no next-hop was received with the route (or we could not successfully process any), - // install the route anyway with an action drop. This is better than not installing the - // route as that could break consistency (e.g. resolving via a default) and cause a loop. + // If no next-hop was received with the route, or none of them could be processed, the + // route is still installed -- with an action drop, which `Vrf::nhops_or_drop` substitutes. + // Not installing it would break consistency (e.g. resolving via a default) and cause a + // loop. Warn here rather than there, since only this layer can tell the two cases apart. if nhops.is_empty() { - warn!("Route to {prefix} from RPC would have no next-hop. Will inject DROP next-hop"); - nhops.push(RouteNhop::default()); + warn!("Route to {prefix} from RPC has no usable next-hop: will be a DROP route"); } // N.B. route and next-hops are passed separately @@ -245,3 +233,387 @@ impl Vrf { self.del_route(prefix, vrf0, rstore); } } + +/// Properties over the translation from control-plane messages into routing state. +/// +/// This is where the routing stack parses input it does not control: `IpRoute`s and their next-hops +/// arrive from FRR over the CPI, and everything downstream is built from whatever this module makes +/// of them. It had no test coverage at all. +/// +/// The oracle throughout is the wire message: what the key should hold, and which next-hops should +/// be refused, worked out from the fields rather than by rerunning the conversion. +#[cfg(test)] +mod rpc_properties { + use super::*; + use crate::fib::fibtype::{FibKey, FibWriter}; + use crate::rib::vrf::RouterVrfConfig; + use bolero::{Driver, ValueGenerator}; + use dplane_rpc::proto::{Ifindex, MaskLen, VrfId}; + use std::net::IpAddr; + use std::ops::Bound::Included; + use std::str::FromStr; + + const NUM_PREFIXES: u8 = 5; + const NUM_ADDRESSES: u8 = 3; + const NUM_IFINDEXES: u8 = 4; + const NUM_VNIS: u8 = 3; + const NUM_RTYPES: u8 = 7; + const MAX_NHOPS: u8 = 3; + + /// The prefix of the last entry in [`prefixes`] cannot be built, so the "message names a prefix + /// we cannot parse" path is generated. + const BAD_PREFIX: usize = 4; + + /// `(address, mask length)` pairs as they arrive on the wire, including one that is not a + /// prefix at all. + fn prefixes() -> Vec<(IpAddr, MaskLen)> { + vec![ + ( + IpAddr::from_str("10.0.0.0").unwrap_or_else(|_| unreachable!()), + 8, + ), + ( + IpAddr::from_str("10.1.0.0").unwrap_or_else(|_| unreachable!()), + 16, + ), + // a host prefix, which turns a connected route into a local one + ( + IpAddr::from_str("10.1.2.3").unwrap_or_else(|_| unreachable!()), + 32, + ), + ( + IpAddr::from_str("2001:db8::").unwrap_or_else(|_| unreachable!()), + 32, + ), + // 33 bits of an ipv4 address: no such prefix + ( + IpAddr::from_str("10.0.0.0").unwrap_or_else(|_| unreachable!()), + 33, + ), + ] + } + + /// Next-hop addresses. `None` is on the wire too, for an interface-only next-hop or a drop. + fn addresses() -> Vec> { + vec![ + None, + Some(IpAddr::from_str("10.0.0.1").unwrap_or_else(|_| unreachable!())), + Some(IpAddr::from_str("7.0.0.1").unwrap_or_else(|_| unreachable!())), + ] + } + + /// Next-hop interface indices, as raw wire values: absent, the invalid zero, one the interface + /// table knows, and one it does not. + fn ifindexes() -> Vec> { + vec![None, Some(0), Some(2), Some(99)] + } + + /// Encapsulation vnis: absent, the invalid zero, and a usable one. + fn vnis() -> Vec> { + vec![None, Some(0), Some(3000)] + } + + fn rtypes() -> Vec { + vec![ + RouteType::Local, + RouteType::Connected, + RouteType::Static, + RouteType::Ospf, + RouteType::Isis, + RouteType::Bgp, + RouteType::Other, + ] + } + + /// One next-hop as it arrives, over indices into the pools above. + #[derive(Debug, Clone)] + struct NhopSpec { + drop: bool, + address: usize, + ifindex: usize, + vni: usize, + vrfid: VrfId, + } + + /// One route as it arrives. + #[derive(Debug, Clone)] + struct RouteSpec { + prefix: usize, + rtype: usize, + distance: u8, + metric: u32, + nhops: Vec, + } + + /// Draws [`RouteSpec`]s. + #[derive(Debug, Clone, Copy, Default)] + struct Routes; + + fn index(driver: &mut D, count: u8) -> Option { + driver + .gen_u8(Included(&0), Included(&(count - 1))) + .map(usize::from) + } + + impl ValueGenerator for Routes { + type Output = RouteSpec; + + fn generate(&self, driver: &mut D) -> Option { + let prefix = index(driver, NUM_PREFIXES)?; + let rtype = index(driver, NUM_RTYPES)?; + let distance = driver.produce::()?; + let metric = driver.produce::()?; + // deliberately able to draw none: a route with no next-hops is on the wire, and the + // comment in `add_route_rpc` is about exactly that + let count = driver.gen_u8(Included(&0), Included(&MAX_NHOPS))?; + let mut nhops = Vec::with_capacity(usize::from(count)); + for _ in 0..count { + nhops.push(NhopSpec { + drop: driver.produce::()?, + address: index(driver, NUM_ADDRESSES)?, + ifindex: index(driver, NUM_IFINDEXES)?, + vni: index(driver, NUM_VNIS)?, + vrfid: 0, + }); + } + Some(RouteSpec { + prefix, + rtype, + distance, + metric, + nhops, + }) + } + } + + fn wire_nhop(spec: &NhopSpec) -> NextHop { + NextHop { + fwaction: if spec.drop { + ForwardAction::Drop + } else { + ForwardAction::Forward + }, + address: addresses()[spec.address], + ifindex: ifindexes()[spec.ifindex], + vrfid: spec.vrfid, + encap: vnis()[spec.vni].map(|vni| NextHopEncap::VXLAN(VxlanEncap { vni })), + } + } + + fn wire_route(spec: &RouteSpec) -> IpRoute { + let (prefix, prefix_len) = prefixes()[spec.prefix]; + IpRoute { + prefix, + prefix_len, + vrfid: 0, + tableid: 254, + rtype: rtypes()[spec.rtype], + distance: spec.distance, + metric: spec.metric, + nhops: spec.nhops.iter().map(wire_nhop).collect(), + } + } + + /// The origin the route should be recorded with. + /// + /// The pairs are written out rather than deferred to `From`, so that a wrong pairing + /// is visible. The one rule that is not a pairing: a *connected* route to a single host is the + /// address of one of our own interfaces, so it is recorded as `Local` -- which is what makes + /// `build_pkt_instructions` emit a local-delivery instruction instead of an egress. + fn expected_origin(rtype: RouteType, prefix: &Prefix) -> RouteOrigin { + if rtype == RouteType::Connected && prefix.is_host() { + return RouteOrigin::Local; + } + match rtype { + RouteType::Local => RouteOrigin::Local, + RouteType::Connected => RouteOrigin::Connected, + RouteType::Static => RouteOrigin::Static, + RouteType::Ospf => RouteOrigin::Ospf, + RouteType::Isis => RouteOrigin::Isis, + RouteType::Bgp => RouteOrigin::Bgp, + RouteType::Other => RouteOrigin::Other, + } + } + + /// The key the next-hop should produce, or `None` if it should be refused. + /// + /// Four reasons to refuse, each worked out from the wire fields: interface index zero, a vni + /// that is not one, a vxlan next-hop that does not say which vtep to send to, and a forwarding + /// next-hop with neither an interface nor an address -- which is nowhere to send anything. + fn expected_key(spec: &NhopSpec, origin: RouteOrigin) -> Option { + let raw = ifindexes()[spec.ifindex]; + if raw == Some(0) { + return None; + } + let address = addresses()[spec.address]; + + let encap = match vnis()[spec.vni] { + None => None, + Some(vni) => Some(Encapsulation::Vxlan(VxlanEncapsulation { + vni: Vni::new_checked(vni).ok()?, + remote: address?, + dmac: None, + })), + }; + + // an encapsulated next-hop is reached by the underlay, so whatever interface the message + // named for it is ignored + let ifindex = if encap.is_some() { + None + } else { + raw.and_then(|i| InterfaceIndex::try_new(i).ok()) + }; + + let fwaction = if spec.drop { + FwAction::Drop + } else { + FwAction::Forward + }; + if fwaction == FwAction::Forward && ifindex.is_none() && address.is_none() { + return None; + } + + // no interface name: the conversion has no interface table to look one up in, which is + // what keeps one wire next-hop from keying two ways + Some(NhopKey::new( + origin, address, ifindex, encap, fwaction, None, + )) + } + + fn test_vrf() -> Vrf { + let config = RouterVrfConfig::new(1, "test"); + let mut vrf = Vrf::new(&config); + let (fibw, _fibr) = FibWriter::new(FibKey::from_vrfid(1)); + vrf.set_fibw(fibw); + vrf + } + + /// The pools and the constants that index them agree, and each refusal is reachable. + #[test] + fn the_pools_are_the_size_the_generator_thinks() { + assert_eq!(prefixes().len(), usize::from(NUM_PREFIXES)); + assert_eq!(addresses().len(), usize::from(NUM_ADDRESSES)); + assert_eq!(ifindexes().len(), usize::from(NUM_IFINDEXES)); + assert_eq!(vnis().len(), usize::from(NUM_VNIS)); + assert_eq!(rtypes().len(), usize::from(NUM_RTYPES)); + + let (prefix, len) = prefixes()[BAD_PREFIX]; + assert!( + Prefix::try_from((prefix, len)).is_err(), + "the bad prefix must not parse" + ); + assert!( + Vni::new_checked(0).is_err(), + "vni zero must not be a valid vni" + ); + assert!( + InterfaceIndex::try_new(0).is_err(), + "interface index zero must not be valid" + ); + // the delete property relies on no pool prefix being a root, since a root route is reset + // rather than removed + for (address, len) in prefixes() { + assert!(Prefix::try_from((address, len)).is_ok_and(|p| !p.is_root()) || len > 32); + } + } + + /// A next-hop off the wire is refused for exactly the reasons it should be, and otherwise + /// yields the key the message describes. + #[test] + fn a_wire_next_hop_becomes_the_key_the_message_describes() { + bolero::check!() + .with_generator(Routes) + .cloned() + .for_each(|spec: RouteSpec| { + for origin in [RouteOrigin::Local, RouteOrigin::Bgp, RouteOrigin::Connected] { + for nhop in &spec.nhops { + let got = RouteNhop::from_rpc_nhop(&wire_nhop(nhop), origin); + match expected_key(nhop, origin) { + Some(want) => { + let got = got.unwrap_or_else(|e| { + panic!("refused {nhop:?} with {e}, expected {want:?}") + }); + assert_eq!(got.key, want, "for {nhop:?} origin {origin:?}"); + assert_eq!(got.vrfid, nhop.vrfid, "vrfid for {nhop:?}"); + } + None => assert!(got.is_err(), "accepted {nhop:?}, expected refusal"), + } + } + } + }); + } + + /// A route off the wire is installed as the message describes, with the next-hops that survived + /// translation -- or a drop next-hop if none did. + #[test] + fn a_wire_route_is_installed_as_the_message_describes() { + bolero::check!() + .with_generator(Routes) + .cloned() + .for_each(|spec: RouteSpec| { + let rstore = RmacStore::new(); + let mut vrf = test_vrf(); + vrf.add_route_rpc(&wire_route(&spec), None, &rstore); + + let (raw, len) = prefixes()[spec.prefix]; + let Ok(prefix) = Prefix::try_from((raw, len)) else { + // a prefix we cannot parse installs nothing: only the two preset root routes + assert_eq!(vrf.len_v4() + vrf.len_v6(), 2, "for {spec:?}"); + return; + }; + + let origin = expected_origin(rtypes()[spec.rtype], &prefix); + let route = vrf + .get_route(prefix) + .unwrap_or_else(|| panic!("no route for {prefix}, for {spec:?}")); + + assert_eq!(route.origin, origin, "origin for {spec:?}"); + assert_eq!(route.distance, spec.distance, "distance for {spec:?}"); + assert_eq!(route.metric, spec.metric, "metric for {spec:?}"); + + let mut want: Vec = spec + .nhops + .iter() + .filter_map(|nhop| expected_key(nhop, origin)) + .collect(); + if want.is_empty() { + // nothing usable: the route is still installed, as a drop + want.push(NhopKey::with_drop()); + } + let got: Vec = route.s_nhops.iter().map(|s| s.rc.key.clone()).collect(); + assert_eq!(got, want, "next-hops for {spec:?}"); + }); + } + + /// Deleting the route the message names removes it; a prefix we cannot parse removes nothing. + #[test] + fn a_wire_delete_removes_what_the_message_names() { + bolero::check!() + .with_generator(Routes) + .cloned() + .for_each(|spec: RouteSpec| { + let rstore = RmacStore::new(); + let mut vrf = test_vrf(); + let route = wire_route(&spec); + vrf.add_route_rpc(&route, None, &rstore); + vrf.del_route_rpc(&route, None, &rstore); + + let (raw, len) = prefixes()[spec.prefix]; + if let Ok(prefix) = Prefix::try_from((raw, len)) { + assert!( + vrf.get_route(prefix).is_none(), + "route to {prefix} survived deletion, for {spec:?}" + ); + } + // no pool prefix is a root, so nothing but the two preset root routes is left + assert_eq!(vrf.len_v4() + vrf.len_v6(), 2, "for {spec:?}"); + // and the next-hop store is left holding only what the root routes name + let keys: Vec = vrf.nhstore.iter().map(|rc| rc.key.clone()).collect(); + assert_eq!( + keys, + vec![NhopKey::with_drop()], + "leftover next-hops for {spec:?}" + ); + }); + } +} diff --git a/scripts/dump-core.sh b/scripts/dump-core.sh new file mode 100755 index 0000000000..e9c398a362 --- /dev/null +++ b/scripts/dump-core.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# Copyright Open Network Fabric Authors + +# Dump a core from a running process without ending it. +# +# Attaching stops the process, so the window between attach and detach is time +# the dataplane is not forwarding. Keep the command list to the dump itself. + +set -euo pipefail + +if [ "$#" -lt 1 ] || [ "$#" -gt 2 ]; then + >&2 echo "usage: ${0##*/} [core-file]" + exit 2 +fi + +declare -r pid="$1" +declare -r core="${2:-/tmp/dataplane.core}" + +if [[ ! "${pid}" =~ ^[0-9]+$ ]]; then + >&2 echo "${0##*/}: '${pid}' is not a pid" + exit 2 +fi + +if [ ! -d "/proc/${pid}" ]; then + >&2 echo "${0##*/}: no process ${pid}" + exit 1 +fi + +# ptrace across uids needs CAP_SYS_PTRACE, and a dataplane does not run as you. +# Checking here turns a confusing gdb error into a clear one. +if [ "$(id -u)" -ne 0 ]; then + >&2 echo "${0##*/}: must run as root to attach to ${pid}" + exit 1 +fi + +# The static gdb is the point of this script: prefer one sitting beside it, so +# that scp'ing the pair to a machine with no nix store is enough. +declare gdb="${GDB:-}" +if [ -z "${gdb}" ]; then + if [ -x "$(dirname -- "$(readlink -f -- "$0")")/gdb" ]; then + gdb="$(dirname -- "$(readlink -f -- "$0")")/gdb" + else + gdb="$(command -v gdb || true)" + fi +fi +declare -r gdb +if [ -z "${gdb}" ]; then + >&2 echo "${0##*/}: no gdb; set GDB, or put one next to this script" + exit 1 +fi + +# `--nx` because an operator's ~/.gdbinit must not decide what a core contains, +# and `auto-load off` because the only thing gdb would auto-load here is +# libthread_db, which it does not need: threads are enumerated from /proc, and +# a static gdb cannot dlopen it anyway. Without this it prints a paragraph of +# safe-path advice on every run. +"${gdb}" --nx --quiet --batch \ + -iex 'set auto-load off' \ + -ex "generate-core-file ${core}" \ + -ex detach \ + -p "${pid}" + +if [ ! -s "${core}" ]; then + >&2 echo "${0##*/}: gdb wrote no core to ${core}" + exit 1 +fi + +# `detach` resumes, but say so from the process's own state rather than from +# gdb's exit status: a core you can open is worthless if the dataplane is still +# sitting in ptrace-stop. 't' is TASK_TRACED. +# The comm field is parenthesised and may contain spaces, so cut past the last +# ')' rather than counting whitespace-separated fields. +declare state +state="$(sed -e 's/^.*) //' -e 's/ .*//' "/proc/${pid}/stat" 2>/dev/null || echo gone)" +declare -r state +case "${state}" in + t) >&2 echo "${0##*/}: ${pid} is still stopped; detach it by hand"; exit 1 ;; + gone) >&2 echo "${0##*/}: ${pid} did not survive"; exit 1 ;; +esac + +echo "${0##*/}: ${core} ($(stat -c %s -- "${core}") bytes), ${pid} running (${state})" diff --git a/scripts/spec-interlock.ts b/scripts/spec-interlock.ts new file mode 100755 index 0000000000..cf4dc07323 --- /dev/null +++ b/scripts/spec-interlock.ts @@ -0,0 +1,730 @@ +#!/usr/bin/env -S deno run --allow-read --allow-run --allow-write +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +/** + * Check that a `type=test` citation actually tests its `type=implementation` citation. + * + * duvet checks that a citation *exists*. It cannot check that the test named by one says + * anything about the code named by the other: both are comments, and a refactor can separate + * them without either changing. That gap is what makes a citation decorative, and it is + * invisible in every report the three tools produce on their own. + * + * This closes it by making the tools check each other. For each requirement duvet has matched + * to both an implementation and a test, mutate *only* the cited implementation region and run + * *only* the cited tests. A mutant that survives is a change to the code that claims to + * implement the requirement which the test claiming to check it does not notice. + * + * The unit is a (requirement, implementation, test) triple rather than a file, because that is + * what the claim is about. `cargo mutants -f ` would answer a weaker question -- "is this + * file tested" -- and would drown the signal in mutants belonging to requirements nobody cited. + */ + +// No imports, deliberately. `jsr:@std/...` would be fetched on first run, which puts this tool +// behind the network in exactly the situation it is most wanted -- a CI or nix sandbox with +// none -- for a path join and an argument parse. The same reasoning vendors the specifications +// under `.duvet/`; see development/code/spec-compliance.md. + +const join = (...parts: string[]) => parts.join("/").replaceAll(/\/+/g, "/"); +const REPO = new URL("..", import.meta.url).pathname.replace(/\/$/, ""); + +/** + * duvet emits no `type` key for the default annotation kind, which is the implementation + * citation. Spelling it here keeps the defaulting in one place. + */ +const IMPLEMENTATION = "CITATION"; +const TEST = "TEST"; + +/** A duvet citation line: `//= `, `//= type=`, or the quoted text `//# ...`. */ +const CITATION_LINE = /^\s*\/\/[=#]/; +const ATTRIBUTE_LINE = /^\s*#\[/; +/** `fn name(`, `pub fn name(`, `async fn name(`. The name is what nextest filters on. */ +const FN_LINE = /\bfn\s+([A-Za-z_][A-Za-z0-9_]*)/; +/** A mutant as cargo-mutants names it in `caught.txt` and friends: `path:line:col: what`. */ +const MUTANT_LINE = /^([^:]+):(\d+):(\d+): /; + +/** + * A mutant that survives its requirement's cited tests, and why that is the correct outcome. + * + * `development/code/mutation-testing.md` asks for every mutant to be *classified*, not killed: + * some are equivalent, and the cheapest way to turn one green is to assert whatever the code + * already does, which entrenches the behaviour instead of checking it. Without somewhere to + * record that judgement the interlock reports an equivalent mutant as decorative forever, and + * the pressure is to write the entrenching test. + * + * Kept here rather than in a data file for the reason `.cargo/mutants.toml` gives for holding + * cargo-mutants' exclusions: the reasoning stays in one place, next to what acts on it. + */ +interface Accepted { + /** The requirement, so an accept cannot silently cover a different citation. */ + requirement: string; + /** cargo-mutants' own stable name -- `: `, with no line or column, so that the + * entry survives the function being moved or reformatted. */ + mutant: string; + reason: string; +} + +const ACCEPTED: Accepted[] = [ + { + requirement: "https://www.rfc-editor.org/rfc/rfc4787#section-4.1", + mutant: + "nat/src/masquerade/apalloc/alloc.rs: replace match guard e.is_exhaustion() with true in IpAllocator::allocate", + reason: + "Equivalent under the allocator's invariants. `reuse_allocated_ip` skips `NoFreePort` " + + "and loops, so the only error it can return from a well-formed pool is `NoFreeIp`, " + + "which is exhaustion; the non-exhaustion arm is defensive depth against an internal " + + "inconsistency. The whole 210-test nat suite passes with the guard forced to `true`. " + + "The guard is kept because a future allocator that can fail for a non-exhaustion reason " + + "must not draw a second public address for a host that already holds one.", + }, + // The same code answers to both specifications, so the judgement is recorded against each. + ...["https://www.rfc-editor.org/rfc/rfc4787#section-4.2.1", + "https://www.rfc-editor.org/rfc/rfc5382#section-8"].map((requirement) => ({ + requirement, + mutant: + "nat/src/masquerade/apalloc/port_alloc.rs: replace < with <= in Bitmap256::allocate_port_from_bitmap", + reason: + "Equivalent on the second half, and unreachable rather than untested. `allocate_port` " + + "only enters the bitmap when `!is_full()`, and `bitmap_full()` is both halves at " + + "`u128::MAX`. Reaching the second-half branch means the first half is already full, so a " + + "block that is not full has a zero in the second half and `trailing_ones() < 128` always " + + "holds; `ones == 128`, the only value the two operators disagree on, cannot occur. The " + + "whole 210-test nat suite passes with `<=` applied. Note the same mutant on the *first* " + + "half is caught, and correctly: a block with the first half full and the second free is " + + "ordinary, so `ones == 128` is reachable there and `1u128 << 128` overflows.", + })), +]; + +/** cargo-mutants' stable name for a mutant: its `: `, dropping line and column. */ +function stableName(mutant: string): string { + const match = MUTANT_LINE.exec(mutant); + if (!match) return mutant; + return `${match[1]}: ${mutant.slice(match[0].length)}`; +} + +/** A half-open line range `[start, end)`, 1-indexed, in `path`. */ +interface Region { + path: string; + start: number; + end: number; +} + +const showRegion = (r: Region) => `${r.path}:${r.start}-${r.end - 1}`; + +/** One requirement, the code cited as implementing it, and the tests cited as checking it. */ +interface Triple { + index: string; + spec: string; + section: string; + implementations: Region[]; + tests: string[]; + testSites: Region[]; +} + +interface Annotation { + source: string; + target_path: string; + target_section: string; + line: number; + type?: string; +} + +interface Status { + citation?: number; + test?: number; + related?: number[]; +} + +interface Report { + annotations: Annotation[]; + statuses: Record; +} + +async function run( + cmd: string, + args: string[], + env: Record = {}, +): Promise<{ code: number; stdout: string; stderr: string }> { + const output = await new Deno.Command(cmd, { + args, + cwd: REPO, + env, + stdout: "piped", + stderr: "piped", + }).output(); + const decode = (b: Uint8Array) => new TextDecoder().decode(b); + return { + code: output.code, + stdout: decode(output.stdout), + stderr: decode(output.stderr), + }; +} + +/** + * How many times the cited tests executed each line of the files they are cited against. + * + * Cheap next to mutation -- one instrumented run rather than one build and test cycle per + * mutant -- and it answers a question mutation cannot separate out on its own: a mutant that + * survives because the test never reached it needs a different input, while one that survives + * although the test ran straight through it needs a different assertion. + * + * The instrumented build gets its own target directory. Sharing the default one would make + * every ordinary `cargo test` afterwards rebuild from scratch, because the coverage flags + * differ. + */ +async function lineCounts( + pkg: string, + tests: string[], + paths: string[], + outDir: string, + sharedTarget: string, +): Promise | null> { + // Added to the parent environment rather than replacing it: `Deno.Command` inherits unless + // `clearEnv` is set, which keeps PATH and the toolchain selection intact without this tool + // needing `--allow-env`. + const env = { + LLVM_COV: join(REPO, "devroot", "bin", "llvm-cov"), + LLVM_PROFDATA: join(REPO, "devroot", "bin", "llvm-profdata"), + // Shared across requirements, not per-requirement: the instrumented build is the slow part + // and is identical for every citation in the same package. + CARGO_TARGET_DIR: sharedTarget, + }; + // `--output-path` will not create the directory it writes into. + await Deno.mkdir(outDir, { recursive: true }); + + // Returning null rather than an empty map matters: an empty map is indistinguishable from + // "nothing was executed", and would relabel every survivor as unreached -- a confident wrong + // answer, which is worse than no answer. + const step = async (args: string[]) => { + const { code, stderr } = await run("cargo", args, env); + if (code !== 0) { + console.log(` warning: coverage step \`${args.join(" ")}\` failed`); + const tail = stderr.trim().split("\n").slice(-3).join("\n "); + if (tail) console.log(` ${tail}`); + } + return code === 0; + }; + + // Only the profiles, not the build: the build is the slow part and is reusable. + const jsonPath = join(outDir, "coverage.json"); + const ok = + await step(["llvm-cov", "clean", "--workspace", "--profraw-only"]) && + await step(["llvm-cov", "--no-report", "--branch", "nextest", "-p", pkg, ...tests]) && + await step(["llvm-cov", "report", "--json", "--output-path", jsonPath]); + if (!ok) return null; + + const counts = new Map(); + let report; + try { + report = JSON.parse(await Deno.readTextFile(jsonPath)); + } catch (e) { + console.log(` warning: coverage report unreadable: ${e}`); + return null; + } + for (const file of report.data?.[0]?.files ?? []) { + const match = paths.find((p) => file.filename.endsWith(p)); + if (!match) continue; + // A segment is [line, column, count, hasCount, ...]. One line can carry several, so the + // largest wins: the question here is only whether execution ever got there. + for (const [line, _col, count, hasCount] of file.segments ?? []) { + if (!hasCount) continue; + const key = `${match}:${line}`; + counts.set(key, Math.max(counts.get(key) ?? 0, count)); + } + } + return counts; +} + +/** + * Run `duvet report` and read back its JSON. + * + * Always regenerated rather than cached: a stale report would silently check the previous + * commit's citations, which is the failure this tool exists to catch. + */ +async function duvetReport(jsonPath: string): Promise { + const { code } = await run("duvet", ["report", "--json", jsonPath]); + if (code !== 0) throw new Error("duvet report failed"); + return JSON.parse(await Deno.readTextFile(jsonPath)); +} + +/** + * The first index at or after `start` that is not part of a citation comment. + * + * Ordinary `//` comments between the citation and the code it annotates are prose about the + * citation -- every existing site has some -- so they are skipped too, as are the attributes + * and doc comments that precede an item. + */ +function skipCitationBlock(lines: string[], start: number): number { + let i = start; + while (i < lines.length) { + const line = lines[i]; + const stripped = line.trim(); + if ( + stripped === "" || CITATION_LINE.test(line) || + stripped.startsWith("//") || ATTRIBUTE_LINE.test(line) + ) { + i += 1; + continue; + } + break; + } + return i; +} + +/** + * The line index just past the item or statement beginning at `start`. + * + * Brace matching rather than a Rust parser: the cited construct is a statement or a single + * item, and the alternative -- taking the whole enclosing function -- would attribute mutants + * to a requirement that does not cover them. + */ +function itemExtent(lines: string[], start: number): number { + let depth = 0; + let opened = false; + let i = start; + while (i < lines.length) { + // Strings and char literals containing braces would break this. None of the cited sites + // has one, and a miscount shows up as a region that fails to bound a mutant, not as a + // silently wrong verdict. + const code = lines[i].split("//")[0]; + const opens = (code.match(/\{/g) ?? []).length; + depth += opens - (code.match(/\}/g) ?? []).length; + if (opens > 0) opened = true; + i += 1; + if (opened && depth <= 0) return i; + if (!opened && code.trimEnd().endsWith(";")) return i; + } + return i; +} + +async function readLines(path: string): Promise { + return (await Deno.readTextFile(path)).split("\n"); +} + +/** The code a `type=implementation` citation at `line` annotates. */ +async function implementationRegion( + source: string, + line: number, +): Promise { + const lines = await readLines(join(REPO, source)); + const start = skipCitationBlock(lines, line - 1); + return { path: source, start: start + 1, end: itemExtent(lines, start) + 1 }; +} + +/** The name of the test function a `type=test` citation at `line` annotates. */ +async function testName( + source: string, + line: number, +): Promise<[string, Region]> { + const lines = await readLines(join(REPO, source)); + const start = skipCitationBlock(lines, line - 1); + for (let i = start; i < Math.min(start + 5, lines.length); i += 1) { + const match = FN_LINE.exec(lines[i]); + if (match) { + return [match[1], { + path: source, + start: i + 1, + end: itemExtent(lines, i) + 1, + }]; + } + } + throw new Error(`${source}:${line}: a type=test citation does not precede a function`); +} + +/** Every requirement duvet has matched to both an implementation and a test. */ +async function collect(report: Report): Promise { + const triples: Triple[] = []; + for (const [index, status] of Object.entries(report.statuses)) { + if (status.citation === undefined || status.test === undefined) continue; + let triple: Triple | undefined; + for (const j of status.related ?? []) { + const annotation = report.annotations[j]; + triple ??= { + index, + spec: annotation.target_path, + section: annotation.target_section, + implementations: [], + tests: [], + testSites: [], + }; + const kind = annotation.type ?? IMPLEMENTATION; + if (kind === IMPLEMENTATION) { + triple.implementations.push( + await implementationRegion(annotation.source, annotation.line), + ); + } else if (kind === TEST) { + const [name, site] = await testName(annotation.source, annotation.line); + triple.tests.push(name); + triple.testSites.push(site); + } + } + if (triple && triple.implementations.length && triple.tests.length) { + triples.push(triple); + } + } + return triples; +} + +/** The cargo package owning a workspace-relative source path. */ +async function packageOf(path: string): Promise { + const manifest = join(REPO, path.split("/")[0], "Cargo.toml"); + for (const line of (await Deno.readTextFile(manifest)).split("\n")) { + if (line.startsWith("name")) { + return line.split("=")[1].trim().replaceAll('"', ""); + } + } + throw new Error(`${manifest}: no package name`); +} + +/** + * A `cargo mutants --re` pattern narrowing the run towards `regions`. + * + * Best effort only, for cost. It cannot be trusted as the selector: cargo-mutants applies + * neither `--re` nor `--exclude-re` nor `.cargo/mutants.toml` to `StructField`-genre mutants, + * so a pattern matching nothing still yields every "delete field X from struct Y" in the + * package. `selectRegions` is what actually decides which mutants count. + */ +function mutantFilter(regions: Region[]): string { + const byFile = new Map>(); + for (const region of regions) { + const lines = byFile.get(region.path) ?? new Set(); + for (let n = region.start; n < region.end; n += 1) lines.add(n); + byFile.set(region.path, lines); + } + const parts = [...byFile.entries()].sort().map(([path, lines]) => { + const escaped = path.replaceAll(".", "\\."); + return `${escaped}:(${[...lines].sort((a, b) => a - b).join("|")}):`; + }); + return `^(?:${parts.join("|")})`; +} + +/** + * The mutants that fall inside a cited implementation region. + * + * The verdict is computed here rather than delegated to cargo-mutants' own filters, because + * those leak (see `mutantFilter`). A mutant outside every cited region says nothing about the + * citation under test and is discarded rather than counted either way. + */ +function selectRegions(mutants: string[], regions: Region[]): string[] { + return mutants.filter((mutant) => { + const match = MUTANT_LINE.exec(mutant); + if (!match) return false; + const [, path, line] = match; + return regions.some((r) => + r.path === path && r.start <= Number(line) && Number(line) < r.end + ); + }); +} + +/** + * Whether the cited test names match at least one test nextest will run. + * + * Without this the tool's central failure mode is silent: a renamed test makes the filter match + * nothing, nextest exits 0 having run nothing, every mutant survives, and a citation that is + * merely stale is reported as decorative. + */ +async function selectsATest(pkg: string, tests: string[]): Promise { + const { code, stdout } = await run("cargo", [ + "nextest", + "list", + "-p", + pkg, + ...tests, + ]); + return code === 0 && tests.some((name) => stdout.includes(name)); +} + +interface Result { + outcome: + | "held" + | "decorative" + | "uncovered" + | "stale" + | "unsupported" + | "no-mutants"; + detail?: string; + caught?: string[]; + missed?: string[]; + accepted?: Accepted[]; + /** Survivors on a line the cited tests never executed: the test needs a different input. */ + unreached?: string[]; + /** Survivors on a line they did execute: the test needs a different assertion. */ + tolerated?: string[]; + unviable?: string[]; + timeout?: string[]; +} + +/** Mutate the cited implementation, run only the cited tests, and report survivors. */ +async function runTriple( + triple: Triple, + output: string, + jobs: number, + used: Set, +): Promise { + const pkg = await packageOf(triple.implementations[0].path); + const testPackages = new Set( + await Promise.all(triple.testSites.map((s) => packageOf(s.path))), + ); + if (testPackages.size !== 1 || !testPackages.has(pkg)) { + return { + outcome: "unsupported", + detail: `implementation in ${pkg}, tests in ${[...testPackages].join(", ")}`, + }; + } + if (!await selectsATest(pkg, triple.tests)) { + return { + outcome: "stale", + detail: `no test in ${pkg} matches ${triple.tests.join(", ")}`, + }; + } + + const outDir = join(output, `requirement-${triple.index}`); + const files = [...new Set(triple.implementations.map((r) => r.path))].sort(); + + // Coverage first, because it is cheap and can settle the question outright. A cited test that + // never executes a single line of the cited region cannot be testing the requirement, whatever + // the mutants would have said, and skipping them saves the build-and-test cycle per mutant. + const counts = await lineCounts( + pkg, + triple.tests, + files, + outDir, + join(output, "cov-target"), + ); + const regionLines = triple.implementations.flatMap((r) => + Array.from({ length: r.end - r.start }, (_, i) => `${r.path}:${r.start + i}`) + ); + const executable = regionLines.filter((key) => counts?.has(key)); + const executed = executable.filter((key) => (counts?.get(key) ?? 0) > 0); + if (counts && executable.length && !executed.length) { + return { + outcome: "uncovered", + detail: `${triple.tests.join(", ")} never executes ${ + triple.implementations.map(showRegion).join(", ") + }; the citation cannot be testing this requirement`, + }; + } + await run("cargo", [ + "mutants", + "--package", + pkg, + "--test-tool", + "nextest", + ...files.flatMap((path) => ["--file", path]), + "--re", + mutantFilter(triple.implementations), + "--output", + outDir, + "--jobs", + String(jobs), + "--no-times", + "--", + ...triple.tests, + ]); + + const read = async (name: string): Promise => { + const path = join(outDir, "mutants.out", `${name}.txt`); + let text: string; + try { + text = await Deno.readTextFile(path); + } catch { + return []; + } + return selectRegions( + text.split("\n").filter((l) => l.trim() !== ""), + triple.implementations, + ); + }; + + const [caught, survived, unviable, timeout] = await Promise.all( + ["caught", "missed", "unviable", "timeout"].map(read), + ); + + // Split the survivors into the ones somebody has judged equivalent and the ones nobody has. + // Only the second kind is a finding. + const accepted: Accepted[] = []; + const missed = survived.filter((mutant) => { + const entry = ACCEPTED.find((a) => + a.requirement === `${triple.spec}#${triple.section}` && + a.mutant === stableName(mutant) + ); + if (entry) { + accepted.push(entry); + used.add(entry); + return false; + } + return true; + }); + // A region whose every mutant is unviable or timed out is not evidence either way: nothing + // ran that the test could have noticed. Reporting that as "held" would credit the citation + // for a check that never happened, which is the exact failure this tool exists to catch. + if (!caught.length && !missed.length) { + const why = unviable.length || timeout.length + ? `${unviable.length} unviable and ${timeout.length} timed out, so none could be tested` + : "generated no mutants"; + return { + outcome: "no-mutants", + detail: `the cited region ${ + triple.implementations.map(showRegion).join(", ") + } ${why}; the citation cannot be checked this way`, + unviable, + timeout, + }; + } + // Split the survivors by whether the cited tests got to them at all. The two need opposite + // fixes, and telling them apart by hand meant reading a coverage report anyway. + // With no coverage there is no basis to split them, and guessing would relabel every + // survivor as unreached. Leave both buckets empty; the reporter then lists them plainly. + const reached = (mutant: string) => { + const match = MUTANT_LINE.exec(mutant); + return match ? (counts!.get(`${match[1]}:${match[2]}`) ?? 0) > 0 : false; + }; + return { + outcome: missed.length ? "decorative" : "held", + caught, + missed, + accepted, + unreached: counts ? missed.filter((m) => !reached(m)) : [], + tolerated: counts ? missed.filter(reached) : [], + unviable, + timeout, + }; +} + +/** `--flag value` and `--flag`, which is all this tool needs. */ +function parseArgs(argv: string[]) { + const args = { + list: false, + help: false, + only: [] as string[], + jobs: 4, + output: join(REPO, "target", "spec-interlock"), + json: "/tmp/duvet-interlock.json", + }; + for (let i = 0; i < argv.length; i += 1) { + const flag = argv[i]; + const value = () => { + const next = argv[++i]; + if (next === undefined) throw new Error(`${flag} needs a value`); + return next; + }; + switch (flag) { + case "--list": args.list = true; break; + case "--help": args.help = true; break; + case "--only": args.only.push(value()); break; + case "--jobs": args.jobs = Number(value()); break; + case "--output": args.output = value(); break; + case "--json": args.json = value(); break; + default: throw new Error(`unknown argument ${flag}`); + } + } + return args; +} + +async function main(): Promise { + const args = parseArgs(Deno.args); + if (args.help) { + console.log( + "usage: spec-interlock.ts [--list] [--only ]... [--jobs N]", + ); + return 0; + } + + const report = await duvetReport(args.json); + let triples = await collect(report); + if (args.only.length) triples = triples.filter((t) => args.only.includes(t.index)); + + if (args.list) { + for (const triple of triples) { + console.log(`${triple.spec}#${triple.section} (requirement ${triple.index})`); + for (const region of triple.implementations) { + console.log(` implementation ${showRegion(region)}`); + } + triple.tests.forEach((name, i) => { + console.log(` test ${name} [${showRegion(triple.testSites[i])}]`); + }); + console.log(); + } + console.log( + `${triples.length} requirements carry both an implementation and a test`, + ); + return 0; + } + + await Deno.mkdir(args.output, { recursive: true }); + const used = new Set(); + let failures = 0; + for (const triple of triples) { + console.log(`==> ${triple.spec}#${triple.section} (requirement ${triple.index})`); + const result = await runTriple(triple, args.output, Number(args.jobs), used); + if (result.outcome === "held") { + console.log( + ` held: ${result.caught!.length} mutants in the cited region, all caught by ${ + triple.tests.join(", ") + }`, + ); + } else if (result.outcome === "decorative") { + failures += 1; + console.log( + ` DECORATIVE: ${result.missed!.length} mutants survive ${triple.tests.join(", ")}`, + ); + if (result.unreached?.length) { + console.log( + ` unreached (${result.unreached.length}) -- the test never runs these lines, so`, + ); + console.log(` closing them means changing what it feeds, not what it asserts:`); + for (const mutant of result.unreached) console.log(` ${mutant}`); + } + if (result.tolerated?.length) { + console.log( + ` tolerated (${result.tolerated.length}) -- the test runs these lines and passes`, + ); + console.log(` anyway, so either an assertion is missing or the mutant is equivalent:`); + for (const mutant of result.tolerated) console.log(` ${mutant}`); + } + if (!result.unreached?.length && !result.tolerated?.length) { + console.log(` (unclassified -- coverage was not collected)`); + for (const mutant of result.missed!) console.log(` ${mutant}`); + } + } else { + failures += 1; + console.log(` ${result.outcome.toUpperCase()}: ${result.detail}`); + } + // Counted but not listed: an unviable mutant is one that did not compile, which says + // nothing about the test. The count is kept because a region that is *entirely* unviable + // is the `no-mutants` case above, and that distinction is worth being able to see. + const skipped = (result.unviable?.length ?? 0) + (result.timeout?.length ?? 0); + if (skipped && result.outcome !== "no-mutants") { + console.log(` (${skipped} unviable or timed out, not counted either way)`); + } + // Printed on every run, not hidden. An accepted mutant is a judgement somebody made, and it + // should be as visible as the finding it replaced -- otherwise the list only ever grows. + for (const entry of result.accepted ?? []) { + console.log(` accepted: ${entry.mutant}`); + console.log(` ${entry.reason}`); + } + } + + // An accept that matches nothing is worse than no accept: it reads as a considered judgement + // while silently covering a mutant that no longer exists, and it would go on hiding whatever + // takes that name next. + const stale = ACCEPTED.filter((entry) => !used.has(entry)); + const checkedAll = !args.only.length; + if (stale.length && checkedAll) { + failures += stale.length; + console.log(); + for (const entry of stale) { + console.log(`STALE ACCEPT: no surviving mutant matches`); + console.log(` requirement ${entry.requirement}`); + console.log(` mutant ${entry.mutant}`); + } + } + + console.log(); + console.log( + `${triples.length - failures}/${triples.length} requirements hold their citations`, + ); + return failures ? 1 : 0; +} + +if (import.meta.main) Deno.exit(await main());