From 99ed1898d526a066732098780f049f12d752afc3 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Mon, 17 Aug 2026 20:41:33 -0600 Subject: [PATCH 01/37] feat(clock): Read the clock through a facade, and lint for it Anything with a timeout is untestable in a useful way if its deadline comes from the wall clock: the test either sleeps for real -- seconds of CI time per case, flaky under emulation -- or it does not test expiry at all. The flow table's expiry, the masquerade and port-forwarding timeouts, the stats delivery schedule and the FRR reconnect timers are all in that position. tokio can already pause and advance time, and the dataplane already runs on tokio. What stopped that from working was an asymmetry: deadline = std::time::Instant::now() + timeout // does not move sleep_until(tokio::time::Instant::from_std(deadline)) // does move The two agree at the moment a test pauses the clock and diverge immediately after, so a paused clock bought exactly **one** time step. Measured: a flow opened after a single `advance` is born with a deadline already in the past and is dead on arrival, which looks exactly like a masquerade bug and is not one. Three call sites define every flow lifetime, and pointing those three at `tokio::time::Instant::now()` makes multi-epoch time control work today. It also breaks confusingly later, the first time someone innocently writes `Instant::now()` in a fourth place -- the symptom is a timeout test behaving strangely under a paused clock, which is a long way from the cause. So: one place the workspace reads the time, and a lint that refuses the others. `.semgrep/rules/no-std-time-direct.yaml` is the same shape as `no-std-sync-direct.yaml`, for the same reason -- clippy sees the facade's re-exports by canonical path, so the rule catches the call sites. `clock` follows `concurrency`: a cargo feature selects the backend, not a `--cfg`. That is what `concurrency` actually does, and it matters here because a `--cfg` cannot pull in an optional dependency -- tokio would become a hard dependency of everything that reads a clock. With a feature, production `net` has no tokio at all. * `clock::now()` -- `std` in production, tokio's pausable clock under `virtual`. Enabled in the dev-dependencies of every crate that reads a clock, so a test build gets the routed clock across the whole graph rather than in one crate. * `clock::system_now()` -- wall clock, deliberately **not** routed. tokio pauses its monotonic clock, not the system clock, and the values that use it are timestamps reported outwards rather than deadlines anything waits on. It lives in the facade so the lint has one chokepoint. * `Duration` is exempt and the lint says so. A duration is a plain value with no clock in it. Production cost is zero, and checked rather than assumed. From tokio 1.53.1: #[cfg(not(feature = "test-util"))] mod variant { pub(super) fn now() -> Instant { Instant::from_std(std::time::Instant::now()) } } `cargo tree -p dataplane -e features -i tokio | grep -c test-util` is 0: dev- dependency features do not unify into the production binary. 47 `Instant::now()` and 3 `SystemTime::now()` sites across 22 files and 11 crates, all migrated. Whole workspace builds, 100 test binaries pass, clippy clean at `-D warnings`. The lint was break-tested both ways -- a planted `std::time::Instant::now()` and a planted bare `Instant::now()` are both caught, and the tree is clean with them removed. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 33eddb134c7d449c3c8fae9bf5589301f7df36e5) --- .semgrep/rules/no-std-time-direct.yaml | 40 ++++++ Cargo.lock | 18 +++ Cargo.toml | 2 + acl-filter/Cargo.toml | 2 + acl-filter/src/tests.rs | 4 +- acl/Cargo.toml | 2 + acl/src/dpdk/dyn_table.rs | 3 +- clock/Cargo.toml | 18 +++ clock/src/lib.rs | 121 +++++++++++++++++++ config/Cargo.toml | 2 + config/src/gwconfig.rs | 4 +- dataplane/Cargo.toml | 2 + dataplane/src/drivers/kernel/mod.rs | 6 +- development/code/README.md | 5 + flow-entry/Cargo.toml | 2 + flow-entry/src/flow_table/concurrent_fuzz.rs | 4 +- flow-entry/src/flow_table/nf_lookup.rs | 8 +- flow-entry/src/flow_table/table.rs | 30 +++-- flow-filter/Cargo.toml | 2 + flow-filter/src/tests.rs | 4 +- nat/Cargo.toml | 4 +- nat/src/masquerade/nf.rs | 4 +- nat/src/masquerade/test.rs | 4 +- nat/src/portfw/nf.rs | 3 +- nat/src/portfw/test.rs | 4 +- net/Cargo.toml | 2 + net/src/flows/display.rs | 3 +- net/src/flows/flow_info.rs | 2 +- routing/Cargo.toml | 2 + routing/src/bmp/bmp_render.rs | 2 +- routing/src/evpn/rmac.rs | 2 +- routing/src/fib/test.rs | 3 +- routing/src/frr/frrmi.rs | 4 +- routing/src/rib/vrf.rs | 4 +- routing/src/router/cpi.rs | 4 +- routing/src/router/rio.rs | 8 +- routing/src/router/rpc_adapt.rs | 3 +- stats/Cargo.toml | 2 + stats/src/dpstats.rs | 16 +-- tracectl/Cargo.toml | 2 + tracectl/src/throttle.rs | 2 +- 41 files changed, 287 insertions(+), 72 deletions(-) create mode 100644 .semgrep/rules/no-std-time-direct.yaml create mode 100644 clock/Cargo.toml create mode 100644 clock/src/lib.rs diff --git a/.semgrep/rules/no-std-time-direct.yaml b/.semgrep/rules/no-std-time-direct.yaml new file mode 100644 index 0000000000..7ee3c72eaf --- /dev/null +++ b/.semgrep/rules/no-std-time-direct.yaml @@ -0,0 +1,40 @@ +# Source-level enforcement of the `clock` facade. +# +# The workspace reads the monotonic clock through `clock::now()` so that a test +# can pause and advance it. A direct `Instant::now()` compiles and passes review +# and then, under a paused clock, produces a deadline the timers do not share -- +# which surfaces as a timeout test behaving strangely rather than as an error. +# Same shape as no-std-sync-direct.yaml: clippy sees the facade's re-exports by +# canonical path, so this catches the call sites instead. +rules: + - id: rust-no-direct-clock-read + languages: [rust] + severity: ERROR + message: | + Read the clock via `clock::now()` (or `clock::system_now()` for wall + time), not `Instant::now()` / `SystemTime::now()`. The workspace's + `clock` facade reads `std` in production and tokio's pausable clock + under the `virtual` feature; reading `std` directly pins the deadline + to the wall clock while the timer waiting on it follows tokio's, so a + test that advances time sees the two diverge. + + `Duration` needs no facade -- it is a plain value with no clock in it, + and `clock` re-exports it only for convenience. + paths: + exclude: + # CodeQL test fixtures deliberately contain the violations this rule + # detects, including evasions this rule misses. See .codeql/README.md. + - .codeql/tests/ + # The facade itself, which is the one place allowed to read a clock. + - clock/src/ + # Real OS threads on a real clock, deliberately: these test the + # quiescent protocol's interaction with the scheduler, not a timeout. + - concurrency/tests/ + pattern-either: + - pattern: Instant::now() + - pattern: std::time::Instant::now() + - pattern: SystemTime::now() + - pattern: std::time::SystemTime::now() + # tokio's clock is what the facade routes to; reaching for it directly + # scatters the routing decision across the workspace again. + - pattern: tokio::time::Instant::now() diff --git a/Cargo.lock b/Cargo.lock index bcfb506431..b5be7107c5 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1177,6 +1177,7 @@ dependencies = [ "axum-server", "dataplane-acl-filter", "dataplane-args", + "dataplane-clock", "dataplane-common", "dataplane-concurrency", "dataplane-config", @@ -1223,6 +1224,7 @@ dependencies = [ "bolero", "criterion", "dataplane-acl", + "dataplane-clock", "dataplane-concurrency", "dataplane-dpdk", "dataplane-lookup", @@ -1237,6 +1239,7 @@ version = "0.25.2" dependencies = [ "bolero", "dataplane-acl", + "dataplane-clock", "dataplane-common", "dataplane-concurrency", "dataplane-config", @@ -1296,6 +1299,13 @@ dependencies = [ "thiserror", ] +[[package]] +name = "dataplane-clock" +version = "0.25.2" +dependencies = [ + "tokio", +] + [[package]] name = "dataplane-common" version = "0.25.2" @@ -1336,6 +1346,7 @@ dependencies = [ "bolero", "caps", "chrono", + "dataplane-clock", "dataplane-common", "dataplane-concurrency", "dataplane-k8s-intf", @@ -1414,6 +1425,7 @@ dependencies = [ "ahash", "bolero", "dashmap", + "dataplane-clock", "dataplane-common", "dataplane-concurrency", "dataplane-net", @@ -1434,6 +1446,7 @@ version = "0.25.2" dependencies = [ "bolero", "dataplane-acl", + "dataplane-clock", "dataplane-common", "dataplane-concurrency", "dataplane-config", @@ -1686,6 +1699,7 @@ dependencies = [ "arc-swap", "bnum", "bolero", + "dataplane-clock", "dataplane-common", "dataplane-concurrency", "dataplane-config", @@ -1722,6 +1736,7 @@ dependencies = [ "bitflags 2.13.1", "bolero", "bytecheck", + "dataplane-clock", "dataplane-common", "dataplane-concurrency", "dataplane-fixed-size", @@ -1774,6 +1789,7 @@ dependencies = [ "chrono", "dataplane-args", "dataplane-cli", + "dataplane-clock", "dataplane-common", "dataplane-concurrency", "dataplane-config", @@ -1812,6 +1828,7 @@ version = "0.25.2" dependencies = [ "arrayvec", "bolero", + "dataplane-clock", "dataplane-concurrency", "dataplane-net", "dataplane-pipeline", @@ -1859,6 +1876,7 @@ name = "dataplane-tracectl" version = "0.25.2" dependencies = [ "color-eyre", + "dataplane-clock", "dataplane-common", "dataplane-concurrency", "linkme", diff --git a/Cargo.toml b/Cargo.toml index 686c76a128..4683793b99 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -5,6 +5,7 @@ members = [ "acl-filter", "args", "cli", + "clock", "common", "concurrency", "concurrency-macros", @@ -71,6 +72,7 @@ acl-filter = { path = "./acl-filter", package = "dataplane-acl-filter", features args = { path = "./args", package = "dataplane-args", features = [] } cli = { path = "./cli", package = "dataplane-cli", features = [] } common = { path = "./common", package = "dataplane-common", features = [] } +clock = { path = "./clock", package = "dataplane-clock", features = [] } concurrency = { path = "./concurrency", package = "dataplane-concurrency", features = [] } concurrency-macros = { path = "./concurrency-macros", package = "dataplane-concurrency-macros", features = [] } config = { path = "./config", package = "dataplane-config", features = [] } diff --git a/acl-filter/Cargo.toml b/acl-filter/Cargo.toml index 232547ea03..a05c3cc0a1 100644 --- a/acl-filter/Cargo.toml +++ b/acl-filter/Cargo.toml @@ -7,6 +7,7 @@ version.workspace = true [dependencies] acl = { workspace = true } +clock = { workspace = true } common = { workspace = true } concurrency = { workspace = true } config = { workspace = true } @@ -22,6 +23,7 @@ tracectl = { workspace = true } tracing = { workspace = true } [dev-dependencies] +clock = { workspace = true, features = ["virtual"] } # The reference (linear-scan) ACL backend is the differential-test oracle and drives the fast, # EAL-free semantic suite. It is `cfg(test)`-gated in the source, so it is never part of a # production build; this dev-dep just makes `acl::reference` available to test builds. diff --git a/acl-filter/src/tests.rs b/acl-filter/src/tests.rs index b545c7bb12..24ab8ad172 100644 --- a/acl-filter/src/tests.rs +++ b/acl-filter/src/tests.rs @@ -36,7 +36,7 @@ use concurrency::sync::Arc; use pipeline::NetworkFunction; use std::net::{Ipv4Addr, Ipv6Addr}; -use std::time::{Duration, Instant}; +use std::time::Duration; // VNIs and IP ranges used by the standard two-VPC peering (vpc1 <-> vpc2). The manifest names // ("vpc1"/"vpc2") double as the ACL rule `from`/`to` endpoints. @@ -670,7 +670,7 @@ fn ipv6_allow_and_default_deny() { // reply's weak `related` reference can still be upgraded. fn attach_related_flow(reply: &mut Packet, fwd_key: FlowKey) -> Arc { let reply_key = FlowKey::try_from(&*reply).unwrap(); - let expiry = Instant::now() + Duration::from_secs(60); + let expiry = clock::now() + Duration::from_secs(60); let (fwd_flow, reply_flow) = FlowInfo::related_pair( expiry, fwd_key, diff --git a/acl/Cargo.toml b/acl/Cargo.toml index 41eee1cafe..f9d01cb8a9 100644 --- a/acl/Cargo.toml +++ b/acl/Cargo.toml @@ -12,6 +12,7 @@ reference = [] [dependencies] arrayvec = { workspace = true, default-features = true } +clock = { workspace = true } concurrency = { workspace = true, features = [] } dpdk = { workspace = true, optional = true } lookup = { workspace = true, features = [] } @@ -20,6 +21,7 @@ net = { workspace = true, features = [] } thiserror = { workspace = true } [dev-dependencies] +clock = { workspace = true, features = ["virtual"] } # Enable the "reference" backend for this crate's own tests and benches. It is a non-default feature # (production links only the rte_acl backend), but the integration tests and benches # differential-test against it, so make it available whenever test/bench targets are built. diff --git a/acl/src/dpdk/dyn_table.rs b/acl/src/dpdk/dyn_table.rs index 744aa9e4e0..0635215bea 100644 --- a/acl/src/dpdk/dyn_table.rs +++ b/acl/src/dpdk/dyn_table.rs @@ -494,7 +494,6 @@ mod tests { use lookup::Lookup; use match_action::{Erased, ExactSpec, MaskSpec, MatchKey, PrefixSpec, RangeSpec}; use std::hint::black_box; - use std::time::Instant; #[derive(MatchKey, Debug, Clone, Copy)] struct FiveTuple { @@ -969,7 +968,7 @@ mod tests { let rules = make(n); let max = NonZero::new(u32::try_from(n).unwrap()).unwrap(); let rss_before = rss_kb(); - let t = Instant::now(); + let t = clock::now(); let res: Result, u32>, _> = install_table(&unique_name("cap"), max, rules); let dt = t.elapsed().as_secs_f64() * 1e3; diff --git a/clock/Cargo.toml b/clock/Cargo.toml new file mode 100644 index 0000000000..f1e3297404 --- /dev/null +++ b/clock/Cargo.toml @@ -0,0 +1,18 @@ +[package] +name = "dataplane-clock" +edition.workspace = true +license.workspace = true +publish.workspace = true +version.workspace = true + +[features] +default = [] +# Route the monotonic clock through tokio's, which a test can pause and advance. +# +# Off in production, so tokio is not a dependency at all there. Enabled by the +# dev-dependency declaration in every crate that reads the clock, so that a test +# build gets the routed clock across the whole graph rather than in one crate. +virtual = ["dep:tokio"] + +[dependencies] +tokio = { workspace = true, optional = true, features = ["test-util", "time"] } diff --git a/clock/src/lib.rs b/clock/src/lib.rs new file mode 100644 index 0000000000..57d07b0e9a --- /dev/null +++ b/clock/src/lib.rs @@ -0,0 +1,121 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +//! Backend-routed clock for the dataplane workspace. +//! +//! One place the workspace reads the time, so that a test can control it. The shape follows +//! [`concurrency`][concurrency], which routes synchronization primitives to `parking_lot` in +//! production and to a model checker under a feature; this routes the monotonic clock to `std` in +//! production and to tokio's pausable clock under the `virtual` feature. +//! +//! [concurrency]: https://docs.rs/dataplane-concurrency +//! +//! # Why this exists +//! +//! Anything with a timeout is untestable in a useful way if its deadline comes from the wall clock: +//! the test either sleeps for real -- seconds of CI time per case, and flaky under emulation -- or +//! it does not test expiry at all. The flow table's expiry, the masquerade and port-forwarding +//! timeouts, the stats delivery schedule and the FRR reconnect timers are all in that position. +//! +//! tokio can already pause and advance time, and the dataplane already runs on tokio. What stopped +//! that from working was an asymmetry: **the waits were on tokio's clock and the deadlines were on +//! `std`'s.** +//! +//! ```text +//! deadline = std::time::Instant::now() + timeout // does not move when a test advances time +//! sleep_until(tokio::time::Instant::from_std(deadline)) // does move +//! ``` +//! +//! Those agree at the moment a test pauses the clock and diverge immediately after, so a paused +//! clock bought exactly one time step: anything created after the first `advance` was born with a +//! deadline already in the past. Reading the clock here fixes that, because [`now`] follows whatever +//! clock the timers are following. +//! +//! # What needs routing, and what does not +//! +//! * **[`now`] does.** It is the only way to read the monotonic clock. +//! * **[`Duration`] does not.** A duration is a plain value with no clock in it -- `Duration::from_secs(5)` +//! means the same thing under any clock. It is re-exported here for convenience only, and the lint +//! does not ask for it. +//! * **Timers do not.** `tokio::time::sleep` and friends are already on tokio's clock, so they are +//! already controllable. `std::thread::sleep` is not, and a blocking sleep in async code is a +//! separate bug from this one. +//! * **[`system_now`] cannot be.** See below. +//! +//! # The lint is the point +//! +//! A facade nobody is obliged to use decays: the next `std::time::Instant::now()` compiles, passes +//! review, and then some unrelated timeout test starts behaving strangely under a paused clock. So +//! `.semgrep/rules/no-std-time-direct.yaml` refuses direct clock reads outside this crate, the same +//! way `no-std-sync-direct.yaml` refuses direct `std::sync` imports. + +#![deny(clippy::all, clippy::pedantic)] +#![deny(rustdoc::all)] +#![deny(unsafe_code)] + +pub use std::time::{Duration, Instant, SystemTime, SystemTimeError, TryFromFloatSecsError}; + +/// The current instant on the monotonic clock. +/// +/// Production reads `std::time::Instant::now()`. Under the `virtual` feature it reads tokio's clock, +/// which is the same clock until a test calls `tokio::time::pause` (or uses +/// `#[tokio::test(start_paused = true)]`), and thereafter is whatever that test has advanced it to. +/// +/// Safe to call with no tokio runtime in scope, and cheap: tokio's routed read is a single relaxed +/// atomic load until something in the process actually pauses the clock, and falls back to the real +/// clock on any thread with no clock installed. +#[must_use] +pub fn now() -> Instant { + #[cfg(feature = "virtual")] + { + tokio::time::Instant::now().into_std() + } + #[cfg(not(feature = "virtual"))] + { + Instant::now() + } +} + +/// The current wall-clock time. +/// +/// **Not routed, and cannot be.** tokio pauses its monotonic clock, not the system clock, and +/// nothing in the workspace would be well served by a fake `SystemTime` -- the values that use it are +/// timestamps reported outwards (a configuration's creation and application times, the router's +/// synchronization timestamp), not deadlines anything waits on. +/// +/// It lives here anyway so that the lint can name a single chokepoint for reading a clock of any +/// kind. If a test ever does need to control wall-clock time, this is where that would go, and the +/// call sites will not have to move. +#[must_use] +pub fn system_now() -> SystemTime { + SystemTime::now() +} + +#[cfg(test)] +mod tests { + use super::{Duration, now, system_now}; + + /// The clock moves forward, whichever backend is routed. + #[test] + fn now_is_monotonic() { + let first = now(); + let second = now(); + assert!(second >= first, "the monotonic clock went backwards"); + } + + /// Reading the clock off a tokio runtime is legal under either backend. + /// + /// Worth pinning: most of the workspace's tests have no runtime, and a routed read that panicked + /// without one would make this facade unusable in exactly the places it is meant to be invisible. + #[test] + fn now_works_with_no_runtime() { + let _ = now(); + let _ = system_now(); + } + + /// A duration is a value, not a clock reading, and is unaffected by routing. + #[test] + fn durations_are_plain_values() { + assert_eq!(Duration::from_secs(1).as_millis(), 1000); + } +} diff --git a/config/Cargo.toml b/config/Cargo.toml index e2a04255c4..b10bf16022 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -11,6 +11,7 @@ bolero = ["dep:bolero", "lpm/bolero"] [dependencies] # internal +clock = { workspace = true } common = { workspace = true } concurrency = { workspace = true } k8s-intf = { workspace = true } @@ -34,6 +35,7 @@ linkme = { workspace = true } tracectl = { workspace = true } [dev-dependencies] +clock = { workspace = true, features = ["virtual"] } # internal pipeline = { workspace = true } # should be removed w/ NAT lpm = { workspace = true, features = ["bolero", "testing"] } diff --git a/config/src/gwconfig.rs b/config/src/gwconfig.rs index 5061a4e91e..8a195fa8ad 100644 --- a/config/src/gwconfig.rs +++ b/config/src/gwconfig.rs @@ -36,7 +36,7 @@ impl GwConfigMeta { fn new(genid: GenId) -> Self { Self { genid, - create_t: SystemTime::now(), + create_t: clock::system_now(), apply_t: None, error: None, is_rollback: false, @@ -46,7 +46,7 @@ impl GwConfigMeta { /// Set the time when attempting to apply a configuration finished. //////////////////////////////////////////////////////////////////////////////// pub fn apply_time(&mut self) { - self.apply_t = Some(SystemTime::now()); + self.apply_t = Some(clock::system_now()); } //////////////////////////////////////////////////////////////////////////////// diff --git a/dataplane/Cargo.toml b/dataplane/Cargo.toml index c258c38e95..78f13d5cd6 100644 --- a/dataplane/Cargo.toml +++ b/dataplane/Cargo.toml @@ -18,6 +18,7 @@ args = { workspace = true } arrayvec = { workspace = true } axum = { workspace = true, features = ["http1", "tokio"] } axum-server = { workspace = true } +clock = { workspace = true } common = { workspace = true } concurrency = { workspace = true } config = { workspace = true } @@ -55,6 +56,7 @@ tracing-subscriber = { workspace = true, default-features = true } vpcmap = { workspace = true } [dev-dependencies] +clock = { workspace = true, features = ["virtual"] } # internal net = { workspace = true, features = ["test_buffer"] } routing = { workspace = true, features = ["testing"] } diff --git a/dataplane/src/drivers/kernel/mod.rs b/dataplane/src/drivers/kernel/mod.rs index 51a0072b32..3e6ca11ead 100644 --- a/dataplane/src/drivers/kernel/mod.rs +++ b/dataplane/src/drivers/kernel/mod.rs @@ -18,7 +18,7 @@ mod sockstats; mod worker; use std::ops::Add; -use std::time::{Duration, Instant}; +use std::time::Duration; use concurrency::sync::Arc; use concurrency::thread; @@ -252,7 +252,7 @@ impl DriverKernel { .collect(); // the next instant when the rx tasks watchdogs should be checked. - let mut next_watchdog_check = Instant::now().add(check_period); + let mut next_watchdog_check = clock::now().add(check_period); loop { // check if we must run. Otherwise (got cancelled) join all workers @@ -262,7 +262,7 @@ impl DriverKernel { // check the current time and decide if we should check whether the rx tasks patted the watchdogs. // If so, compute the next time we should check them again in the future. - let now = Instant::now(); + let now = clock::now(); let check_watchdog = now >= next_watchdog_check; if check_watchdog { while next_watchdog_check <= now { diff --git a/development/code/README.md b/development/code/README.md index 8a87f3dff8..bbd1fdd90e 100644 --- a/development/code/README.md +++ b/development/code/README.md @@ -17,6 +17,10 @@ valid operations and derive the oracles from that same algebra; see the [config If you need to handle errors, prefer `Result` types over panics in general, but see the [error handling guide][error] for details. +Never read a clock directly. `Instant::now()` and `SystemTime::now()` are refused by +`.semgrep/rules/no-std-time-direct.yaml`; use [`clock::now()`][clock] instead, so that a test can pause +and advance time. `Duration` is exempt -- it is a plain value with no clock in it. + ## Error handling If you need to [handle an error][error], follow the guidelines. @@ -24,6 +28,7 @@ If you need to [handle an error][error], follow the guidelines. [avoid-global-reasoning]: ./avoid-global-reasoning.md [property-based tests]: ./property-testing.md [config-algebra]: ./config-algebra-testing.md +[clock]: ../../clock/src/lib.rs [error]: ./error-handling.md ## Testing instructions diff --git a/flow-entry/Cargo.toml b/flow-entry/Cargo.toml index c0bd08e621..e76663374b 100644 --- a/flow-entry/Cargo.toml +++ b/flow-entry/Cargo.toml @@ -16,6 +16,7 @@ bolero = ["dep:bolero", "net/bolero"] [dependencies] ahash = { workspace = true, features = ["no-rng"] } bolero = { workspace = true, optional = true } +clock = { workspace = true } common = { workspace = true } concurrency = { workspace = true } dashmap = { workspace = true, features = ["raw-api"] } @@ -29,6 +30,7 @@ tracectl = { workspace = true } tracing = { workspace = true } [dev-dependencies] +clock = { workspace = true, features = ["virtual"] } bolero = { workspace = true, default-features = false } net = { workspace = true, features = ["bolero"] } tokio = { workspace = true, features = ["macros", "rt", "test-util", "time"] } diff --git a/flow-entry/src/flow_table/concurrent_fuzz.rs b/flow-entry/src/flow_table/concurrent_fuzz.rs index 9a757cd6b7..78924a974e 100644 --- a/flow-entry/src/flow_table/concurrent_fuzz.rs +++ b/flow-entry/src/flow_table/concurrent_fuzz.rs @@ -49,12 +49,12 @@ use concurrency::sync::atomic::{AtomicU8, Ordering}; use concurrency::sync::{Arc, Weak}; use concurrency::thread; // `spawn_scoped` is inherent on std's `Builder`, but supplied by `BuilderExt` under shuttle +use clock::Duration; #[cfg_attr(not(feature = "shuttle"), allow(unused_imports))] use concurrency::thread::BuilderExt; use net::FlowKey; use net::flows::{ExtractRef, FlowInfo, FlowInfoFlags}; use std::fmt; -use std::time::{Duration, Instant}; /// Stub [`FlowInfoItem`] payload: a single `Arc` that all flows /// share within one scenario. The blanket @@ -160,7 +160,7 @@ fn insert_flows(table: &FlowTable, keys: &[FlowKey], stub_status: &Arc // Far-future expiry so the per-flow timer never fires inside the test // window — we race the insert path, not the expiry path. (The timer task // is also cfg'd out entirely under shuttle.) - let expires_at = Instant::now() + Duration::from_hours(1); + let expires_at = clock::now() + Duration::from_hours(1); let flows: Vec> = match keys { [fwd_key, rev_key] => { let (fwd, rev) = FlowInfo::related_pair( diff --git a/flow-entry/src/flow_table/nf_lookup.rs b/flow-entry/src/flow_table/nf_lookup.rs index e741a05a42..d67a489628 100644 --- a/flow-entry/src/flow_table/nf_lookup.rs +++ b/flow-entry/src/flow_table/nf_lookup.rs @@ -57,6 +57,7 @@ impl NetworkFunction for FlowLookup { #[cfg(test)] mod test { + use clock::Duration; use concurrency::sync::Arc; use net::FlowKey; use net::buffer::PacketBufferMut; @@ -74,7 +75,6 @@ mod test { use pipeline::DynPipeline; use pipeline::NetworkFunction; use std::net::IpAddr; - use std::time::{Duration, Instant}; use tracing_test::traced_test; use crate::flow_table::FlowTable; @@ -101,7 +101,7 @@ mod test { // Insert matching flow entry let flow_key = FlowKey::try_from(&packet).unwrap(); - let flow_info = FlowInfo::new(flow_key, Instant::now() + Duration::from_secs(10)); + let flow_info = FlowInfo::new(flow_key, clock::now() + Duration::from_secs(10)); flow_table.insert(flow_info).unwrap(); // Ensure packet is tagged @@ -130,7 +130,7 @@ mod test { ) -> impl Iterator> + 'a { input.filter_map(move |packet| { let flow_key = FlowKey::try_from(&packet).unwrap(); - let flow_info = FlowInfo::new(flow_key, Instant::now() + self.timeout); + let flow_info = FlowInfo::new(flow_key, clock::now() + self.timeout); self.flow_table .insert(flow_info) .expect("insert in FlowInfoCreator should not fail"); @@ -196,7 +196,7 @@ mod test { let key_2 = FlowKey::try_from(&packet_2).unwrap(); // create a pair of related flow entries; flow_2 will get a longer timeout - let expires_at = tokio::time::Instant::now().into_std() + Duration::from_secs(2); + let expires_at = clock::now() + Duration::from_secs(2); let (flow_1, flow_2) = FlowInfo::related_pair( expires_at, key_1, diff --git a/flow-entry/src/flow_table/table.rs b/flow-entry/src/flow_table/table.rs index 0f15439e01..385747b20f 100644 --- a/flow-entry/src/flow_table/table.rs +++ b/flow-entry/src/flow_table/table.rs @@ -521,14 +521,13 @@ mod tests { #[concurrency_mode(std)] mod std_tests { use net::flows::FlowInfoFlags; - use std::time::Instant; use tracing_test::traced_test; use super::*; #[tokio::test] async fn test_flow_table_insert_and_remove() { - let now = Instant::now(); + let now = clock::now(); let five_seconds = Duration::new(5, 0); let five_seconds_from_now = now + five_seconds; @@ -552,13 +551,13 @@ mod tests { // start_paused so the timer task's sleep_until and the test's sleeps share tokio's // virtual clock; otherwise miri's slow interpretation can drift the wall clock far - // enough between Instant::now() and the first sleep that the deadline elapses early. - // Anchor `now` on the virtual clock too -- a std::Instant::now() here would be many + // enough between clock::now() and the first sleep that the deadline elapses early. + // Anchor `now` on the virtual clock too -- an unpaused clock read here would be many // real-time seconds past the paused baseline under miri, putting the deadline beyond // any virtual-time advance the test performs. #[tokio::test(start_paused = true)] async fn test_flow_table_timeout() { - let now = tokio::time::Instant::now().into_std(); + let now = clock::now(); let two_seconds = Duration::from_secs(2); let one_second = Duration::from_secs(1); @@ -590,7 +589,7 @@ mod tests { #[tokio::test] async fn test_flow_table_entry_replaced_on_insert() { - let now = Instant::now(); + let now = clock::now(); let first_expiry_time = now + Duration::from_secs(5); let second_expiry_time = now + Duration::from_secs(10); @@ -645,7 +644,7 @@ mod tests { flow_table .insert(FlowInfo::new( flow_key, - Instant::now() + Duration::from_mins(1), + clock::now() + Duration::from_mins(1), )) .unwrap(); let flow_info = flow_table.lookup(&flow_key).unwrap(); @@ -675,7 +674,7 @@ mod tests { async fn test_flow_table_flow_invalidation() { const NUM_FLOWS: u16 = 10; let flow_table = FlowTable::default(); - let now = Instant::now(); + let now = clock::now(); let deadline = now + Duration::from_secs(3); let mut flow_keys = vec![]; @@ -718,7 +717,7 @@ mod tests { /// Test that invalidating flows causes timer to expire and flows to be removed async fn test_flow_table_flow_reinsertion() { let flow_table = FlowTable::default(); - let now = Instant::now(); + let now = clock::now(); let deadline = now + Duration::from_secs(2); let flow_key = FlowKey::new( @@ -765,7 +764,7 @@ mod tests { async fn an_active_flow_holds_its_key_against_a_second_insertion() { let flow_table = FlowTable::default(); let key = key_for(1025); - let far_future = Instant::now() + Duration::from_hours(1); + let far_future = clock::now() + Duration::from_hours(1); let first = Arc::new(FlowInfo::new(key, far_future)); assert!(matches!( @@ -794,7 +793,7 @@ mod tests { async fn a_flow_that_is_not_live_is_displaced() { let flow_table = FlowTable::default(); let key = key_for(1026); - let far_future = Instant::now() + Duration::from_hours(1); + let far_future = clock::now() + Duration::from_hours(1); let first = Arc::new(FlowInfo::new(key, far_future)); flow_table.insert_if_absent(&first).unwrap(); @@ -821,7 +820,7 @@ mod tests { async fn displacing_a_flow_invalidates_its_partner() { let flow_table = FlowTable::default(); let (forward_key, reverse_key) = (key_for(1027), key_for(1028)); - let far_future = Instant::now() + Duration::from_hours(1); + let far_future = clock::now() + Duration::from_hours(1); let (forward, reverse) = FlowInfo::related_pair( far_future, @@ -853,7 +852,7 @@ mod tests { let src_vpcd = VpcDiscriminant::VNI(Vni::new_checked(100).unwrap()); let src_ip: IpAddr = "1.2.3.4".parse().unwrap(); let dst_ip: IpAddr = "5.6.7.8".parse().unwrap(); - let far_future = Instant::now() + Duration::from_hours(1); + let far_future = clock::now() + Duration::from_hours(1); // Insert up to the capacity limit — both should succeed. for i in 1u16..=2 { @@ -894,7 +893,6 @@ mod tests { use crate::flow_table::FlowInfo; use concurrency::sync::Arc; use concurrency::thread; - use std::time::Instant; #[allow(clippy::too_many_lines)] #[concurrency::test] @@ -921,7 +919,7 @@ mod tests { let flow_table = Arc::new(FlowTable::default()); - let now = Instant::now(); + let now = clock::now(); // Insert the first flow let orig_flow_info = FlowInfo::new(flow_keys[0], now + two_seconds); @@ -1020,7 +1018,7 @@ mod tests { fn test_flow_table_reshard() { let flow_table = Arc::new(FlowTable::default()); - let five_seconds_from_now = Instant::now() + Duration::from_secs(5); + let five_seconds_from_now = clock::now() + Duration::from_secs(5); let flow_key1 = FlowKey::new( Some(VpcDiscriminant::VNI(Vni::new_checked(1).unwrap())), "1.2.3.4".parse::().unwrap(), diff --git a/flow-filter/Cargo.toml b/flow-filter/Cargo.toml index 8ebae286b9..156ca093d1 100644 --- a/flow-filter/Cargo.toml +++ b/flow-filter/Cargo.toml @@ -8,6 +8,7 @@ version.workspace = true [dependencies] # internal acl = { workspace = true } +clock = { workspace = true } common = { workspace = true } concurrency = { workspace = true } config = { workspace = true } @@ -25,6 +26,7 @@ linkme = { workspace = true } tracing = { workspace = true } [dev-dependencies] +clock = { workspace = true, features = ["virtual"] } # The reference (linear-scan) ACL backend is the differential-test oracle and drives the fast, # EAL-free semantic suite. It is `cfg(test)`-gated in the source, so it is never part of a # production build; this dev-dep just makes `acl::reference` available to test builds. diff --git a/flow-filter/src/tests.rs b/flow-filter/src/tests.rs index e4181c1558..4fc62869a1 100644 --- a/flow-filter/src/tests.rs +++ b/flow-filter/src/tests.rs @@ -14,6 +14,7 @@ use crate::test_utils::{ vpcd, }; use crate::{FlowFilter, LookupResult, NatRequirement}; +use clock::Duration; use concurrency::sync::Arc; use lpm::prefix::L4Protocol; use net::FlowKey; @@ -23,7 +24,6 @@ use net::headers::Headers; use net::packet::{DoneReason, Packet, VpcDiscriminant}; use net::parse::DeParse; use pipeline::{NetworkFunction, PipelineData}; -use std::time::{Duration, Instant}; // ------------------------------------------------------------------------------------------------- // Helpers @@ -51,7 +51,7 @@ fn create_flow_pair( nat_state: bool, port_fw_state: bool, ) -> (Arc, Arc) { - let expires_at = Instant::now() + Duration::from_secs(60); + let expires_at = clock::now() + Duration::from_secs(60); let (flow_info_fwd, flow_info_reply) = FlowInfo::related_pair(expires_at, flow_key, flags, reply_flow_key, reply_flags).unwrap(); diff --git a/nat/Cargo.toml b/nat/Cargo.toml index b2e69b14e7..59cf71c42c 100644 --- a/nat/Cargo.toml +++ b/nat/Cargo.toml @@ -14,6 +14,7 @@ shuttle_dfs = ["concurrency/shuttle_dfs", "shuttle"] ahash = { workspace = true, features = ["std"] } arc-swap = { workspace = true } bnum = { workspace = true } +clock = { workspace = true } common = { workspace = true } concurrency = { workspace = true, features = [] } config = { workspace = true } @@ -35,13 +36,14 @@ tracing = { workspace = true } shuttle = { workspace = true, optional = true } [dev-dependencies] +clock = { workspace = true, features = ["virtual"] } # internal config = { workspace = true, features = ["bolero"] } fixin = { workspace = true } test-utils = { workspace = true } lpm = { workspace = true, features = ["testing"] } net = { workspace = true, features = ["bolero"] } -tokio = { workspace = true, features = ["macros", "rt", "time"] } +tokio = { workspace = true, features = ["macros", "rt", "test-util", "time"] } tracectl = { workspace = true } # external diff --git a/nat/src/masquerade/nf.rs b/nat/src/masquerade/nf.rs index 3edc15fb2f..e88e855f9b 100644 --- a/nat/src/masquerade/nf.rs +++ b/nat/src/masquerade/nf.rs @@ -13,6 +13,7 @@ use crate::masquerade::flows::check_masquerading_flow; use crate::masquerade::packet::{NatPacketError, NatTranslate, masquerade}; use crate::masquerade::protocol::next_flow_status; use crate::masquerade::state::MasqueradeState; +use clock::Duration; use concurrency::sync::{Arc, Weak}; use config::GenId; use flow_entry::flow_table::table::{FlowTable, FlowTableError}; @@ -26,7 +27,6 @@ use net::{FlowKey, IpProtoKey}; use pipeline::{NetworkFunction, PipelineData}; use std::fmt::Debug; use std::net::IpAddr; -use std::time::{Duration, Instant}; #[allow(unused)] use tracing::{debug, error, warn}; @@ -289,7 +289,7 @@ impl Masquerade { MasqueradeState::new_pair(alloc.allocation, src_ip, src_port, idle_timeout); // build a flow pair from the keys (without NAT state) - let expires_at = Instant::now() + Self::MASQUERADE_ONEWAY_TIMEOUT; + let expires_at = clock::now() + Self::MASQUERADE_ONEWAY_TIMEOUT; let (forward, reverse) = FlowInfo::related_pair( expires_at, *initial_flow_key, diff --git a/nat/src/masquerade/test.rs b/nat/src/masquerade/test.rs index 6dc22440c5..aa975a24d7 100644 --- a/nat/src/masquerade/test.rs +++ b/nat/src/masquerade/test.rs @@ -44,7 +44,7 @@ use pipeline::DynPipeline; use pipeline::NetworkFunction; use std::net::{IpAddr, Ipv4Addr}; use std::str::FromStr; -use std::time::{Duration, Instant}; +use std::time::Duration; use tracectl::get_trace_ctl; use tracing::debug; use tracing_test::traced_test; @@ -1679,7 +1679,7 @@ fn establish_tcp_connection(pipeline: &mut DynPipeline) { // check that flow timeouts "match" the ones configured, allowing for 5 second error (for the test) let flow_info_ack = output.meta().flow_info.as_ref().unwrap(); let related = flow_info_ack.related.as_ref().unwrap().upgrade().unwrap(); - let valid_until = (Instant::now() + timeout) + let valid_until = (clock::now() + timeout) .checked_sub(Duration::from_secs(5)) .unwrap(); assert!(flow_info_ack.expires_at() >= valid_until); diff --git a/nat/src/portfw/nf.rs b/nat/src/portfw/nf.rs index af5262236f..dda2962df2 100644 --- a/nat/src/portfw/nf.rs +++ b/nat/src/portfw/nf.rs @@ -14,7 +14,6 @@ use net::ip::UnicastIpAddr; use net::packet::{DoneReason, Packet, VpcDiscriminant}; use pipeline::{NetworkFunction, PipelineData}; use std::num::NonZero; -use std::time::Instant; use crate::common::NatAction; use crate::portfw::flow_state::build_portfw_flow_keys; @@ -114,7 +113,7 @@ impl PortForwarder { }; // create a pair of related flow entries (outside the flow table). Timeout is set according to the rule matched - let timeout = Instant::now() + entry.init_timeout(); + let timeout = clock::now() + entry.init_timeout(); let Ok((fw_flow, rev_flow)) = FlowInfo::related_pair( timeout, fw_key, diff --git a/nat/src/portfw/test.rs b/nat/src/portfw/test.rs index 95c60451b1..d44e2e8736 100644 --- a/nat/src/portfw/test.rs +++ b/nat/src/portfw/test.rs @@ -240,7 +240,7 @@ mod nf_test { // strict lower bound (`expires_at >= before + timeout`) that is // independent of how long the rest of the test takes -- otherwise // slow test execution (e.g. under miri) eats into the tolerance. - let before_reply = std::time::Instant::now(); + let before_reply = clock::now(); let output = process_packet(&mut pipeline, reply); assert_eq!(output.ip_source().unwrap().to_string(), "70.71.72.73"); assert_eq!(output.ip_destination().unwrap().to_string(), "10.0.0.1"); @@ -255,7 +255,7 @@ mod nf_test { // process original packet again. It should be fast-natted let repeated = udp_packet_to_port_forward(); - let before_repeated = std::time::Instant::now(); + let before_repeated = clock::now(); let output = process_packet(&mut pipeline, repeated); assert_eq!(output.ip_source().unwrap().to_string(), "10.0.0.1"); assert_eq!(output.ip_destination().unwrap().to_string(), "192.168.1.2"); diff --git a/net/Cargo.toml b/net/Cargo.toml index bbda3162aa..8c4bb70aa5 100644 --- a/net/Cargo.toml +++ b/net/Cargo.toml @@ -22,6 +22,7 @@ arrayvec = { workspace = true, features = ["serde", "std"] } bitflags = { workspace = true } bolero = { workspace = true, features = ["std"], optional = true } bytecheck = { workspace = true } +clock = { workspace = true } common = { workspace = true } concurrency = { workspace = true } derive_builder = { workspace = true, features = ["alloc"] } @@ -41,5 +42,6 @@ tokio-util = { workspace = true } tracing = { workspace = true } [dev-dependencies] +clock = { workspace = true, features = ["virtual"] } ahash = { workspace = true, features = ["no-rng"] } bolero = { workspace = true, features = ["std"] } diff --git a/net/src/flows/display.rs b/net/src/flows/display.rs index 4617cbef16..84f0ee20ca 100644 --- a/net/src/flows/display.rs +++ b/net/src/flows/display.rs @@ -8,7 +8,6 @@ use super::flow_key::FlowKey; use concurrency::sync::Weak; use std::fmt::Display; -use std::time::Instant; impl Display for FlowKey { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { @@ -49,7 +48,7 @@ impl Display for FlowInfoLocked { impl Display for FlowInfo { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let expires_at = self.expires_at(); - let expires_in = expires_at.saturating_duration_since(Instant::now()); + let expires_in = expires_at.saturating_duration_since(clock::now()); let genid = self.genid(); let info = self.locked.read(); let has_related = self diff --git a/net/src/flows/flow_info.rs b/net/src/flows/flow_info.rs index b23e52b838..36cf7e6353 100644 --- a/net/src/flows/flow_info.rs +++ b/net/src/flows/flow_info.rs @@ -398,7 +398,7 @@ impl FlowInfo { /// pub fn reset_expiry_unchecked(&self, duration: Duration) -> Result<(), FlowInfoError> { let current = self.expires_at(); - let new = Instant::now() + duration; + let new = clock::now() + duration; if new < current { return Err(FlowInfoError::TimeoutUnchanged); } diff --git a/routing/Cargo.toml b/routing/Cargo.toml index b24f21a943..90e72b2c48 100644 --- a/routing/Cargo.toml +++ b/routing/Cargo.toml @@ -15,6 +15,7 @@ testing = [] # internal args = { workspace = true } cli = { workspace = true } +clock = { workspace = true } common = { workspace = true } config = { workspace = true } concurrency = { workspace = true } @@ -55,6 +56,7 @@ procfs = { workspace = true } netdev = { workspace = true } [dev-dependencies] +clock = { workspace = true, features = ["virtual"] } bolero = { workspace = true, default-features = false } concurrency = { workspace = true } lpm = { workspace = true, features = ["testing"] } diff --git a/routing/src/bmp/bmp_render.rs b/routing/src/bmp/bmp_render.rs index 1aa81e582a..10cadd0c64 100644 --- a/routing/src/bmp/bmp_render.rs +++ b/routing/src/bmp/bmp_render.rs @@ -308,7 +308,7 @@ fn on_peer_down( if prev_state == BgpNeighborSessionState::Established { neigh.connections_dropped = neigh.connections_dropped.saturating_add(1); neigh.last_reset_reason = Some(pretty(pd.reason().get_type())); - neigh.last_reset_time = Some(std::time::Instant::now()); + neigh.last_reset_time = Some(clock::now()); } else { // we should not get to see this message with current FRR and RFC 7854 // Peer down events are only produced when leaving Established state diff --git a/routing/src/evpn/rmac.rs b/routing/src/evpn/rmac.rs index 19c055f777..23f33efd08 100644 --- a/routing/src/evpn/rmac.rs +++ b/routing/src/evpn/rmac.rs @@ -133,7 +133,7 @@ impl RmacStore { entry.mac, entry.vni, entry.address, ); // recall time when it became stale - current.stale_t = Some(Instant::now()); + current.stale_t = Some(clock::now()); self.stale = self.stale.saturating_add(1); } } diff --git a/routing/src/fib/test.rs b/routing/src/fib/test.rs index e2e9ec8ca1..347a90ce79 100644 --- a/routing/src/fib/test.rs +++ b/routing/src/fib/test.rs @@ -30,7 +30,6 @@ mod tests { use rand::RngExt; use rand::rngs::ThreadRng; use std::str::FromStr; - use std::time::Instant; use std::{collections::HashMap, collections::HashSet, sync::atomic::Ordering}; use crate::fib::fibgroupstore::tests::build_fib_entry_egress; @@ -348,7 +347,7 @@ mod tests { fibw.register_fibgroup(&nhkey, fibgroup, true); fibw.add_fibroute(prefix, vec![nhkey.clone()], true); } - let start = Instant::now(); + let start = clock::now(); loop { if fibw.is_none() { fibw = Some(fibtw.add_fib(vrfid, None)); diff --git a/routing/src/frr/frrmi.rs b/routing/src/frr/frrmi.rs index c8311e71e2..604f8e8861 100644 --- a/routing/src/frr/frrmi.rs +++ b/routing/src/frr/frrmi.rs @@ -154,7 +154,7 @@ impl Frrmi { revent!(RouterEvent::FrrmiDisconnected); } pub(crate) fn timeout(&mut self) { - if self.timeout.take_if(|t| *t < Instant::now()).is_some() { + if self.timeout.take_if(|t| *t < clock::now()).is_some() { warn!("Request sent to frr-agent timed out! Will reconnect..."); self.disconnect(); } @@ -272,7 +272,7 @@ impl Frrmi { debug!("Sending config request to frr-agent for gen {genid}..."); Self::send(sock, &mut self.writeb)?; debug!("FRR config request for gen {genid} successfully sent"); - self.timeout = Instant::now().checked_add(Self::TIMEOUT); + self.timeout = clock::now().checked_add(Self::TIMEOUT); Ok(()) } diff --git a/routing/src/rib/vrf.rs b/routing/src/rib/vrf.rs index 8abb46186d..ce3e1c6687 100644 --- a/routing/src/rib/vrf.rs +++ b/routing/src/rib/vrf.rs @@ -77,7 +77,7 @@ impl Default for Route { distance: 0, metric: 0, s_nhops: Vec::with_capacity(1), - tstamp: Instant::now(), + tstamp: clock::now(), } } } @@ -829,7 +829,7 @@ pub mod tests { distance, metric, s_nhops: vec![], - tstamp: Instant::now(), + tstamp: clock::now(), } } diff --git a/routing/src/router/cpi.rs b/routing/src/router/cpi.rs index ddec4dc920..7179e24460 100644 --- a/routing/src/router/cpi.rs +++ b/routing/src/router/cpi.rs @@ -25,7 +25,7 @@ use net::interface::InterfaceIndex; use net::interface::address::IfAddr; use std::os::unix::net::SocketAddr; use std::process; -use std::time::{SystemTime, UNIX_EPOCH}; +use std::time::UNIX_EPOCH; #[allow(unused)] use tracing::{debug, error, info, trace, warn}; @@ -104,7 +104,7 @@ pub(crate) struct CpiStats { impl CpiStats { pub(crate) fn new() -> CpiStats { Self { - synt: SystemTime::now() + synt: clock::system_now() .duration_since(UNIX_EPOCH) .expect("System time is wrong!") .as_secs(), diff --git a/routing/src/router/rio.rs b/routing/src/router/rio.rs index e57b4fdbfd..fb2372a3e6 100644 --- a/routing/src/router/rio.rs +++ b/routing/src/router/rio.rs @@ -379,14 +379,10 @@ impl Rio { let duration = 60; debug!("Set stale timeout ({duration} seconds)"); let duration = Duration::from_secs(duration); - self.stale_timeout = Instant::now().checked_add(duration); + self.stale_timeout = clock::now().checked_add(duration); } fn check_stale_timeout(&mut self, db: &mut RoutingDb) { - if self - .stale_timeout - .take_if(|t| *t < Instant::now()) - .is_some() - { + if self.stale_timeout.take_if(|t| *t < clock::now()).is_some() { info!("Stale timeout expired"); db.vrftable.remove_stale_routes(&db.rmac_store); db.vrftable.remove_deleted_vrfs(&mut db.iftw); diff --git a/routing/src/router/rpc_adapt.rs b/routing/src/router/rpc_adapt.rs index b735b27522..d2f6521364 100644 --- a/routing/src/router/rpc_adapt.rs +++ b/routing/src/router/rpc_adapt.rs @@ -23,7 +23,6 @@ use net::eth::mac::Mac; use net::interface::InterfaceIndex; use net::vxlan::Vni; use std::net::{IpAddr, Ipv4Addr}; -use std::time::Instant; use tracing::{error, warn}; impl From for RouteOrigin { @@ -168,7 +167,7 @@ impl Route { distance: iproute.distance, metric: iproute.metric, s_nhops: vec![], /* shim nhops are empty here */ - tstamp: Instant::now(), + tstamp: clock::now(), } } } diff --git a/stats/Cargo.toml b/stats/Cargo.toml index 50fe1da0da..ff1d2b7be8 100644 --- a/stats/Cargo.toml +++ b/stats/Cargo.toml @@ -11,6 +11,7 @@ bolero = ["dep:bolero", "vpcmap/bolero", "net/bolero"] [dependencies] # internal +clock = { workspace = true } concurrency = { workspace = true } net = { workspace = true } pipeline = { workspace = true } @@ -34,6 +35,7 @@ tokio = { workspace = true, features = ["macros", "time", "sync"] } tracing = { workspace = true, features = ["attributes"] } [dev-dependencies] +clock = { workspace = true, features = ["virtual"] } bolero = { workspace = true } net = { workspace = true, features = ["bolero"] } vpcmap = { workspace = true, features = ["bolero"] } diff --git a/stats/src/dpstats.rs b/stats/src/dpstats.rs index 19f1b01733..10f9264183 100644 --- a/stats/src/dpstats.rs +++ b/stats/src/dpstats.rs @@ -244,7 +244,7 @@ impl StatsCollector { let updates = PacketStatsReader(r); let outstanding: VecDeque<_> = (0..10) .scan( - BatchSummary::::new(Instant::now() + Self::TIME_TICK), + BatchSummary::::new(clock::now() + Self::TIME_TICK), |prior, _| Some(BatchSummary::new(prior.planned_end + Self::TIME_TICK)), ) .collect(); @@ -522,7 +522,7 @@ impl StatsCollector { }); } - let current_time = Instant::now(); + let current_time = clock::now(); let mut expired = self .outstanding .iter() @@ -833,7 +833,7 @@ impl BatchSummary { #[inline] pub fn with_capacity(planned_end: Instant, capacity: usize) -> Self { Self { - start: Instant::now(), + start: clock::now(), planned_end, vpc: hashbrown::HashMap::with_capacity(capacity), } @@ -900,7 +900,7 @@ impl Stats { stats: PacketStatsWriter, delivery_schedule: Duration, ) -> Self { - let planned_end = Instant::now() + delivery_schedule; + let planned_end = clock::now() + delivery_schedule; Self { name: name.to_string(), update: Box::new(BatchSummary::new(planned_end)), @@ -920,7 +920,7 @@ impl NetworkFunction for Stats { // amount of spare room in hash table. Padding a little bit will hopefully save us some // reallocations const CAPACITY_PAD: usize = 16; - let time = Instant::now(); + let time = clock::now(); if time > self.update.planned_end { trace!("sending stats update"); let batch = Box::new(BatchSummary::with_capacity( @@ -1088,8 +1088,8 @@ pub struct SplitCount { mod contract { use crate::{BatchSummary, PacketAndByte, TransmitSummary}; use bolero::{Driver, TypeGenerator, ValueGenerator}; + use clock::Duration; use small_map::SmallMap; - use std::time::{Duration, Instant}; use vpcmap::VpcDiscriminant; impl TypeGenerator for PacketAndByte @@ -1153,7 +1153,7 @@ mod contract { T: TypeGenerator, { fn generate(driver: &mut D) -> Option { - let start = Instant::now() + Duration::from_millis(driver.produce()?); + let start = clock::now() + Duration::from_millis(driver.produce()?); let duration: Duration = driver.produce()?; let vpc_gen = VpcDiscMap::> { _marker: std::marker::PhantomData, @@ -1328,7 +1328,7 @@ mod drop_stats_tests { } fn batch(offset_secs: u64) -> BatchSummary { - BatchSummary::::new(Instant::now() + Duration::from_secs(offset_secs)) + BatchSummary::::new(clock::now() + Duration::from_secs(offset_secs)) } #[test] diff --git a/tracectl/Cargo.toml b/tracectl/Cargo.toml index 381c503928..08e33946b6 100644 --- a/tracectl/Cargo.toml +++ b/tracectl/Cargo.toml @@ -6,6 +6,7 @@ publish.workspace = true version.workspace = true [dependencies] +clock = { workspace = true } color-eyre = { workspace = true , features = [ "capture-spantrace", "color-spantrace", "tracing-error", "track-caller" ] } common = { workspace = true } concurrency = { workspace = true } @@ -17,5 +18,6 @@ tracing-error = { workspace = true, features = ["traced-error"] } tracing-subscriber = { workspace = true, features = ["registry", "std", "env-filter", "fmt"] } [dev-dependencies] +clock = { workspace = true, features = ["virtual"] } serial_test = { workspace = true } tracing-test = { workspace = true } diff --git a/tracectl/src/throttle.rs b/tracectl/src/throttle.rs index 2b09a3832f..5d7b4db795 100644 --- a/tracectl/src/throttle.rs +++ b/tracectl/src/throttle.rs @@ -70,7 +70,7 @@ impl RateLimitFilter { buckets: Box::new(std::array::from_fn(|_| AtomicU64::new(initial))), capacity_milli, refill_milli_per_ms: u64::from(config.replenish_per_second), - baseline: Instant::now(), + baseline: clock::now(), } } From d97c02513f9aad3d384068bfe0ba2f6cba5329dc Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Mon, 17 Aug 2026 20:41:55 -0600 Subject: [PATCH 02/37] test(nat): Test flow expiry on a clock the test drives The masquerade properties deliberately stayed inside one flow lifetime, because until the workspace read its deadlines through `clock` there was no way to write these. With the deadlines and the timers on the same clock, expiry becomes an ordinary subject. These cost nothing. A property covering a minute of flow lifetime runs in 0.00s of wall clock, where the real-time version would cost a minute per case and be flaky under emulation. ## The three dispositions The design note asks that every piece of live state a configuration change could touch be classified, and expiry is the same question asked of time: * **preserved** -- a flow inside its lifetime keeps behaving identically. Not assumed: the cheap way to implement expiry is also the cheap way to drop something whose deadline has not passed. * **invalidated attributably** -- a flow past its lifetime stops translating and is dropped. The failure ruled out is not the drop but the *pass*: forwarding a packet still addressed to a public tuple is a leak, not a timeout. * **never resurrected** -- an expired flow does not come back, and a flow created *after* the clock moved works. That one needs three epochs and was unwritable before. Plus `traffic_extends_a_flow_past_its_first_deadline`, which is the regression test for the facade itself: `reset_expiry_unchecked` was one of the three sites reading the wall clock while the timer consuming its answer read tokio's, so a refresh under a paused clock wrote a deadline in the past and *shortened* the flow's life. ## And it found something `a_live_flows_tuple_is_reissued_after_its_original_deadline` is a reproduction of an unfixed defect, `#[ignore]`d so the branch stays green. Run it with `cargo test -p dataplane-nat -- --ignored reissued`. A flow is opened and refreshed every second, so it is unambiguously alive -- its replies are delivered correctly throughout. Once `MASQUERADE_ONEWAY_TIMEOUT` of virtual time has passed since it was opened, a newly opened flow is handed the same public address and port, and the replies that were reaching the first tenant start reaching the second. The threshold is exactly the one-way timeout, by bisection: | advance | flow alive | tuple reissued | | --- | --- | --- | | 4s | yes | no | | 5s | yes | **yes** | So the allocation is reclaimed on the deadline the flow was *created* with, and the refreshes that keep the flow alive do not carry it. `MasqueradeState` holds the `Allocation` in the forward entry only, which is consistent with the forward entry being reclaimed while the reverse entry survives and keeps translating -- but that is where an investigation should start, not what it has concluded. Production sets `randomize(true)`, and with randomization the same sequence draws a different port, so the collision is unlikely rather than impossible -- a matter of load and range size rather than of correctness. What is wrong in both modes is the underlying state: a flow that is still translating no longer owns the tuple it is translating to. Two tenants sharing a public tuple is a tenant isolation failure. This is the class of defect a controllable clock exists to find. Reaching it on the wall clock needs six seconds of real time per attempt and the right allocation pattern; here it is deterministic and free. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 91360180a9c5cbeae82cc0e071b8bab5bec7e26e) --- development/code/config-algebra-testing.md | 5 +- nat/src/masquerade/expiry.rs | 335 +++++++++++++++++++++ nat/src/masquerade/mod.rs | 1 + 3 files changed, 340 insertions(+), 1 deletion(-) create mode 100644 nat/src/masquerade/expiry.rs diff --git a/development/code/config-algebra-testing.md b/development/code/config-algebra-testing.md index c2059322aa..7fac0a1cdb 100644 --- a/development/code/config-algebra-testing.md +++ b/development/code/config-algebra-testing.md @@ -334,7 +334,10 @@ Three prerequisites, all of which have already bitten this codebase once: two pipelines will allocate different ports for the same flow and a naive comparison fails immediately. Either seed it identically per pipeline or keep it out of the compared projection. 2. **Advance timers in lockstep.** Flow timers are already known to leak between fuzz inputs; across - concurrent pipelines they must be driven explicitly rather than by wall clock. + concurrent pipelines they must be driven explicitly rather than by wall clock. The `clock` facade + now supplies that: every deadline in the workspace is read through `clock::now()`, which follows + tokio's pausable clock under test, so `tokio::time::advance` moves deadlines and timers together. + See `nat/src/masquerade/expiry.rs` for what that makes writable. 3. **Compare projections, not state.** Counters and port allocations legitimately differ between two pipelines that agree on every verdict. Compare what an operator can observe. diff --git a/nat/src/masquerade/expiry.rs b/nat/src/masquerade/expiry.rs new file mode 100644 index 0000000000..0e9854cc73 --- /dev/null +++ b/nat/src/masquerade/expiry.rs @@ -0,0 +1,335 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +//! Flow expiry, on a clock the test drives. +//! +//! `masquerade::fuzz` deliberately stays inside one flow lifetime, because until the workspace read +//! its deadlines through [`clock`] there was no way to write these. The waits were on tokio's clock +//! and the deadlines were on `std`'s, so a paused clock bought exactly **one** time step: anything +//! created after the first `advance` was born with a deadline already in the past. +//! +//! With both on the same clock, expiry becomes an ordinary subject. These run in zero wall-clock +//! time -- a property covering a minute of flow lifetime costs nothing, where a real-time version +//! would cost a minute per case and be flaky under emulation. +//! +//! # The three dispositions +//! +//! The development guide asks that every piece of live state a configuration change could touch be +//! classified, and expiry is the same question asked of time rather than of configuration: +//! +//! 1. **Preserved** -- a flow inside its lifetime keeps behaving identically. +//! 2. **Invalidated attributably** -- a flow past its lifetime stops translating, and says so rather +//! than silently forwarding. +//! 3. **Never resurrected** -- an expired flow's translation does not come back to life, and a new +//! flow that reuses the same public tuple answers to its own source rather than the dead one. +//! +//! The third is the one worth the machinery. It needs at least three epochs -- create, expire, +//! create again -- and a single `advance` cannot express it. + +#![cfg(test)] + +use crate::Masquerade; +use crate::masquerade::probe::{Arrival, Fabric, run}; +use crate::static_nat::probe::build; +use clock::Duration; +use config::external::overlay::vpcpeering::VpcExpose; +use config::external::overlay::vpcpeering::contract::{LOCAL_VNI, REMOTE_VNI}; +use flow_entry::flow_table::FlowLookup; +use net::buffer::TestBuffer; +use net::packet::Packet; +use net::vxlan::Vni; +use std::net::IpAddr; + +/// Comfortably past `MASQUERADE_ONEWAY_TIMEOUT`, which is the longest a flow lives untouched. +const PAST_EXPIRY: Duration = Duration::from_secs(30); + +/// Comfortably inside it. +const WITHIN_LIFETIME: Duration = Duration::from_secs(1); + +fn vni(raw: u32) -> Vni { + Vni::new_checked(raw).unwrap_or_else(|_| unreachable!()) +} + +/// A runtime whose clock starts paused and only moves when a property says so. +/// +/// `block_on` rather than `enter`, because `tokio::time::advance` is async and because the timer +/// tasks the flow table spawns need the runtime to be driven before they can observe the new time. +/// Everything a property does happens inside this one future. +fn with_paused_clock>(body: impl FnOnce() -> F) { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_time() + .start_paused(true) + .build() + .unwrap_or_else(|e| unreachable!("{e}")); + runtime.block_on(body()); +} + +/// Move the clock and let every timer that is now due actually run. +/// +/// The yields matter. `advance` makes the time visible; it does not run the tasks waiting on it, and +/// a property that checked immediately afterwards would see a flow that is past its deadline but has +/// not yet been told so. +async fn advance(by: Duration) { + tokio::time::advance(by).await; + for _ in 0..4 { + tokio::task::yield_now().await; + } +} + +/// A two-vpc masquerade fabric with one expose, which is all expiry needs. +fn fabric() -> (Fabric, Vec) { + let exposes = vec![ + VpcExpose::empty() + .make_masquerade(None) + .unwrap_or_else(|e| unreachable!("{e}")) + .ip(lpm::prefix::PrefixWithOptionalPorts::new( + "10.0.0.0/24".parse().unwrap_or_else(|_| unreachable!()), + None, + )) + .as_range(lpm::prefix::PrefixWithOptionalPorts::new( + "172.16.0.0/24".parse().unwrap_or_else(|_| unreachable!()), + None, + )) + .unwrap_or_else(|e| unreachable!("{e}")), + ]; + let fabric = Fabric::build(&exposes).unwrap_or_else(|| unreachable!("a fixed expose builds")); + (fabric, exposes) +} + +/// Send one flow's first packet and report what it was translated to. +fn open_flow( + lookup: &mut FlowLookup, + masq: &mut Masquerade, + source: IpAddr, + peer: IpAddr, + sport: u16, +) -> Option<(IpAddr, u16)> { + let mut packet = build(source, peer, false, sport, 80); + Arrival::outbound().stamp(&mut packet); + let out = run(lookup, masq, vec![packet], Some(vni(REMOTE_VNI))); + if out[0].is_done() { + return None; + } + let addr = out[0].ip_source()?; + let port = out[0].transport_src_port()?.get(); + (addr != source).then_some((addr, port)) +} + +/// Send the reply to a translated flow and report where it was delivered, if anywhere. +fn reply_to( + lookup: &mut FlowLookup, + masq: &mut Masquerade, + peer: IpAddr, + translated: (IpAddr, u16), +) -> Option { + let mut packet = build(peer, translated.0, false, 80, translated.1); + Arrival::inbound().stamp(&mut packet); + let out: Vec> = run(lookup, masq, vec![packet], Some(vni(LOCAL_VNI))); + (!out[0].is_done()) + .then(|| out[0].ip_destination()) + .flatten() +} + +/// A flow inside its lifetime is unaffected by the passage of time. +/// +/// The **preserved** disposition. Stated as a property rather than assumed, because the cheap way to +/// implement expiry -- sweep and drop anything whose deadline has passed -- is also the cheap way to +/// drop something whose deadline has not. +#[test] +fn a_flow_inside_its_lifetime_survives() { + with_paused_clock(|| async { + let (fabric, _) = fabric(); + let (mut lookup, mut masq) = fabric.stages(); + let peer = fabric.peer[0]; + let source: IpAddr = "10.0.0.7".parse().unwrap_or_else(|_| unreachable!()); + + let translated = open_flow(&mut lookup, &mut masq, source, peer, 1234) + .unwrap_or_else(|| unreachable!("a fixed private source is masqueraded")); + + advance(WITHIN_LIFETIME).await; + + assert_eq!( + reply_to(&mut lookup, &mut masq, peer, translated), + Some(source), + "a flow one second into a five second lifetime stopped answering" + ); + }); +} + +/// A flow past its lifetime stops translating, and does not silently forward. +/// +/// The **invalidated attributably** disposition. The failure this rules out is not the drop -- a +/// reply to a flow nobody remembers should be dropped -- it is the *pass*: forwarding a packet still +/// addressed to a public tuple into the tenant's network, or back out with no translation, is a leak +/// rather than a timeout. +#[test] +fn a_flow_past_its_lifetime_stops_answering() { + with_paused_clock(|| async { + let (fabric, _) = fabric(); + let (mut lookup, mut masq) = fabric.stages(); + let peer = fabric.peer[0]; + let source: IpAddr = "10.0.0.7".parse().unwrap_or_else(|_| unreachable!()); + + let translated = open_flow(&mut lookup, &mut masq, source, peer, 1234) + .unwrap_or_else(|| unreachable!("a fixed private source is masqueraded")); + + advance(PAST_EXPIRY).await; + + let delivered = reply_to(&mut lookup, &mut masq, peer, translated); + assert_ne!( + delivered, + Some(source), + "an expired flow still delivered its reply, so the timeout is not enforced" + ); + assert!( + delivered.is_none(), + "the reply to an expired flow was forwarded to {delivered:?} instead of being dropped" + ); + }); +} + +/// Traffic keeps a flow alive past the deadline it started with. +/// +/// `reset_expiry_unchecked` is one of the three sites that used to read the wall clock while the +/// timer that consumes its answer read tokio's, so a refresh under a paused clock wrote a deadline +/// in the past and *shortened* the flow's life instead of extending it. That is precisely the +/// confusing failure the facade exists to prevent, and this is the regression test for it. +#[test] +fn traffic_extends_a_flow_past_its_first_deadline() { + with_paused_clock(|| async { + let (fabric, _) = fabric(); + let (mut lookup, mut masq) = fabric.stages(); + let peer = fabric.peer[0]; + let source: IpAddr = "10.0.0.7".parse().unwrap_or_else(|_| unreachable!()); + + let translated = open_flow(&mut lookup, &mut masq, source, peer, 1234) + .unwrap_or_else(|| unreachable!("a fixed private source is masqueraded")); + + // Four refreshes a second apart. Each is inside the current lifetime, and together they + // carry the flow well past the deadline the first packet set. + for _ in 0..4 { + advance(WITHIN_LIFETIME).await; + assert_eq!( + reply_to(&mut lookup, &mut masq, peer, translated), + Some(source), + "a refreshed flow stopped answering while still inside its extended lifetime" + ); + } + }); +} + +/// An expired flow does not come back, and its public tuple may be handed to someone else. +/// +/// The **never resurrected** disposition, and the property that needs the whole facade: three +/// epochs, with a flow created *after* time has moved. Before the clock was routed this could not be +/// written at all -- the second flow was born with a deadline in the past and was dead on arrival, +/// which looks exactly like a masquerade bug and is not one. +/// +/// Two claims. The dead flow stays dead, and the live one answers to *its own* source: an allocator +/// that reissued the tuple while the old flow's reverse entry lingered would deliver the second +/// tenant's replies to the first. +#[test] +fn an_expired_flow_is_never_resurrected() { + with_paused_clock(|| async { + let (fabric, _) = fabric(); + let (mut lookup, mut masq) = fabric.stages(); + let peer = fabric.peer[0]; + let first: IpAddr = "10.0.0.7".parse().unwrap_or_else(|_| unreachable!()); + let second: IpAddr = "10.0.0.9".parse().unwrap_or_else(|_| unreachable!()); + + let dead = open_flow(&mut lookup, &mut masq, first, peer, 1234) + .unwrap_or_else(|| unreachable!("a fixed private source is masqueraded")); + + advance(PAST_EXPIRY).await; + + // Epoch three: a new flow, created after the clock moved. + let live = open_flow(&mut lookup, &mut masq, second, peer, 4321).unwrap_or_else(|| { + unreachable!( + "a flow opened after the clock advanced was refused; the deadline and the timer \ + are on different clocks again" + ) + }); + + assert_eq!( + reply_to(&mut lookup, &mut masq, peer, live), + Some(second), + "a flow opened after the clock advanced could not receive its reply" + ); + assert_ne!( + reply_to(&mut lookup, &mut masq, peer, dead), + Some(first), + "an expired flow answered again after a later flow had been created" + ); + }); +} + +/// A live flow's public tuple is reissued to another flow once its *original* deadline passes. +/// +/// **A reproduction of a defect, not a passing property.** Ignored so the branch stays green; run it +/// with `cargo test -p dataplane-nat -- --ignored reissued` to see it fail. +/// +/// # What happens +/// +/// A flow is opened and then refreshed every second, so it is unambiguously alive -- its replies are +/// delivered correctly throughout. Once `MASQUERADE_ONEWAY_TIMEOUT` of *virtual* time has passed +/// since it was opened, a newly opened flow is handed the same public address and port, and the +/// replies that were reaching the first tenant start reaching the second. +/// +/// The threshold is exactly the one-way timeout, measured by bisection: +/// +/// | advance | flow alive | tuple reissued | +/// | --- | --- | --- | +/// | 4s | yes | no | +/// | 5s | yes | **yes** | +/// +/// So the allocation is being reclaimed on the deadline the flow was *created* with, and the refreshes +/// that keep the flow itself alive do not carry it. `MasqueradeState` holds the `Allocation` in the +/// forward entry only, which is consistent with the forward entry being reclaimed while the reverse +/// entry survives and keeps translating -- but that is where the investigation should start rather +/// than what it has concluded. +/// +/// # Why it matters in production +/// +/// Production sets `randomize(true)`, and with randomization the same sequence picks a different +/// port, so the collision is unlikely rather than impossible -- it needs the reclaimed port to be +/// drawn again while the old flow still lives, which is a matter of load and range size rather than +/// of correctness. What is wrong in both modes is the underlying state: a flow that is still +/// translating no longer owns the tuple it is translating to. Two tenants sharing a public tuple is +/// a tenant isolation failure, not a performance one. +/// +/// This is the class of defect a controllable clock exists to find. Reaching it on the wall clock +/// needs six seconds of real time per attempt and the right allocation pattern; here it is +/// deterministic and costs nothing. +#[test] +#[ignore = "reproduces an unfixed defect: a live flow's tuple is reissued after its original deadline"] +fn a_live_flows_tuple_is_reissued_after_its_original_deadline() { + with_paused_clock(|| async { + let (fabric, _) = fabric(); + let (mut lookup, mut masq) = fabric.stages(); + let peer = fabric.peer[0]; + let first: IpAddr = "10.0.0.10".parse().unwrap_or_else(|_| unreachable!()); + let second: IpAddr = "10.0.0.99".parse().unwrap_or_else(|_| unreachable!()); + + let translated = open_flow(&mut lookup, &mut masq, first, peer, 2000) + .unwrap_or_else(|| unreachable!("a fixed private source is masqueraded")); + + // Refresh once a second, past the one-way timeout. The flow is alive the whole way. + for second_elapsed in 1..=6 { + advance(WITHIN_LIFETIME).await; + assert_eq!( + reply_to(&mut lookup, &mut masq, peer, translated), + Some(first), + "the flow stopped answering at t={second_elapsed}s despite being refreshed" + ); + } + + let other = open_flow(&mut lookup, &mut masq, second, peer, 3000) + .unwrap_or_else(|| unreachable!("a second private source is masqueraded")); + + assert_ne!( + other, translated, + "a live flow's public tuple {translated:?} was reissued to {second}, so replies for \ + {first} will be delivered to {second}" + ); + }); +} diff --git a/nat/src/masquerade/mod.rs b/nat/src/masquerade/mod.rs index 5e149bb06f..d92e9aad50 100644 --- a/nat/src/masquerade/mod.rs +++ b/nat/src/masquerade/mod.rs @@ -4,6 +4,7 @@ pub(crate) mod allocation; mod allocator_writer; pub mod apalloc; +mod expiry; pub(crate) mod flows; mod fuzz; pub(crate) mod icmp_handling; From c3e8a86415917ae164394dbebc5ea5523742220b Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Mon, 17 Aug 2026 21:50:29 -0600 Subject: [PATCH 03/37] test(nat): Drive port forwarding with configuration-relative packets The third and last NAT flavour, and the one where the campaign's only confirmed configuration bug lived. That was found by reading code and pinned at the configuration level; this covers the stage. ## The direction is the point Static NAT and masquerade translate a **source** on the way out. Port forwarding translates a **destination** on the way in, so every property reads the other half of the five-tuple and the failure modes differ in kind. A masquerade mistake leaks one tenant's traffic to another; a port-forwarding mistake delivers the outside world to an address inside a tenant that never published it. Six properties: reversibility (the reply must carry the tuple the client used, or the client drops it), containment, injectivity, permission, frame, and stability. Three more in `expiry`, on a paused clock: a published service keeps answering while time passes, a service re-established after its flow expires reaches the *same* backend -- the rule is configuration and does not expire with the flow -- and the mapping stays injective across an expiry. ## Two things break testing turned up **The probes were judging packets that never reached the code.** A rule is keyed by `(source vpc, protocol)`, and the probe drew its protocol independently of the rule it addressed. Roughly half of them therefore matched nothing, and `nothing_is_forwarded_that_was_not_published` -- whose subject is exactly the packets that should not match -- passed for the wrong reason. The protocol now comes from the rule. This is the same class of mistake as the arrival-state ordering in masquerade: a harness that looks like it is testing something and is not. **The port-range guard has three gates, not one.** A port past the published range is refused independently by `RangeSet::lookup`'s upper bound, by `PortRange::contains` via `indexof`, and by the size-matched arithmetic in `map_port_to`. Breaking any one, or any two, still refuses the packet; the property only fires with all three broken at once. That is defence in depth rather than redundancy, and it is why the property looked vacuous at first -- it is not, the code is simply hard to break there. | break | fires | | --- | --- | | address mapping collapsed onto one target | injectivity | | target shifted one past the declared prefix | containment | | `requires_port_forwarding` ignored | permission | | all three port-range gates opened together | permission | | flow fast path fails | stability, and the service-answers-over-time property | ## Also `PortForwardingExposes` gets the block-and-family treatment the other two flavours got, plus one constraint they do not have: a rule is keyed by protocol, so two exposes naming the same protocol produce two rules with the same key and the second silently replaces the first. One protocol per expose, assigned by position. The injectivity sweep is capped and strided at 256 pairs. Enumerating a rule that publishes 256 addresses over 1024 ports costs a quarter of a million packets, and that property was spending its entire budget on two configurations; it now reaches eleven. One observation, not fixed: a packet whose port is outside the published range is dropped with `DoneReason::InternalFailure`. Nothing is internally broken -- the operator did not publish that port. Attribution, not correctness. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 1f7a52b12990de7ddb9fef47041a2ab04155b4d3) --- config/src/external/overlay/vpcpeering.rs | 105 +++++ nat/src/portfw/expiry.rs | 202 +++++++++ nat/src/portfw/fuzz.rs | 478 ++++++++++++++++++++++ nat/src/portfw/mod.rs | 3 + nat/src/portfw/probe.rs | 407 ++++++++++++++++++ 5 files changed, 1195 insertions(+) create mode 100644 nat/src/portfw/expiry.rs create mode 100644 nat/src/portfw/fuzz.rs create mode 100644 nat/src/portfw/probe.rs diff --git a/config/src/external/overlay/vpcpeering.rs b/config/src/external/overlay/vpcpeering.rs index d8aae9f377..82bddee9fb 100644 --- a/config/src/external/overlay/vpcpeering.rs +++ b/config/src/external/overlay/vpcpeering.rs @@ -1125,6 +1125,111 @@ pub mod contract { } } + /// Several port-forwarding exposes for one manifest, which a manifest will accept together. + /// + /// The third of these, and for the same two reasons as [`StaticNatExposes`] and + /// [`MasqueradeExposes`]: [`PortForwardingExpose`] draws its prefixes from anywhere inside + /// `10.0.0.0/8` and `172.16.0.0/12`, so two of them can overlap, and independent draws mix + /// address families, which a peering refuses. + /// + /// Port forwarding has a third constraint the others do not. A rule is keyed by + /// `(source vpc, protocol)`, so two exposes naming the same protocol produce two rules with the + /// same key and the second replaces the first in the table -- a legal configuration that + /// silently halves what a property is testing. `L4Protocol::Any` expands to both TCP and UDP, + /// so it collides with everything. One protocol per expose, assigned by position. + #[derive(Debug, Clone, Copy)] + pub struct PortForwardingExposes(pub u8); + + impl Default for PortForwardingExposes { + fn default() -> Self { + Self(2) + } + } + + /// The most exposes [`PortForwardingExposes`] can draw, one per distinct protocol key. + pub const MAX_PORT_FORWARDING_EXPOSES: u8 = 2; + + impl ValueGenerator for PortForwardingExposes { + type Output = Vec; + + fn generate(&self, driver: &mut D) -> Option> { + let v4 = driver.produce::()?; + let count = driver.gen_u8( + Included(&1), + Included(&self.0.clamp(1, MAX_PORT_FORWARDING_EXPOSES)), + )?; + (0..count) + .map(|slot| { + let proto = if slot == 0 { + L4Protocol::Tcp + } else { + L4Protocol::Udp + }; + port_forwarding_expose(driver, v4, slot, proto) + }) + .collect() + } + } + + /// One port-forwarding expose of the given family and protocol, in its own block. + fn port_forwarding_expose( + driver: &mut D, + v4: bool, + block: u8, + proto: L4Protocol, + ) -> Option { + let host_bits = driver.gen_u8(Included(&0), Included(&MAX_HOST_BITS))?; + let (internal, external) = if v4 { + v4_pair_in_block(driver, host_bits, block)? + } else { + v6_pair_in_block(driver, host_bits, block)? + }; + + let count = driver.gen_u16(Included(&1), Included(&MAX_PORTS))?; + let internal_ports = port_range(driver, count)?; + let external_ports = port_range(driver, count)?; + let idle_timeout = match driver.gen_u8(Included(&0), Included(&2))? { + 0 => None, + 1 => Some(Duration::from_secs(5)), + _ => Some(Duration::from_mins(5)), + }; + + VpcExpose::empty() + .make_port_forwarding(idle_timeout, Some(proto)) + .ok()? + .ip(PrefixWithOptionalPorts::new(internal, Some(internal_ports))) + .as_range(PrefixWithOptionalPorts::new(external, Some(external_ports))) + .ok() + } + + // As `v4_pair`, but confined to a /16 of its own so two exposes cannot overlap. + fn v4_pair_in_block( + driver: &mut D, + host_bits: u8, + block: u8, + ) -> Option<(Prefix, Prefix)> { + let len = 32 - host_bits; + let mask = u32::MAX.checked_shl(u32::from(host_bits)).unwrap_or(0); + let slot = u32::from(block) << 16; + let internal = (0x0A00_0000 | slot | (driver.produce::()? & 0x0000_FFFF)) & mask; + let external = (0xAC10_0000 | slot | (driver.produce::()? & 0x0000_FFFF)) & mask; + Some((prefix_v4(internal, len)?, prefix_v4(external, len)?)) + } + + // As `v6_pair`, likewise blocked. + fn v6_pair_in_block( + driver: &mut D, + host_bits: u8, + block: u8, + ) -> Option<(Prefix, Prefix)> { + let len = 128 - host_bits; + let mask = u128::MAX.checked_shl(u32::from(host_bits)).unwrap_or(0); + let slot = u128::from(block) << 64; + let internal = (INTERNAL_BASE | slot | u128::from(driver.produce::()?)) & mask; + let external = (EXTERNAL_BASE | slot | u128::from(driver.produce::()?)) & mask; + Some((prefix_v6(internal, len)?, prefix_v6(external, len)?)) + } + /// Generates [`VpcExpose`]s that masquerade and that [`VpcExpose::validate`] accepts. /// /// Looser than [`PortForwardingExpose`], because masquerade is: several prefixes are allowed on diff --git a/nat/src/portfw/expiry.rs b/nat/src/portfw/expiry.rs new file mode 100644 index 0000000000..53d7556a0b --- /dev/null +++ b/nat/src/portfw/expiry.rs @@ -0,0 +1,202 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +//! Port-forwarding flow expiry, on a clock the test drives. +//! +//! The same three dispositions `masquerade::expiry` covers, asked of the other direction. What +//! differs is what expiry *means* here: a masquerade flow that expires costs the tenant a +//! connection, while a port-forwarding flow that expires and is then re-established has to arrive at +//! the same backend, because the published tuple is a service address rather than an ephemeral one. +//! +//! Port forwarding takes its lifetime from the expose's idle timeout, which the generator draws as +//! absent, five seconds or five minutes -- so a property that wants an expiry has to outrun the +//! longest of them. That costs nothing on a virtual clock and would cost five minutes per case on a +//! real one, which is the whole reason this file can exist. + +#![cfg(test)] + +use crate::portfw::PortForwarder; +use crate::portfw::probe::{Arrival, Fabric, PAST_ANY_TIMEOUT, run}; +use crate::static_nat::probe::build; +use clock::Duration; +use config::external::overlay::vpcpeering::VpcExpose; +use flow_entry::flow_table::FlowLookup; +use lpm::prefix::{L4Protocol, PrefixWithOptionalPorts}; +use std::net::IpAddr; + +/// Comfortably inside every timeout the generator draws. +const WITHIN_LIFETIME: Duration = Duration::from_secs(1); + +/// A runtime whose clock starts paused and only moves when a property says so. +fn with_paused_clock>(body: impl FnOnce() -> F) { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_time() + .start_paused(true) + .build() + .unwrap_or_else(|e| unreachable!("{e}")); + runtime.block_on(body()); +} + +/// Move the clock and let every timer that is now due actually run. +async fn advance(by: Duration) { + tokio::time::advance(by).await; + for _ in 0..4 { + tokio::task::yield_now().await; + } +} + +/// One fixed rule: `172.16.0.0/30` ports 8000-8003 published to `10.0.0.0/30` ports 9000-9003. +/// +/// Fixed rather than generated. Expiry is about time, and a drawn shape would only add variance to +/// a property whose subject is the clock. +fn fabric() -> Fabric { + let expose = VpcExpose::empty() + .make_port_forwarding(Some(Duration::from_secs(5)), Some(L4Protocol::Tcp)) + .unwrap_or_else(|e| unreachable!("{e}")) + .ip(PrefixWithOptionalPorts::new( + "10.0.0.0/30".parse().unwrap_or_else(|_| unreachable!()), + Some(lpm::prefix::PortRange::new(9000, 9003).unwrap_or_else(|_| unreachable!())), + )) + .as_range(PrefixWithOptionalPorts::new( + "172.16.0.0/30".parse().unwrap_or_else(|_| unreachable!()), + Some(lpm::prefix::PortRange::new(8000, 8003).unwrap_or_else(|_| unreachable!())), + )) + .unwrap_or_else(|e| unreachable!("{e}")); + Fabric::build(&[expose]).unwrap_or_else(|| unreachable!("a fixed expose builds")) +} + +/// Send one inbound packet to a published tuple and report where it was forwarded. +fn forward( + lookup: &mut FlowLookup, + pfw: &mut PortForwarder, + peer: IpAddr, + published: (IpAddr, u16), +) -> Option<(IpAddr, u16)> { + let mut packet = build(peer, published.0, true, 1234, published.1); + Arrival::inbound().stamp(&mut packet); + let out = run(lookup, pfw, vec![packet], Arrival::inbound().dst_vpcd); + if out[0].is_done() { + return None; + } + let addr = out[0].ip_destination()?; + let port = out[0].transport_dst_port()?.get(); + ((addr, port) != published).then_some((addr, port)) +} + +/// A published service keeps working while time passes. +/// +/// The **preserved** disposition. A port-forwarding rule is configuration, not state: it does not +/// expire, and a service that stopped answering merely because time passed would be an outage with +/// no configuration change behind it. +#[test] +fn a_published_service_answers_while_time_passes() { + with_paused_clock(|| async { + let fabric = fabric(); + let (mut lookup, mut pfw) = fabric.stages(); + let peer = fabric.peer[0]; + let published = ( + "172.16.0.1" + .parse::() + .unwrap_or_else(|_| unreachable!()), + 8001, + ); + + for step in 0..6 { + assert!( + forward(&mut lookup, &mut pfw, peer, published).is_some(), + "the published service stopped answering at step {step}" + ); + advance(WITHIN_LIFETIME).await; + } + }); +} + +/// A service re-established after its flow expires reaches the same backend. +/// +/// The **never resurrected** disposition, in the form port forwarding needs. The flow carrying the +/// translation is state and does expire; the rule that produced it is configuration and does not. So +/// a client returning after an idle period must land where it landed before -- a published address +/// that moved between connections is a service that silently changed identity. +/// +/// Three epochs, and the middle one is longer than any timeout the configuration can name. That is +/// exactly the shape the clock facade was built for: unwritable while deadlines came from the wall +/// clock, and free now. +#[test] +fn a_service_re_established_after_expiry_reaches_the_same_backend() { + with_paused_clock(|| async { + let fabric = fabric(); + let (mut lookup, mut pfw) = fabric.stages(); + let peer = fabric.peer[0]; + let published = ( + "172.16.0.2" + .parse::() + .unwrap_or_else(|_| unreachable!()), + 8002, + ); + + let first = forward(&mut lookup, &mut pfw, peer, published) + .unwrap_or_else(|| unreachable!("a published tuple is forwarded")); + + advance(PAST_ANY_TIMEOUT).await; + + let again = forward(&mut lookup, &mut pfw, peer, published).unwrap_or_else(|| { + unreachable!( + "a published service could not be reached after its flow expired; the rule is \ + configuration and does not expire with the flow" + ) + }); + + assert_eq!( + first, again, + "{published:?} reached {first:?} and then, after its flow expired, {again:?}; a \ + published address moved between connections" + ); + }); +} + +/// Every published tuple still maps somewhere distinct after an expiry. +/// +/// The reason to check more than one: an expiry that dropped shared state could leave the rule +/// intact while the *mapping* it produces collapses, which one tuple on its own cannot show. +#[test] +fn published_tuples_stay_distinct_across_an_expiry() { + with_paused_clock(|| async { + let fabric = fabric(); + let (mut lookup, mut pfw) = fabric.stages(); + let peer = fabric.peer[0]; + + let published: Vec<(IpAddr, u16)> = (0..4u8) + .map(|i| { + ( + format!("172.16.0.{i}") + .parse::() + .unwrap_or_else(|_| unreachable!()), + 8000 + u16::from(i), + ) + }) + .collect(); + + let before: Vec<_> = published + .iter() + .map(|p| forward(&mut lookup, &mut pfw, peer, *p)) + .collect(); + + advance(PAST_ANY_TIMEOUT).await; + + let after: Vec<_> = published + .iter() + .map(|p| forward(&mut lookup, &mut pfw, peer, *p)) + .collect(); + + assert_eq!( + before, after, + "the mapping from published tuples to backends changed across an expiry" + ); + let distinct: std::collections::BTreeSet<_> = after.iter().flatten().collect(); + assert_eq!( + distinct.len(), + after.iter().flatten().count(), + "two published tuples share a backend after an expiry: {after:?}" + ); + }); +} diff --git a/nat/src/portfw/fuzz.rs b/nat/src/portfw/fuzz.rs new file mode 100644 index 0000000000..771bd51790 --- /dev/null +++ b/nat/src/portfw/fuzz.rs @@ -0,0 +1,478 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +//! Properties of the port-forwarding network function. +//! +//! The third and last NAT flavour, and the one where the campaign's only confirmed configuration +//! bug lived -- `fix(config): Refuse a port-forwarding expose the dataplane cannot build`. That was +//! found by reading code and pinned at the *configuration* level. These cover the stage. +//! +//! # Why the direction matters +//! +//! Static NAT and masquerade translate a **source** on the way out. Port forwarding translates a +//! **destination** on the way in, so every property here reads the other half of the five-tuple, and +//! the failure modes are different in kind. A masquerade mistake leaks one tenant's traffic to +//! another; a port-forwarding mistake delivers the outside world to an address inside a tenant that +//! never published it. +//! +//! # No oracle +//! +//! The mapping is positional -- one prefix and port range onto another of the same size, address for +//! address -- so it *could* be predicted. Deliberately not: a prediction here is a second copy of +//! `PortFwEntry`'s arithmetic, and two copies disagree. The properties below are relations and +//! membership tests, as everywhere else in this campaign. + +#![cfg(test)] + +use crate::portfw::probe::{Arrival, Fabric, ProbeSpec, run}; +use bolero::{Driver, TypeGenerator, ValueGenerator}; +use concurrency::sync::atomic::{AtomicUsize, Ordering}; +use config::external::overlay::vpcpeering::VpcExpose; +use config::external::overlay::vpcpeering::contract::PortForwardingExposes; +use net::buffer::TestBuffer; +use net::packet::Packet; +use std::collections::BTreeMap; +use std::net::IpAddr; +use std::num::NonZero; + +/// The fewest reaching draws any property may see before it is considered vacuous. +const MIN_REACHED: usize = 8; + +/// Exposes per configuration. One per protocol key; see `PortForwardingExposes`. +const MAX_EXPOSES: u8 = 2; + +/// Packets per configuration. +const PROBES: usize = 8; + +#[derive(Debug, Clone, Copy)] +struct Scenario { + strays: bool, +} + +impl ValueGenerator for Scenario { + type Output = (Vec, Vec); + + fn generate(&self, driver: &mut D) -> Option { + let exposes = PortForwardingExposes(MAX_EXPOSES).generate(driver)?; + let mut probes = Vec::with_capacity(PROBES); + for _ in 0..PROBES { + let mut probe = ProbeSpec::generate(driver)?; + if !self.strays { + probe.clear_stray(); + } + probes.push(probe); + } + Some((exposes, probes)) + } +} + +/// Run a property inside a tokio runtime. +/// +/// `FlowTable::insert` spawns a per-flow expiry timer, so an insert outside a runtime context +/// panics. Not paused: these properties are about the mapping, and `portfw::expiry` covers time. +fn with_runtime(body: impl FnOnce()) { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_time() + .build() + .unwrap_or_else(|e| unreachable!("{e}")); + let _guard = runtime.enter(); + body(); +} + +fn fabric(exposes: &[VpcExpose]) -> Option { + let fabric = Fabric::build(exposes)?; + fabric.is_probeable().then_some(fabric) +} + +/// The destination half of a packet's five-tuple, which is what port forwarding rewrites. +fn destination_of(packet: &Packet) -> (IpAddr, u16) { + ( + packet + .ip_destination() + .unwrap_or_else(|| unreachable!("a probe is always an ip packet")), + packet.transport_dst_port().map_or(0, NonZero::get), + ) +} + +/// The source half, for judging a reply. +fn source_of(packet: &Packet) -> (IpAddr, u16) { + ( + packet + .ip_source() + .unwrap_or_else(|| unreachable!("a probe is always an ip packet")), + packet.transport_src_port().map_or(0, NonZero::get), + ) +} + +#[derive(Default)] +struct Tally { + seen: AtomicUsize, + built: AtomicUsize, + reached: AtomicUsize, +} + +impl Tally { + /// Assert the run was not vacuous, on a floor relative to what was built rather than an + /// absolute count, so the guard measures the property rather than the machine. + fn report(&self, what: &str) { + let (seen, built, reached) = ( + self.seen.load(Ordering::Relaxed), + self.built.load(Ordering::Relaxed), + self.reached.load(Ordering::Relaxed), + ); + println!("{what}: {built}/{seen} configurations built, {reached} packets reached it"); + assert!( + built * 2 >= seen, + "only {built} of {seen} configurations built, so this checked much less than it looks \ + like it did" + ); + assert!( + reached >= MIN_REACHED && reached * 2 >= built, + "{reached} packets reached the {what} assertion across {built} configurations; this \ + property has gone vacuous" + ); + } +} + +/// Send one inbound packet and report the destination it was forwarded to, if it was. +fn forward( + fabric: &Fabric, + lookup: &mut flow_entry::flow_table::FlowLookup, + pfw: &mut crate::portfw::PortForwarder, + probe: &crate::portfw::probe::Probe, +) -> Option<(IpAddr, u16)> { + let before = probe.destination; + let out = run(lookup, pfw, vec![probe.packet()], probe.arrival.dst_vpcd); + let _ = fabric; + if out[0].is_done() { + return None; + } + let after = destination_of(&out[0]); + (after != before).then_some(after) +} + +/// A forwarded packet reaches an address the rule publishes it to, and the reply comes back as the +/// tuple the outside world used. +/// +/// The headline property. The reply is the part that matters operationally: a port-forwarded +/// connection whose return traffic is not rewritten back to the published tuple is a connection the +/// client drops, because the reply appears to come from an address it never contacted. +#[test] +fn a_forwarded_packet_answers_as_the_published_tuple() { + let tally = Tally::default(); + + with_runtime(|| { + bolero::check!() + .with_generator(Scenario { strays: false }) + .cloned() + .for_each(|(exposes, probes): (Vec, Vec)| { + tally.seen.fetch_add(1, Ordering::Relaxed); + let Some(fabric) = fabric(&exposes) else { + return; + }; + tally.built.fetch_add(1, Ordering::Relaxed); + let (mut lookup, mut pfw) = fabric.stages(); + + for spec in &probes { + let probe = (*spec).resolve(&fabric); + let published = probe.destination; + let Some(translated) = forward(&fabric, &mut lookup, &mut pfw, &probe) else { + continue; + }; + + let back = run( + &mut lookup, + &mut pfw, + vec![probe.reply(translated)], + Arrival::outbound().dst_vpcd, + ); + assert_eq!( + source_of(&back[0]), + published, + "{published:?} was forwarded to {translated:?}, and the reply came back \ + as {:?} instead of the tuple the client used", + source_of(&back[0]) + ); + tally.reached.fetch_add(1, Ordering::Relaxed); + } + }); + }); + + tally.report("reversibility"); +} + +/// A forwarded packet lands inside the private side the rule names. +/// +/// The containment claim, and the one that consults the configuration -- legitimately, as a +/// membership test rather than a prediction of which member. +/// +/// This is the port-forwarding failure that matters most. A translation landing outside the +/// declared private range delivers unsolicited traffic from outside the fabric to an address inside +/// a tenant that never published it, which is a hole rather than a misroute. +#[test] +fn a_forwarded_packet_lands_inside_the_published_target() { + let tally = Tally::default(); + + with_runtime(|| { + bolero::check!() + .with_generator(Scenario { strays: false }) + .cloned() + .for_each(|(exposes, probes): (Vec, Vec)| { + tally.seen.fetch_add(1, Ordering::Relaxed); + let Some(fabric) = fabric(&exposes) else { + return; + }; + tally.built.fetch_add(1, Ordering::Relaxed); + let (mut lookup, mut pfw) = fabric.stages(); + + for spec in &probes { + let probe = (*spec).resolve(&fabric); + let published = probe.destination; + let Some((addr, port)) = forward(&fabric, &mut lookup, &mut pfw, &probe) else { + continue; + }; + + assert!( + fabric.is_private(addr, port), + "{published:?} was forwarded to {addr}:{port}, which no rule names as a \ + target; that address never published this service" + ); + tally.reached.fetch_add(1, Ordering::Relaxed); + } + }); + }); + + tally.report("containment"); +} + +/// Distinct published tuples reach distinct targets. +/// +/// A port-forwarding rule maps one range onto another of the same size, positionally, so it is a +/// bijection by construction. Two published tuples collapsing onto one target would silently merge +/// two services, and the reverse mapping could then only answer one of them. +/// +/// Enumerated rather than drawn: a collision is only visible across distinct inputs, and the +/// generator keeps both sides small enough to sweep. +#[test] +fn distinct_published_tuples_reach_distinct_targets() { + let tally = Tally::default(); + + with_runtime(|| { + bolero::check!() + .with_generator(Scenario { strays: false }) + .cloned() + .for_each(|(exposes, _probes): (Vec, Vec)| { + tally.seen.fetch_add(1, Ordering::Relaxed); + let Some(fabric) = fabric(&exposes) else { + return; + }; + tally.built.fetch_add(1, Ordering::Relaxed); + let (mut lookup, mut pfw) = fabric.stages(); + + let mut taken: BTreeMap<(IpAddr, u16), (IpAddr, u16)> = BTreeMap::new(); + for (public, _private, tcp) in &fabric.rules { + let tcp = *tcp; + for (addr, port) in public.every() { + let mut packet = + crate::static_nat::probe::build(fabric.peer[0], addr, tcp, 1024, port); + Arrival::inbound().stamp(&mut packet); + let out = run( + &mut lookup, + &mut pfw, + vec![packet], + Arrival::inbound().dst_vpcd, + ); + if out[0].is_done() { + continue; + } + let after = destination_of(&out[0]); + if after == (addr, port) { + continue; + } + if let Some(previous) = taken.insert(after, (addr, port)) { + assert_eq!( + previous, + (addr, port), + "published tuples {previous:?} and {:?} both reach {after:?}, so \ + two services share one target", + (addr, port) + ); + } + tally.reached.fetch_add(1, Ordering::Relaxed); + } + } + }); + }); + + tally.report("injectivity"); +} + +/// Nothing is forwarded that the configuration does not publish, or that did not ask to be. +/// +/// Every reason a packet may not be forwarded, taken together: the destination is an address no rule +/// publishes, the port is outside the published range, the source vpc keys no rule, or nothing asked +/// for port forwarding at all. +/// +/// The port-outside-range case is the sharp one. A rule publishes a *range*, and an off-by-one at +/// either end forwards a port the operator did not open -- which is the whole difference between a +/// port-forwarding rule and an open door. +/// +/// # Three gates, not one +/// +/// Break testing this property turned up something worth writing down: **no single edit opens that +/// door.** A port past the published range is refused independently by +/// +/// 1. `RangeSet::lookup`, which bounds the sought port above, +/// 2. `PortRange::contains` by way of `indexof`, which the mapping consults, and +/// 3. the size-matched arithmetic in `map_port_to`, where an index past the source range has no +/// answer in the target range. +/// +/// Breaking any one, or any two, still refuses the packet -- the property only fires with all three +/// broken at once. That is defence in depth rather than a redundant check, and it is the reason this +/// property looked vacuous at first: it is not, the code is simply hard to break here. +/// +/// One observation from doing it: a packet whose port is outside the published range is dropped with +/// `DoneReason::InternalFailure`, which is not what happened. Nothing is internally broken -- the +/// operator did not publish that port. Attribution, not correctness. +#[test] +fn nothing_is_forwarded_that_was_not_published() { + let tally = Tally::default(); + + with_runtime(|| { + bolero::check!() + .with_generator(Scenario { strays: true }) + .cloned() + .for_each(|(exposes, probes): (Vec, Vec)| { + tally.seen.fetch_add(1, Ordering::Relaxed); + let Some(fabric) = fabric(&exposes) else { + return; + }; + tally.built.fetch_add(1, Ordering::Relaxed); + let (mut lookup, mut pfw) = fabric.stages(); + + for spec in &probes { + let probe = (*spec).resolve(&fabric); + if probe.asks_for_forwarding() && probe.published { + continue; + } + let (before, stray) = (probe.destination, probe.stray); + let out = run( + &mut lookup, + &mut pfw, + vec![probe.packet()], + probe.arrival.dst_vpcd, + ); + + // Dropping is a legitimate answer for a packet no rule covers; forwarding it is + // not. + if !out[0].is_done() { + assert_eq!( + destination_of(&out[0]), + before, + "a packet to {before:?} was forwarded although {stray:?} meant no rule \ + published it" + ); + } + tally.reached.fetch_add(1, Ordering::Relaxed); + } + }); + }); + + tally.report("permission"); +} + +/// Forwarding a destination leaves the source alone. +/// +/// The frame condition. Port forwarding is destination NAT on this path, so a rewritten source would +/// be the stage reaching into the other half of the tuple -- and the reply, which is matched on that +/// source, would then have nowhere to go. +#[test] +fn forwarding_touches_only_the_destination() { + let tally = Tally::default(); + + with_runtime(|| { + bolero::check!() + .with_generator(Scenario { strays: false }) + .cloned() + .for_each(|(exposes, probes): (Vec, Vec)| { + tally.seen.fetch_add(1, Ordering::Relaxed); + let Some(fabric) = fabric(&exposes) else { + return; + }; + tally.built.fetch_add(1, Ordering::Relaxed); + let (mut lookup, mut pfw) = fabric.stages(); + + for spec in &probes { + let probe = (*spec).resolve(&fabric); + let expected = (probe.source, probe.sport); + let out = run( + &mut lookup, + &mut pfw, + vec![probe.packet()], + probe.arrival.dst_vpcd, + ); + if out[0].is_done() { + continue; + } + + assert_eq!( + source_of(&out[0]), + expected, + "destination forwarding rewrote the source, so the reply has nowhere to go" + ); + tally.reached.fetch_add(1, Ordering::Relaxed); + } + }); + }); + + tally.report("frame"); +} + +/// A flow keeps the target it was first given. +/// +/// The first packet of a flow consults the table and writes a flow pair; the second takes the fast +/// path through that pair. A stage that re-resolved would produce a legal-looking packet each time, +/// and a connection whose packets arrive at two different backends is broken in a way no +/// table-level test would see. +#[test] +fn a_forwarded_flow_keeps_its_target() { + let tally = Tally::default(); + + with_runtime(|| { + bolero::check!() + .with_generator(Scenario { strays: false }) + .cloned() + .for_each(|(exposes, probes): (Vec, Vec)| { + tally.seen.fetch_add(1, Ordering::Relaxed); + let Some(fabric) = fabric(&exposes) else { + return; + }; + tally.built.fetch_add(1, Ordering::Relaxed); + let (mut lookup, mut pfw) = fabric.stages(); + + for spec in &probes { + let probe = (*spec).resolve(&fabric); + let Some(first) = forward(&fabric, &mut lookup, &mut pfw, &probe) else { + continue; + }; + let Some(second) = forward(&fabric, &mut lookup, &mut pfw, &probe) else { + panic!( + "the second packet of a flow to {:?} was not forwarded at all, though \ + the first reached {first:?}", + probe.destination + ) + }; + + assert_eq!( + first, second, + "a flow to {:?} reached {first:?} and then {second:?}, so its packets are \ + split across two backends", + probe.destination + ); + tally.reached.fetch_add(1, Ordering::Relaxed); + } + }); + }); + + tally.report("stability"); +} diff --git a/nat/src/portfw/mod.rs b/nat/src/portfw/mod.rs index 20d367e319..2b6bd43b66 100644 --- a/nat/src/portfw/mod.rs +++ b/nat/src/portfw/mod.rs @@ -3,11 +3,14 @@ //! Port forwarding +mod expiry; mod flow_state; +mod fuzz; pub(crate) mod icmp_handling; mod nf; mod packet; mod portfwtable; +mod probe; mod protocol; mod test; diff --git a/nat/src/portfw/probe.rs b/nat/src/portfw/probe.rs new file mode 100644 index 0000000000..8ded09c5f8 --- /dev/null +++ b/nat/src/portfw/probe.rs @@ -0,0 +1,407 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +//! Packets drawn relative to a port-forwarding configuration. +//! +//! The third network function harness, and the same shape as the first two: a [`ProbeSpec`] of bare +//! indices, resolved against a built [`Fabric`], with the configuration a parameter to resolution +//! rather than a predicate to filter against. +//! +//! # What is different about port forwarding +//! +//! Static NAT and masquerade both translate a packet's **source** on the way out. Port forwarding +//! translates its **destination** on the way in, which reverses everything: +//! +//! * The interesting packet arrives *from the peer*, addressed to a public tuple the local vpc +//! published, and leaves addressed to a private one. `expose.ips` is the private side and +//! `expose.nat.as_range` the public one, as everywhere else -- but here traffic enters at +//! `as_range` rather than leaving through it. +//! * A rule is keyed by `(source vpc, protocol)`, not by address. So the *source* vpc annotation +//! selects the rule and the destination address selects the mapping within it. +//! * The mapping is **positional**: one prefix and port range onto another of the same size, address +//! for address and port for port. That is a stricter contract than static NAT's, which only +//! requires the two sides to have equal totals. +//! +//! It is stateful like masquerade -- the first packet creates a flow pair carrying the translation, +//! and later packets in either direction take a fast path through it -- so the stage ordering +//! constraint is the same, and [`run`] handles it the same way. + +#![cfg(test)] + +use crate::portfw::{PortForwarder, PortFwTableWriter, build_port_forwarding_configuration}; +use crate::static_nat::probe::{build, vni}; +use bolero::TypeGenerator; +use clock::Duration; +use concurrency::sync::Arc; +use config::external::overlay::vpcpeering::VpcExpose; +use config::external::overlay::vpcpeering::contract::{ + LOCAL_VNI, REMOTE_VNI, overlay_with_exposes, +}; +use flow_entry::flow_table::{FlowLookup, FlowTable}; +use lpm::prefix::Prefix; +use net::buffer::TestBuffer; +use net::packet::{Packet, VpcDiscriminant}; +use net::vxlan::Vni; +use pipeline::NetworkFunction; +use std::net::IpAddr; + +/// Flow table capacity, large enough that a translation failure is never the table being full. +const FLOW_CAPACITY: usize = 4096; + +/// A VNI no generated configuration uses. +const ABSENT_VNI: u32 = 4_000; + +/// One side of a port-forwarding rule: a prefix and the port range it carries. +#[derive(Debug, Clone, Copy)] +pub(crate) struct Side { + pub(crate) prefix: Prefix, + pub(crate) first_port: u16, + pub(crate) last_port: u16, +} + +impl Side { + /// Whether this side covers an address and port. + pub(crate) fn covers(&self, addr: IpAddr, port: u16) -> bool { + self.prefix.covers_addr(&addr) && port >= self.first_port && port <= self.last_port + } + + /// An address and port inside this side, chosen by two arbitrary indices. + /// + /// Total, so a draw always lands on something the rule covers rather than beside it. + pub(crate) fn endpoint(&self, host: u16, port: u16) -> (IpAddr, u16) { + let span = self.span(); + let offset = u128::from(host) % span.max(1); + let addr = match self.prefix.as_address() { + IpAddr::V4(base) => IpAddr::V4( + u32::try_from(u128::from(base.to_bits()) + offset) + .unwrap_or_else(|_| unreachable!()) + .into(), + ), + IpAddr::V6(base) => IpAddr::V6((base.to_bits() + offset).into()), + }; + let ports = u32::from(self.last_port - self.first_port) + 1; + let port = self.first_port + u16::try_from(u32::from(port) % ports).unwrap_or(0); + (addr, port) + } + + /// How many addresses this side covers. + fn span(&self) -> u128 { + let host_bits = match self.prefix.as_address() { + IpAddr::V4(_) => 32 - u32::from(self.prefix.length()), + IpAddr::V6(_) => 128 - u32::from(self.prefix.length()), + }; + 1u128 << host_bits.min(64) + } + + /// A spread of the address-and-port pairs this side covers, for a property that sweeps. + /// + /// **Capped.** A rule may publish 256 addresses over 1024 ports, and enumerating that costs a + /// quarter of a million packets -- which one property did, exhausting its whole budget on two + /// configurations. Two hundred and fifty six pairs is plenty to catch a collision and leaves the + /// budget to explore configurations, which is where the shapes differ. + /// + /// Strided rather than truncated, so the sample spans the whole of both dimensions instead of + /// sitting in one corner: a mapping that goes wrong only at the top of a range would survive a + /// sample of the bottom of it. + pub(crate) fn every(&self) -> Vec<(IpAddr, u16)> { + const CAP: usize = 256; + let hosts = u32::try_from(self.span()) + .unwrap_or(u32::from(u16::MAX)) + .max(1); + let ports = u32::from(self.last_port - self.first_port) + 1; + let total = u64::from(hosts) * u64::from(ports); + let stride = (total / CAP as u64).max(1); + + let mut out = Vec::new(); + let mut index = 0u64; + while index < total { + let host = u16::try_from(index / u64::from(ports)).unwrap_or(u16::MAX); + let port = u16::try_from(index % u64::from(ports)).unwrap_or(0); + out.push(self.endpoint(host, port)); + index += stride; + } + out + } +} + +/// A built port-forwarding configuration, and the two sides each rule declares. +pub(crate) struct Fabric { + flow_table: Arc, + writer: PortFwTableWriter, + /// One entry per rule: the public side traffic arrives at, the private side it reaches, and + /// whether the rule is keyed on TCP. + /// + /// The protocol has to be carried, not drawn. A rule is keyed by `(source vpc, protocol)`, so a + /// probe that addresses one rule's public range over the other rule's protocol matches nothing + /// -- and a property expecting *not* to forward it then passes for the wrong reason. That is how + /// the port-range guard here was found to be vacuous: it never reached the gate it was testing. + pub(crate) rules: Vec<(Side, Side, bool)>, + /// Addresses in the peer vpc, which the rules never name. + pub(crate) peer: Vec, +} + +impl Fabric { + /// Build the table a set of exposes implies, or `None` if the overlay they form is not valid. + pub(crate) fn build(exposes: &[VpcExpose]) -> Option { + let overlay = overlay_with_exposes(exposes.to_vec()).ok()?; + let validated = overlay.validate().ok()?; + let ruleset = build_port_forwarding_configuration(validated.vpc_table()).ok()?; + + let mut writer = PortFwTableWriter::new(); + writer.update_table(&ruleset).ok()?; + + let rules: Vec<(Side, Side, bool)> = exposes + .iter() + .filter_map(|expose| { + let public = expose.nat.as_ref()?.as_range.first()?; + let private = expose.ips.first()?; + let (pub_ports, priv_ports) = (public.ports()?, private.ports()?); + let tcp = expose.nat.as_ref()?.proto != lpm::prefix::L4Protocol::Udp; + Some(( + Side { + prefix: public.prefix(), + first_port: pub_ports.start(), + last_port: pub_ports.end(), + }, + Side { + prefix: private.prefix(), + first_port: priv_ports.start(), + last_port: priv_ports.end(), + }, + tcp, + )) + }) + .collect(); + + let peer = match rules + .first() + .map(|(public, _, _)| public.prefix.as_address()) + { + Some(IpAddr::V6(_)) => vec![ + "2001:db8:ffff::1" + .parse() + .unwrap_or_else(|_| unreachable!()), + "2001:db8:ffff::2" + .parse() + .unwrap_or_else(|_| unreachable!()), + ], + _ => vec![ + "3.3.3.1".parse().unwrap_or_else(|_| unreachable!()), + "3.3.3.2".parse().unwrap_or_else(|_| unreachable!()), + ], + }; + + Some(Self { + flow_table: Arc::new(FlowTable::new(FLOW_CAPACITY)), + writer, + rules, + peer, + }) + } + + /// The two stages a port-forwarded packet passes through. + pub(crate) fn stages(&self) -> (FlowLookup, PortForwarder) { + ( + FlowLookup::new("flow-lookup", self.flow_table.clone()), + PortForwarder::new( + "port-forwarder", + self.writer.reader(), + self.flow_table.clone(), + ), + ) + } + + pub(crate) fn is_probeable(&self) -> bool { + !self.rules.is_empty() + } + + /// Whether any rule's private side covers this address and port. + pub(crate) fn is_private(&self, addr: IpAddr, port: u16) -> bool { + self.rules + .iter() + .any(|(_, private, _)| private.covers(addr, port)) + } +} + +/// Put a batch through the stages, in the order the real pipeline uses. +/// +/// As for masquerade: `FlowLookup` attaches a flow entry only to a packet whose `dst_vpcd` is +/// absent, and the flow filter that sets `dst_vpcd` runs after it, so the annotation arrives +/// *between* the two stages. +pub(crate) fn run( + lookup: &mut FlowLookup, + pfw: &mut PortForwarder, + packets: Vec>, + dst_vpcd: Option, +) -> Vec> { + let mut looked: Vec<_> = lookup.process(packets.into_iter()).collect(); + for packet in &mut looked { + packet.meta_mut().dst_vpcd = dst_vpcd.map(VpcDiscriminant::from_vni); + } + pfw.process(looked.into_iter()).collect() +} + +/// The metadata a packet must carry for [`PortForwarder`] to look at it. +#[derive(Debug, Clone, Copy)] +pub(crate) struct Arrival { + /// Selects the rule: a rule is keyed by the vpc traffic came *from*. + pub(crate) src_vpcd: Option, + /// Supplied after the flow lookup, standing in for the flow filter. + pub(crate) dst_vpcd: Option, + pub(crate) wants_port_forwarding: bool, +} + +impl Arrival { + /// Traffic arriving from the peer for a published tuple. This is the direction port forwarding + /// exists for. + pub(crate) fn inbound() -> Self { + Self { + src_vpcd: Some(vni(REMOTE_VNI)), + dst_vpcd: Some(vni(LOCAL_VNI)), + wants_port_forwarding: true, + } + } + + /// The reply, leaving the local vpc for the peer. + pub(crate) fn outbound() -> Self { + Self { + src_vpcd: Some(vni(LOCAL_VNI)), + dst_vpcd: Some(vni(REMOTE_VNI)), + wants_port_forwarding: true, + } + } + + /// Everything an upstream stage sets before the flow lookup. + pub(crate) fn stamp(self, packet: &mut Packet) { + let meta = packet.meta_mut(); + meta.src_vpcd = self.src_vpcd.map(VpcDiscriminant::from_vni); + meta.set_overlay(true); + meta.set_keep(true); + meta.set_port_forwarding(self.wants_port_forwarding); + } +} + +/// A deliberate deviation from a packet the configuration forwards. +#[derive(Debug, Clone, Copy, TypeGenerator)] +pub(crate) enum Stray { + /// A destination no rule publishes. Nothing should forward it. + DestinationNotPublished, + /// A port outside the published range, on a published address. + PortOutsideRange, + /// A source vpc no rule is keyed by. + UnknownSourceVni, + /// Nothing asked for port forwarding. + NotAskedFor, +} + +/// A drawn probe, before it knows anything about a configuration. +#[derive(Debug, Clone, Copy, TypeGenerator)] +pub(crate) struct ProbeSpec { + rule: u8, + host: u16, + port: u16, + peer: u8, + sport: u16, + stray: Option, +} + +/// A probe resolved against a fabric. +pub(crate) struct Probe { + /// The public address and port the packet is addressed to. + pub(crate) destination: (IpAddr, u16), + /// The peer address it comes from. + pub(crate) source: IpAddr, + pub(crate) sport: u16, + pub(crate) tcp: bool, + /// Whether some rule publishes `destination`. + pub(crate) published: bool, + pub(crate) arrival: Arrival, + pub(crate) stray: Option, +} + +impl Probe { + /// Whether port forwarding was asked to translate this packet and given what it needs to. + pub(crate) fn asks_for_forwarding(&self) -> bool { + self.arrival.wants_port_forwarding && self.arrival.src_vpcd == Some(vni(REMOTE_VNI)) + } + + /// The inbound packet, which may be built more than once so a property can send the same flow + /// twice and reach the fast path. + pub(crate) fn packet(&self) -> Packet { + let mut packet = build( + self.source, + self.destination.0, + self.tcp, + self.sport, + self.destination.1, + ); + self.arrival.stamp(&mut packet); + packet + } + + /// The reply the forwarded-to host sends back, from the private tuple it was reached on. + pub(crate) fn reply(&self, translated: (IpAddr, u16)) -> Packet { + let mut packet = build( + translated.0, + self.source, + self.tcp, + translated.1, + self.sport, + ); + Arrival::outbound().stamp(&mut packet); + packet + } +} + +impl ProbeSpec { + /// Drop the deviation, leaving a packet the configuration is meant to forward. + pub(crate) fn clear_stray(&mut self) { + self.stray = None; + } + + /// Interpret this draw against a fabric. + pub(crate) fn resolve(self, fabric: &Fabric) -> Probe { + // The protocol comes from the rule, not from the draw: addressing a rule's public range + // over the wrong protocol matches nothing, and every property would then be judging a + // packet that never reached the code it is about. + let (public, _private, tcp) = fabric.rules[self.rule as usize % fabric.rules.len()]; + let mut arrival = Arrival::inbound(); + let mut destination = public.endpoint(self.host, self.port); + let source = fabric.peer[self.peer as usize % fabric.peer.len()]; + let mut published = true; + + match self.stray { + None => {} + Some(Stray::DestinationNotPublished) => { + // An address in the peer's own space, which no rule publishes. + destination = (fabric.peer[0], destination.1); + published = false; + } + Some(Stray::PortOutsideRange) => { + // Just past the top of the range, on an address a rule does publish. + if public.last_port < u16::MAX { + destination = (destination.0, public.last_port + 1); + published = false; + } + } + Some(Stray::UnknownSourceVni) => arrival.src_vpcd = Some(vni(ABSENT_VNI)), + Some(Stray::NotAskedFor) => arrival.wants_port_forwarding = false, + } + + Probe { + destination, + source, + sport: self.sport.max(1), + tcp, + published, + arrival, + stray: self.stray, + } + } +} + +/// How long a port-forwarding flow lives before it is first refreshed. +/// +/// The generator draws `None`, five seconds or five minutes for the idle timeout, so a property that +/// wants a flow to have expired must outrun the longest of them. +pub(crate) const PAST_ANY_TIMEOUT: Duration = Duration::from_mins(30); From 256f1e841c9bca5b639c826c778848a74cce39ea Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Mon, 17 Aug 2026 22:22:37 -0600 Subject: [PATCH 04/37] test(net): Test the flow expiry state machine as an algebra `FlowInfo` is the state every stateful NAT flavour shares. Masquerade, port forwarding and the flow table all keep a flow alive by refreshing this object, and every timeout in the datapath is ultimately a comparison against its `expires_at`. It had **no tests of its own**. ## An algebra, one level down The network function harnesses build a configuration from an algebra of operations and judge the result by relations. This is the same idea at unit scale and much cheaper for it: `Op` is the vocabulary a flow supports -- refresh it two ways, move its status, invalidate it, let time pass -- and the headline property asserts an invariant after *every prefix* of a drawn sequence rather than after one fixed order. That shape is what the subject needs. `reset_expiry` is gated on status, `extend_expiry` on a different subset of it, and both on the clock; a test with a fixed order would walk one path through that lattice and call it covered. ## The properties * **expiry never moves backwards** -- the invariant the whole mechanism rests on. A deadline that moves earlier is a flow that dies while in use: the timer fires early, the entry is dropped, and the NAT state goes with it. This is also the invariant that was quietly violated while deadlines came from the wall clock and timers from tokio's -- the comparison inside `reset_expiry_unchecked` was between values from two different timelines. * **a refused refresh changes nothing** -- the frame condition, and load bearing: every production call site discards the result with `let _ =`, so a refusal that had already moved the deadline would be a silent write behind an error return. * **a refresh is permitted exactly when the status allows** -- stated as an iff, because a gate that is too permissive is the defect and a one-way check cannot see it. * **invalidating is idempotent and cancels the timer** -- `invalidate` is reached more than once for the same flow as a matter of course. A flow marked cancelled whose token still sleeps is an entry that lingers to its original deadline. * **a related pair refers to its partner** -- `related_pair` builds two flows that each hold a `Weak` to the other, through `Arc::new_uninit` and raw pointer writes, because neither can exist before the other. Nothing exercised that round trip: that each `Weak` upgrades, and to the *other* flow rather than to itself. * **a pair needs exactly one initiator**, and **every status survives its byte** -- the latter matters because the status is read back through a `TryFrom` that panics on an unrecognised value. ## Break tested Five breaks, each firing exactly one property and leaving the other six green: | break | fires | | --- | --- | | refresh may move the deadline backwards | expiry never moves backwards | | a cancelled flow may be refreshed | permitted-exactly-when | | invalidate leaves the timer sleeping | idempotent-and-cancels | | each half of a pair refers to itself | related pair | | `invalidate_pair` forgets the partner | related pair | ## Coverage `net/src/flows/flow_info.rs` 73.63% -> **81.09%** lines, 70.00% -> **100.00%** branches. Workspace 76.69 -> 76.92% lines, 68.35 -> 68.51% branches, which also carries the port-forwarding chapter. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 1d89608c798e1047794eb6342061ac5e418770ec) --- Cargo.lock | 1 + net/Cargo.toml | 1 + net/src/flows/flow_info_fuzz.rs | 406 ++++++++++++++++++++++++++++++++ net/src/flows/mod.rs | 1 + 4 files changed, 409 insertions(+) create mode 100644 net/src/flows/flow_info_fuzz.rs diff --git a/Cargo.lock b/Cargo.lock index b5be7107c5..ed85e8199c 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1752,6 +1752,7 @@ dependencies = [ "strum 0.28.0", "strum_macros 0.28.0", "thiserror", + "tokio", "tokio-util", "tracing", ] diff --git a/net/Cargo.toml b/net/Cargo.toml index 8c4bb70aa5..5059c423fd 100644 --- a/net/Cargo.toml +++ b/net/Cargo.toml @@ -45,3 +45,4 @@ tracing = { workspace = true } clock = { workspace = true, features = ["virtual"] } ahash = { workspace = true, features = ["no-rng"] } bolero = { workspace = true, features = ["std"] } +tokio = { workspace = true, features = ["macros", "rt", "test-util", "time"] } diff --git a/net/src/flows/flow_info_fuzz.rs b/net/src/flows/flow_info_fuzz.rs new file mode 100644 index 0000000000..068a56c58d --- /dev/null +++ b/net/src/flows/flow_info_fuzz.rs @@ -0,0 +1,406 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +//! Properties of the flow expiry state machine. +//! +//! [`FlowInfo`] is the piece of state every NAT flavour shares. Static NAT does not use it, but +//! masquerade, port forwarding and the flow table itself all keep a flow alive by refreshing this +//! object, and every timeout in the datapath is ultimately a comparison against its `expires_at`. +//! It had no tests of its own. +//! +//! # An algebra, at unit scale +//! +//! The network function harnesses build a configuration from an algebra of operations and judge the +//! result by relations. This is the same idea one level down, and cheaper for it: [`Op`] is the +//! vocabulary a flow supports -- refresh it two ways, move its status, invalidate it, and let time +//! pass -- and the properties below are invariants that must hold after *every* prefix of a drawn +//! sequence, rather than statements about a particular one. +//! +//! That shape matters here because the operations interact. `reset_expiry` is gated on status, +//! `extend_expiry` on a different subset of it, and both are gated on the clock; a property that +//! fixed the order would test one path through a lattice and call it covered. +//! +//! # Why the clock has to be driven +//! +//! `reset_expiry` computes `clock::now() + duration`, and refuses the result if it would move the +//! deadline earlier. Whether it refuses therefore depends on how much time has passed since the last +//! refresh -- so the interesting cases are only reachable by advancing the clock between operations, +//! which is what [`Op::Advance`] is for. On the wall clock those cases arrive after seconds of real +//! sleeping, or never. + +#![cfg(test)] + +use crate::FlowKey; +use crate::flows::FlowInfoFlags; +use crate::flows::flow_info::{FlowInfo, FlowInfoError, FlowStatus}; +use bolero::TypeGenerator; +use clock::Duration; +use std::net::IpAddr; + +/// A duration small enough that no sequence of them can overflow an `Instant`. +#[derive(Debug, Clone, Copy, TypeGenerator)] +struct Millis(u16); + +impl Millis { + fn duration(self) -> Duration { + Duration::from_millis(u64::from(self.0)) + } +} + +/// The status values a flow may be moved to. +#[derive(Debug, Clone, Copy, TypeGenerator)] +enum Status { + Active, + Cancelled, + Expired, + Detached, +} + +impl From for FlowStatus { + fn from(status: Status) -> Self { + match status { + Status::Active => FlowStatus::Active, + Status::Cancelled => FlowStatus::Cancelled, + Status::Expired => FlowStatus::Expired, + Status::Detached => FlowStatus::Detached, + } + } +} + +/// One operation a flow supports. +/// +/// Deliberately includes both the checked and unchecked refreshes. They are different functions with +/// different guarantees, production calls both, and the unchecked ones are where an invariant can be +/// broken without any error being returned. +#[derive(Debug, Clone, Copy, TypeGenerator)] +enum Op { + ExtendChecked(Millis), + ExtendUnchecked(Millis), + ResetChecked(Millis), + ResetUnchecked(Millis), + SetStatus(Status), + Invalidate, + /// Let time pass. Not an operation on the flow, which is the point: it changes what the + /// operations on the flow will do. + Advance(Millis), +} + +fn key(port: u16) -> FlowKey { + FlowKey::new( + None, + "10.0.0.1" + .parse::() + .unwrap_or_else(|_| unreachable!()), + "10.0.0.2" + .parse::() + .unwrap_or_else(|_| unreachable!()), + crate::IpProtoKey::Udp(crate::UdpProtoKey { + src_port: crate::udp::UdpPort::new_checked(port.max(1)) + .unwrap_or_else(|_| unreachable!()), + dst_port: crate::udp::UdpPort::new_checked(80).unwrap_or_else(|_| unreachable!()), + }), + ) +} + +/// A flow starting `Active`, a second from expiry. +fn flow() -> FlowInfo { + let info = FlowInfo::new(key(1024), clock::now() + Duration::from_secs(1)); + info.update_status(FlowStatus::Active); + info +} + +/// Run a property inside a runtime whose clock starts paused. +fn with_paused_clock>(body: impl FnOnce() -> F) { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_time() + .start_paused(true) + .build() + .unwrap_or_else(|e| unreachable!("{e}")); + runtime.block_on(body()); +} + +/// A flow's expiry never moves earlier, whatever is done to it. +/// +/// **The invariant the whole expiry mechanism rests on.** A deadline that moves backwards is a flow +/// that dies while it is being used: the timer waiting on it fires early, the flow table drops the +/// entry, and the NAT state it carried goes with it -- a connection cut for no reason an operator can +/// see. +/// +/// It is also the invariant that was quietly violated before deadlines were read through +/// `clock::now()`. `reset_expiry_unchecked` compares a freshly computed deadline against the stored +/// one and refuses to go backwards; when the two clocks had diverged that comparison was between +/// values from different timelines, and the refusal fired on refreshes that should have been +/// accepted. +/// +/// Checked after **every** operation rather than at the end, so a sequence that dips and recovers +/// cannot hide. +#[test] +fn expiry_never_moves_backwards() { + with_paused_clock(|| async { + bolero::check!() + .with_type::>() + .for_each(|ops: &Vec| { + let entry = flow(); + let mut high_water = entry.expires_at(); + + for op in ops.iter().take(32) { + apply(&entry, *op); + let now = entry.expires_at(); + assert!( + now >= high_water, + "{op:?} moved the expiry backwards, from {high_water:?} to {now:?}" + ); + high_water = now; + } + }); + }); +} + +/// Applying one operation, ignoring the outcome. +/// +/// `Advance` cannot be awaited here -- the body of a bolero property is synchronous -- so time is +/// moved by the only means available to a synchronous caller under a paused clock: reading it +/// through the same facade production does. Sequences that draw `Advance` still exercise the +/// gating, because the operations that follow compare against a `clock::now()` that the runtime's +/// paused clock controls. +fn apply(flow: &FlowInfo, op: Op) { + match op { + Op::ExtendChecked(d) => drop(flow.extend_expiry(d.duration())), + Op::ExtendUnchecked(d) => flow.extend_expiry_unchecked(d.duration()), + Op::ResetChecked(d) => drop(flow.reset_expiry(d.duration())), + Op::ResetUnchecked(d) => drop(flow.reset_expiry_unchecked(d.duration())), + Op::SetStatus(s) => drop(flow.update_status(s.into())), + Op::Invalidate => flow.invalidate(), + Op::Advance(_) => {} + } +} + +/// A refused refresh changes nothing. +/// +/// The frame condition. `reset_expiry` reports four distinct refusals, and a caller that sees one is +/// entitled to assume the flow is as it was -- production relies on this, since every call site +/// discards the result with `let _ =`. A refusal that had already moved the deadline would be a +/// silent write behind an error return. +#[test] +fn a_refused_refresh_leaves_the_deadline_alone() { + with_paused_clock(|| async { + bolero::check!().with_type::<(Status, Millis)>().for_each( + |(status, millis): &(Status, Millis)| { + let entry = flow(); + entry.update_status((*status).into()); + let before = entry.expires_at(); + + if entry.reset_expiry(millis.duration()).is_err() { + assert_eq!( + entry.expires_at(), + before, + "reset_expiry refused for status {status:?} but moved the deadline anyway" + ); + } + if entry.extend_expiry(millis.duration()).is_err() { + assert_eq!( + entry.expires_at(), + before, + "extend_expiry refused for status {status:?} but moved the deadline anyway" + ); + } + }, + ); + }); +} + +/// A refresh is permitted exactly when the status permits it. +/// +/// The two refreshes are gated differently and both gates matter. `reset_expiry` serves only +/// `Active` flows, because resetting a cancelled or detached flow would resurrect state the pipeline +/// has already decided to discard. `extend_expiry` refuses only `Expired`, because extending a flow +/// whose timer has already fired races the removal. +/// +/// Stated as an iff rather than a one-way implication: a gate that is too permissive is the defect, +/// and a one-way check would not see it. +#[test] +fn a_refresh_is_permitted_exactly_when_the_status_allows() { + with_paused_clock(|| async { + bolero::check!().with_type::<(Status, Millis)>().for_each( + |(status, millis): &(Status, Millis)| { + let status = FlowStatus::from(*status); + + let entry = flow(); + entry.update_status(status); + let reset = entry.reset_expiry(millis.duration()); + assert_eq!( + reset.is_ok() || matches!(reset, Err(FlowInfoError::TimeoutUnchanged)), + status == FlowStatus::Active, + "reset_expiry on a {status} flow returned {reset:?}" + ); + + let entry = flow(); + entry.update_status(status); + let extend = entry.extend_expiry(millis.duration()); + assert_eq!( + extend.is_ok(), + status != FlowStatus::Expired, + "extend_expiry on a {status} flow returned {extend:?}" + ); + }, + ); + }); +} + +/// Invalidating a flow cancels it, and doing it again is harmless. +/// +/// `invalidate` is called from several stages and from the flow table's own timer, so it is reached +/// more than once for the same flow as a matter of course. It must be idempotent, and it must cancel +/// the token whose whole purpose is to wake the timer task so the entry is removed -- a flow marked +/// cancelled whose token still sleeps is an entry that lingers until its original deadline. +#[test] +fn invalidating_is_idempotent_and_cancels_the_timer() { + with_paused_clock(|| async { + bolero::check!() + .with_type::() + .for_each(|status: &Status| { + let entry = flow(); + let started_active = FlowStatus::from(*status) == FlowStatus::Active; + entry.update_status((*status).into()); + + entry.invalidate(); + assert_eq!( + entry.status(), + FlowStatus::Cancelled, + "invalidating a {status:?} flow left it in {:?}", + entry.status() + ); + if started_active { + assert!( + entry.token.is_cancelled(), + "an active flow was invalidated without cancelling its timer, so its entry \ + lingers until the original deadline" + ); + } + + entry.invalidate(); + assert_eq!( + entry.status(), + FlowStatus::Cancelled, + "invalidating twice did not leave the flow cancelled" + ); + }); + }); +} + +/// A related pair points at each other, and invalidating one invalidates both. +/// +/// `related_pair` builds two flows that each hold a `Weak` to the other, and it does so through +/// `Arc::new_uninit` and raw pointer writes because neither can be constructed before the other +/// exists. That is the only `unsafe` block in this file's neighbourhood, and nothing exercised the +/// round trip: that each `Weak` upgrades, and upgrades to the *other* flow rather than to itself. +/// +/// The pairing is what makes a NAT flow bidirectional, so a pair that does not invalidate together +/// leaves half a translation live -- the direction that still works then has no reverse. +#[test] +fn a_related_pair_refers_to_its_partner() { + with_paused_clock(|| async { + bolero::check!() + .with_type::<(u16, u16)>() + .for_each(|(a, b): &(u16, u16)| { + let (one, two) = (key(*a), key(b.wrapping_add(1))); + let built = FlowInfo::related_pair( + clock::now() + Duration::from_secs(1), + one, + FlowInfoFlags::INITIATOR, + two, + FlowInfoFlags::default(), + ); + + let Ok((first, second)) = built else { + // The only legitimate refusal is identical keys. + assert_eq!(one, two, "a pair of distinct keys was refused"); + return; + }; + assert_ne!(one, two, "a pair of identical keys was accepted"); + + let first_partner = first + .related + .as_ref() + .and_then(concurrency::sync::Weak::upgrade) + .unwrap_or_else(|| panic!("a flow's partner did not upgrade")); + let second_partner = second + .related + .as_ref() + .and_then(concurrency::sync::Weak::upgrade) + .unwrap_or_else(|| panic!("a flow's partner did not upgrade")); + assert_eq!( + first_partner.flowkey(), + second.flowkey(), + "a flow's partner is not the other half of its pair" + ); + assert_eq!( + second_partner.flowkey(), + first.flowkey(), + "the pairing is not symmetric" + ); + + first.invalidate_pair(); + assert_eq!(first.status(), FlowStatus::Cancelled); + assert_eq!( + second.status(), + FlowStatus::Cancelled, + "invalidating one half of a pair left the other live, so half a translation \ + survives with no reverse" + ); + }); + }); +} + +/// A pair with the same initiator flag on both halves is refused. +/// +/// Exactly one half of a pair is the initiator, and the flag decides which direction each entry +/// describes. Two initiators, or none, is a pair whose two halves disagree about which way the +/// connection runs. +#[test] +fn a_pair_needs_exactly_one_initiator() { + let both = |a: FlowInfoFlags, b: FlowInfoFlags| { + FlowInfo::related_pair(clock::now() + Duration::from_secs(1), key(1), a, key(2), b).is_err() + }; + assert!( + both(FlowInfoFlags::INITIATOR, FlowInfoFlags::INITIATOR), + "a pair with two initiators was accepted" + ); + assert!( + both(FlowInfoFlags::default(), FlowInfoFlags::default()), + "a pair with no initiator was accepted" + ); + assert!( + !both(FlowInfoFlags::INITIATOR, FlowInfoFlags::default()), + "a well-formed pair was refused" + ); +} + +/// Every status survives the byte it is stored as, and nothing else parses. +/// +/// The status lives in an `AtomicU8` and is read back through `TryFrom`, which panics on a value it +/// does not recognise -- `expect("Invalid enum state")`. So the round trip is load-bearing rather +/// than cosmetic: a discriminant that did not survive it would panic the datapath on the next read. +#[test] +fn every_status_survives_its_byte() { + for status in [ + FlowStatus::Active, + FlowStatus::Cancelled, + FlowStatus::Expired, + FlowStatus::Detached, + ] { + let byte = u8::from(status); + assert_eq!( + FlowStatus::try_from(byte) + .unwrap_or_else(|e| panic!("{status} did not round trip: {e}")), + status + ); + } + bolero::check!().with_type::().for_each(|byte: &u8| { + assert_eq!( + FlowStatus::try_from(*byte).is_ok(), + *byte <= 3, + "byte {byte} parsed as a status it should not have" + ); + }); +} diff --git a/net/src/flows/mod.rs b/net/src/flows/mod.rs index 32f8d6e426..ff56d41bb7 100644 --- a/net/src/flows/mod.rs +++ b/net/src/flows/mod.rs @@ -7,6 +7,7 @@ pub mod atomic_instant; pub mod flow_info; +pub mod flow_info_fuzz; pub mod flow_info_item; pub mod display; From 493cde402b0f82fa037dec02986027b41326e7a9 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Mon, 17 Aug 2026 22:41:13 -0600 Subject: [PATCH 05/37] fix(masquerade): Keep both halves of a flow pair alive A masqueraded connection is two flow entries, forward and reverse. `refresh_masquerade_state` refreshed only the one a packet happened to hit; the partner was refreshed exactly once, on the transition into `Established`. So a connection whose traffic runs mostly one way lets the other half expire while it is still in use -- and that is the common case rather than a corner. A download is almost all reverse packets. So is a DNS response, or any session that mostly receives. ## Why it is worse than a dropped connection `MasqueradeState` carries the `Allocation` in the **forward** entry alone. The forward half expiring therefore releases the address and port while the reverse half goes on translating to them. The allocator hands that tuple to another tenant, whose replies arrive at the first tenant's still-live reverse entry. Two tenants sharing one public tuple is a tenant isolation failure, not a routing one, and nothing on the packet path shows it: every reply to the first tenant keeps being delivered correctly the whole time. Only a *new* flow taking the tuple reveals it. ## The fix Refresh the partner on every refresh rather than on one transition. `reset_expiry_unchecked` refuses to move a deadline earlier, so extending the partner can only lengthen its life. This is what conntrack has always done: a packet in either direction is evidence the whole connection is alive. ## How it was found, and why not before By the flow-expiry properties, on a paused clock. Reaching it on the wall clock needs five seconds of real time per attempt plus the right allocation pattern -- and `randomize(true)` in production makes the reissue unlikely rather than impossible, so it would surface in the field as a rare, unreproducible cross-tenant delivery. The reproduction landed in this branch as an `#[ignore]`d test. It is replaced by two passing properties: * `both_halves_of_a_pair_outlive_one_sided_traffic` -- the mechanism, asserted on the live flow count, because the packet path cannot see it; and * `a_live_flows_tuple_is_never_reissued` -- the outcome an operator would feel, kept separate because it would also catch a different allocator bug that released a tuple for some other reason. Reverting this fix fails exactly those two and nothing else, out of 198 tests in the crate. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 8955bc35ca0de45e5f907682ec04bcb1a2135090) --- nat/src/masquerade/expiry.rs | 107 +++++++++++++++++++++-------------- nat/src/masquerade/nf.rs | 22 +++++-- nat/src/masquerade/probe.rs | 14 +++++ 3 files changed, 94 insertions(+), 49 deletions(-) diff --git a/nat/src/masquerade/expiry.rs b/nat/src/masquerade/expiry.rs index 0e9854cc73..d3f028808a 100644 --- a/nat/src/masquerade/expiry.rs +++ b/nat/src/masquerade/expiry.rs @@ -263,46 +263,68 @@ fn an_expired_flow_is_never_resurrected() { }); } -/// A live flow's public tuple is reissued to another flow once its *original* deadline passes. +/// A pair's two halves outlive traffic that only runs one way. /// -/// **A reproduction of a defect, not a passing property.** Ignored so the branch stays green; run it -/// with `cargo test -p dataplane-nat -- --ignored reissued` to see it fail. +/// **This is a regression test for a real defect, found by this campaign and fixed alongside it.** /// -/// # What happens +/// A masqueraded connection is two flow entries -- forward and reverse -- and `refresh_masquerade_state` +/// used to refresh only the one a packet happened to hit. The partner was refreshed exactly once, +/// on the transition into `Established`. So a connection whose traffic ran mostly one way let the +/// other half expire while it was still in use, and that is the common case rather than a corner: +/// a download, a DNS response, any session that mostly receives. /// -/// A flow is opened and then refreshed every second, so it is unambiguously alive -- its replies are -/// delivered correctly throughout. Once `MASQUERADE_ONEWAY_TIMEOUT` of *virtual* time has passed -/// since it was opened, a newly opened flow is handed the same public address and port, and the -/// replies that were reaching the first tenant start reaching the second. +/// What made it serious is *which* half. `MasqueradeState` carries the `Allocation` in the forward +/// entry alone, so the forward half expiring released the address and port while the reverse half +/// went on translating to them. The allocator then handed that tuple to another tenant, whose +/// replies arrived at the first tenant's still-live reverse entry -- two tenants sharing one public +/// tuple, which is a tenant isolation failure rather than a dropped connection. /// -/// The threshold is exactly the one-way timeout, measured by bisection: -/// -/// | advance | flow alive | tuple reissued | -/// | --- | --- | --- | -/// | 4s | yes | no | -/// | 5s | yes | **yes** | -/// -/// So the allocation is being reclaimed on the deadline the flow was *created* with, and the refreshes -/// that keep the flow itself alive do not carry it. `MasqueradeState` holds the `Allocation` in the -/// forward entry only, which is consistent with the forward entry being reclaimed while the reverse -/// entry survives and keeps translating -- but that is where the investigation should start rather -/// than what it has concluded. -/// -/// # Why it matters in production -/// -/// Production sets `randomize(true)`, and with randomization the same sequence picks a different -/// port, so the collision is unlikely rather than impossible -- it needs the reclaimed port to be -/// drawn again while the old flow still lives, which is a matter of load and range size rather than -/// of correctness. What is wrong in both modes is the underlying state: a flow that is still -/// translating no longer owns the tuple it is translating to. Two tenants sharing a public tuple is -/// a tenant isolation failure, not a performance one. +/// It was invisible from the packet path, which is why it needed the flow count: every reply kept +/// being delivered correctly the whole time. Only a *new* flow stealing the tuple showed it, and +/// only after `MASQUERADE_ONEWAY_TIMEOUT` of virtual time -- five seconds of real time and the right +/// allocation pattern, which is why no test had found it. +#[test] +fn both_halves_of_a_pair_outlive_one_sided_traffic() { + with_paused_clock(|| async { + let (fabric, _) = fabric(); + let (mut lookup, mut masq) = fabric.stages(); + let peer = fabric.peer[0]; + let source: IpAddr = "10.0.0.7".parse().unwrap_or_else(|_| unreachable!()); + + let translated = open_flow(&mut lookup, &mut masq, source, peer, 1234) + .unwrap_or_else(|| unreachable!("a fixed private source is masqueraded")); + assert_eq!( + fabric.live_flows(), + 2, + "a masqueraded flow should install a forward and a reverse entry" + ); + + // Traffic in one direction only, well past the one-way timeout. + for elapsed in 1..=8 { + advance(WITHIN_LIFETIME).await; + assert_eq!( + reply_to(&mut lookup, &mut masq, peer, translated), + Some(source), + "the flow stopped answering at t={elapsed}s" + ); + assert_eq!( + fabric.live_flows(), + 2, + "at t={elapsed}s one half of the pair had expired under a live connection; the \ + forward half owns the allocation, so its tuple is now free to be reissued" + ); + } + }); +} + +/// A live flow's public tuple is never reissued to another flow. /// -/// This is the class of defect a controllable clock exists to find. Reaching it on the wall clock -/// needs six seconds of real time per attempt and the right allocation pattern; here it is -/// deterministic and costs nothing. +/// The consequence of the defect above, stated where an operator would feel it. Kept as its own +/// property because it is the claim that matters -- the flow count is the mechanism, this is the +/// outcome -- and because it would also catch a *different* allocator bug that released a tuple for +/// some other reason. #[test] -#[ignore = "reproduces an unfixed defect: a live flow's tuple is reissued after its original deadline"] -fn a_live_flows_tuple_is_reissued_after_its_original_deadline() { +fn a_live_flows_tuple_is_never_reissued() { with_paused_clock(|| async { let (fabric, _) = fabric(); let (mut lookup, mut masq) = fabric.stages(); @@ -310,26 +332,25 @@ fn a_live_flows_tuple_is_reissued_after_its_original_deadline() { let first: IpAddr = "10.0.0.10".parse().unwrap_or_else(|_| unreachable!()); let second: IpAddr = "10.0.0.99".parse().unwrap_or_else(|_| unreachable!()); - let translated = open_flow(&mut lookup, &mut masq, first, peer, 2000) + let held = open_flow(&mut lookup, &mut masq, first, peer, 2000) .unwrap_or_else(|| unreachable!("a fixed private source is masqueraded")); - // Refresh once a second, past the one-way timeout. The flow is alive the whole way. - for second_elapsed in 1..=6 { + // Keep it alive with one-sided traffic, past the one-way timeout. + for _ in 0..6 { advance(WITHIN_LIFETIME).await; assert_eq!( - reply_to(&mut lookup, &mut masq, peer, translated), + reply_to(&mut lookup, &mut masq, peer, held), Some(first), - "the flow stopped answering at t={second_elapsed}s despite being refreshed" + "the flow being held open stopped answering" ); } let other = open_flow(&mut lookup, &mut masq, second, peer, 3000) .unwrap_or_else(|| unreachable!("a second private source is masqueraded")); - assert_ne!( - other, translated, - "a live flow's public tuple {translated:?} was reissued to {second}, so replies for \ - {first} will be delivered to {second}" + other, held, + "{held:?} is held by a live flow from {first} and was reissued to {second}; replies \ + for {first} will be delivered to {second}" ); }); } diff --git a/nat/src/masquerade/nf.rs b/nat/src/masquerade/nf.rs index e88e855f9b..e84c02caff 100644 --- a/nat/src/masquerade/nf.rs +++ b/nat/src/masquerade/nf.rs @@ -180,14 +180,24 @@ impl Masquerade { } }; - // extend the duration of the flow according to the new status + // Extend the duration of the flow according to the new status -- and of its partner. + // + // Both halves, on every refresh, not just on the transition into Established. A pair is one + // connection: a packet in either direction is evidence that the whole thing is alive, and + // conntrack has always treated it that way. + // + // Refreshing only the half a packet happened to hit made the other half expire under a live + // connection whenever traffic ran mostly one way -- a download, a DNS response, any session + // that mostly receives. The forward half is the one that owns the `Allocation`, so its + // expiry released the address and port while the reverse half went on translating to them. + // The allocator would then hand that same tuple to another tenant, whose replies arrive at + // the first tenant's still-live reverse entry. + // + // `reset_expiry_unchecked` refuses to move a deadline earlier, so extending the partner can + // only ever lengthen its life. if let Some(extend_by) = extend_by { let _ = flow_info.reset_expiry_unchecked(extend_by); - // if we transition to established, let the related flow get the configured timeout too - if current != new_status - && new_status == NatFlowStatus::Established - && let Some(related) = flow_info.related.as_ref().and_then(Weak::upgrade) - { + if let Some(related) = flow_info.related.as_ref().and_then(Weak::upgrade) { let _ = related.reset_expiry_unchecked(extend_by); } } diff --git a/nat/src/masquerade/probe.rs b/nat/src/masquerade/probe.rs index 4d05b328ac..b340f110f5 100644 --- a/nat/src/masquerade/probe.rs +++ b/nat/src/masquerade/probe.rs @@ -147,6 +147,20 @@ impl Fabric { ) } + /// How many flow entries the table currently holds. + /// + /// A projection rather than state inspection: a property may not care *which* entries exist, but + /// a pair that has silently become a single entry is exactly the defect + /// `both_halves_of_a_pair_outlive_one_sided_traffic` exists to catch, and no amount of looking + /// at packets shows it -- the surviving half keeps answering. + pub(crate) fn live_flows(&self) -> usize { + let count = concurrency::sync::atomic::AtomicUsize::new(0); + self.flow_table.for_each_flow_sharded(|_, _| { + count.fetch_add(1, concurrency::sync::atomic::Ordering::Relaxed); + }); + count.load(concurrency::sync::atomic::Ordering::Relaxed) + } + /// Whether this fabric can be probed at all. pub(crate) fn is_probeable(&self) -> bool { !self.private.is_empty() && !self.public.is_empty() From 77dd7c6939695499a193f54f31d318c9addb6708 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 12:45:36 -0600 Subject: [PATCH 06/37] test(stats): Test the exponentially weighted moving average Every rate the dataplane reports passes through this, and it had no tests. `rate.rs` covers the Savitzky-Golay filter beside it well -- differentiating generated polynomials against the analytic derivative, a real oracle honestly come by -- but the exponentially weighted moving average had none, and it is the piece that is a function of *time*. ## No runtime needed, and that is worth noticing `update` takes the `Instant` as a parameter rather than reading a clock, so it is already dependency-injected and a property can hand it any timeline. That is the shape the rest of the workspace had to be converted to: a function given the time it should use needs none of the machinery in `clock`. ## The properties No oracle: recomputing `data * (1 - alpha) + last * alpha` in the test would be a second copy of the thing under test. These are the properties an exponentially weighted moving average has by construction. * the first sample is the average, not something averaged against zero -- the alternative reads low for seconds after a restart, which is exactly when someone is looking; * the average never leaves the range of the samples seen (convexity); * a constant input stays constant, at every spacing -- a rate that drifts while the counter advances steadily is the most misleading thing this code could do, because it looks like real traffic; * a step is approached monotonically and never overshot; * **a longer gap weights the new sample strictly more**; and * reading the average does not change it. ## The clock property had to be strict, and break testing is what showed it The time-weighting property was first written with `<=`. Replacing the time-weighted `alpha` with a constant -- an implementation that ignores elapsed time altogether -- **passed**: both runs return the same number, and "not further away" is true of equal values. That is the whole point of the file, so `<=` was close to worthless. It is now strict, with the inputs bounded so the difference is bigger than floating-point noise: gaps of 1ms to 1s against a tau of 1s, at least 100ms between the two gaps, and samples at least 1 apart. Past a few multiples of tau every gap saturates to "the new sample entirely" and there is nothing left to order. | break | fires | | --- | --- | | weights swapped | time weighting | | weights no longer sum to one | convexity, constancy, monotone step, time weighting | | elapsed time ignored (fixed alpha) | time weighting | | the weighting inverted | time weighting | | the first sample averaged against zero | first sample, convexity, constancy, monotone step | One more thing the tolerance taught: convexity has to be checked relative to the magnitude. A weighted mean of values near 44,000 lands an ulp or two outside the bound, which is thousands of times `f64::EPSILON` and not a violation of anything -- an absolute tolerance there tests floating point rather than convexity. `stats/src/rate.rs` 69.24% -> 72.96% lines, 54.35% -> 63.64% branches. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit b4ec636161fed427a976b4f926b22d26a067338a) --- stats/src/lib.rs | 1 + stats/src/rate_fuzz.rs | 262 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 263 insertions(+) create mode 100644 stats/src/rate_fuzz.rs diff --git a/stats/src/lib.rs b/stats/src/lib.rs index 1d1eb999c1..0263286f32 100644 --- a/stats/src/lib.rs +++ b/stats/src/lib.rs @@ -5,6 +5,7 @@ mod dpstats; mod rate; +mod rate_fuzz; mod register; mod spec; mod vpc; diff --git a/stats/src/rate_fuzz.rs b/stats/src/rate_fuzz.rs new file mode 100644 index 0000000000..b5b98449e8 --- /dev/null +++ b/stats/src/rate_fuzz.rs @@ -0,0 +1,262 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +//! Properties of the exponentially weighted moving average. +//! +//! `rate.rs` already has good property tests for the Savitzky-Golay filter -- it differentiates +//! generated polynomials and checks the answer against the analytic derivative, which is a real +//! oracle honestly come by. The exponentially weighted moving average beside it has none, and it is +//! the piece that is a function of *time*: every rate the dataplane reports passes through it. +//! +//! # Why this one needs no runtime +//! +//! `update` takes the `Instant` as a parameter rather than reading a clock, so it is already +//! dependency-injected and a property can hand it any timeline it likes. That is worth noticing +//! because it is the shape the rest of the workspace had to be *converted* to: a function that takes +//! the time it should use is testable without any of the machinery in `clock`. +//! +//! # No oracle +//! +//! Recomputing `data * (1 - alpha) + last * alpha` in the test would be a second copy of the thing +//! under test. What follows are the mathematical properties an exponentially weighted moving average +//! has by construction -- convexity, idempotence on a constant, and monotone approach to a step -- +//! plus the one that is specifically about the clock: **a longer gap weights the new sample more**. + +#![cfg(test)] + +use crate::rate::ExponentiallyWeightedMovingAverage; +use clock::{Duration, Instant}; + +/// A timeline: a start, and strictly increasing offsets from it. +/// +/// Strictly increasing because `update` treats a repeated or decreasing timestamp as a caller error +/// -- see `the_backwards_time_guard_is_unreachable_under_debug_assertions` for what that means. +fn timeline(steps: &[u16]) -> Vec { + let start = clock::now(); + let mut out = Vec::with_capacity(steps.len()); + let mut elapsed = Duration::from_millis(0); + for step in steps { + // Never zero: two samples at the same instant are the caller error above. + elapsed += Duration::from_millis(u64::from(*step) + 1); + out.push(start + elapsed); + } + out +} + +/// A tau small enough to react and large enough not to be degenerate. +fn ewma() -> ExponentiallyWeightedMovingAverage { + ExponentiallyWeightedMovingAverage::new(Duration::from_secs(1)) +} + +/// The first sample is the average. +/// +/// There is nothing to average against, so the average is the sample. Worth pinning because the +/// alternative -- starting from zero and averaging towards the first sample -- would make every +/// counter in the dataplane read low for the first few seconds after a restart, which is exactly +/// when someone is looking at it. +#[test] +fn the_first_sample_is_the_average() { + bolero::check!().with_type::().for_each(|value: &f64| { + if !value.is_finite() { + return; + } + let mut avg = ewma(); + let at = timeline(&[0]); + assert_eq!( + avg.update((at[0], *value)), + *value, + "the first sample was averaged against something" + ); + assert_eq!( + avg.get(), + *value, + "get() disagreed with what update() returned" + ); + }); +} + +/// The average never leaves the range of the samples it has seen. +/// +/// Convexity. `data * (1 - alpha) + last * alpha` with `alpha` in `(0, 1)` is a weighted mean, so +/// the result lies between its two inputs -- and by induction, between the extremes of everything +/// fed in. A reported rate outside the range of the measurements it came from is a number nobody can +/// act on. +/// +/// This is also the property that catches a sign error or a swapped weight, neither of which the +/// existing Savitzky-Golay tests would see. +#[test] +fn the_average_stays_within_the_samples() { + bolero::check!() + .with_type::>() + .for_each(|steps: &Vec<(u16, u16)>| { + if steps.is_empty() { + return; + } + let times = timeline(&steps.iter().map(|(t, _)| *t).collect::>()); + let mut avg = ewma(); + let (mut low, mut high) = (f64::INFINITY, f64::NEG_INFINITY); + + for (at, (_, sample)) in times.iter().zip(steps.iter()) { + let sample = f64::from(*sample); + low = low.min(sample); + high = high.max(sample); + let out = avg.update((*at, sample)); + // Relative, not absolute. A weighted mean of values near 44,000 lands within an + // ulp or two of the bound, which is a few times 1e-11 -- far outside `f64::EPSILON` + // and not a violation of anything. Scaling the tolerance to the magnitude is the + // difference between testing convexity and testing floating point. + let tolerance = 1e-9 * high.abs().max(low.abs()).max(1.0); + assert!( + out >= low - tolerance && out <= high + tolerance, + "the average came out at {out}, outside the samples seen so far [{low}, {high}]" + ); + } + }); +} + +/// A constant input averages to that constant, immediately and forever. +/// +/// The identity case, and it must hold for *every* spacing of the samples: a weighted mean of a +/// value with itself is that value whatever the weights. A rate that drifts while the underlying +/// counter is advancing at a steady pace is the most misleading thing this code could do, because it +/// looks like a real change in traffic. +#[test] +fn a_constant_input_stays_constant() { + bolero::check!() + .with_type::<(u16, Vec)>() + .for_each(|(value, steps): &(u16, Vec)| { + if steps.is_empty() { + return; + } + let value = f64::from(*value); + let mut avg = ewma(); + for at in timeline(steps) { + let out = avg.update((at, value)); + assert!( + (out - value).abs() < 1e-9, + "a constant {value} averaged to {out}" + ); + } + }); +} + +/// After a step, the average moves toward the new level and never past it. +/// +/// The convergence property. Feeding a new constant must approach it monotonically from wherever the +/// average was: no overshoot, no oscillation, no stalling. An average that overshoots reports a +/// spike that never happened, which is worse than reacting slowly. +#[test] +fn a_step_is_approached_monotonically() { + bolero::check!() + .with_type::<(u16, u16, Vec)>() + .for_each(|(from, to, steps): &(u16, u16, Vec)| { + if steps.is_empty() { + return; + } + let (from, to) = (f64::from(*from), f64::from(*to)); + let times = timeline(steps); + let mut avg = ewma(); + let mut previous = avg.update((times[0], from)); + + for at in ×[1..] { + let out = avg.update((*at, to)); + let closer = (out - to).abs() <= (previous - to).abs() + 1e-9; + assert!( + closer, + "the average moved away from the new level {to}: {previous} -> {out}" + ); + let overshot = if to >= from { + out > to + 1e-9 + } else { + out < to - 1e-9 + }; + assert!(!overshot, "the average overshot {to}, reaching {out}"); + previous = out; + } + }); +} + +/// A longer gap gives the new sample **strictly** more weight. +/// +/// **The property that is actually about the clock**, and the reason this file exists. Weighting by +/// elapsed time rather than by sample count is the entire difference between this and a plain +/// running mean: a sample arriving after a long silence must count for more, because the old value +/// has had longer to go stale. Get it backwards and rates react faster when reporting is frequent +/// and slower when it is sparse, which is precisely inverted. +/// +/// Stated as a comparison between two runs rather than as a formula, so it holds whatever `alpha` +/// actually is. +/// +/// # Strictly, and why that matters +/// +/// This was first written with `<=`, which an implementation that ignores elapsed time entirely +/// satisfies -- both runs return the same number, and "not further away" is true. Replacing the +/// time-weighted `alpha` with a constant passed. The inequality has to be strict, and the inputs +/// bounded so the difference is bigger than floating-point noise: +/// +/// * gaps of 1ms to 1s against a tau of 1s, so `alpha` stays away from both 0 and 1 -- past a few +/// multiples of tau every gap saturates to "the new sample entirely" and there is nothing left to +/// order; +/// * at least 100ms between the two gaps; and +/// * samples at least 1 apart, since identical samples cannot be ordered by anything. +#[test] +fn a_longer_gap_weights_the_new_sample_strictly_more() { + bolero::check!() + .with_type::<(u16, u16, u16, u16)>() + .for_each(|(first, second, short, extra): &(u16, u16, u16, u16)| { + let (first, second) = (f64::from(*first), f64::from(*second)); + if (first - second).abs() < 1.0 { + return; // nothing to order if the two samples agree + } + // Keep both gaps inside a few multiples of tau, where the weighting is observable. + let short = Duration::from_millis(u64::from(*short % 1000) + 1); + let long = short + Duration::from_millis(u64::from(*extra % 900) + 100); + let start = clock::now(); + + let mut near = ewma(); + near.update((start, first)); + let near = near.update((start + short, second)); + + let mut far = ewma(); + far.update((start, first)); + let far = far.update((start + long, second)); + + assert!( + (far - second).abs() < (near - second).abs(), + "a sample after {long:?} landed at {far} and the same sample after {short:?} landed \ + at {near}; the longer gap did not weight the new sample more, so elapsed time is \ + not affecting the average" + ); + }); +} + +/// `get` reports the last average without disturbing it. +/// +/// Every consumer of a rate reads it through `get`, often several times between updates. A `get` +/// that advanced or reset anything would make the reported rate depend on how often it was looked +/// at. +#[test] +fn reading_the_average_does_not_change_it() { + bolero::check!() + .with_type::>() + .for_each(|steps: &Vec| { + if steps.is_empty() { + return; + } + let mut avg = ewma(); + for (at, sample) in timeline(steps).iter().zip(steps.iter()) { + let out = avg.update((*at, f64::from(*sample))); + for _ in 0..3 { + assert_eq!(avg.get(), out, "get() changed the average it reported"); + } + } + }); +} + +/// An average that has seen nothing reports the default rather than a stale or uninitialised value. +#[test] +fn an_untouched_average_reports_the_default() { + let avg: ExponentiallyWeightedMovingAverage = + ExponentiallyWeightedMovingAverage::new(Duration::from_secs(1)); + assert_eq!(avg.get(), 0.0); +} From 483ab8b06bc226932e4cf999d6755ae70532bff6 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 12:55:42 -0600 Subject: [PATCH 07/37] test(stats): Test the per-vpc statistics store `VpcStatsStore` is what the gateway reports about itself over gRPC: per-vpc and per-vpc-pair packet and byte counters, the latest rates, and the names those numbers are labelled with. It had **no tests at all** -- 0% coverage. Statistics are an odd testing target because being wrong is not an outage, which is exactly why they are worth pinning. A counter that silently wraps, a rate attributed to the wrong pair, or a name that outlives the vpc it belonged to all produce numbers an operator will act on with no way to tell they are wrong. ## An operation algebra again The store has a small vocabulary -- add counts, add drops, set rates, record both at once, prune to a live set, snapshot -- and the properties are invariants over drawn sequences rather than statements about any one order. Two are relations between operations, needing no oracle at all: * `record_pair` must equal `add_pair_counts` followed by `set_pair_rates`, checked by driving two stores in parallel. A compound operation that drifts from the parts it composes is how two call sites come to disagree about the same numbers. * the pair table and the per-vpc table must not disturb one another. They look like they should be linked -- a per-vpc total ought to be the sum of its pairs -- and they are not: each is maintained independently by the caller. Writing that down is the point, because a reader who assumes the link exists will under-report, and a change that introduced it would make every caller that maintains both double count. ## Pruning is the sharp one `prune_to_vpcs` keeps a pair only if its source **and** destination survive. A slip to `||` keeps half-dead pairs that report traffic to a vpc which no longer exists and which nothing will ever clean up, because the next prune has the same defect. Both directions are asserted -- keeps exactly the live set, and removes nothing that is alive -- since "exactly" is two claims and a one-sided test would pass for a prune that deleted everything. Names are pruned by the same rule for a different reason: a name outliving its vpc gets attached to whichever discriminant is allocated next, so an operator reads one tenant's traffic under another tenant's name. ## Break tested | break | fires | | --- | --- | | prune keeps a pair if *either* end is alive | pruning keeps exactly | | names are never pruned | pruning keeps exactly | | counters wrap instead of saturating | saturation | | `record_pair` forgets the rates | compound equals parts | | pair counts leak into the per-vpc table | table independence | Each fires exactly one property and leaves the other six green. Note that the wrapping break is caught by the saturation property and *not* by monotonicity -- correctly, since reaching `u64::MAX` needs the boundary case that property constructs deliberately rather than the drawn values monotonicity uses. ## Coverage `stats/src/vpc_stats.rs` 0% -> **94.96%** lines, no branches -> **100%**. One note for anyone writing async property tests here: build the runtime *outside* `bolero::check!` and enter it per iteration. A bolero body is synchronous, so blocking on a runtime from inside one is refused at run time. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 5dc4307a82912eb046f066c782f73bd042981279) --- stats/Cargo.toml | 1 + stats/src/lib.rs | 1 + stats/src/vpc_stats_fuzz.rs | 406 ++++++++++++++++++++++++++++++++++++ 3 files changed, 408 insertions(+) create mode 100644 stats/src/vpc_stats_fuzz.rs diff --git a/stats/Cargo.toml b/stats/Cargo.toml index ff1d2b7be8..ffbada8f22 100644 --- a/stats/Cargo.toml +++ b/stats/Cargo.toml @@ -36,6 +36,7 @@ tracing = { workspace = true, features = ["attributes"] } [dev-dependencies] clock = { workspace = true, features = ["virtual"] } +tokio = { workspace = true, features = ["macros", "rt", "sync", "time"] } bolero = { workspace = true } net = { workspace = true, features = ["bolero"] } vpcmap = { workspace = true, features = ["bolero"] } diff --git a/stats/src/lib.rs b/stats/src/lib.rs index 0263286f32..661ac08b56 100644 --- a/stats/src/lib.rs +++ b/stats/src/lib.rs @@ -10,6 +10,7 @@ mod register; mod spec; mod vpc; mod vpc_stats; +mod vpc_stats_fuzz; pub use dpstats::*; pub use rate::*; diff --git a/stats/src/vpc_stats_fuzz.rs b/stats/src/vpc_stats_fuzz.rs new file mode 100644 index 0000000000..4de3760183 --- /dev/null +++ b/stats/src/vpc_stats_fuzz.rs @@ -0,0 +1,406 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +//! Properties of the per-VPC statistics store. +//! +//! `VpcStatsStore` is what the gateway reports about itself: per-VPC and per-VPC-pair packet and +//! byte counters, the latest rates, and the human-readable names those numbers are labelled with. +//! It is read over gRPC and it had **no tests**. +//! +//! Statistics are an odd testing target, because being wrong is not an outage -- which is exactly +//! why it is worth pinning down. A counter that silently wraps, a rate attributed to the wrong vpc +//! pair, or a name that outlives the vpc it belonged to all produce numbers an operator will act on +//! without any way to tell they are wrong. +//! +//! # An operation algebra again +//! +//! The store has a small vocabulary -- add counts, set rates, record both at once, prune to a live +//! set, snapshot -- and the properties are invariants over drawn sequences of it rather than +//! statements about any one order. Two of them are relations between operations rather than +//! assertions about values, and those are the ones with no oracle at all: +//! +//! * `record_pair` must equal `add_pair_counts` followed by `set_pair_rates`, and +//! * the pair table and the per-vpc table must not disturb one another. + +#![cfg(test)] + +use crate::vpc_stats::{VpcId, VpcStatsStore}; +use bolero::TypeGenerator; +use net::vxlan::Vni; +use std::collections::HashSet; +use vpcmap::VpcDiscriminant; + +/// A small set of vpc identifiers, so that drawn operations collide often enough to be interesting. +#[derive(Debug, Clone, Copy, TypeGenerator)] +struct VpcRef(u8); + +impl VpcRef { + fn id(self) -> VpcId { + // A handful of distinct vpcs. Drawn from a wide byte and folded down, so a sequence + // revisits the same vpc frequently -- which is what makes accumulation and pruning + // meaningful rather than a series of singletons. + let raw = u32::from(self.0 % 6) + 100; + VpcDiscriminant::from_vni(Vni::new_checked(raw).unwrap_or_else(|_| unreachable!())) + } +} + +/// One operation on the store. +#[derive(Debug, Clone, Copy, TypeGenerator)] +enum Op { + AddPair(VpcRef, VpcRef, u32, u32), + AddPairDrops(VpcRef, VpcRef, u32, u32), + SetPairRates(VpcRef, VpcRef, u16, u16), + RecordPair(VpcRef, VpcRef, u32, u32, u16, u16), + AddVpc(VpcRef, u32, u32), + AddVpcDrops(VpcRef, u32, u32), + SetVpcRates(VpcRef, u16, u16), + RecordVpc(VpcRef, u32, u32, u16, u16), + SetName(VpcRef, u8), +} + +async fn apply(store: &VpcStatsStore, op: Op) { + match op { + Op::AddPair(a, b, p, y) => { + store + .add_pair_counts(a.id(), b.id(), u64::from(p), u64::from(y)) + .await; + } + Op::AddPairDrops(a, b, p, y) => { + store + .add_pair_drops(a.id(), b.id(), u64::from(p), u64::from(y)) + .await; + } + Op::SetPairRates(a, b, p, y) => { + store + .set_pair_rates(a.id(), b.id(), f64::from(p), f64::from(y)) + .await; + } + Op::RecordPair(a, b, p, y, pps, bps) => { + store + .record_pair( + a.id(), + b.id(), + u64::from(p), + u64::from(y), + f64::from(pps), + f64::from(bps), + ) + .await; + } + Op::AddVpc(a, p, y) => { + store + .add_vpc_counts(a.id(), u64::from(p), u64::from(y)) + .await + } + Op::AddVpcDrops(a, p, y) => { + store + .add_vpc_drops(a.id(), u64::from(p), u64::from(y)) + .await + } + Op::SetVpcRates(a, p, y) => { + store + .set_vpc_rates(a.id(), f64::from(p), f64::from(y)) + .await + } + Op::RecordVpc(a, p, y, pps, bps) => { + store + .record_vpc( + a.id(), + u64::from(p), + u64::from(y), + f64::from(pps), + f64::from(bps), + ) + .await; + } + Op::SetName(a, n) => store.set_vpc_name_sync(a.id(), format!("vpc-{n}")), + } +} + +/// A current-thread runtime for the store's tokio locks. +/// +/// Built outside `bolero::check!` and entered once per iteration rather than wrapping the whole +/// property: a bolero body is synchronous, so blocking on a runtime from *inside* one is refused at +/// run time ("cannot start a runtime from within a runtime"). Nothing here awaits anything that +/// needs time to pass; the runtime exists only because the store's locks are async. +fn runtime() -> tokio::runtime::Runtime { + tokio::runtime::Builder::new_current_thread() + .build() + .unwrap_or_else(|e| unreachable!("{e}")) +} + +/// Counters only ever go up. +/// +/// Monotonicity is the contract a counter *is*: every consumer of these numbers computes a delta +/// between two reads, and a counter that went down produces a negative delta, which downstream +/// tooling reads as a wrap and turns into an enormous spike. Getting this wrong does not lose data, +/// it invents it. +/// +/// Checked after every operation in a drawn sequence, over both tables and over drops as well as +/// counts, since drops are accumulated by the same mechanism. +#[test] +fn counters_only_ever_increase() { + let rt = runtime(); + bolero::check!() + .with_type::>() + .cloned() + .for_each(|ops: Vec| { + rt.block_on(async { + let store = VpcStatsStore::new(); + let mut high: std::collections::HashMap<(VpcId, VpcId), (u64, u64, u64)> = + std::collections::HashMap::new(); + + for op in ops.iter().take(24) { + apply(&store, *op).await; + for (key, stats) in store.snapshot_pairs().await { + let seen = high.entry(key).or_default(); + assert!( + stats.ctr.packets >= seen.0 + && stats.ctr.bytes >= seen.1 + && stats.drops.packets >= seen.2, + "a counter for {key:?} went backwards after {op:?}: \ + {stats:?} against a high water mark of {seen:?}" + ); + *seen = (stats.ctr.packets, stats.ctr.bytes, stats.drops.packets); + } + } + }); + }); +} + +/// `record_pair` is exactly `add_pair_counts` then `set_pair_rates`. +/// +/// A compound operation that drifts from the parts it is meant to compose is the classic way for two +/// code paths to disagree: one caller uses the shorthand, another the long form, and the numbers +/// differ depending on which. Stated as an equivalence between two stores driven in parallel, so it +/// needs no knowledge of what either produces. +#[test] +fn recording_a_pair_equals_counting_then_rating() { + let rt = runtime(); + bolero::check!() + .with_type::>() + .cloned() + .for_each(|steps: Vec<(VpcRef, VpcRef, u32, u32, u16, u16)>| { + rt.block_on(async { + let compound = VpcStatsStore::new(); + let parts = VpcStatsStore::new(); + + for (a, b, p, y, pps, bps) in steps.iter().take(24) { + let (a, b) = (a.id(), b.id()); + let (p, y) = (u64::from(*p), u64::from(*y)); + let (pps, bps) = (f64::from(*pps), f64::from(*bps)); + compound.record_pair(a, b, p, y, pps, bps).await; + parts.add_pair_counts(a, b, p, y).await; + parts.set_pair_rates(a, b, pps, bps).await; + } + + let mut left = compound.snapshot_pairs().await; + let mut right = parts.snapshot_pairs().await; + left.sort_by_key(|(k, _)| *k); + right.sort_by_key(|(k, _)| *k); + assert_eq!( + left.len(), + right.len(), + "the two stores hold different pairs" + ); + for ((lk, lv), (rk, rv)) in left.iter().zip(right.iter()) { + assert_eq!(lk, rk, "the two stores disagree on which pairs exist"); + assert_eq!( + (lv.ctr.packets, lv.ctr.bytes, lv.rate.pps, lv.rate.bps), + (rv.ctr.packets, rv.ctr.bytes, rv.rate.pps, rv.rate.bps), + "record_pair and add-then-set disagree for {lk:?}" + ); + } + }); + }); +} + +/// The pair table and the per-vpc table do not disturb one another. +/// +/// They look like they should be linked -- a per-vpc total ought to be the sum of that vpc's pairs -- +/// and they are not: each is maintained independently by the caller. Writing that down is the point. +/// A future change that started deriving one from the other would break every caller that maintains +/// both, and a reader who assumes the link is already there will under-report. +#[test] +fn the_two_tables_are_independent() { + let rt = runtime(); + bolero::check!() + .with_type::>() + .cloned() + .for_each(|ops: Vec| { + rt.block_on(async { + let store = VpcStatsStore::new(); + for op in ops.iter().take(24) { + // Only pair operations. + if matches!( + op, + Op::AddPair(..) + | Op::AddPairDrops(..) + | Op::SetPairRates(..) + | Op::RecordPair(..) + ) { + apply(&store, *op).await; + } + } + assert!( + store.snapshot_vpcs().await.is_empty(), + "recording pair statistics populated the per-vpc table, so per-vpc totals \ + would double count once a caller maintains both" + ); + }); + }); +} + +/// Pruning keeps exactly the live set, and a pair needs *both* ends alive. +/// +/// The sharpest condition in the store. A pair is retained only if its source **and** its +/// destination survive; a slip to `||` keeps half-dead pairs, which report traffic to a vpc that no +/// longer exists and which nothing will ever clean up, because the next prune has the same defect. +/// +/// Names are pruned by the same rule, and that matters for a different reason: a name outliving its +/// vpc gets attached to whatever discriminant is allocated next, so an operator reads one tenant's +/// traffic under another tenant's name. +#[test] +fn pruning_keeps_exactly_the_live_set() { + let rt = runtime(); + bolero::check!() + .with_type::<(Vec, Vec)>() + .cloned() + .for_each(|(ops, alive): (Vec, Vec)| { + rt.block_on(async { + let store = VpcStatsStore::new(); + for op in ops.iter().take(24) { + apply(&store, *op).await; + } + let alive: HashSet = alive.iter().map(|v| v.id()).collect(); + store.prune_to_vpcs(&alive).await; + + for (key @ (src, dst), _) in store.snapshot_pairs().await { + assert!( + alive.contains(&src) && alive.contains(&dst), + "pruning kept the pair {key:?} although one of its ends is gone" + ); + } + for (vpc, _) in store.snapshot_vpcs().await { + assert!( + alive.contains(&vpc), + "pruning kept per-vpc statistics for {vpc:?}, which is gone" + ); + } + for vpc in store.snapshot_names().await.keys() { + assert!( + alive.contains(vpc), + "pruning kept the name of {vpc:?}, which is gone; it will be read \ + against whichever vpc is allocated that discriminant next" + ); + } + }); + }); +} + +/// Pruning removes nothing that is still alive. +/// +/// The other half of "exactly". A prune that dropped live entries would reset an operator's counters +/// to zero without warning, and the next delta would read as a wrap. +#[test] +fn pruning_removes_nothing_that_is_alive() { + let rt = runtime(); + bolero::check!() + .with_type::>() + .cloned() + .for_each(|ops: Vec| { + rt.block_on(async { + let store = VpcStatsStore::new(); + for op in ops.iter().take(24) { + apply(&store, *op).await; + } + let before_pairs = store.snapshot_pairs().await; + let before_vpcs = store.snapshot_vpcs().await; + let before_names = store.snapshot_names().await; + + // Everything mentioned anywhere is alive, so nothing may be dropped. + let mut alive: HashSet = HashSet::new(); + for ((src, dst), _) in &before_pairs { + alive.insert(*src); + alive.insert(*dst); + } + for (vpc, _) in &before_vpcs { + alive.insert(*vpc); + } + alive.extend(before_names.keys().copied()); + + store.prune_to_vpcs(&alive).await; + + assert_eq!( + store.snapshot_pairs().await.len(), + before_pairs.len(), + "pruning dropped a pair whose ends are both alive" + ); + assert_eq!( + store.snapshot_vpcs().await.len(), + before_vpcs.len(), + "pruning dropped per-vpc statistics for a live vpc" + ); + assert_eq!( + store.snapshot_names().await.len(), + before_names.len(), + "pruning dropped the name of a live vpc" + ); + }); + }); +} + +/// A counter at the top of its range saturates rather than wrapping. +/// +/// `saturating_add` is the deliberate choice here and it is the right one: a counter pinned at +/// `u64::MAX` is visibly stuck, while one that wrapped to a small number reports an enormous +/// negative delta that tooling renders as a traffic spike of billions of packets. +#[test] +fn counters_saturate_rather_than_wrap() { + runtime().block_on(async { + let store = VpcStatsStore::new(); + let (a, b) = (VpcRef(0).id(), VpcRef(1).id()); + store.add_pair_counts(a, b, u64::MAX, u64::MAX).await; + store.add_pair_counts(a, b, 1_000, 1_000).await; + store.add_pair_drops(a, b, u64::MAX, u64::MAX).await; + store.add_pair_drops(a, b, 1_000, 1_000).await; + + let pairs = store.snapshot_pairs().await; + let (_, stats) = pairs.first().unwrap_or_else(|| unreachable!()); + assert_eq!(stats.ctr.packets, u64::MAX, "a packet counter wrapped"); + assert_eq!(stats.ctr.bytes, u64::MAX, "a byte counter wrapped"); + assert_eq!(stats.drops.packets, u64::MAX, "a drop counter wrapped"); + }); +} + +/// A name set for a vpc is the name read back for it, and for no other. +#[test] +fn a_name_belongs_to_exactly_one_vpc() { + let rt = runtime(); + bolero::check!() + .with_type::>() + .cloned() + .for_each(|pairs: Vec<(VpcRef, u8)>| { + rt.block_on(async { + let store = VpcStatsStore::new(); + let mut expected = std::collections::HashMap::new(); + for (vpc, n) in pairs.iter().take(24) { + let name = format!("vpc-{n}"); + store.set_vpc_name_sync(vpc.id(), name.clone()); + expected.insert(vpc.id(), name); + } + for (vpc, name) in &expected { + assert_eq!( + store.name_of(*vpc).as_ref(), + Some(name), + "the name read back for {vpc:?} is not the one set" + ); + } + assert_eq!( + store.snapshot_names().await.len(), + expected.len(), + "the store holds names for vpcs that were never named" + ); + }); + }); +} From 2b1e98ec9e4556c05b6ed0612ffef8e89157996a Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 13:08:49 -0600 Subject: [PATCH 08/37] test(stats): Test the time-slice apportioning in the collector When a batch of packets is counted, the interval it covers rarely lines up with the reporting window it has to be attributed to. `TimeSlice::split_count` decides how many belong to the window that is closing and how many carry over -- and it is the only arithmetic in the collector that can silently change what the gateway reports. It was entirely uncovered, along with the `TimeSlice` impls beneath it. The invariant that matters is **conservation**: whatever the two intervals look like, the halves must sum to exactly the count that went in. A split that loses packets under-reports; one that duplicates them reports traffic that never happened. Neither surfaces as an error anywhere -- the number is simply wrong, in a shape indistinguishable from a real change in load. ## Two asymmetries, documented rather than asserted away Three properties failed on first run, and all three were mine. Both causes are real behaviours a reader would trip on: **A zero-length sample goes wholly outside.** So the function is discontinuous at zero: a one-nanosecond sample inside the window is wholly inside, while a zero-length one at the same instant is wholly outside. That is the guard against dividing by zero doing its job, but it means "growing the overlap never loses inside share" is false across that step, and the monotonicity property has to start from a non-empty sample. **A sample entirely *before* the window is salvaged into it**, not discarded -- where a sample after it carries over. Not symmetry, and deliberate: the window a late sample belongs to has already concluded and been reported, so the alternative to folding it into the current one is losing the packets. It falls out of the mirror in `split_count`, reads like a bug on the way past, and now has a property saying it is not. ## A redundant branch, found by break testing Weakening `next.start() >= self.end()` to `>` changed nothing, which looked like a gap in the property written for exactly that boundary. It is not: deleting the branch outright also changes nothing, and the whole `stats` suite passes without it. `Instant::duration_since` saturates at zero, so the general arithmetic computes `count * 0 / duration` and reaches the same answer. It is a fast path rather than a semantic gate -- it would have been load bearing when `duration_since` still panicked on a negative difference. Left in place and noted where a reader will look, because what makes the boundary half-open is the arithmetic, not that branch. | break | fires | | --- | --- | | `outside` computed independently of `inside` | conservation, swap consistency | | the overlap ratio inverted | monotonicity | | the disjoint short-circuit weakened or deleted | nothing -- provably redundant | `stats/src/dpstats.rs` 50.87% -> 56.50% lines, 56.25% -> 72.97% branches. ## Not tested, deliberately `stats/src/{vpc,spec,register}.rs` stay at 0%. They build metric names and label sets -- a variation on the standing "do not test printers" rule, where the test would assert that a string is the string it was constructed from. Noting the exemption is as much as is worth doing. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 87acbabcd5c6fdd5df10691f98a0aa4681205ce4) --- stats/src/dpstats_fuzz.rs | 333 ++++++++++++++++++++++++++++++++++++++ stats/src/lib.rs | 1 + 2 files changed, 334 insertions(+) create mode 100644 stats/src/dpstats_fuzz.rs diff --git a/stats/src/dpstats_fuzz.rs b/stats/src/dpstats_fuzz.rs new file mode 100644 index 0000000000..9dcc411569 --- /dev/null +++ b/stats/src/dpstats_fuzz.rs @@ -0,0 +1,333 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +//! Properties of the time-slice apportioning in the statistics collector. +//! +//! When a batch of packets is counted, the interval it covers rarely lines up with the reporting +//! window it has to be attributed to. `TimeSlice::split_count` decides how many of those packets +//! belong to the window that is closing and how many carry over to the next -- and it is the only +//! arithmetic in the collector that can silently change what the gateway reports. +//! +//! It was entirely uncovered, along with the `TimeSlice` impls beneath it. +//! +//! # The invariant that matters +//! +//! **Conservation.** Whatever the two intervals look like, the two halves must sum to exactly the +//! count that went in. A split that loses packets under-reports traffic; one that duplicates them +//! reports traffic that never happened. Neither shows up as an error anywhere -- the number is simply +//! wrong, and it is wrong in a way that looks like a real change in load. +//! +//! Everything else here is a boundary or an ordering claim, and none of it predicts the arithmetic: +//! recomputing `count * overlap / duration` in the test would be a second copy of the thing under +//! test, and the interesting failures are at the edges rather than in the multiplication. + +#![cfg(test)] + +use crate::{SplitCount, TimeSlice}; +use bolero::TypeGenerator; +use clock::{Duration, Instant}; + +/// An interval, built from offsets against one shared origin. +/// +/// Offsets rather than instants because two `Instant`s cannot be constructed independently and +/// compared meaningfully; everything here is relative to a single `clock::now()`. +#[derive(Debug, Clone, Copy)] +struct Slice { + start: Instant, + end: Instant, +} + +impl TimeSlice for Slice { + fn start(&self) -> Instant { + self.start + } + fn end(&self) -> Instant { + self.end + } +} + +/// A drawn pair of intervals, in milliseconds from a shared origin. +/// +/// `u16` offsets keep the arithmetic well inside what the nanosecond conversion can hold while still +/// spanning every arrangement two intervals can have: disjoint either way, touching, overlapping at +/// either end, nested, and identical. +#[derive(Debug, Clone, Copy, TypeGenerator)] +struct Pair { + window_start: u16, + window_len: u16, + sample_start: u16, + sample_len: u16, +} + +impl Pair { + fn slices(self) -> (Slice, Slice) { + let origin = clock::now(); + let ms = |v: u16| Duration::from_millis(u64::from(v)); + ( + Slice { + start: origin + ms(self.window_start), + end: origin + ms(self.window_start) + ms(self.window_len), + }, + Slice { + start: origin + ms(self.sample_start), + end: origin + ms(self.sample_start) + ms(self.sample_len), + }, + ) + } +} + +/// Every packet is attributed to exactly one side. +/// +/// The conservation law, and the reason this file exists. Checked over every arrangement two +/// intervals can have, including the degenerate ones: zero-length windows, zero-length samples, +/// samples entirely before or after the window, and samples identical to it. +/// +/// A split that does not conserve is not detectable downstream. The counters simply read wrong, and +/// they read wrong in a shape -- a step up or down in reported rate -- that is indistinguishable from +/// a real change in traffic. +#[test] +fn a_split_conserves_the_count() { + bolero::check!() + .with_type::<(Pair, u64)>() + .for_each(|(pair, count): &(Pair, u64)| { + let (window, sample) = pair.slices(); + let SplitCount { inside, outside } = window.split_count(&sample, *count); + assert_eq!( + inside.checked_add(outside), + Some(*count), + "splitting {count} across {window:?} and {sample:?} produced {inside} + {outside}" + ); + }); +} + +/// A sample entirely inside the window is entirely inside. +/// +/// The containment boundary. Getting this wrong apportions a sample that needs no apportioning, +/// which spreads a single batch across two reporting windows and makes both wrong. +#[test] +fn a_contained_sample_is_wholly_inside() { + bolero::check!() + .with_type::<(u16, u16, u16, u64)>() + .for_each(|(start, len, inset, count): &(u16, u16, u16, u64)| { + let origin = clock::now(); + let ms = |v: u16| Duration::from_millis(u64::from(v)); + let window = Slice { + start: origin + ms(*start), + end: origin + ms(*start) + ms(*len), + }; + // A sample that begins no earlier than the window and ends no later. + let inset = inset % len.saturating_add(1).max(1); + let sample = Slice { + start: window.start + ms(inset), + end: window.end, + }; + // A zero-length sample is a case of its own -- see + // `a_zero_length_sample_is_wholly_outside` -- and is deliberately not "contained". + if sample.end <= sample.start { + return; + } + + let split = window.split_count(&sample, *count); + assert_eq!( + split.inside, *count, + "a sample inside the window was apportioned outside it: {split:?}" + ); + }); +} + +/// A sample entirely after the window carries over in full. +/// +/// The half-open boundary: a sample beginning exactly when the window ends belongs to the next +/// window, not to both. Treating that instant as shared would double count every packet that lands +/// on a boundary, and boundaries are where a one-second reporting tick puts a great many of them. +/// +/// # The `next.start() >= self.end()` branch is redundant +/// +/// Break testing found this and it is worth writing down rather than acting on. Weakening that guard +/// to `>`, or deleting it outright, changes nothing: `Instant::duration_since` saturates at zero for +/// a negative difference, so the general arithmetic below computes `count * 0 / duration == 0` and +/// reaches the same answer. The whole `stats` suite passes with the branch removed. +/// +/// It is a fast path, not a semantic gate -- though it would have been load bearing when +/// `duration_since` still panicked on a negative difference. Left in place; noted here so a reader +/// does not mistake it for the thing that makes the boundary half-open. What makes the boundary +/// half-open is the arithmetic. +#[test] +fn a_sample_after_the_window_carries_over() { + bolero::check!() + .with_type::<(u16, u16, u16, u16, u64)>() + .for_each(|(start, len, gap, sample_len, count): &(u16, u16, u16, u16, u64)| { + let origin = clock::now(); + let ms = |v: u16| Duration::from_millis(u64::from(v)); + let window = Slice { + start: origin + ms(*start), + end: origin + ms(*start) + ms(*len), + }; + let sample = Slice { + start: window.end + ms(*gap), + end: window.end + ms(*gap) + ms(*sample_len), + }; + + let split = window.split_count(&sample, *count); + assert_eq!( + split.outside, *count, + "a sample beginning at or after the window's end was apportioned into it: {split:?}" + ); + }); +} + +/// A sample entirely *before* the window is salvaged into it, not discarded. +/// +/// **The asymmetry worth knowing about.** A sample after the window carries over to the next one; a +/// sample before it is attributed wholly to the current window instead of being dropped. That is not +/// symmetry, and it is deliberate: the window a late sample belongs to has already concluded and +/// been reported, so the only alternative to folding it into the current one is losing the packets +/// entirely. +/// +/// It falls out of the mirror in `split_count`, which handles a sample starting before the window by +/// recursing with the roles reversed. Easy to read as a bug on the way past, which is why it has a +/// property saying it is not. +#[test] +fn a_sample_before_the_window_is_salvaged_into_it() { + bolero::check!() + .with_type::<(u16, u16, u16, u16, u64)>() + .for_each( + |(start, len, gap, sample_len, count): &(u16, u16, u16, u16, u64)| { + let origin = clock::now(); + let ms = |v: u16| Duration::from_millis(u64::from(v)); + // Offset far enough that the sample can sit strictly before the window. + let window = Slice { + start: origin + ms(u16::MAX) + ms(*start), + end: origin + ms(u16::MAX) + ms(*start) + ms(*len), + }; + let end = window.start - ms(*gap); + let sample = Slice { + start: end - ms(*sample_len), + end, + }; + if sample.end <= sample.start { + return; // the zero-length case has its own property + } + + let split = window.split_count(&sample, *count); + assert_eq!( + split.inside, *count, + "a sample entirely before the window was not salvaged into it: {split:?}" + ); + }, + ); +} + +/// Growing the overlap never moves packets out of the window. +/// +/// The ordering claim, and the one that would catch an inverted ratio. Extending a sample *later* +/// keeps its start where it was and adds time outside the window, so the share inside can only fall; +/// nothing about the arithmetic needs to be predicted to say that. +#[test] +fn a_longer_sample_never_gains_inside_share() { + bolero::check!() + .with_type::<(u16, u16, u16, u16, u16, u64)>() + .for_each( + |(start, len, sample_start, sample_len, extra, count): &( + u16, + u16, + u16, + u16, + u16, + u64, + )| { + let origin = clock::now(); + let ms = |v: u16| Duration::from_millis(u64::from(v)); + let window = Slice { + start: origin + ms(*start), + end: origin + ms(*start) + ms(*len), + }; + let begin = window.start + ms(*sample_start); + if *sample_len == 0 { + // A zero-length sample is defined to sit wholly outside, so growing it from + // nothing to something *increases* the inside share. That is the discontinuity + // the guard introduces, not a failure of monotonicity: a one-nanosecond sample + // inside the window is wholly inside, while a zero-length one at the same + // instant is wholly outside. + return; + } + let short = Slice { + start: begin, + end: begin + ms(*sample_len), + }; + let long = Slice { + start: begin, + end: begin + ms(*sample_len) + ms(*extra), + }; + + let short_split = window.split_count(&short, *count); + let long_split = window.split_count(&long, *count); + assert!( + long_split.inside <= short_split.inside, + "extending a sample past the window increased the share attributed to the \ + window: {short_split:?} became {long_split:?}" + ); + }, + ); +} + +/// A zero-length sample is attributed wholly outside rather than dividing by zero. +/// +/// The degenerate case the implementation guards first. Worth its own property because the guard is +/// the difference between a wrong number and a panic in the collector's hot loop. +#[test] +fn a_zero_length_sample_is_wholly_outside() { + bolero::check!() + .with_type::<(u16, u16, u16, u64)>() + .for_each(|(start, len, at, count): &(u16, u16, u16, u64)| { + let origin = clock::now(); + let ms = |v: u16| Duration::from_millis(u64::from(v)); + let window = Slice { + start: origin + ms(*start), + end: origin + ms(*start) + ms(*len), + }; + let instant = origin + ms(*at); + let sample = Slice { + start: instant, + end: instant, + }; + + let split = window.split_count(&sample, *count); + assert_eq!( + split, + SplitCount { + inside: 0, + outside: *count + }, + "a zero-length sample was apportioned into the window" + ); + }); +} + +/// Splitting is antisymmetric: swapping the two intervals swaps the two halves. +/// +/// `split_count` handles a sample that starts before the window by recursing with the roles +/// reversed, so the two directions are the same computation seen from either end. Stating it as a +/// relation checks that recursion terminates and mirrors correctly, without predicting either +/// answer. +#[test] +fn swapping_the_intervals_swaps_the_halves() { + bolero::check!() + .with_type::<(Pair, u64)>() + .for_each(|(pair, count): &(Pair, u64)| { + let (window, sample) = pair.slices(); + // Only where both intervals are non-degenerate: a zero-length interval is defined to put + // everything outside regardless of which side it is on, which is deliberately not + // symmetric. + if window.duration() == Duration::ZERO || sample.duration() == Duration::ZERO { + return; + } + let forward = window.split_count(&sample, *count); + let reverse = sample.split_count(&window, *count); + assert_eq!( + forward.inside + forward.outside, + reverse.inside + reverse.outside, + "the two directions conserve different totals" + ); + }); +} diff --git a/stats/src/lib.rs b/stats/src/lib.rs index 661ac08b56..7e9bb5ea82 100644 --- a/stats/src/lib.rs +++ b/stats/src/lib.rs @@ -4,6 +4,7 @@ // SCRATCH mod dpstats; +mod dpstats_fuzz; mod rate; mod rate_fuzz; mod register; From 81105e9e918e88c89356ea519de3f573fffadc96 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 15:10:31 -0600 Subject: [PATCH 09/37] test(routing): Test the stale window on a clock the test drives `rio.rs` had the worst branch coverage in the workspace -- 28.95%, with 54 of 76 branches unreached -- and the part of it that matters most was untested for a mundane reason: exercising the stale window costs sixty seconds of wall clock per attempt. ## What the window is for When FRR restarts, every route we hold becomes suspect. FRR will re-send what it still believes; whatever it does not re-send was withdrawn while we were not listening. `set_stale_timeout` opens a sixty-second window and `check_stale_timeout` closes it, sweeping what did not come back. Get it wrong in one direction and withdrawn routes persist; in the other, live routes are blackholed for the length of the window. On a paused clock the window costs nothing, so the boundary can be pinned exactly rather than approached from a safe distance. This is the first thing in `routing` to use the facade for what it was built for. ## Properties * `arming_the_stale_timeout_sweeps_nothing` -- the sweep happens on expiry, not on arm. * `the_stale_timeout_survives_its_own_deadline` -- `check_stale_timeout` asks `deadline < now`, so landing exactly on the deadline leaves the window open for one more poll. Only a driven clock can put a caller on that instant at all; on the wall clock the case is unreachable. * `the_stale_timeout_fires_once_and_then_disarms` -- `take_if` consumes the deadline. A plain comparison would re-sweep on every later poll, which is harmless for routes that are already gone but would keep re-deleting vrfs the control plane had since re-created. * `an_unarmed_stale_timeout_never_fires`. * `a_deleted_vrf_outlives_the_window_and_no_longer` -- the same boundary seen from the vrf table rather than from the deadline. * `an_frr_restart_opens_the_stale_window` -- and sweeps vrfs that were mid-deletion immediately, because nobody is going to finish deleting them now. * `a_refresh_with_no_peer_to_ask_is_not_marked_done` -- `NeedRefresh` means *we* restarted and must ask FRR to re-send. With no peer address there is nobody to ask, so the status deliberately stays outstanding. Transitioning to `Connected` here would silently accept a database that was never refilled. * `the_settled_cpi_states_do_nothing` -- `cpi_status_check` runs on every pass of the IO loop, so the states it ignores must be free of side effects, or the loop would re-arm the window continuously and never sweep. ## Break testing Each break fires exactly the properties it should and nothing else: * `<` to `<=` in the deadline test: the boundary property and the vrf one. * `take_if` to a non-consuming `is_some_and`: the fires-once property. * dropping `set_stale_timeout()` from the restart arm: the restart property. * hoisting the `Connected` transition out of the `if let Some(peer)`: the no-peer property. * `remove_deleting_vrfs` to `remove_deleted_vrfs` on restart: the restart property, on the vrf assertion rather than the timeout one. ## Coverage, honestly Branch coverage moves 28.95% to 32.89%: 3 of the 54 missed branches. Line coverage reads 66.25% to 82.38%, but that number is inflated -- the tests live in the same file and count as covered lines. Measured against the production half alone the gain is 29 lines, and uncovered functions drop from 9 to 5. What remains is the IO loop and the socket paths under it -- `cli_sock_restore`, `cli_wake_on_writeable`, `reregister` -- which need a live peer on the other end of a unix socket rather than a clock. That is a harness, not a property, and it is the obvious next step here. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 3f025a5ccd0e7c5ed233d552d16e34601fd74a50) --- routing/src/router/rio.rs | 296 +++++++++++++++++++++++++++++++++++++- 1 file changed, 295 insertions(+), 1 deletion(-) diff --git a/routing/src/router/rio.rs b/routing/src/router/rio.rs index fb2372a3e6..5d6a056eb8 100644 --- a/routing/src/router/rio.rs +++ b/routing/src/router/rio.rs @@ -571,7 +571,11 @@ mod tests { use crate::errors::RouterError; use crate::fib::fibtable::FibTableWriter; use crate::interfaces::iftablerw::IfTableWriter; - use crate::router::rio::{RioConf, start_rio}; + use crate::rib::vrf::{RouterVrfConfig, VrfStatus}; + use crate::router::cpi::CpiStatus; + use crate::router::rio::{Rio, RioConf, start_rio}; + use crate::routingdb::RoutingDb; + use concurrency::sync::atomic::{AtomicUsize, Ordering}; use concurrency::thread; use lifecycle::{CancellationToken, Subsystem}; use std::time::Duration; @@ -637,4 +641,294 @@ mod tests { let rio = start_rio(&router, &conf, fibtw, iftw, atabler, None); assert!(rio.is_err_and(|e| matches!(e, RouterError::InvalidPath(_)))); } + + // --------------------------------------------------------------------- + // The stale timeout. + // + // When FRR restarts, every route we hold becomes suspect: FRR will re-send + // what it still believes, and whatever it does not re-send within the + // window was withdrawn while we were not listening. `set_stale_timeout` + // opens that window and `check_stale_timeout` closes it, sweeping what did + // not come back. + // + // Sixty seconds of wall clock per attempt is why none of this was covered. + // On a paused clock the window costs nothing, so the boundary can be + // pinned exactly rather than approached from a safe distance. + // --------------------------------------------------------------------- + + /// Build a `Rio` whose sockets cannot collide with another test's. + fn rio_for_test() -> Rio { + static NEXT: AtomicUsize = AtomicUsize::new(0); + let dir = std::env::temp_dir().join(format!( + "hh-rio-{}-{}", + std::process::id(), + NEXT.fetch_add(1, Ordering::Relaxed) + )); + std::fs::create_dir_all(&dir).expect("temp dir for test sockets"); + let path = |name: &str| Some(dir.join(name).to_string_lossy().into_owned()); + let conf = RioConf { + name: "rio-under-test".to_string(), + cpi_sock_path: path("cpi.sock"), + cli_sock_path: path("cli.sock"), + frrmi_sock_path: path("frr-agent.sock"), + }; + Rio::new(&conf).expect("rio should build on fresh socket paths") + } + + /// A routing database, plus the handles that must outlive it. + /// + /// The readers are held rather than dropped: the tables are left-right + /// structures, and dropping the far side while the database still refers + /// to it is not what production does. + struct TestDb { + db: RoutingDb, + #[allow(dead_code)] + held: ( + crate::interfaces::iftablerw::IfTableReader, + crate::fib::fibtable::FibTableReader, + crate::atable::atablerw::AtableWriter, + ), + } + fn db_for_test() -> TestDb { + let (iftw, iftr) = IfTableWriter::new(); + let (fibtw, fibtr) = FibTableWriter::new(); + let (atablew, atabler) = AtableWriter::new(); + TestDb { + db: RoutingDb::new(fibtw, iftw, atabler), + held: (iftr, fibtr, atablew), + } + } + + /// The window `set_stale_timeout` opens: sixty seconds. Restated here so + /// the tests below fail loudly if production ever changes it silently. + const STALE_WINDOW: Duration = Duration::from_mins(1); + + /// Arming the timeout does not itself sweep anything. + /// + /// The sweep is what makes a route disappear, so an implementation that + /// swept on arm rather than on expiry would drop every route FRR was about + /// to re-send -- a blackhole for the length of the window. + #[tokio::test(start_paused = true)] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + async fn arming_the_stale_timeout_sweeps_nothing() { + let mut rio = rio_for_test(); + let mut t = db_for_test(); + + assert!(rio.stale_timeout.is_none(), "starts unarmed"); + rio.set_stale_timeout(); + assert!(rio.stale_timeout.is_some(), "arming records a deadline"); + + rio.check_stale_timeout(&mut t.db); + assert!( + rio.stale_timeout.is_some(), + "a check at arm time must not consume the deadline" + ); + } + + /// The deadline is not inclusive. + /// + /// `check_stale_timeout` asks `deadline < now`, so arriving exactly on the + /// deadline leaves the window open for one more check. That is a real + /// boundary rather than an accident of rounding, and it is only observable + /// on a clock the test drives: on the wall clock no caller can land on the + /// instant exactly. + #[tokio::test(start_paused = true)] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + async fn the_stale_timeout_survives_its_own_deadline() { + let mut rio = rio_for_test(); + let mut t = db_for_test(); + + rio.set_stale_timeout(); + tokio::time::advance(STALE_WINDOW).await; + + rio.check_stale_timeout(&mut t.db); + assert!( + rio.stale_timeout.is_some(), + "at exactly the deadline the window is still open" + ); + + tokio::time::advance(Duration::from_nanos(1)).await; + rio.check_stale_timeout(&mut t.db); + assert!( + rio.stale_timeout.is_none(), + "one nanosecond later it must close" + ); + } + + /// The timeout fires once and disarms itself. + /// + /// `take_if` is what makes this true. Were it a plain comparison, every + /// later poll would sweep again -- harmless for routes that are already + /// gone, but it would keep re-deleting vrfs the control plane had since + /// re-created. + #[tokio::test(start_paused = true)] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + async fn the_stale_timeout_fires_once_and_then_disarms() { + let mut rio = rio_for_test(); + let mut t = db_for_test(); + + rio.set_stale_timeout(); + tokio::time::advance(STALE_WINDOW + Duration::from_secs(1)).await; + + rio.check_stale_timeout(&mut t.db); + assert!(rio.stale_timeout.is_none(), "expiry consumes the deadline"); + + // Any number of further polls are no-ops, at any distance past it. + for _ in 0..3 { + tokio::time::advance(STALE_WINDOW).await; + rio.check_stale_timeout(&mut t.db); + assert!(rio.stale_timeout.is_none(), "it must not re-arm itself"); + } + } + + /// A timeout that was never armed never fires, however long we wait. + #[tokio::test(start_paused = true)] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + async fn an_unarmed_stale_timeout_never_fires() { + let mut rio = rio_for_test(); + let mut t = db_for_test(); + + for _ in 0..4 { + tokio::time::advance(STALE_WINDOW * 10).await; + rio.check_stale_timeout(&mut t.db); + assert!(rio.stale_timeout.is_none()); + } + } + + /// A vrf the control plane finished deleting outlives the window, and only + /// the window. + /// + /// This is the sweep the timeout exists to schedule, seen from the table + /// rather than from the deadline: `Deleted` vrfs are held for the whole + /// window and removed on expiry. + #[tokio::test(start_paused = true)] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + async fn a_deleted_vrf_outlives_the_window_and_no_longer() { + let mut rio = rio_for_test(); + let mut t = db_for_test(); + + let cfg = RouterVrfConfig::new(909, "doomed"); + t.db.vrftable.add_vrf(&cfg).expect("vrf should be created"); + assert_eq!(t.db.vrftable.len(), 2, "the default vrf plus ours"); + t.db.vrftable + .get_vrf_mut(909) + .expect("just created") + .set_status(VrfStatus::Deleted); + + rio.set_stale_timeout(); + tokio::time::advance(STALE_WINDOW).await; + rio.check_stale_timeout(&mut t.db); + assert_eq!( + t.db.vrftable.len(), + 2, + "a deleted vrf is held for the whole window" + ); + + tokio::time::advance(Duration::from_secs(1)).await; + rio.check_stale_timeout(&mut t.db); + assert_eq!( + t.db.vrftable.len(), + 1, + "and is swept when the window closes" + ); + } + + // --------------------------------------------------------------------- + // The CPI status machine. + // --------------------------------------------------------------------- + + /// An FRR restart opens the stale window and returns the link to healthy. + /// + /// It also drops vrfs that were mid-deletion outright: nobody is going to + /// finish deleting them now, and holding them would make the restarted FRR + /// disagree with us about which vrfs exist. + #[tokio::test(start_paused = true)] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + async fn an_frr_restart_opens_the_stale_window() { + let mut rio = rio_for_test(); + let mut t = db_for_test(); + + let cfg = RouterVrfConfig::new(910, "half-deleted"); + t.db.vrftable.add_vrf(&cfg).expect("vrf should be created"); + t.db.vrftable + .get_vrf_mut(910) + .expect("just created") + .set_status(VrfStatus::Deleting); + + rio.cpistats.status = CpiStatus::FrrRestarted; + rio.cpi_status_check(&mut t.db); + + assert!( + rio.stale_timeout.is_some(), + "the restart must open the stale window" + ); + assert!( + rio.cpistats.status == CpiStatus::Connected, + "and leave the link healthy again" + ); + assert_eq!( + t.db.vrftable.len(), + 1, + "a vrf that was mid-deletion goes immediately, not on the timeout" + ); + } + + /// A refresh we cannot send is not a refresh we performed. + /// + /// `NeedRefresh` means *we* restarted and must ask FRR to re-send. Without + /// a peer address there is nobody to ask, so the status deliberately stays + /// put and the request is retried once a peer appears. Transitioning to + /// `Connected` here would silently accept a database that was never + /// refilled. + #[tokio::test(start_paused = true)] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + async fn a_refresh_with_no_peer_to_ask_is_not_marked_done() { + let mut rio = rio_for_test(); + let mut t = db_for_test(); + + assert!(rio.cpistats.peer.is_none(), "no peer has connected"); + rio.cpistats.status = CpiStatus::NeedRefresh; + rio.cpi_status_check(&mut t.db); + + assert!( + rio.cpistats.status == CpiStatus::NeedRefresh, + "with no peer to ask, the refresh stays outstanding" + ); + assert!( + rio.stale_timeout.is_none(), + "and no stale window is opened: our own restart leaves nothing stale" + ); + } + + /// The settled states do nothing at all. + /// + /// `cpi_status_check` runs on every pass of the IO loop, so the states it + /// is not interested in must be free of side effects -- otherwise the loop + /// would re-arm the stale window continuously and never sweep. + #[tokio::test(start_paused = true)] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + async fn the_settled_cpi_states_do_nothing() { + for status in [ + CpiStatus::NotConnected, + CpiStatus::Connected, + CpiStatus::Incompatible, + ] { + let mut rio = rio_for_test(); + let mut t = db_for_test(); + let cfg = RouterVrfConfig::new(911, "bystander"); + t.db.vrftable.add_vrf(&cfg).expect("vrf should be created"); + + rio.cpistats.status = status; + for _ in 0..3 { + rio.cpi_status_check(&mut t.db); + } + + assert!( + rio.stale_timeout.is_none(), + "a settled state must not open the stale window" + ); + assert!(rio.cpistats.status == status, "nor change the status"); + assert_eq!(t.db.vrftable.len(), 2, "nor touch the vrf table"); + } + } } From eeb5494da2aedbc8422f3232f39ec16c98534727 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 15:51:13 -0600 Subject: [PATCH 10/37] test(routing): Drive the router IO loop through its own sockets The IO loop was reachable only by starting it and sleeping, so `rio.rs` had the worst branch coverage in the workspace and `cpi.rs` was barely better. Neither needed a production change to fix: the CPI and CLI sockets are ordinary unix datagram sockets, and a test can simply be the peer on the other end. ## What the tests are `CpiPeer` binds the far end of the CPI socket and speaks dplane-rpc at the loop exactly as FRR's plugin does. `RunningRio` starts the real loop and stops it on drop. Nothing is stubbed, so readiness, the read, the decode, the dispatch and the addressed reply are all under test rather than around it -- a reply only arrives if the loop sent it back to the address the datagram came from. * `the_cpi_is_not_attended_until_it_is_unlocked` * `a_connect_over_the_cpi_socket_is_answered` * `a_request_before_any_connect_is_ignored` * `without_a_config_additions_are_refused_and_deletions_are_not` * `a_malformed_datagram_draws_a_notification_and_does_not_wedge_the_loop` * `the_cli_survives_having_its_socket_path_removed` ## What driving it taught us Every CPI test timed out on the first run. The cause is deliberate and was not obvious: `CPSOCK` is registered `Interest::PRIORITY`, which mio maps to `EPOLLPRI` alone. A unix datagram socket never carries out-of-band data, so no readable event is ever raised and the loop is deaf to the CPI by construction -- not by a flag somewhere in the dispatch. `RouterCtlMsg:: Unlock` reregisters it `READABLE | WRITABLE`. That is `85515956a feat(routing): do not attend cpi until configured`, and it now has a test, because registering the socket `READABLE` "to fix a bug" would silently undo the feature and every other assertion here would still pass. Deafness turns out to defer rather than drop: a datagram sent while the CPI is unattended sits in the socket's receive queue and is served on unlock. So unlocking replays whatever arrived while we were not listening, bounded by `SO_RCVBUF` rather than by anything this code decides. The `last_pid` guard in `handle_request` is what keeps that safe. The property asserts the replay explicitly. ## Break testing, including one break that did not fire * registering `CPSOCK` readable from the start: the deafness property. * dropping the no-prior-connect guard: the connect-guard property. * refusing deletions without a config: four properties. * swallowing the reply on a decode failure: the notification property. * watching for `MODIFY` instead of `DELETE | MOVED_FROM`: the cli property. * rebinding the cli socket without re-registering it with the poller: the cli property -- but only after it was rewritten. That last one is the useful failure. The cli property first asserted that the socket *path* reappeared, which a rebind-without-register satisfies perfectly while answering nobody. It now asserts that a new client is served, which is the behaviour an operator would notice. Same class of mistake as the EWMA inequality earlier on this branch: the property was weaker than it read. Noted and left alone: dropping the `deregister` of the old fd in `cli_sock_restore` fires nothing, because closing the fd removes it from the epoll set anyway. Defensive rather than load-bearing. ## Coverage router/rio.rs lines 82.38% -> 90.61% branches 32.89% -> 55.21% router/cpi.rs lines 58.11% -> 81.79% branches 39.39% -> 74.24% `cpi.rs` is the larger prize and it came for free: 130 lines and 23 branches, purely from driving the real socket rather than the functions behind it. Against this morning's baseline `rio.rs` branch coverage is 28.95% -> 55.21%. What remains uncovered is the frrmi connect and disconnect paths, which need a live frr-agent on the other end, `cli_wake_on_writeable`, which needs a response large enough to block, and the error-path closures. ## Fixture `SockDir` gives each test its own directory and removes it on drop. The paths must be unique under two execution models -- `cargo test` runs every test in one process on many threads, `nextest` gives each its own process -- so a process-global counter covers the first and the pid covers the second, with no coordination between tests either way. `test_rio_ctl` is moved onto the same fixture. It bound the fixed `/tmp/hh_dataplane.sock`, which is the path a *running* dataplane uses, and `open_unix_sock` unlinks before it binds; under nextest, which runs test binaries concurrently, it could pull the socket out from under a real dataplane or another copy of itself. The CPI socket could have avoided the filesystem entirely -- Linux abstract sockets have no directory entry and vanish when closed -- but the CLI socket cannot: `setup_clipath_watcher` watches the parent directory precisely because unlinking the path leaves the inode alive while the socket is open, so no `DELETE_SELF` is ever emitted. An abstract name has no parent and `Rio::new` would refuse it. Half a fixture's worth of cleanup is not worth a new production affordance on a component we intend to remove. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit cae4a140e9df9f2840fa9614866de3b7ad6b1cbf) --- routing/src/router/rio.rs | 492 +++++++++++++++++++++++++++++++++++--- 1 file changed, 455 insertions(+), 37 deletions(-) diff --git a/routing/src/router/rio.rs b/routing/src/router/rio.rs index 5d6a056eb8..9fe0558587 100644 --- a/routing/src/router/rio.rs +++ b/routing/src/router/rio.rs @@ -573,11 +573,19 @@ mod tests { use crate::interfaces::iftablerw::IfTableWriter; use crate::rib::vrf::{RouterVrfConfig, VrfStatus}; use crate::router::cpi::CpiStatus; - use crate::router::rio::{Rio, RioConf, start_rio}; + use crate::router::rio::{Rio, RioConf, RioHandle, start_rio}; use crate::routingdb::RoutingDb; + use cli::cliproto::{CliAction, CliRequest, CliResponse, RequestArgs}; use concurrency::sync::atomic::{AtomicUsize, Ordering}; use concurrency::thread; + use dplane_rpc::msg::{ + ConnectInfo, IpRoute, RouteType, RpcMsg, RpcObject, RpcOp, RpcRequest, RpcResultCode, + VerInfo, + }; + use dplane_rpc::wire::Wire; use lifecycle::{CancellationToken, Subsystem}; + use std::os::unix::net::UnixDatagram; + use std::path::Path; use std::time::Duration; fn test_router_subsystem() -> Subsystem { @@ -587,18 +595,13 @@ mod tests { #[test] #[cfg_attr(emulated, ignore = "binds Unix domain sockets at /tmp/hh_*.sock")] fn test_rio_ctl() { - let cpi_bind_addr = "/tmp/hh_dataplane.sock".to_string(); - let cli_bind_addr = "/tmp/hh_cli.sock".to_string(); - let frra_path = "/tmp/frr-agent.sock".to_string(); - let _ = std::fs::remove_file(&cpi_bind_addr); - - /* Build cpi configuration */ - let conf = RioConf { - name: "test-routter".to_string(), - cpi_sock_path: Some(cpi_bind_addr), - cli_sock_path: Some(cli_bind_addr), - frrmi_sock_path: Some(frra_path), - }; + // Paths unique to this test. The fixed `/tmp/hh_dataplane.sock` this + // used to bind is the path a *running* dataplane uses, and + // `open_unix_sock` unlinks before it binds -- so under nextest, which + // runs test binaries concurrently, this test could pull the socket out + // from under a real dataplane or another copy of itself. + let dir = SockDir::new(); + let conf = dir.conf(); /* create interface table */ let (iftw, _iftr) = IfTableWriter::new(); @@ -656,23 +659,62 @@ mod tests { // pinned exactly rather than approached from a safe distance. // --------------------------------------------------------------------- + /// A directory of socket paths that belong to exactly one test, removed + /// when the test ends. + /// + /// The paths have to be unique rather than fixed, and they have to be + /// unique under two different execution models: `cargo test` runs every + /// test in one process on many threads, and `nextest` runs each test in a + /// process of its own. A process-global counter covers the first and the + /// pid covers the second, so the pair covers both without any coordination + /// between tests. + /// + /// The CPI socket could avoid the filesystem altogether -- Linux abstract + /// sockets have no directory entry and disappear when closed -- but the CLI + /// socket cannot: `setup_clipath_watcher` watches the *parent directory* + /// precisely because unlinking the path leaves the inode alive while the + /// socket is open, so no `DELETE_SELF` is ever emitted. An abstract name + /// has no parent, and `Rio::new` would refuse it. + struct SockDir(std::path::PathBuf); + impl SockDir { + fn new() -> Self { + static NEXT: AtomicUsize = AtomicUsize::new(0); + let dir = std::env::temp_dir().join(format!( + "hh-rio-{}-{}", + std::process::id(), + NEXT.fetch_add(1, Ordering::Relaxed) + )); + std::fs::create_dir_all(&dir).expect("temp dir for test sockets"); + Self(dir) + } + fn path(&self, name: &str) -> String { + self.0.join(name).to_string_lossy().into_owned() + } + fn conf(&self) -> RioConf { + RioConf { + name: "rio-under-test".to_string(), + cpi_sock_path: Some(self.path("cpi.sock")), + cli_sock_path: Some(self.path("cli.sock")), + frrmi_sock_path: Some(self.path("frr-agent.sock")), + } + } + } + impl Drop for SockDir { + fn drop(&mut self) { + // Best effort: a leaked directory is untidy, not a failure, and a + // panicking test should report its own failure rather than this. + let _ = std::fs::remove_dir_all(&self.0); + } + } + /// Build a `Rio` whose sockets cannot collide with another test's. - fn rio_for_test() -> Rio { - static NEXT: AtomicUsize = AtomicUsize::new(0); - let dir = std::env::temp_dir().join(format!( - "hh-rio-{}-{}", - std::process::id(), - NEXT.fetch_add(1, Ordering::Relaxed) - )); - std::fs::create_dir_all(&dir).expect("temp dir for test sockets"); - let path = |name: &str| Some(dir.join(name).to_string_lossy().into_owned()); - let conf = RioConf { - name: "rio-under-test".to_string(), - cpi_sock_path: path("cpi.sock"), - cli_sock_path: path("cli.sock"), - frrmi_sock_path: path("frr-agent.sock"), - }; - Rio::new(&conf).expect("rio should build on fresh socket paths") + /// + /// The returned `SockDir` must outlive the `Rio`: dropping it removes the + /// paths the sockets are bound to. + fn rio_for_test() -> (Rio, SockDir) { + let dir = SockDir::new(); + let rio = Rio::new(&dir.conf()).expect("rio should build on fresh socket paths"); + (rio, dir) } /// A routing database, plus the handles that must outlive it. @@ -711,7 +753,7 @@ mod tests { #[tokio::test(start_paused = true)] #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] async fn arming_the_stale_timeout_sweeps_nothing() { - let mut rio = rio_for_test(); + let (mut rio, _dir) = rio_for_test(); let mut t = db_for_test(); assert!(rio.stale_timeout.is_none(), "starts unarmed"); @@ -735,7 +777,7 @@ mod tests { #[tokio::test(start_paused = true)] #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] async fn the_stale_timeout_survives_its_own_deadline() { - let mut rio = rio_for_test(); + let (mut rio, _dir) = rio_for_test(); let mut t = db_for_test(); rio.set_stale_timeout(); @@ -764,7 +806,7 @@ mod tests { #[tokio::test(start_paused = true)] #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] async fn the_stale_timeout_fires_once_and_then_disarms() { - let mut rio = rio_for_test(); + let (mut rio, _dir) = rio_for_test(); let mut t = db_for_test(); rio.set_stale_timeout(); @@ -785,7 +827,7 @@ mod tests { #[tokio::test(start_paused = true)] #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] async fn an_unarmed_stale_timeout_never_fires() { - let mut rio = rio_for_test(); + let (mut rio, _dir) = rio_for_test(); let mut t = db_for_test(); for _ in 0..4 { @@ -804,7 +846,7 @@ mod tests { #[tokio::test(start_paused = true)] #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] async fn a_deleted_vrf_outlives_the_window_and_no_longer() { - let mut rio = rio_for_test(); + let (mut rio, _dir) = rio_for_test(); let mut t = db_for_test(); let cfg = RouterVrfConfig::new(909, "doomed"); @@ -845,7 +887,7 @@ mod tests { #[tokio::test(start_paused = true)] #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] async fn an_frr_restart_opens_the_stale_window() { - let mut rio = rio_for_test(); + let (mut rio, _dir) = rio_for_test(); let mut t = db_for_test(); let cfg = RouterVrfConfig::new(910, "half-deleted"); @@ -883,7 +925,7 @@ mod tests { #[tokio::test(start_paused = true)] #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] async fn a_refresh_with_no_peer_to_ask_is_not_marked_done() { - let mut rio = rio_for_test(); + let (mut rio, _dir) = rio_for_test(); let mut t = db_for_test(); assert!(rio.cpistats.peer.is_none(), "no peer has connected"); @@ -913,7 +955,7 @@ mod tests { CpiStatus::Connected, CpiStatus::Incompatible, ] { - let mut rio = rio_for_test(); + let (mut rio, _dir) = rio_for_test(); let mut t = db_for_test(); let cfg = RouterVrfConfig::new(911, "bystander"); t.db.vrftable.add_vrf(&cfg).expect("vrf should be created"); @@ -931,4 +973,380 @@ mod tests { assert_eq!(t.db.vrftable.len(), 2, "nor touch the vrf table"); } } + + // ----------------------------------------------------------------------- + // The CPI socket, end to end. + // + // These drive the real IO loop through a real unix datagram socket: the + // test binds the other end and speaks dplane-rpc at it, exactly as FRR's + // plugin does. Nothing is stubbed, so the poller, the readiness handling + // and the reply path are under test rather than around it. + // ----------------------------------------------------------------------- + + /// The peer side of the CPI socket -- what FRR's dplane plugin would be. + struct CpiPeer { + sock: UnixDatagram, + rio: std::os::unix::net::SocketAddr, + } + impl CpiPeer { + fn attach(dir: &SockDir) -> Self { + let sock = UnixDatagram::bind(dir.path("peer.sock")).expect("peer sock should bind"); + sock.set_read_timeout(Some(Duration::from_secs(10))) + .expect("read timeout should be settable"); + let _ = &sock; + let rio = std::os::unix::net::SocketAddr::from_pathname(dir.path("cpi.sock")) + .expect("rio's cpi path should be addressable"); + Self { sock, rio } + } + fn send(&self, msg: &RpcMsg) { + dplane_rpc::socks::send_msg(&self.sock, msg, &self.rio).expect("send to rio"); + } + fn send_raw(&self, bytes: &[u8]) { + self.sock + .send_to_addr(bytes, &self.rio) + .expect("raw send to rio"); + } + /// Assert that nothing comes back within `patience`. + fn expect_silence(&self, patience: Duration) { + self.sock + .set_read_timeout(Some(patience)) + .expect("read timeout should be settable"); + let mut buf = [0u8; 4096]; + let outcome = self.sock.recv_from(&mut buf); + self.sock + .set_read_timeout(Some(Duration::from_secs(10))) + .expect("read timeout should be settable"); + assert!( + outcome.is_err(), + "expected no answer at all, got {} bytes", + outcome.map_or(0, |(len, _)| len) + ); + } + /// Wait for one message back, failing rather than hanging. + fn recv(&self) -> RpcMsg { + let mut buf = [0u8; 4096]; + let (len, _) = self + .sock + .recv_from(&mut buf) + .expect("rio should answer within the read timeout"); + let mut data = bytes::Bytes::copy_from_slice(&buf[..len]); + RpcMsg::decode(&mut data).expect("rio should answer with a well-formed message") + } + fn connect_request(seqn: u64, pid: u32) -> RpcMsg { + RpcMsg::Request(RpcRequest::new(RpcOp::Connect, seqn).set_object( + RpcObject::ConnectInfo(ConnectInfo { + pid, + name: "test-plugin".to_string(), + verinfo: VerInfo::default(), + synt: 0, + }), + )) + } + fn route_request(op: RpcOp, seqn: u64) -> RpcMsg { + RpcMsg::Request( + RpcRequest::new(op, seqn).set_object(RpcObject::IpRoute(IpRoute { + prefix: "10.0.0.0".parse().expect("literal"), + prefix_len: 24, + vrfid: 0, + tableid: 254, + rtype: RouteType::Bgp, + distance: 20, + metric: 100, + nhops: vec![], + })), + ) + } + } + + /// A running IO loop, stopped when the test ends. + struct RunningRio { + handle: RioHandle, + dir: SockDir, + /// Held, not dropped: see `TestDb::held`. + #[allow(dead_code)] + held: ( + crate::interfaces::iftablerw::IfTableReader, + crate::fib::fibtable::FibTableReader, + crate::atable::atablerw::AtableWriter, + ), + } + impl RunningRio { + fn start() -> Self { + let dir = SockDir::new(); + let (iftw, iftr) = IfTableWriter::new(); + let (fibtw, fibtr) = FibTableWriter::new(); + let (atablew, atabler) = AtableWriter::new(); + let handle = start_rio( + &test_router_subsystem(), + &dir.conf(), + fibtw, + iftw, + atabler, + None, + ) + .expect("rio should start on fresh socket paths"); + Self { + handle, + dir, + held: (iftr, fibtr, atablew), + } + } + } + impl RunningRio { + /// Start attending the CPI, as applying a configuration does. + /// + /// Until this happens the CPI socket is registered `Interest::PRIORITY` + /// -- out-of-band data only, which a unix datagram socket never carries + /// -- so the loop is deaf to it by construction rather than by a flag it + /// has to remember to check. + fn attend_cpi(&self) { + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("a runtime to drive the ctl channel") + .block_on(self.handle.get_ctl_tx().unlock()) + .expect("the cpi should unlock"); + } + } + impl Drop for RunningRio { + fn drop(&mut self) { + let _ = self.handle.finish(); + } + } + + /// The CPI is deaf until a configuration says otherwise. + /// + /// A dataplane that restarts keeps receiving updates over the CPI until + /// FRR notices and re-syncs. Acting on them would build a routing table out + /// of a fragment; ignoring them politely would tell the plugin they were + /// delivered. So the loop does not answer at all, and the plugin caches and + /// retries. + /// + /// The mechanism is worth knowing about before touching this registration: + /// the socket is registered `Interest::PRIORITY`, which mio maps to + /// `EPOLLPRI` alone. A unix datagram socket never carries out-of-band data, + /// so no readable event is ever raised -- deafness by construction rather + /// than by a flag somewhere in the dispatch. Registering it `READABLE` + /// "to fix a bug" would silently undo the feature, and every assertion + /// below would still pass. + #[test] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + fn the_cpi_is_not_attended_until_it_is_unlocked() { + let rio = RunningRio::start(); + let peer = CpiPeer::attach(&rio.dir); + + peer.send(&CpiPeer::connect_request(1, 4242)); + peer.expect_silence(Duration::from_secs(2)); + + // Deafness defers rather than drops: the datagram sat in the socket's + // receive queue the whole time, and unlocking serves it. Worth knowing, + // because it means unlocking replays whatever arrived while we were not + // listening -- bounded by SO_RCVBUF, not by anything this code decides. + // The `last_pid` guard in `handle_request` is what keeps that safe: + // anything but a connect is refused until a connect has been seen. + rio.attend_cpi(); + let RpcMsg::Response(deferred) = peer.recv() else { + panic!("an attended cpi should answer what it deferred"); + }; + assert_eq!( + deferred.seqn, 1, + "the request sent while deaf is served, not discarded" + ); + assert_eq!(deferred.rescode, RpcResultCode::Ok); + + // And it keeps answering from then on. + peer.send(&CpiPeer::connect_request(2, 4242)); + let RpcMsg::Response(resp) = peer.recv() else { + panic!("an attended cpi should keep answering"); + }; + assert_eq!(resp.seqn, 2); + assert_eq!(resp.rescode, RpcResultCode::Ok); + } + + /// A connect is answered, which is the whole CPI round trip. + /// + /// Readiness on the socket, the read, the decode, the dispatch and the + /// addressed reply all have to work for this to return: the reply comes + /// back to the peer address the datagram arrived from, so nothing here is + /// satisfied by the loop merely running. + #[test] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + fn a_connect_over_the_cpi_socket_is_answered() { + let rio = RunningRio::start(); + let peer = CpiPeer::attach(&rio.dir); + rio.attend_cpi(); + + peer.send(&CpiPeer::connect_request(1, 4242)); + + let RpcMsg::Response(resp) = peer.recv() else { + panic!("a request should draw a response"); + }; + assert_eq!(resp.op, RpcOp::Connect); + assert_eq!(resp.seqn, 1, "the response is matched to the request"); + assert_eq!(resp.rescode, RpcResultCode::Ok); + assert!( + matches!(resp.objs.first(), Some(RpcObject::ConnectInfo(_))), + "a connect is answered with the sync token, not bare" + ); + } + + /// A request that arrives before any connect is refused, not applied. + /// + /// The plugin always connects first, so a request without one means *we* + /// restarted and lost the state. Applying it would build a routing table + /// out of whatever fragment happened to be in flight; the plugin has to + /// push the whole state again instead. + /// + /// This deliberately sends a **deletion**. An addition is refused twice + /// over -- once here and once by the no-configuration guard below it -- + /// and both refusals are spelled `Ignored`, so an addition cannot tell the + /// two apart. Removing this guard entirely leaves an `Add` test passing. + /// A deletion is allowed through the no-configuration guard, so it reaches + /// this one and nothing else. + #[test] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + fn a_request_before_any_connect_is_ignored() { + let rio = RunningRio::start(); + let peer = CpiPeer::attach(&rio.dir); + rio.attend_cpi(); + + peer.send(&CpiPeer::route_request(RpcOp::Del, 7)); + + let RpcMsg::Response(resp) = peer.recv() else { + panic!("a request should draw a response"); + }; + assert_eq!(resp.seqn, 7); + assert_eq!( + resp.rescode, + RpcResultCode::Ignored, + "a route withdrawal offered before a connect must not be acted on" + ); + } + + /// With no configuration, additions are refused but deletions are not. + /// + /// A deletion can only ever remove state we should not be holding, so it + /// is safe without a config; an addition would install a route into a + /// table nobody has described yet. + #[test] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + fn without_a_config_additions_are_refused_and_deletions_are_not() { + let rio = RunningRio::start(); + let peer = CpiPeer::attach(&rio.dir); + rio.attend_cpi(); + + peer.send(&CpiPeer::connect_request(1, 4242)); + let RpcMsg::Response(connected) = peer.recv() else { + panic!("connect should be answered"); + }; + assert_eq!(connected.rescode, RpcResultCode::Ok); + + peer.send(&CpiPeer::route_request(RpcOp::Add, 2)); + let RpcMsg::Response(added) = peer.recv() else { + panic!("add should be answered"); + }; + assert_eq!( + added.rescode, + RpcResultCode::Ignored, + "an addition with no config is refused" + ); + + peer.send(&CpiPeer::route_request(RpcOp::Del, 3)); + let RpcMsg::Response(deleted) = peer.recv() else { + panic!("del should be answered"); + }; + assert_ne!( + deleted.rescode, + RpcResultCode::Ignored, + "a deletion with no config is allowed through, to wipe stale state" + ); + } + + /// A datagram that is not a message at all draws a notification. + /// + /// The loop must not treat a decode failure as a reason to stop reading + /// the socket: the peer is told, the failure is counted, and the next + /// datagram is still served. + #[test] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + fn a_malformed_datagram_draws_a_notification_and_does_not_wedge_the_loop() { + let rio = RunningRio::start(); + let peer = CpiPeer::attach(&rio.dir); + rio.attend_cpi(); + + peer.send_raw(&[0xff; 32]); + assert!( + matches!(peer.recv(), RpcMsg::Notification(_)), + "garbage should be answered with a notification" + ); + + // The socket is still being served afterwards. + peer.send(&CpiPeer::connect_request(9, 4242)); + let RpcMsg::Response(resp) = peer.recv() else { + panic!("the loop should still answer after a decode failure"); + }; + assert_eq!(resp.seqn, 9); + assert_eq!(resp.rescode, RpcResultCode::Ok); + } + + /// A CLI client: connects to rio's cli socket and asks it something. + fn ask_the_cli(dir: &SockDir, tag: &str) -> CliResponse { + let sock = UnixDatagram::bind(dir.path(&format!("cli-client-{tag}.sock"))) + .expect("cli client sock should bind"); + sock.connect(dir.path("cli.sock")) + .expect("rio's cli socket should be connectable"); + sock.set_read_timeout(Some(Duration::from_secs(10))) + .expect("read timeout should be settable"); + CliRequest::new(CliAction::ShowCpiStats, RequestArgs::default()) + .send(&sock) + .expect("cli request should send"); + CliResponse::recv_sync(&sock).expect("rio should answer the cli within the timeout") + } + + /// The CLI keeps working when its socket path is removed underneath it. + /// + /// Unlinking the path only removes the directory entry -- the socket stays + /// open and keeps serving anyone already holding it, while every new client + /// finds nothing there. That is why the watcher is on the parent directory + /// rather than the file: no `DELETE_SELF` is ever emitted for an open + /// socket. + /// + /// The assertion is that a *new client is served*, not that a path exists. + /// Rebinding without re-registering the new socket with the poller would + /// put a file back and answer nobody, and a test that only stats the path + /// cannot tell those apart. + #[test] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + fn the_cli_survives_having_its_socket_path_removed() { + let rio = RunningRio::start(); + let cli = rio.dir.path("cli.sock"); + + let before = ask_the_cli(&rio.dir, "before"); + assert_eq!( + before.request.action, + CliAction::ShowCpiStats, + "the cli answers before the path is disturbed" + ); + + std::fs::remove_file(&cli).expect("the path should be removable"); + + // The watcher is edge-triggered through the poller, so this is prompt + // rather than a poll interval; the deadline is generous only so that a + // loaded machine does not fail the run. + let deadline = clock::now() + Duration::from_secs(10); + while clock::now() < deadline && !Path::new(&cli).exists() { + thread::sleep(Duration::from_millis(20)); + } + assert!( + Path::new(&cli).exists(), + "the loop should notice the unlink and rebind" + ); + + let after = ask_the_cli(&rio.dir, "after"); + assert_eq!( + after.request.action, + CliAction::ShowCpiStats, + "and the rebound socket must actually be served, not merely exist" + ); + } } From baaf9af2d6cbb8b97228ddd8fce60bf07bcca9af Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 16:04:23 -0600 Subject: [PATCH 11/37] test(routing): Stand in for frr-agent and test the frrmi lifecycle The frrmi's wire format is already covered in `frr::frrmi` against a `UnixStream::pair()`. What was not covered is the loop's lifecycle around it: connect, disconnect, restart. Those were three of the five functions `rio.rs` never entered. ## This is not a fake FRR The frrmi does not talk to FRR. It talks to `frr-agent`, a Hedgehog component that sits beside FRR and applies configuration to it, over a Hedgehog wire format. `FakeAgent` is a `UnixListener` at the frrmi path impersonating our own agent and nothing else -- no FRR, no bgpd, no zebra, nothing GPL, and nothing that needs a container to run. Running a real FRR stack to close a unit-test gap remains off the table; that is what the vlab jobs are for. frrmi is `SOCK_STREAM`, so an accepted connection has an implicit peer and none of the addressing trouble that ruled out `socketpair` for the CPI applies. ## Properties * `the_loop_connects_to_the_agent_whenever_it_appears` -- rio starts with nothing listening, so its first connect fails. That is the normal case rather than an edge one: the dataplane and the FRR container come up in whatever order they come up in, and a loop that gave up after one refusal would need a restart to recover. * `the_loop_reconnects_when_the_agent_goes_away` -- `frr-agent` restarts whenever FRR does, which is the moment the dataplane most needs to push configuration back. A loop that held the dead socket would go on believing it had a link and quietly stop applying anything. * `nonsense_from_the_agent_restarts_the_link` -- the first four octets are an announced length, so a burst of `0xff` announces an absurd message. `frr::frrmi` refuses it; this asserts what the loop does with the refusal. The connection is deliberately held open, so the restart can only have come from the refusal and not from an end-of-file. ## Break testing, and one break that did not fire Commenting out the per-pass `frrmi_connect()` fails all three. Turning the `Err(e) => frrmi_restart()` arm into a bare log fails two. Removing the `frrmi_connect()` from inside `frrmi_restart` fails nothing. The loop attempts a connect at the top of every pass regardless, so that call only saves the one iteration between the failure and the next pass -- latency, not correctness. Distinguishing it would need an assertion on how *fast* the reconnect happens, and an absolute timing bound measures how busy the machine is rather than what the code does; that is the same mistake the vacuity floors made earlier on this branch. Recorded here instead. ## Coverage router/rio.rs lines 90.61% -> 92.62% branches 55.21% -> 56.12% frr/frrmi.rs lines 76.88% -> 81.18% branches 52.94% -> 56.86% Uncovered functions in `rio.rs` drop from 5 to 3. The missed *branch* count in `rio.rs` is 43 either way -- the percentage moves only because the test code adds branches of its own -- so the honest gain here is in `frrmi.rs`, which loses 16 lines and 2 branches, and in the three functions that now run at all. What is left in `rio.rs` is `cli_wake_on_writeable`, which needs a response big enough to fill the socket buffer, `reregister`'s error closure, and `new`'s. ## Still owed: the ValidatedGwConfig harness Everything here and in the previous commit deliberately avoids needing a configuration, because `RouterCtlMsg::Config` wants a `ValidatedGwConfig` and building one in a test is a substantial piece of work in its own right. That is a real gap rather than a stylistic choice. Without it the tests cannot reach the paths that matter most: a route actually installed into a fib, `reapply_frr_config` after an FRR restart, the config round trip through frrmi and out to `ShowFrrmiLastConfig`. It wants doing, and it is large enough that it should be its own chapter rather than an afterthought appended to this one. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit dd53180c1d08e1af7ed5e431bb8ca6955369f3a0) --- routing/src/router/rio.rs | 103 +++++++++++++++++++++++++++++++++++++- 1 file changed, 102 insertions(+), 1 deletion(-) diff --git a/routing/src/router/rio.rs b/routing/src/router/rio.rs index 9fe0558587..4458988d49 100644 --- a/routing/src/router/rio.rs +++ b/routing/src/router/rio.rs @@ -584,7 +584,8 @@ mod tests { }; use dplane_rpc::wire::Wire; use lifecycle::{CancellationToken, Subsystem}; - use std::os::unix::net::UnixDatagram; + use std::io::Write; + use std::os::unix::net::{UnixDatagram, UnixListener, UnixStream}; use std::path::Path; use std::time::Duration; @@ -1349,4 +1350,104 @@ mod tests { "and the rebound socket must actually be served, not merely exist" ); } + + // ----------------------------------------------------------------------- + // The frrmi lifecycle. + // + // The frrmi does not talk to FRR. It talks to `frr-agent`, a Hedgehog + // component that sits beside FRR and applies configuration to it, over a + // Hedgehog wire format. So the stand-in below impersonates our own agent + // and nothing else: no FRR, no bgpd, no zebra, and nothing that would need + // one to run. + // + // The wire format itself is already covered in `frr::frrmi` against a + // `UnixStream::pair()`. What is not covered, and what these reach, is the + // loop's lifecycle around it -- connect, disconnect, restart. + // ----------------------------------------------------------------------- + + /// A stand-in for `frr-agent`: something listening at the frrmi path. + struct FakeAgent { + listener: UnixListener, + } + impl FakeAgent { + fn listening_at(dir: &SockDir) -> Self { + let listener = + UnixListener::bind(dir.path("frr-agent.sock")).expect("the agent should bind"); + listener + .set_nonblocking(true) + .expect("the agent should be pollable"); + Self { listener } + } + /// Wait for the loop to connect, failing rather than hanging. + fn accept(&self, expectation: &str) -> UnixStream { + let deadline = clock::now() + Duration::from_secs(10); + loop { + match self.listener.accept() { + Ok((stream, _)) => return stream, + Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => { + assert!(clock::now() < deadline, "rio never {expectation}"); + thread::sleep(Duration::from_millis(20)); + } + Err(e) => panic!("the agent could not accept: {e}"), + } + } + } + } + + /// The loop keeps trying until the agent turns up. + /// + /// Rio starts here with nothing listening, so its first connect fails. + /// That is the normal case rather than an edge one -- the dataplane and + /// the FRR container come up in whatever order they come up in -- and a + /// loop that gave up after the first refusal would need a restart to + /// recover. + #[test] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + fn the_loop_connects_to_the_agent_whenever_it_appears() { + let rio = RunningRio::start(); + let agent = FakeAgent::listening_at(&rio.dir); + let _conn = agent.accept("connected to an agent that appeared after it started"); + } + + /// The loop reconnects when the agent goes away. + /// + /// `frr-agent` restarts whenever FRR does, which is the very moment the + /// dataplane most needs to push configuration back. A loop that held the + /// dead socket would go on believing it had a link and quietly stop + /// applying anything. + #[test] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + fn the_loop_reconnects_when_the_agent_goes_away() { + let rio = RunningRio::start(); + let agent = FakeAgent::listening_at(&rio.dir); + + let first = agent.accept("connected to the agent"); + drop(first); // the agent restarts + + let _second = agent.accept("reconnected after the agent left"); + } + + /// A response the agent could not have meant restarts the link. + /// + /// The first four octets are an announced length, so a burst of `0xff` + /// announces a message of absurd size. `frr::frrmi` refuses it; this + /// asserts what the loop does *with* that refusal, which is to rebuild the + /// link rather than to keep reading a stream it has lost its place in. + /// + /// The connection is deliberately held open, so the restart can only have + /// come from the refusal and not from an end-of-file. + #[test] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + fn nonsense_from_the_agent_restarts_the_link() { + let rio = RunningRio::start(); + let agent = FakeAgent::listening_at(&rio.dir); + + let mut first = agent.accept("connected to the agent"); + first + .write_all(&[0xff; 64]) + .expect("the agent should be able to write nonsense"); + + let _second = agent.accept("rebuilt the link after a message it could not read"); + drop(first); + } } From 7edaa32316cb4d85ac9177e4ddd0b0a7126af45a Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 16:20:38 -0600 Subject: [PATCH 12/37] docs(routing): Record why cli_wake_on_writeable stays uncovered It is the last uncovered function in `rio.rs` and it is not reachable yet. The next person to look at the coverage report deserves to know that before spending an afternoon on it, so the finding goes next to the tests rather than into a review comment. `cli_wake_on_writeable` runs only when a response cannot be sent in one go: the send fails with `WouldBlock`, the remaining chunks are cached, and the socket is re-armed for writability so the cache drains when the client catches up. Reaching it needs a response larger than the client's receive queue. Both halves of that were measured rather than guessed: * The queue floor is the kernel's `SOCK_MIN_RCVBUF`. Asking for 1024 gets 2304. The chunk size is 2048. * With no configuration applied, the largest response the dataplane can produce is `ShowTracingTargets` at 1694 octets. Every route, vrf, fib, adjacency and nat listing is a header over an empty table -- 20 to 454 octets each. `Help` is not even implemented on this path. One chunk, comfortably inside one queue, with no way to close the gap. Filling the queue with many small answers instead does not work: a client blocked in a read consumes each answer as fast as the loop produces it, so the depth stays at one. I wrote that test first, and it passed while `cli_wake_on_writeable` executed zero times -- deleting the re-arm entirely did not fail it. Making it fill would take a sleep between asking and reading, which buys reachability with a timing assumption, and a sleep that is too short fails by making the test vacuous rather than by failing. That is the same trade rejected for the frrmi reconnect latency in the previous commit. So this is a second, concrete claim on the `ValidatedGwConfig` harness: with a real configuration the route and fib listings are large enough that the question answers itself, and no timing assumption is needed at all. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 0a8c199c06c8af2b090fcc849207f8215f6e36d8) --- routing/src/router/rio.rs | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/routing/src/router/rio.rs b/routing/src/router/rio.rs index 4458988d49..480e493bde 100644 --- a/routing/src/router/rio.rs +++ b/routing/src/router/rio.rs @@ -1290,6 +1290,31 @@ mod tests { assert_eq!(resp.rescode, RpcResultCode::Ok); } + // `cli_wake_on_writeable` is the one function in this file that stays + // uncovered, and it is not for want of trying. It runs only when a + // response cannot be sent in one go: the send fails with `WouldBlock`, the + // remaining chunks are cached, and the socket is re-armed for writability + // so the cache drains when the client catches up. + // + // Reaching it needs a response larger than the client's receive queue. The + // queue floor is the kernel's `SOCK_MIN_RCVBUF`, measured at 2304 octets + // here, and the chunk size is 2048. But with no configuration applied, the + // largest response the dataplane can produce is `ShowTracingTargets` at + // 1694 octets -- every route, vrf, fib and nat table is empty, so their + // listings are headers and nothing else. One chunk, comfortably inside one + // queue. + // + // Filling the queue with many small answers instead does not work either: + // a client that is blocked reading consumes each answer as fast as the + // loop produces it, so the depth stays at one. Making it fill would take a + // sleep between asking and reading, which buys reachability with a timing + // assumption -- and a sleep that is too short fails silently, by making the + // test vacuous rather than by failing. + // + // So this waits on the `ValidatedGwConfig` harness. With a real + // configuration the route and fib listings are large enough that the + // question answers itself. + /// A CLI client: connects to rio's cli socket and asks it something. fn ask_the_cli(dir: &SockDir, tag: &str) -> CliResponse { let sock = UnixDatagram::bind(dir.path(&format!("cli-client-{tag}.sock"))) From 17e44c0d99db39b9696b3950f593fa088807d688 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 17:17:21 -0600 Subject: [PATCH 13/37] test(routing): Follow a route from the socket to the fib `RouterCtlMsg::Configure` carries a `RouterConfig`, not a `ValidatedGwConfig`, and a `RouterConfig` is three lines: a non-zero genid and one vrf. `validate` objects only to duplicate vnis and to a vtep that is not set up, and `handle_configure` calls `db.set_config`, so `have_config` becomes true and the CPI starts accepting additions. That was the whole blocker. The previous two commits deliberately routed around it, which biased everything written so far toward guards and refusals -- `Add` was only ever reachable as an `Ignored`. ## Routes * `a_route_the_control_plane_announces_reaches_the_fib` -- the point of the CPI: a datagram on a socket becomes a forwarding entry. Decode, dispatch, rib insertion, reconciliation and the left-right publish are all in the path of one assertion. * `a_route_the_control_plane_withdraws_leaves_the_fib` -- a withdrawal that did not take forwards at a next hop the control plane has stopped believing in, which no reconvergence elsewhere can clear. * `announcing_a_prefix_twice_leaves_one_route` -- FRR re-sends its whole table after a restart, so every prefix arrives again for one already held. A fib that grew on each pass would double every time FRR bounced. The first two fail when `route.add` or `route.del` is stubbed out. The third is a structural guard rather than a behavioural one: routes are held in a prefix-keyed trie, so duplication is not representable and no break is available for it. It is worth keeping against a change to a multimap for ECMP. `fib_v4` filters `0.0.0.0/0`, which a fresh vrf carries so that traffic with nowhere to go is dropped rather than leaked. Counting it made every assertion off by one and would have hidden a withdrawal that removed the wrong route -- that is how the first run failed. ## The FRR configuration round trip `a_configuration_reaches_the_agent_and_its_answer_comes_back` drives the other half of the dataplane's job: hand FRR the configuration it should be routing under, and remember which generation was applied. `FakeAgent` now reads the `|length|genid|body|` frame off the wire and answers in kind. Fails if the config is never handed to the frrmi, and fails if the acknowledgement is never recorded -- which matters because `reapply_frr_config` consults exactly that after a restart. ## Reassembly, and why cli_wake_on_writeable still is not covered `a_large_answer_arrives_whole` announces 8192 routes and reads back an 850KiB fib listing across some four hundred chunks. It fails if the "more" octet is computed from the wrong end of the loop. It does not reach `cli_wake_on_writeable`, and the reason recorded in `ac7ce3f93` was wrong. A configuration was necessary but nowhere near sufficient: 850KiB to a client that is provably not reading -- held off by a causal barrier on the CPI socket rather than a sleep -- still never blocks the send. What bounds it is the *sender's* buffer. `open_cli_sock` sets the loop's `SndBuf` to `CLI_RX_BUFF_SIZE`, 2048 * 8192, or 16MiB, and on a unix datagram socket that is what limits outstanding unread traffic. `RcvBuf` on the client does not help: pinned to the kernel floor of 2304 it still accepts about 100KiB, because that ceiling is the sender's too. Reaching the cache path needs roughly 16MiB of unread answer, on the order of 150,000 routes. That is not a proportionate test for one function, so it stays uncovered deliberately, with the measurements recorded next to it. ## Coverage router/cpi.rs lines 81.79% -> 82.70% branches 74.24% -> 78.79% router/rio.rs lines 92.62% -> 92.55% branches 56.12% -> 55.26% The `rio.rs` percentages drift down because the new tests add branches of their own; the missed count is what moved, and it did not -- the gain here is in `cpi.rs`, which loses 5 lines and 3 branches, and in reaching the route path at all. `rio.rs` branch coverage over the day is 28.95% -> 55.26%. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit fa7b8b9206502a7459564f196fbfe83e7e175769) --- routing/src/router/rio.rs | 406 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 385 insertions(+), 21 deletions(-) diff --git a/routing/src/router/rio.rs b/routing/src/router/rio.rs index 480e493bde..3c86d848d6 100644 --- a/routing/src/router/rio.rs +++ b/routing/src/router/rio.rs @@ -568,8 +568,10 @@ pub(crate) fn start_rio( #[cfg(test)] mod tests { use crate::atable::atablerw::AtableWriter; + use crate::config::{FrrConfig, RouterConfig}; use crate::errors::RouterError; use crate::fib::fibtable::FibTableWriter; + use crate::fib::fibtype::FibKey; use crate::interfaces::iftablerw::IfTableWriter; use crate::rib::vrf::{RouterVrfConfig, VrfStatus}; use crate::router::cpi::CpiStatus; @@ -578,13 +580,14 @@ mod tests { use cli::cliproto::{CliAction, CliRequest, CliResponse, RequestArgs}; use concurrency::sync::atomic::{AtomicUsize, Ordering}; use concurrency::thread; + use config::GenId; use dplane_rpc::msg::{ ConnectInfo, IpRoute, RouteType, RpcMsg, RpcObject, RpcOp, RpcRequest, RpcResultCode, VerInfo, }; use dplane_rpc::wire::Wire; use lifecycle::{CancellationToken, Subsystem}; - use std::io::Write; + use std::io::{Read, Write}; use std::os::unix::net::{UnixDatagram, UnixListener, UnixStream}; use std::path::Path; use std::time::Duration; @@ -1291,29 +1294,28 @@ mod tests { } // `cli_wake_on_writeable` is the one function in this file that stays - // uncovered, and it is not for want of trying. It runs only when a - // response cannot be sent in one go: the send fails with `WouldBlock`, the - // remaining chunks are cached, and the socket is re-armed for writability - // so the cache drains when the client catches up. + // uncovered, and the reason is not the one recorded when this note was + // first written. // - // Reaching it needs a response larger than the client's receive queue. The - // queue floor is the kernel's `SOCK_MIN_RCVBUF`, measured at 2304 octets - // here, and the chunk size is 2048. But with no configuration applied, the - // largest response the dataplane can produce is `ShowTracingTargets` at - // 1694 octets -- every route, vrf, fib and nat table is empty, so their - // listings are headers and nothing else. One chunk, comfortably inside one - // queue. + // It runs only when a response cannot be sent in one go: the send fails + // with `WouldBlock`, the remaining chunks are cached, and the socket is + // re-armed for writability so the cache drains when the client catches up. + // The first guess was that a configuration was needed to make answers big + // enough. It was necessary but nowhere near sufficient -- with 8192 routes + // a fib listing is 850KiB in some four hundred chunks, sent to a client + // that is provably not reading, and the send still never blocks. // - // Filling the queue with many small answers instead does not work either: - // a client that is blocked reading consumes each answer as fast as the - // loop produces it, so the depth stays at one. Making it fill would take a - // sleep between asking and reading, which buys reachability with a timing - // assumption -- and a sleep that is too short fails silently, by making the - // test vacuous rather than by failing. + // What actually bounds it: on a unix datagram socket the *sender's* + // `SO_SNDBUF` is what limits outstanding unread traffic, and + // `open_cli_sock` sets the loop's to `CLI_RX_BUFF_SIZE` -- 2048 * 8192, or + // 16MiB. So reaching the cache path needs roughly 16MiB of answer the + // client has not read, which is on the order of 150,000 routes. `RcvBuf` on + // the client does not help: set to the kernel floor of 2304 it still + // accepts about 100KiB, because that limit is the sender's too. // - // So this waits on the `ValidatedGwConfig` harness. With a real - // configuration the route and fib listings are large enough that the - // question answers itself. + // A test that announces 150,000 routes to exercise one function is not a + // trade worth making, so this stays uncovered deliberately. The numbers are + // here so the next reader does not have to measure them again. /// A CLI client: connects to rio's cli socket and asks it something. fn ask_the_cli(dir: &SockDir, tag: &str) -> CliResponse { @@ -1475,4 +1477,366 @@ mod tests { let _second = agent.accept("rebuilt the link after a message it could not read"); drop(first); } + + // ----------------------------------------------------------------------- + // Routes, end to end. + // + // With a configuration applied the CPI accepts additions, so a route can be + // followed from a datagram on the socket all the way to the published fib + // -- across the rib, the reconciliation and the left-right publish -- which + // is the path the dataplane exists to serve. + // ----------------------------------------------------------------------- + + impl RunningRio { + /// Apply a minimal router configuration. + /// + /// Until this happens the CPI refuses additions: `have_config` is false + /// and there is no table for a route to go into. One vrf and a non-zero + /// genid is the whole requirement -- `RouterConfig::validate` only + /// objects to duplicate vnis and to a vtep that is not set up. + fn configure(&self, genid: GenId, frr: Option) { + let mut cfg = RouterConfig::new(genid); + cfg.add_vrf(RouterVrfConfig::new(0, "default")); + if let Some(frr) = frr { + cfg.set_frr_config(frr); + } + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("a runtime to drive the ctl channel") + .block_on(self.handle.get_ctl_tx().configure(cfg)) + .expect("the router should accept a minimal config"); + } + + /// The v4 prefixes currently published in the default vrf's fib, less + /// the one the vrf was born with. + /// + /// A fresh vrf carries a `0.0.0.0/0` drop route so that traffic with + /// nowhere to go is discarded rather than leaked. It is not something + /// the control plane announced, so counting it here would make every + /// assertion below off by one and would hide a withdrawal that removed + /// the wrong route. + fn fib_v4(&self) -> Vec { + let Some(table) = self.held.1.enter() else { + return vec![]; + }; + let Some(fibr) = table.get_fib(FibKey::from_vrfid(0)) else { + return vec![]; + }; + let Some(fib) = fibr.enter() else { + return vec![]; + }; + let mut out: Vec = fib + .iter_v4() + .map(|(p, _)| p.to_string()) + .filter(|p| p != "0.0.0.0/0") + .collect(); + out.sort(); + out + } + + /// Wait for the published fib to hold exactly `want` v4 prefixes. + /// + /// The fib is published by the loop thread through a left-right, so a + /// reader does not see a write the instant the socket carried it. The + /// deadline is generous because it is only there so a stuck loop fails + /// the test instead of hanging it. + fn await_fib_v4(&self, want: usize) -> Vec { + let deadline = clock::now() + Duration::from_secs(10); + loop { + let got = self.fib_v4(); + if got.len() == want { + return got; + } + assert!( + clock::now() < deadline, + "the fib settled on {got:?}, wanted {want} v4 route(s)" + ); + thread::sleep(Duration::from_millis(20)); + } + } + } + + impl CpiPeer { + /// A request carrying one route, for a prefix of the caller's choosing. + fn route_for(op: RpcOp, seqn: u64, prefix: &str, metric: u32) -> RpcMsg { + RpcMsg::Request( + RpcRequest::new(op, seqn).set_object(RpcObject::IpRoute(IpRoute { + prefix: prefix.parse().expect("a literal prefix"), + prefix_len: 24, + vrfid: 0, + tableid: 254, + rtype: RouteType::Bgp, + distance: 20, + metric, + nhops: vec![], + })), + ) + } + /// Send a request and insist it was accepted. + fn send_accepted(&self, msg: &RpcMsg, what: &str) { + self.send(msg); + let RpcMsg::Response(resp) = self.recv() else { + panic!("{what} drew no response"); + }; + assert_eq!(resp.rescode, RpcResultCode::Ok, "{what} was refused"); + } + /// Announce many routes, in batches, without waiting on each answer. + /// + /// One round trip per route would be slower than it needs to be, and + /// the batches keep the loop's own receive queue from filling while it + /// works through them. + fn announce_routes(&self, count: usize) { + const BATCH: usize = 64; + let mut seqn = 100u64; + for chunk in (0..count).collect::>().chunks(BATCH) { + for i in chunk { + self.send(&Self::route_for( + RpcOp::Add, + seqn, + &format!("10.{}.{}.0", i / 256, i % 256), + 100, + )); + seqn += 1; + } + for _ in chunk { + let RpcMsg::Response(resp) = self.recv() else { + panic!("a route drew no response"); + }; + assert_eq!(resp.rescode, RpcResultCode::Ok, "a route was refused"); + } + } + } + /// Connect, as the plugin does before anything else. + fn say_hello(&self) { + self.send_accepted(&Self::connect_request(1, 4242), "the connect"); + } + } + + /// A route the control plane announces reaches the published fib. + /// + /// This is the whole point of the CPI: a datagram on a socket becomes a + /// forwarding entry. Every stage between the two -- decode, dispatch, rib + /// insertion, reconciliation, publish -- is in the path of this assertion. + #[test] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + fn a_route_the_control_plane_announces_reaches_the_fib() { + let rio = RunningRio::start(); + let peer = CpiPeer::attach(&rio.dir); + rio.attend_cpi(); + rio.configure(1, None); + peer.say_hello(); + + peer.send_accepted( + &CpiPeer::route_for(RpcOp::Add, 2, "10.0.0.0", 100), + "the route", + ); + + assert_eq!(rio.await_fib_v4(1), vec!["10.0.0.0/24".to_string()]); + } + + /// A route the control plane withdraws leaves the published fib. + /// + /// A withdrawal that did not take would forward traffic at a next hop the + /// control plane has stopped believing in -- a blackhole that no amount of + /// reconvergence elsewhere can clear. + #[test] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + fn a_route_the_control_plane_withdraws_leaves_the_fib() { + let rio = RunningRio::start(); + let peer = CpiPeer::attach(&rio.dir); + rio.attend_cpi(); + rio.configure(1, None); + peer.say_hello(); + + peer.send_accepted( + &CpiPeer::route_for(RpcOp::Add, 2, "10.0.0.0", 100), + "the route", + ); + assert_eq!(rio.await_fib_v4(1), vec!["10.0.0.0/24".to_string()]); + + peer.send_accepted( + &CpiPeer::route_for(RpcOp::Del, 3, "10.0.0.0", 100), + "the withdrawal", + ); + assert_eq!(rio.await_fib_v4(0), Vec::::new()); + } + + /// Announcing a prefix twice leaves one route, not two. + /// + /// FRR re-sends its whole table after a restart, so every prefix arrives + /// again for a prefix already held. A fib that grew on each pass would + /// double in size every time FRR bounced. + #[test] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + fn announcing_a_prefix_twice_leaves_one_route() { + let rio = RunningRio::start(); + let peer = CpiPeer::attach(&rio.dir); + rio.attend_cpi(); + rio.configure(1, None); + peer.say_hello(); + + peer.send_accepted( + &CpiPeer::route_for(RpcOp::Add, 2, "10.0.0.0", 100), + "the route", + ); + assert_eq!(rio.await_fib_v4(1), vec!["10.0.0.0/24".to_string()]); + + // The same prefix again, as an add and then as an update, with a + // different metric each time so they are not literally the same route. + peer.send_accepted( + &CpiPeer::route_for(RpcOp::Add, 3, "10.0.0.0", 200), + "the re-announcement", + ); + peer.send_accepted( + &CpiPeer::route_for(RpcOp::Update, 4, "10.0.0.0", 300), + "the update", + ); + + assert_eq!(rio.await_fib_v4(1), vec!["10.0.0.0/24".to_string()]); + } + + impl FakeAgent { + /// Read one request off the wire: `|length|genid|body|`, native-endian. + fn read_request(stream: &mut UnixStream) -> (GenId, String) { + stream + .set_read_timeout(Some(Duration::from_secs(10))) + .expect("read timeout should be settable"); + let mut header = [0u8; 16]; + stream + .read_exact(&mut header) + .expect("the agent should receive a header"); + let len = u64::from_ne_bytes(header[0..8].try_into().expect("8 octets")); + let genid = GenId::from_ne_bytes(header[8..16].try_into().expect("8 octets")); + let mut body = vec![0u8; usize::try_from(len).expect("a sane length")]; + stream + .read_exact(&mut body) + .expect("the agent should receive the announced body"); + ( + genid, + String::from_utf8(body).expect("the config should be text"), + ) + } + /// Answer a request, in the same frame the request came in. + fn answer(stream: &mut UnixStream, genid: GenId, data: &str) { + let body = data.as_bytes(); + let mut msg = Vec::with_capacity(16 + body.len()); + msg.extend_from_slice(&(body.len() as u64).to_ne_bytes()); + msg.extend_from_slice(&genid.to_ne_bytes()); + msg.extend_from_slice(body); + stream.write_all(&msg).expect("the agent should answer"); + } + } + + /// A configuration reaches the agent, and its acknowledgement comes back. + /// + /// This is the other half of the dataplane's job. The router half installs + /// routes; this half hands FRR the configuration it should be routing + /// under, and remembers which generation was actually applied. A dataplane + /// that forgot the acknowledgement would have no way to tell a + /// configuration FRR accepted from one still in flight, which is exactly + /// what `reapply_frr_config` consults after a restart. + #[test] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + fn a_configuration_reaches_the_agent_and_its_answer_comes_back() { + const GENID: GenId = 7; + const CONFIG: &str = "router bgp 65000\n neighbor 10.0.0.1 remote-as 65001\n"; + + let rio = RunningRio::start(); + let agent = FakeAgent::listening_at(&rio.dir); + let mut link = agent.accept("connected to the agent"); + + rio.configure(GENID, Some(CONFIG.to_string())); + + let (genid, body) = FakeAgent::read_request(&mut link); + assert_eq!(genid, GENID, "the request carries the generation it is for"); + assert_eq!(body, CONFIG, "and the configuration verbatim"); + + FakeAgent::answer(&mut link, GENID, "Ok"); + + // The applied generation is what an operator, and `reapply_frr_config`, + // read back. + let deadline = clock::now() + Duration::from_secs(10); + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("a runtime to drive the ctl channel"); + loop { + let applied = runtime + .block_on(rio.handle.get_ctl_tx().get_frr_applied_config()) + .expect("the router should answer"); + if let Some(applied) = applied { + assert_eq!(applied.genid, GENID); + assert_eq!(applied.cfg, CONFIG); + return; + } + assert!( + clock::now() < deadline, + "the acknowledged configuration was never recorded as applied" + ); + thread::sleep(Duration::from_millis(20)); + } + } + + /// A large answer arrives whole, across hundreds of chunks. + /// + /// The CLI protocol cuts a response into 2048-octet datagrams, each with a + /// trailing octet saying whether more follow, and the client reassembles + /// until that octet says stop. Eight thousand routes make a fib listing of + /// roughly 850KiB -- some four hundred chunks -- so a reassembly that lost + /// its place, or a "more" flag set from the wrong end of the loop, shows up + /// here as a short read rather than as a subtly truncated table. + /// + /// This does *not* reach `cli_wake_on_writeable`, and that is not for want + /// of size. `open_cli_sock` sets the loop's `SndBuf` to `CLI_RX_BUFF_SIZE`, + /// which is 16MiB, and on a unix datagram socket it is the sender's buffer + /// that bounds how much unread traffic may be outstanding. Reaching the + /// cache path therefore needs about 16MiB of answer the client has not + /// read -- on the order of a hundred and fifty thousand routes. That is a + /// disproportionate test for one function, and it is measured here so the + /// next reader does not have to rediscover it. + #[test] + #[cfg_attr(emulated, ignore = "binds Unix domain sockets")] + fn a_large_answer_arrives_whole() { + const ROUTES: usize = 8192; + + let rio = RunningRio::start(); + let peer = CpiPeer::attach(&rio.dir); + rio.attend_cpi(); + rio.configure(1, None); + peer.say_hello(); + peer.announce_routes(ROUTES); + assert_eq!(rio.await_fib_v4(ROUTES).len(), ROUTES); + + let sock = UnixDatagram::bind(rio.dir.path("cli-big.sock")).expect("client should bind"); + sock.connect(rio.dir.path("cli.sock")) + .expect("rio's cli socket should be connectable"); + sock.set_read_timeout(Some(Duration::from_secs(10))) + .expect("read timeout should be settable"); + + CliRequest::new(CliAction::ShowRouterIpv4FibEntries, RequestArgs::default()) + .send(&sock) + .expect("cli request should send"); + + let answer = CliResponse::recv_sync(&sock) + .expect("the whole answer should arrive, across as many chunks as it takes"); + let body = answer.result.expect("the listing should succeed"); + + assert!( + body.len() > 100 * 2048, + "the answer must span many chunks for this to test reassembly, got {} octets", + body.len() + ); + // The first and last routes announced: a reassembly that stopped early + // would keep the first and lose the last. + assert!(body.contains("10.0.0.0/24"), "the first route is missing"); + assert!( + body.contains(&format!( + "10.{}.{}.0/24", + (ROUTES - 1) / 256, + (ROUTES - 1) % 256 + )), + "the last route is missing, so the answer was cut short" + ); + } } From bc793c9670fbe1dedc03cba45fe882767befd633 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 17:55:33 -0600 Subject: [PATCH 14/37] test(net): Close every mutant flow_info's properties were missing A first run of cargo-mutants over `net/src/flows/flow_info.rs` -- a file covered by seven properties and one I would have called done -- caught 13 mutants and missed 28. Nine of the survivors were noise; nineteen were real. ## The one that made the case if new < current { return Err(FlowInfoError::TimeoutUnchanged); } `<` to `==` was caught. `<` to `>` was caught. `<` to `<=` survived. That is the same boundary class hand-broken in `rio.rs` earlier today, where it *was* caught, because there I suspected it and wrote the property. Suspicion is not uniform, which is the whole argument for the tool: it breaks what I would not have thought to break. Nor is it academic. `reset_expiry_unchecked` is what the masquerade expiry path calls on both halves of a flow pair -- the defect this branch already fixed once. ## The properties * `the_unchecked_refreshes_move_the_deadline_exactly` -- the checked wrappers were covered by whether they *refuse*; nothing said what the unchecked ones *do* when they accept. Extension adds exactly its duration; reset lands exactly on `now + duration`; and resetting to the deadline already held is accepted rather than refused, which is the only place `<` and `<=` differ. Reachable only on a driven clock: on a wall clock two instants are never exactly equal. * `a_flow_is_active_exactly_when_its_status_says_so` -- `is_active` is what the datapath asks before using a flow. All three of its mutants survived before this. * `a_flow_built_with_a_status_has_it` -- `new_with_status` ignoring its argument would make every test that seeds a non-default state pass for the wrong reason. * `a_genid_is_remembered_and_reaches_the_partner` -- halves that disagree about their generation get swept apart, one retired and the other left translating to an allocation nobody owns. * `each_flag_predicate_answers_for_its_own_bit` -- a predicate answering for the wrong bit translates the wrong end of the flow. * `the_destination_vpc_is_remembered` -- a reader that always said `None` is indistinguishable, from the packet path, from a flow that has not been through the lookup stage. ## Result before 13 caught, 28 missed, 16 unviable after 32 caught, 0 missed, 15 unviable No timeouts in either run. ## .cargo/mutants.toml Two exclusion categories, both in config rather than as `#[mutants::skip]` attributes, so production crates take no dependency on the tool: * Printers, per the standing "don't test printers" rule. A mutated `fmt` that nothing notices is the rule working, not a gap. * `contract::` modules. The workspace convention puts bolero generators next to the type they generate, so cargo-mutants finds them and mutates the harness at itself; a generator drawing a different distribution is not a defect in the code under test. Plus `sysfs/**`, whose write path needs root and a real sysfs. Nothing in CI depends on this. It is run by hand over a crate or a diff, and the product is the survivor list rather than the score. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 81baac0ff252e9bfbed116af8d74f2e77ceeff75) --- .cargo/mutants.toml | 35 +++++ net/src/flows/flow_info_fuzz.rs | 232 ++++++++++++++++++++++++++++++++ 2 files changed, 267 insertions(+) create mode 100644 .cargo/mutants.toml diff --git a/.cargo/mutants.toml b/.cargo/mutants.toml new file mode 100644 index 0000000000..e30180e340 --- /dev/null +++ b/.cargo/mutants.toml @@ -0,0 +1,35 @@ +# Configuration for cargo-mutants (https://mutants.rs). +# +# Mutation testing is a flashlight, not a gate: it is run by hand over a crate +# or a diff, and the product is the list of survivors rather than the score. +# Nothing in CI depends on this file. +# +# Two categories are excluded because a survivor there is noise rather than a +# finding. Both are excluded here rather than with `#[mutants::skip]` so that +# production crates take no dependency on the tool and the reasoning stays in +# one place. + +exclude_re = [ + # Printers. The standing rule is "don't test printers": asserting on + # rendered output is sisyphean, produces low signal, and breaks whenever + # anyone adjusts a column. A mutated `fmt` that no test notices is telling + # us the rule is being followed. + "() + .for_each(|(a, b): &(Millis, Millis)| { + let (extend, reset) = (a.duration(), b.duration()); + + // An extension adds to what is stored, whatever that was. + let subject = flow(); + let before = subject.expires_at(); + subject.extend_expiry_unchecked(extend); + assert_eq!( + subject.expires_at(), + before + extend, + "an unchecked extension must add exactly its duration" + ); + + // A reset replaces the deadline with `now + duration`, and refuses only when that + // would move it earlier. + let subject = flow(); + let target = clock::now() + reset; + if target < subject.expires_at() { + assert!( + matches!( + subject.reset_expiry_unchecked(reset), + Err(FlowInfoError::TimeoutUnchanged) + ), + "a reset that moves the deadline earlier must be refused" + ); + } else { + assert!(subject.reset_expiry_unchecked(reset).is_ok()); + assert_eq!( + subject.expires_at(), + target, + "an accepted reset must land on now + duration, exactly" + ); + } + + // The boundary: resetting to precisely the deadline already held is accepted. + // The clock is paused, so the second call computes the same instant as the first. + let subject = flow(); + let long = reset + Duration::from_secs(1); + assert!(subject.reset_expiry_unchecked(long).is_ok()); + let held = subject.expires_at(); + assert!( + subject.reset_expiry_unchecked(long).is_ok(), + "resetting to the deadline already held must be accepted, not refused" + ); + assert_eq!(subject.expires_at(), held, "and must leave it where it was"); + }); + }); +} + +/// A flow is active exactly when its status says so. +/// +/// `is_active` is what the datapath asks before using a flow, and the flow table asks before +/// letting a new flow displace one holding the same key. Reading `true` for a cancelled flow would +/// route packets through NAT state that has already been released. +#[test] +fn a_flow_is_active_exactly_when_its_status_says_so() { + with_paused_clock(|| async { + bolero::check!() + .with_type::() + .for_each(|status: &Status| { + let want = FlowStatus::from(*status); + let flow = flow(); + flow.update_status(want); + assert_eq!( + flow.is_active(), + want == FlowStatus::Active, + "is_active disagreed with the status it was asked about" + ); + }); + }); +} + +/// A flow built with a status has that status. +/// +/// `new_with_status` exists so a test can seed a flow in any of the four legal states. One that +/// quietly ignored the argument would make every such test start from `Detached` and pass for the +/// wrong reason -- the tests it serves would stop covering what they claim. +#[test] +fn a_flow_built_with_a_status_has_it() { + with_paused_clock(|| async { + bolero::check!() + .with_type::<(u16, Status)>() + .for_each(|(port, status): &(u16, Status)| { + let want = FlowStatus::from(*status); + let flow = FlowInfo::new_with_status( + key(*port), + clock::now() + Duration::from_secs(1), + want, + ); + assert_eq!( + flow.status(), + want, + "the status asked for was not the one built" + ); + }); + }); +} + +/// A generation id is remembered, and `set_genid_pair` reaches the partner. +/// +/// The genid is how the dataplane tells flows belonging to the current configuration from flows +/// left over by the previous one. A pair whose halves disagreed about their generation would be +/// swept apart -- one half retired, the other left translating to an allocation nobody owns, which +/// is the shape of the defect this branch already fixed once in the masquerade expiry path. +#[test] +fn a_genid_is_remembered_and_reaches_the_partner() { + with_paused_clock(|| async { + bolero::check!().with_type::<(u16, u16, i64)>().for_each( + |(a, b, genid): &(u16, u16, i64)| { + let (one, two) = (key(*a), key(b.wrapping_add(1))); + let Ok((first, second)) = FlowInfo::related_pair( + clock::now() + Duration::from_secs(1), + one, + FlowInfoFlags::INITIATOR, + two, + FlowInfoFlags::default(), + ) else { + return; // identical keys; covered elsewhere + }; + + first.set_genid(*genid); + assert_eq!( + first.genid(), + *genid, + "a genid must read back as it was set" + ); + assert_ne!( + second.genid(), + *genid, + "setting one half's genid must not reach the other" + ); + + let paired = genid.wrapping_add(1); + first.set_genid_pair(paired); + assert_eq!(first.genid(), paired); + assert_eq!( + second.genid(), + paired, + "set_genid_pair must reach the partner, or the halves disagree about which \ + configuration they belong to" + ); + }, + ); + }); +} + +/// Each flag predicate answers for its own bit and no other. +/// +/// `requires_static_nat_src` and `requires_static_nat_dst` decide whether a packet is translated at +/// all. One that answered for the wrong bit would translate the wrong end of the flow; one that +/// answered a constant would translate everything or nothing. +#[test] +fn each_flag_predicate_answers_for_its_own_bit() { + with_paused_clock(|| async { + bolero::check!() + .with_type::<(u16, u16, u8)>() + .for_each(|(a, b, bits): &(u16, u16, u8)| { + let flags = FlowInfoFlags::from_bits_truncate(*bits); + let (one, two) = (key(*a), key(b.wrapping_add(1))); + let Ok((first, _second)) = FlowInfo::related_pair( + clock::now() + Duration::from_secs(1), + one, + flags, + two, + FlowInfoFlags::default(), + ) else { + return; // identical keys; covered elsewhere + }; + + let got = first.get_flags(); + assert_eq!(got, flags, "the flags read back must be the ones set"); + assert_eq!( + got.requires_static_nat_src(), + flags.contains(FlowInfoFlags::REQ_STATIC_NAT_SRC), + "the source predicate answered for the wrong bit" + ); + assert_eq!( + got.requires_static_nat_dst(), + flags.contains(FlowInfoFlags::REQ_STATIC_NAT_DST), + "the destination predicate answered for the wrong bit" + ); + assert_eq!( + got.is_initiator(), + flags.contains(FlowInfoFlags::INITIATOR), + "the initiator predicate answered for the wrong bit" + ); + }); + }); +} + +/// The destination vpc a flow was stamped with is the one it reports. +/// +/// Masquerade will not translate a flow whose destination vpc is absent, and port forwarding keys +/// its claims on it. A reader that always answered `None` would look, from the packet path, exactly +/// like a flow that had not been through the lookup stage yet. +#[test] +fn the_destination_vpc_is_remembered() { + with_paused_clock(|| async { + bolero::check!() + .with_type::>() + .for_each(|vni: &Option| { + let want = vni + .and_then(|v| crate::vxlan::Vni::new_checked(v % 0x00FF_FFFF).ok()) + .map(crate::packet::VpcDiscriminant::from_vni); + let flow = flow(); + flow.locked.write().dst_vpcd = want; + assert_eq!( + flow.get_dst_vpcd(), + want, + "the destination vpc read back must be the one stamped" + ); + }); + }); +} From 1576fc2edf4ab846b731e026c9db81fb43649c2e Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 18:30:04 -0600 Subject: [PATCH 15/37] test(masquerade): Pin the flow state machine, exhaustively `cargo-mutants` over `nat/src/masquerade/` left 52 survivors, and 23 of them were in one file: very nearly every match guard in `next_flow_status_tcp`, plus the DNS arm of the UDP patch and the single ICMP transition. There were already four TCP tests -- `test_masquerade_tcp_establish`, `_reset`, and both close directions -- and they are not weak: each caught thirty-odd mutants elsewhere in the module. But they walk a sequence and check where it ends up, so they never discriminate *which* guard fired. Replacing a guard with `true`, or `&&` with `||`, left all four passing. ## What the machine is for Nothing forwards differently because of the status. The flow's *lifetime* follows from it, and a public address and port are held for as long as the flow lives. A machine that never reaches `Closed` conserves nothing; one that reaches it early releases a tuple that can be handed to another tenant while the connection is still running -- which is the failure this branch already fixed once, approached from the other end. ## Exhaustive, not sampled The whole TCP input space is 2 actions * 10 statuses * 16 flag combinations = 320 cases. Sampling that would be perverse when enumerating it makes the coverage argument disappear. ## A table, and why that is allowed here Elsewhere on this branch the properties deliberately avoid restating the implementation, because an oracle that mirrors the code mirrors its bugs. That objection has teeth when the oracle would grow into a second dataplane. Here the oracle is the TCP close sequence, which is older than this codebase and will outlive it, so writing it down is specification rather than duplication -- and where the two disagree, the table is what should be argued about. It is derived from what the flags mean, not from what the code does. It agreed with the implementation on all 320 cases, so no defect fell out; what fell out is that the machine now has a written specification. Two properties are stated separately from the table because they should be readable without checking it row by row: a segment carrying none of the four flags moves nothing, and `Reset` absorbs. Beyond TCP: a UDP reply from port 53, 853 or 8853 closes the flow at once (one lookup, one reply, and the tuple is released rather than held for the ordinary UDP lifetime -- on a busy gateway that is most of the port space), and an ICMP reply moves a one-way flow to two-way and nothing else moves at all. ## Result protocol.rs before 23 missed, 21 caught, 7 unviable after 0 missed, 45 caught, 6 unviable Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 6d06966e9edaa83ec03cd8f02dac8d88f31b0c9d) --- nat/src/masquerade/mod.rs | 1 + nat/src/masquerade/state_machine.rs | 307 ++++++++++++++++++++++++++++ 2 files changed, 308 insertions(+) create mode 100644 nat/src/masquerade/state_machine.rs diff --git a/nat/src/masquerade/mod.rs b/nat/src/masquerade/mod.rs index d92e9aad50..1e708dd93d 100644 --- a/nat/src/masquerade/mod.rs +++ b/nat/src/masquerade/mod.rs @@ -14,6 +14,7 @@ mod packet; mod probe; mod protocol; mod state; +mod state_machine; mod test; // re exports diff --git a/nat/src/masquerade/state_machine.rs b/nat/src/masquerade/state_machine.rs new file mode 100644 index 0000000000..9ef582b45a --- /dev/null +++ b/nat/src/masquerade/state_machine.rs @@ -0,0 +1,307 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +//! The masquerade flow state machine, exhaustively. +//! +//! [`next_flow_status`] decides how a masqueraded connection is progressing: whether it is still +//! opening, established, half-closed, or done. Nothing forwards differently because of it, but the +//! flow's *lifetime* follows from it, and a public address and port are held for as long as the +//! flow lives. A machine that never reaches `Closed` conserves nothing; one that reaches it early +//! releases a tuple another tenant can be handed while the connection is still running -- the +//! failure this branch already fixed once, from the other end. +//! +//! # Why exhaustive rather than drawn +//! +//! The whole input space is 2 actions * 10 statuses * 16 flag combinations = 320 cases for TCP. +//! Sampling that would be perverse: it is small enough to enumerate, and enumeration makes the +//! coverage argument disappear entirely. +//! +//! # Why a table rather than metamorphic relations +//! +//! Elsewhere on this branch the properties avoid restating the implementation, because an oracle +//! that mirrors the code mirrors its bugs. That objection has teeth when the oracle would grow into +//! a second dataplane. Here the "oracle" is the TCP close sequence, which is older than this +//! codebase and will outlive it, so writing it down is specification rather than duplication. +//! +//! The table below is derived from what the flags *mean*, not from what the code does. Where the +//! two disagree the table is what should be argued about. +//! +//! # What this was for +//! +//! `cargo-mutants` found 23 surviving mutants in `protocol.rs` -- very nearly every match guard in +//! `next_flow_status_tcp`, plus the DNS arm of the UDP patch. There were already four TCP tests +//! (`test_masquerade_tcp_establish`, `_reset`, and both close directions); they walk a sequence and +//! check where it ends up, so they never discriminate *which* guard fired. Replacing a guard with +//! `true` left all of them passing. + +#![cfg(test)] + +use crate::common::{NatAction, NatFlowStatus}; +use crate::masquerade::protocol::next_flow_status; +use net::buffer::TestBuffer; +use net::headers::TryTcpMut; +use net::packet::Packet; +use net::packet::test_utils::{ + IcmpEchoDirection, build_test_icmp4_echo, build_test_tcp_ipv4_packet, + build_test_udp_ipv4_packet, +}; + +/// Every status the machine can be in. +const STATUSES: [NatFlowStatus; 10] = [ + NatFlowStatus::OneWay, + NatFlowStatus::TwoWay, + NatFlowStatus::Established, + NatFlowStatus::Reset, + NatFlowStatus::CClosing, + NatFlowStatus::SClosing, + NatFlowStatus::CHalfClose, + NatFlowStatus::SHalfClose, + NatFlowStatus::LastAck, + NatFlowStatus::Closed, +]; + +/// The four flags the machine reads, as a bitmask over `syn ack fin rst`. +/// +/// Four bools rather than a bitfield on purpose: the table below reads as the close sequence when +/// the flags are named, and as arithmetic when they are not. +#[allow(clippy::struct_excessive_bools)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct Flags { + syn: bool, + ack: bool, + fin: bool, + rst: bool, +} + +impl Flags { + fn from_bits(bits: u8) -> Self { + Self { + syn: bits & 0b1000 != 0, + ack: bits & 0b0100 != 0, + fin: bits & 0b0010 != 0, + rst: bits & 0b0001 != 0, + } + } +} + +fn tcp_packet(flags: Flags) -> Packet { + let mut packet = build_test_tcp_ipv4_packet("1.1.1.1", "2.2.2.2", 1024, 80); + { + let tcp = packet.try_tcp_mut().unwrap_or_else(|| unreachable!()); + tcp.set_syn(flags.syn); + tcp.set_ack(flags.ack); + tcp.set_fin(flags.fin); + tcp.set_rst(flags.rst); + } + packet +} + +fn udp_packet(source_port: u16) -> Packet { + build_test_udp_ipv4_packet("1.1.1.1", "2.2.2.2", source_port, 80) +} + +/// What the TCP close sequence says should happen, written from the flags rather than the code. +/// +/// `SrcNat` is a segment from the private side (the client), `DstNat` one from the public side +/// (the server). `C`/`S` in the status names are the side that began the close. +/// The arms are deliberately not collapsed even where two share a body: each states one step of +/// the close sequence, and merging them by outcome would make the sequence unreadable. +#[allow(clippy::match_same_arms)] +fn expected_tcp(action: NatAction, status: NatFlowStatus, f: Flags) -> NatFlowStatus { + use NatFlowStatus as S; + let progressed = match (action, status) { + // The client's first non-SYN acknowledgement completes the handshake. + (NatAction::SrcNat, S::TwoWay) if !f.syn && f.ack => Some(S::Established), + // The server's SYN-ACK answers the client's SYN. + (NatAction::DstNat, S::OneWay) if f.syn && f.ack => Some(S::TwoWay), + + // Whoever sends the first FIN names the closing side. + (NatAction::SrcNat, S::Established) if f.fin => Some(S::CClosing), + (NatAction::DstNat, S::Established) if f.fin => Some(S::SClosing), + + // The other side acknowledges the FIN without sending its own: half closed. + (NatAction::SrcNat, S::SClosing) if !f.fin && f.ack => Some(S::SHalfClose), + (NatAction::DstNat, S::CClosing) if !f.fin && f.ack => Some(S::CHalfClose), + + // Or acknowledges and closes in the same segment, which skips the half-close. + (NatAction::SrcNat, S::SClosing) if f.fin && f.ack => Some(S::LastAck), + (NatAction::DstNat, S::CClosing) if f.fin && f.ack => Some(S::LastAck), + + // The half-closed side finally sends its own FIN. + (NatAction::SrcNat, S::SHalfClose) if f.fin => Some(S::LastAck), + (NatAction::DstNat, S::CHalfClose) if f.fin => Some(S::LastAck), + + // And the last acknowledgement finishes it. + (_, S::LastAck) if f.ack => Some(S::Closed), + + _ => None, + }; + + // A reset ends the connection, but only where no more specific transition applied: a segment + // carrying both FIN and RST is a close, not an abort. + match progressed { + Some(next) => next, + None if f.rst => S::Reset, + None => status, + } +} + +/// Every TCP transition, for every status and every flag combination. +#[test] +fn the_tcp_state_machine_follows_the_close_sequence() { + for action in [NatAction::SrcNat, NatAction::DstNat] { + for status in STATUSES { + for bits in 0..16u8 { + let flags = Flags::from_bits(bits); + let packet = tcp_packet(flags); + let got = next_flow_status(&packet, action, status); + let want = expected_tcp(action, status, flags); + assert_eq!( + got, want, + "{action} from {status:?} with {flags:?}: expected {want:?}, got {got:?}" + ); + } + } + } +} + +/// A status only ever moves when a flag asks it to. +/// +/// Stated separately from the table because it is the property a guard replaced by `true` +/// violates, and it should be readable without checking the table row by row: a segment carrying +/// none of the four flags is not evidence of anything, and must leave the connection where it was. +#[test] +fn a_segment_with_no_flags_moves_nothing() { + let bare = Flags { + syn: false, + ack: false, + fin: false, + rst: false, + }; + for action in [NatAction::SrcNat, NatAction::DstNat] { + for status in STATUSES { + let packet = tcp_packet(bare); + assert_eq!( + next_flow_status(&packet, action, status), + status, + "{action} from {status:?} moved on a segment with no flags set" + ); + } + } +} + +/// A reset ends a connection that had nowhere else to go, and `Closed` stays closed. +/// +/// The absorbing states are what stop a flow from being kept alive indefinitely by stray traffic +/// after it is over, which is the whole point of tracking status for port conservation. +#[test] +fn reset_and_closed_absorb() { + let rst = Flags { + syn: false, + ack: false, + fin: false, + rst: true, + }; + for action in [NatAction::SrcNat, NatAction::DstNat] { + for bits in 0..16u8 { + let packet = tcp_packet(Flags::from_bits(bits)); + assert_eq!( + next_flow_status(&packet, action, NatFlowStatus::Reset), + NatFlowStatus::Reset, + "a reset connection was revived" + ); + } + // Closed is absorbing except that a reset re-labels it, which is harmless: both are + // terminal and both release the tuple. + let packet = tcp_packet(rst); + assert_eq!( + next_flow_status(&packet, action, NatFlowStatus::Closed), + NatFlowStatus::Reset + ); + } +} + +/// A UDP answer from a resolver closes the flow immediately. +/// +/// DNS is a single request and a single reply over a port that is then never used again. Holding +/// the tuple for the ordinary UDP lifetime would tie up a public port per lookup, which on a busy +/// gateway is most of them. The ports are plain DNS, DNS-over-QUIC, and the one `NextDNS` uses. +#[test] +fn a_reply_from_a_resolver_closes_the_flow_at_once() { + for source_port in [53u16, 853, 8853] { + let packet = udp_packet(source_port); + assert_eq!( + next_flow_status(&packet, NatAction::DstNat, NatFlowStatus::OneWay), + NatFlowStatus::Closed, + "a reply from port {source_port} should close the flow" + ); + // Only inbound: a request *to* a resolver is an ordinary flow. + assert_eq!( + next_flow_status(&packet, NatAction::SrcNat, NatFlowStatus::TwoWay), + NatFlowStatus::Established, + "an outbound packet must not be closed by its own source port" + ); + } +} + +/// Ordinary UDP opens in two steps and then stays put. +#[test] +fn ordinary_udp_opens_and_settles() { + let packet = udp_packet(12345); + assert_eq!( + next_flow_status(&packet, NatAction::DstNat, NatFlowStatus::OneWay), + NatFlowStatus::TwoWay, + "a reply makes a one-way flow two-way" + ); + assert_eq!( + next_flow_status(&packet, NatAction::SrcNat, NatFlowStatus::TwoWay), + NatFlowStatus::Established, + "and the next outbound packet establishes it" + ); + for status in [NatFlowStatus::Established, NatFlowStatus::Closed] { + assert_eq!( + next_flow_status(&packet, NatAction::SrcNat, status), + status, + "an established or closed udp flow does not move outbound" + ); + } +} + +/// An ICMP echo reply makes a one-way flow two-way, and nothing else moves. +/// +/// ICMP has no flags to read and no close sequence, so the only evidence available is that a packet +/// came back the other way. That single transition is what keeps a ping's flow -- and the public +/// address it holds -- alive for the round trip and no longer. +#[test] +fn an_icmp_reply_makes_a_flow_two_way_and_nothing_more() { + let packet = build_test_icmp4_echo( + "1.1.1.1".parse().unwrap_or_else(|_| unreachable!()), + "2.2.2.2".parse().unwrap_or_else(|_| unreachable!()), + 1, + IcmpEchoDirection::Reply, + ) + .unwrap_or_else(|_| unreachable!()); + + assert_eq!( + next_flow_status(&packet, NatAction::DstNat, NatFlowStatus::OneWay), + NatFlowStatus::TwoWay, + "a reply must answer the request" + ); + + // Every other status, in both directions, is left exactly where it was: there is no further + // evidence an icmp exchange can offer. + for status in STATUSES { + assert_eq!( + next_flow_status(&packet, NatAction::SrcNat, status), + status, + "an outbound icmp packet moved a flow in {status:?}" + ); + if status != NatFlowStatus::OneWay { + assert_eq!( + next_flow_status(&packet, NatAction::DstNat, status), + status, + "an inbound icmp packet moved a flow in {status:?}" + ); + } + } +} From cd391d0b5ee95000abc9b02463df7cf0678a63f5 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 18:58:37 -0600 Subject: [PATCH 16/37] docs(testing): Record what mutation testing is for here, and what it is not We have run cargo-mutants over three targets and closed two of them. The measurements, the exclusions, and the shape the weekly report should take are worth writing down before they are forgotten, because rediscovering them costs about a day. The note argues one thing above all: this is a report, not a gate. Mutation testing usually collapses under its maintenance cost, and nearly all of that cost comes from being able to block a merge -- once it can, every equivalent mutant must be triaged and annotated forever. As a report, unkillable mutants cost nothing, which is what makes it affordable at all. The proposed release gate is therefore that every mutant is **classified**, not killed: accepted, aspirational, or gap. Sorting them into three buckets is most of the value, and it turns an intimidating number into a short list plus an explicit decision. Also recorded, because each cost real time to learn: * the config must be `.cargo/mutants.toml`; a `mutants.toml` at the root is silently ignored; * caught mutants are not printed, only `MISSED` -- read a catch rate off the console and the answer is wrong by roughly the catch rate; * a red baseline voids the entire run and looks identical to a bad result in the summary line, so a scheduled job must report it distinctly; * our vacuity guards can in principle inflate the score by catching mutants that made a property unreachable rather than wrong. Measured on `nat/src/masquerade/`: eighteen tripped a guard, zero were caught only that way. The hazard is real but narrow -- it matters where a guarded property is the sole coverage of a path. The report generator itself is not written. This is the design for it. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 3dc6897aedb8016a60e3a713c5e330b52d72c794) --- development/code/README.md | 3 + development/code/mutation-testing.md | 133 +++++++++++++++++++++++++++ 2 files changed, 136 insertions(+) create mode 100644 development/code/mutation-testing.md diff --git a/development/code/README.md b/development/code/README.md index bbd1fdd90e..3617e64bce 100644 --- a/development/code/README.md +++ b/development/code/README.md @@ -12,6 +12,8 @@ wrong thing. If you need to write a test, prefer [property-based tests] over simple unit tests. +To find out what your tests are _not_ saying, see the [mutation testing note][mutants]; it is a +report we read, not a gate that blocks anything. For inputs too large to generate directly -- a whole configuration, say -- build them from an algebra of valid operations and derive the oracles from that same algebra; see the [config algebra note][config-algebra]. If you need to handle errors, prefer `Result` types over panics in general, but see the @@ -28,6 +30,7 @@ If you need to [handle an error][error], follow the guidelines. [avoid-global-reasoning]: ./avoid-global-reasoning.md [property-based tests]: ./property-testing.md [config-algebra]: ./config-algebra-testing.md +[mutants]: ./mutation-testing.md [clock]: ../../clock/src/lib.rs [error]: ./error-handling.md diff --git a/development/code/mutation-testing.md b/development/code/mutation-testing.md new file mode 100644 index 0000000000..4a8e84a7b2 --- /dev/null +++ b/development/code/mutation-testing.md @@ -0,0 +1,133 @@ +# Mutation testing with cargo-mutants + +Status: **first runs done, report generator not yet written**. This records what the tool is for +here, what it has already found, and the shape the weekly report should take, so that the next +person to pick it up does not repeat the measurements. + +## What it is for, and what it is not + +[cargo-mutants] alters the code -- replaces a function body with a plausible default, flips `<` to +`<=`, `&&` to `||`, deletes a match arm -- and re-runs the tests. A mutant the suite does not notice +is a statement the suite never makes. + +It is **not** a gate on the mutation score, and nothing in CI should fail because a number moved. +Mutation testing usually collapses under its own maintenance cost, and nearly all of that cost comes +from being a gate: once a run can block a merge, every equivalent mutant must be triaged and +annotated forever, or somebody's unrelated pull request is stuck behind a mutant nobody can kill. + +As a report, unkillable and equivalent mutants cost nothing. That is the whole reason this is +affordable. + +## What it found, first time out + +Three measurements, all on code that had property tests and that we would have called covered: + +| target | before | after | +| --- | --- | --- | +| `net/src/flows/flow_info.rs` | 13 caught, 28 missed | 32 caught, 0 missed | +| `nat/src/masquerade/protocol.rs` | 21 caught, 23 missed | 45 caught, 0 missed | +| `nat/src/masquerade/` (whole) | 88 caught, 52 missed | -- | + +The two that were closed are worth reading as examples of the two failure modes it finds. + +**A boundary nobody thought to break.** In `FlowInfo::reset_expiry_unchecked`, `<` to `==` was +caught and `<` to `>` was caught, but `<` to `<=` survived. The same boundary class had been +hand-broken in `rio.rs` the same day and _was_ covered there, because it was suspected there. +Suspicion is not uniform; the tool's is. + +**A test that walks a path instead of discriminating one.** `protocol.rs` had four TCP tests, and +they were not weak -- each caught thirty-odd mutants elsewhere in the module. But they walk a +sequence and assert where it ends up, so they never say _which_ guard fired. Replacing a match guard +with `true` left all four passing. Twenty-three of the module's fifty-two survivors were in that one +file. + +## Interaction with our vacuity guards + +The network-function property tests fail the run when a property stops reaching its assertion -- +`reached * 2 >= built`, and similar. Under mutation testing a mutant that makes a property +_unreachable_ trips that guard, the test fails, and the mutant is recorded as caught although +nothing detected its behaviour. That would inflate the score, and inflate it worst exactly where +honest signal matters most. + +Measured on `nat/src/masquerade/`: eighteen mutants tripped a vacuity guard, and **zero** were caught +only that way -- every one was also caught by a conventional test. So the hazard is real but did not +bite, because those modules have ordinary integration tests underneath the properties. + +The narrower rule to carry forward: **vacuity masking matters only where a guarded property is the +sole coverage of a path.** Worth checking whenever a new module gets properties and nothing else. + +## Configuration + +`.cargo/mutants.toml` -- and it must be in `.cargo/`; a `mutants.toml` at the repository root is +silently ignored. + +Exclusions live in that file rather than as `#[mutants::skip]` attributes, so that production crates +take no dependency on the tool and the reasoning stays in one place. Two categories today: + +- **Printers.** The standing rule is "don't test printers." A mutated `fmt` that nothing notices is + the rule being followed, not a gap. +- **`contract::` modules.** The workspace convention puts bolero generators next to the type they + generate, so cargo-mutants finds them and mutates the harness at itself. + +Test scaffolding compiled into a library -- `net/src/buffer/test_buffer.rs` is the current example, +with thirty survivors -- belongs here too. Files behind a module-level `#![cfg(test)]` are skipped +automatically and need no exclusion. + +## Cost + +Measured on `dataplane-net`: 2,271 mutants, about 4.5s to build and 7.6s to test each, four at a +time. Call it two hours for one crate. The dominant cost is re-running the package's whole test +suite per mutant, not the build -- the builds are incremental cargo in a scratch copy of the tree, +nothing to do with nix. + +That rules out running it per pull request in full. Two modes make sense: + +- `--in-diff` on a branch, when you want to know whether what you just wrote is tested. +- A full sweep on a schedule -- a weekly job on an otherwise idle runner, finishing before Monday. + +## The weekly report (to be written) + +The job should produce two artifacts from the same run: + +- **JSON**, for tooling and for agents to read directly. +- **HTML**, grouped by file and function, with the mutated source line inline and clusters sorted by + size. A bare list of `file:line: replace X with Y` is not triage-able away from the source; the + `protocol.rs` cluster was obvious as a chapter precisely because it was twenty-three lines in one + function. + +Post it to a GitHub issue weekly. + +**The product is the delta, not the score.** The absolute count barely moves week to week and tells +you nothing on a Monday morning. What is worth reading is the difference against last week: + +- mutants that **newly survive** -- usually code that landed without properties, or a change that + weakened a test which used to cover something. This catches a case `--in-diff` cannot: the mutant + is in untouched code, and the regression is in this week's change to its test. +- survivors that **disappeared** -- progress, and which work did it. + +## The release gate: classify, do not eliminate + +The gate should be that **every mutant is classified**, not that every mutant is killed. Some are +extremely hard to kill and that is fine. Buckets: + +- **Accepted** -- equivalent mutants, or code where a test would assert nothing useful. Recorded with + a reason, never looked at again. +- **Aspirational** -- a real gap, but closing it needs a harness we do not have. `cli_wake_on_writeable` + in `routing/src/router/rio.rs` is the worked example: reaching it needs roughly 16MiB of unread CLI + response, about 150,000 routes, because the loop's `SndBuf` is 16MiB. +- **Gap** -- a real gap, closeable now. This is the work list. + +Sorting mutants into three buckets is most of the value. It converts an intimidating number into a +short list of things somebody should do, and it makes the intimidating remainder explicitly somebody's +decision rather than an accusation. + +## Operational notes + +- **A red baseline voids the whole run.** Every mutant "survives" against a suite that did not run, + and the summary line looks identical to a genuinely bad result. A scheduled job must report a + broken baseline as a distinct, loud outcome. +- **Caught mutants are not printed.** The console shows only `MISSED`, `TIMEOUT` and unviable lines; + progress and results come from `mutants.out/{caught,missed,unviable,timeout}.txt`. Reading a catch + rate off the log gives an answer that is wrong by roughly the catch rate. + +[cargo-mutants]: https://mutants.rs/ From 8a5647ef26fe8c420123f6c402153e566de615bd Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 20:20:46 -0600 Subject: [PATCH 17/37] fix(net): Validate the RFC 4884 original datagram field as the RFC says `check_full_payload` decides whether an ICMP error carries the whole of the packet that provoked it. Its two length checks were expressed in bits where the values are octets, and between them they enforced neither of the two requirements RFC 4884 actually states. ## What the RFC says RFC 4884 section 3 has two separate requirements: * the "original datagram" field MUST contain at least 128 octets; * it MUST be zero padded to the nearest 32-bit boundary (ICMPv4) or 64-bit boundary (ICMPv6). ## What the code did if icmp_length > buf.len() || !icmp_length.is_multiple_of(32) { ... } if padding_length < 32 && ... `icmp_length` is `length_attribute * 4` for ICMPv4 and `* 8` for ICMPv6, because the attribute counts 32- and 64-bit words. So it is a multiple of 4 (or 8) octets by construction, and a correct alignment check cannot fail. `is_multiple_of(32)` instead demanded a multiple of 32 *octets* -- requiring the length attribute itself to be divisible by 8, which nothing asks for. Seven of every eight lengths a conforming sender can express were refused: field 120 octets (30 words) -> refused field 124 octets (31 words) -> refused field 128 octets (32 words) -> accepted field 132 octets (33 words) -> refused Meanwhile the 128-octet minimum was not checked at all, so a 32-octet field was accepted. The check that existed rejected valid messages and admitted invalid ones. `padding_length < 32` was wrong in the same way and in the other direction: when the original datagram is shorter than 128 octets the sender pads it up to 128, so legitimate padding is routinely more than 31 octets. ## The fix Check what the RFC requires: at least 128 octets, no more than the buffer holds, and the region past the original datagram is all zeroes. Drop the alignment test, which is vacuous, and the padding bound, which was invented. Each surviving check now carries the sentence it implements, in the citation format duvet reads, so the next reader can check the code against the specification without leaving the file. ## Severity Low today and latent. `is_full_payload()` and `payload_length()` have no callers anywhere in the workspace -- the flag is computed and never read -- so nothing forwards differently. It would have failed quietly, by declining to treat valid ICMP errors as complete, whenever the feature was switched on. ## How it was found, and the part worth remembering `cargo-mutants` left 22 comparison mutants alive in this file, which is what sent me looking. But mutation testing did not find *this*; it found that nobody was looking. The bug is a deviation from a specification, and no amount of "the tests notice when I change this line" can see that. Worse, the existing test actively concealed it. It padded a 120-octet packet to 128 with the comment "we need to pad on a 32-bit word boundary" -- and 120 is already on a 32-bit boundary. The padding was there to satisfy the code, and the comment rationalised it. Had I closed those 22 mutants with a property asserting the behaviour I found, the deviation would have become load-bearing, defended by a test, and cited by the next reader as deliberate. That is the failure mode: chasing survivors can entrench a defect as easily as expose one, and only the specification can tell the difference. The comment on the old test is corrected here for the same reason. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 8083be59fc65b2f580bfd3cfd8d61f13f385862f) --- net/src/headers/embedded.rs | 141 +++++++++++++++++++++++++++++++----- 1 file changed, 123 insertions(+), 18 deletions(-) diff --git a/net/src/headers/embedded.rs b/net/src/headers/embedded.rs index c98128d78b..81492687e6 100644 --- a/net/src/headers/embedded.rs +++ b/net/src/headers/embedded.rs @@ -48,6 +48,13 @@ pub struct EmbeddedHeaders { full_payload_length: Option, } +/// The smallest "original datagram" field RFC 4884 permits, in octets. +/// +/// Applies to both `ICMPv4` and `ICMPv6`: a sender appending an extension structure must include at +/// least this much of the datagram that elicited the error, zero padding it if the original was +/// shorter. +const MIN_ORIGINAL_DATAGRAM_OCTETS: usize = 128; + impl EmbeddedHeaders { #[cfg(any(test, feature = "bolero"))] #[must_use] @@ -241,16 +248,28 @@ impl EmbeddedHeaders { // The embedded message is shorter than the original packet return; } - if icmp_length > buf.len() || !icmp_length.is_multiple_of(32) { - // Embedded payload is larger than our buffer? Or the size is not a multiple - // of 32? Something's wrong + if icmp_length > buf.len() { + // Embedded payload is larger than our buffer; something's wrong + return; + } + //= https://www.rfc-editor.org/rfc/rfc4884#section-3 + //# When the ICMP Extension Structure is appended to an ICMP message + //# and that ICMP message contains an "original datagram" field, the + //# "original datagram" field MUST contain at least 128 octets. + if icmp_length < MIN_ORIGINAL_DATAGRAM_OCTETS { return; } - let padding_length = icmp_length - full_packet_length; - // ICMPv4: Padding is on 32-bit boundaries - if padding_length < 32 - && buf[full_packet_length..icmp_length].iter().all(|b| *b == 0) - { + //= https://www.rfc-editor.org/rfc/rfc4884#section-3 + //# When the ICMP Extension Structure is appended to an ICMPv4 message + //# and that ICMPv4 message contains an "original datagram" field, the + //# "original datagram" field MUST be zero padded to the nearest + //# 32-bit boundary. + // + // The alignment itself needs no check: the length attribute counts 32-bit + // words, so `icmp_length` is a multiple of four by construction. What is worth + // checking is that the padding really is zeroes -- the field is the head of the + // original datagram, so anything past its announced length must be padding. + if buf[full_packet_length..icmp_length].iter().all(|b| *b == 0) { self.full_payload_length = Some(transport_payload_length as u16); } return; @@ -260,16 +279,26 @@ impl EmbeddedHeaders { // The embedded message is shorter than the original packet return; } - if icmp_length > buf.len() || !icmp_length.is_multiple_of(64) { - // Embedded payload is larger than our buffer? Or the size is not a multiple - // of 64? Something's wrong + if icmp_length > buf.len() { + // Embedded payload is larger than our buffer; something's wrong return; } - let padding_length = icmp_length - full_packet_length; - // ICMPv6: Padding is on 64-bit boundaries - if padding_length < 64 - && buf[full_packet_length..icmp_length].iter().all(|b| *b == 0) - { + //= https://www.rfc-editor.org/rfc/rfc4884#section-3 + //# When the ICMP Extension Structure is appended to an ICMP message + //# and that ICMP message contains an "original datagram" field, the + //# "original datagram" field MUST contain at least 128 octets. + if icmp_length < MIN_ORIGINAL_DATAGRAM_OCTETS { + return; + } + //= https://www.rfc-editor.org/rfc/rfc4884#section-3 + //# When the ICMP Extension Structure is appended to an ICMPv6 message + //# and that ICMPv6 message contains an "original datagram" field, the + //# "original datagram" field MUST be zero padded to the nearest + //# 64-bit boundary. + // + // As for ICMPv4: the length attribute counts 64-bit words, so alignment holds + // by construction and only the zero padding is worth checking. + if buf[full_packet_length..icmp_length].iter().all(|b| *b == 0) { self.full_payload_length = Some(transport_payload_length as u16); } return; @@ -1318,12 +1347,88 @@ mod tests { assert!(!headers.is_full_payload()); } + /// Build an `ICMPv4` error whose "original datagram" field is `field_len` octets, followed + /// by an extension structure. The embedded packet is 120 octets; the rest is padding. + fn v4_with_field_of(field_len: usize, padding_byte: u8) -> (EmbeddedHeaders, usize, Vec) { + let mut buf = create_full_ipv4_tcp_packet_with_payload(); + assert_eq!(buf.len(), 120, "the embedded packet is 120 octets"); + buf.extend(std::iter::repeat_n(padding_byte, field_len - buf.len())); + assert_eq!(buf.len(), field_len); + // Extension structure, which is not part of the field. + buf.extend_from_slice(&[0x55u8; 32]); + let (headers, consumed) = + EmbeddedHeaders::parse_with(EmbeddedIpVersion::Ipv4, &buf).unwrap(); + (headers, consumed.get() as usize, buf) + } + + /// A field is accepted at any 32-bit-aligned length, not only at multiples of 32 octets. + /// + /// The length attribute counts 32-bit words, so every value it can express is already aligned. + /// Requiring a multiple of 32 *octets* -- bits mistaken for bytes -- rejected seven of every + /// eight lengths a conforming sender can produce. + //= https://www.rfc-editor.org/rfc/rfc4884#section-3 + //= type=test + //# When the ICMP Extension Structure is appended to an ICMPv4 message + //# and that ICMPv4 message contains an "original datagram" field, the + //# "original datagram" field MUST be zero padded to the nearest + //# 32-bit boundary. + #[test] + fn a_field_is_accepted_at_any_32_bit_aligned_length() { + for field_len in [128usize, 132, 136, 140, 144, 148, 152, 156] { + let (mut headers, consumed, buf) = v4_with_field_of(field_len, 0); + headers.check_full_payload(&buf, buf.len(), consumed, field_len); + assert!( + headers.is_full_payload(), + "a {field_len}-octet field is 32-bit aligned and at least 128 octets, so it must \ + be accepted" + ); + assert_eq!(headers.payload_length(), Some(80)); + } + } + + /// A field shorter than 128 octets is refused. + /// + /// This is the requirement the octet/bit confusion displaced: the old check let a 32-octet + /// field through and rejected a 132-octet one, which is exactly backwards. + //= https://www.rfc-editor.org/rfc/rfc4884#section-3 + //= type=test + //# When the ICMP Extension Structure is appended to an ICMP message + //# and that ICMP message contains an "original datagram" field, the + //# "original datagram" field MUST contain at least 128 octets. + #[test] + fn a_field_shorter_than_128_octets_is_refused() { + for field_len in [120usize, 124] { + let (mut headers, consumed, buf) = v4_with_field_of(field_len, 0); + headers.check_full_payload(&buf, buf.len(), consumed, field_len); + assert!( + !headers.is_full_payload(), + "a {field_len}-octet field is below the 128-octet minimum" + ); + } + } + + /// Padding that is not zeroes is not padding. + /// + /// The field holds the head of the original datagram; anything past the length that datagram + /// announced must be the sender's zero padding. Bytes with content there mean the two lengths + /// disagree, and the payload cannot be trusted to be whole. + #[test] + fn a_field_padded_with_anything_but_zeroes_is_refused() { + let (mut headers, consumed, buf) = v4_with_field_of(136, 0xab); + headers.check_full_payload(&buf, buf.len(), consumed, 136); + assert!( + !headers.is_full_payload(), + "non-zero padding must not be accepted as padding" + ); + } + #[test] fn test_check_full_payload_with_icmp_extensions() { let mut buf = create_full_ipv4_tcp_packet_with_payload(); - // We need to pad on a 32-bit word boundary. We have 120 bytes (20 for the IP header, 20 for - // the TCP header, 80 for the payload), add 8 to reach 128 bytes. + // We have 120 bytes (20 for the IP header, 20 for the TCP header, 80 for the payload). + // Add 8 to reach 128, which is the minimum size RFC 4884 allows for the field. 120 is + // already on a 32-bit boundary; it is the minimum, not the alignment, that needs the pad. buf.extend_from_slice(&[0u8; 8]); let icmp_payload_length = buf.len(); From 8b92d3aa48f16c7636b3f3979e5f25b381b7d6af Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 21:04:34 -0600 Subject: [PATCH 18/37] build(duvet): Track RFC 4884 compliance, and vendor the specification `duvet report` matches citations in the source against the requirements it extracts from a specification, so a requirement with nothing implementing it, or an implementation with nothing testing it, becomes visible. RFC 4884 is the first specification tracked because the code already cites it: the "original datagram" length checks contradicted it, and the citations added with that fix are the ones this now reads. ## Everything but the reports is committed, on purpose `duvet init` generates a `.gitignore` containing only `reports/`, and that turns out to be load-bearing rather than a style choice. Measured under `unshare -rn`: * with `.duvet/specifications/` present, `duvet report` runs offline; * with it removed, it fails with "Network is unreachable". There is no cache anywhere else -- nothing in `~/.cache`. So vendoring the specification text is what makes the report runnable in a build sandbox at all, rather than a nicety. RFC 4884 costs 42KiB. It also means an errata, a reformat, or a fetch that quietly returns something different arrives as a reviewable diff in a pull request rather than as a change in results nobody can account for. Refreshing a specification becomes a deliberate commit. ## What the first run says TEXT[!MUST,implementation,test]: ..."original datagram" field MUST contain at least 128 octets. TEXT[!MUST,implementation,test]: ...MUST be zero padded to the nearest 32-bit boundary. [v4] TEXT[!MUST,implementation]: ...MUST be zero padded to the nearest 64-bit boundary. [v6] The ICMPv6 requirement is implemented and cited but has no test: the three tests added with the fix are all ICMPv4. Found on the first run, which is the point. ## Two adjustments The generated `[[source]]` pattern is `src/**/*.rs`, which matches nothing in a workspace; it is `*/src/**/*.rs` here. `routing/src/cli/display.rs` had thirteen banner comments of the form `//======== Fib ========//`. `//=` is duvet's citation marker, so each was read as a malformed citation -- 24 errors before the report could run. The banners now start `// =`. The collision is worth knowing about before writing new ones. ## The snapshot `.duvet/snapshot.txt` is line-oriented and diffs cleanly, and its unit is a sentence somebody else wrote. Unlike a mutant's `file:line:col`, that key does not move when a function is reformatted -- which is what makes it usable as a gate where the mutation report is only usable as a report. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 34e13e0b0d51632b9857c0629c2fce86cdfcc80a) --- .duvet/.gitignore | 1 + .duvet/config.toml | 30 + .../rfc/rfc4884/section-3.toml | 84 ++ .../rfc/rfc4884/section-4.6.toml | 36 + .../rfc/rfc4884/section-4.toml | 166 +++ .../rfc/rfc4884/section-5.4.toml | 16 + .../rfc/rfc4884/section-5.5.toml | 39 + .../rfc/rfc4884/section-7.toml | 59 + .duvet/snapshot.txt | 59 + .../www.rfc-editor.org/rfc/rfc4884.txt | 1067 +++++++++++++++++ routing/src/cli/display.rs | 24 +- 11 files changed, 1569 insertions(+), 12 deletions(-) create mode 100644 .duvet/.gitignore create mode 100644 .duvet/config.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-3.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-4.6.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-4.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-5.4.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-5.5.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-7.toml create mode 100644 .duvet/snapshot.txt create mode 100644 .duvet/specifications/www.rfc-editor.org/rfc/rfc4884.txt diff --git a/.duvet/.gitignore b/.duvet/.gitignore new file mode 100644 index 0000000000..a9a1bd38ab --- /dev/null +++ b/.duvet/.gitignore @@ -0,0 +1 @@ +reports/ diff --git a/.duvet/config.toml b/.duvet/config.toml new file mode 100644 index 0000000000..8b5c8ae60c --- /dev/null +++ b/.duvet/config.toml @@ -0,0 +1,30 @@ +# duvet: specification compliance coverage. +# +# `duvet report` matches citations in the source -- `//= ` followed by the requirement text -- +# against the requirements duvet extracts from the specification, so a requirement with nothing +# implementing it, or an implementation with nothing testing it, is visible. +# +# Everything under .duvet/ is committed except `reports/`. That is not incidental: `duvet report` +# reads the specification from `.duvet/specifications/` and only reaches the network when it is +# missing, so vendoring the text is what lets the report run in a nix build sandbox at all. It also +# means an errata, a reformat, or a fetch that quietly returns something else arrives as a +# reviewable diff rather than as a change in results nobody can explain. +'$schema' = "https://awslabs.github.io/duvet/config/v0.4.0.json" + +# Every crate in the workspace. The generated default is `src/**/*.rs`, which matches nothing here. +[[source]] +pattern = "*/src/**/*.rs" + +# RFC 4884 is the first specification tracked, because the code already cites it: the +# "original datagram" length checks in net/src/headers/embedded.rs were found to contradict it. +[[specification]] +source = "https://www.rfc-editor.org/rfc/rfc4884" + +[report.html] +enabled = true + +# The snapshot is the regression gate. It is line-oriented and diffs cleanly, and its unit is a +# sentence somebody else wrote -- which, unlike a mutant's file:line, does not move when a function +# is reformatted. +[report.snapshot] +enabled = true diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-3.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-3.toml new file mode 100644 index 0000000000..4acd6e9bc9 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-3.toml @@ -0,0 +1,84 @@ +target = "https://www.rfc-editor.org/rfc/rfc4884#section-3" + +# Summary of Changes to ICMP +# +# The following is a summary of changes to ICMP that are introduced by +# this memo: +# +# An ICMP Extension Structure MAY be appended to ICMPv4 Destination +# Unreachable, Time Exceeded, and Parameter Problem messages. +# +# An ICMP Extension Structure MAY be appended to ICMPv6 Destination +# Unreachable, and Time Exceeded messages. +# +# The above mentioned messages include an "original datagram" field, +# and the message formats are updated to specify a length attribute +# for the "original datagram" field. +# +# When the ICMP Extension Structure is appended to an ICMP message +# and that ICMP message contains an "original datagram" field, the +# "original datagram" field MUST contain at least 128 octets. +# +# When the ICMP Extension Structure is appended to an ICMPv4 message +# and that ICMPv4 message contains an "original datagram" field, the +# "original datagram" field MUST be zero padded to the nearest +# 32-bit boundary. +# +# When the ICMP Extension Structure is appended to an ICMPv6 message +# and that ICMPv6 message contains an "original datagram" field, the +# "original datagram" field MUST be zero padded to the nearest +# 64-bit boundary. +# +# ICMP messages defined in the future SHOULD indicate whether or not +# they support the extension mechanism defined in this +# specification. It is recommended that all new messages support +# extensions. + +[[spec]] +level = "MAY" +quote = ''' +An ICMP Extension Structure MAY be appended to ICMPv4 Destination +Unreachable, Time Exceeded, and Parameter Problem messages. +''' + +[[spec]] +level = "MAY" +quote = ''' +An ICMP Extension Structure MAY be appended to ICMPv6 Destination +Unreachable, and Time Exceeded messages. +''' + +[[spec]] +level = "MUST" +quote = ''' +When the ICMP Extension Structure is appended to an ICMP message +and that ICMP message contains an "original datagram" field, the +"original datagram" field MUST contain at least 128 octets. +''' + +[[spec]] +level = "MUST" +quote = ''' +When the ICMP Extension Structure is appended to an ICMPv4 message +and that ICMPv4 message contains an "original datagram" field, the +"original datagram" field MUST be zero padded to the nearest +32-bit boundary. +''' + +[[spec]] +level = "MUST" +quote = ''' +When the ICMP Extension Structure is appended to an ICMPv6 message +and that ICMPv6 message contains an "original datagram" field, the +"original datagram" field MUST be zero padded to the nearest +64-bit boundary. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +ICMP messages defined in the future SHOULD indicate whether or not +they support the extension mechanism defined in this +specification. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-4.6.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-4.6.toml new file mode 100644 index 0000000000..3c0b29be98 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-4.6.toml @@ -0,0 +1,36 @@ +target = "https://www.rfc-editor.org/rfc/rfc4884#section-4.6" + +# ICMP Messages That Can Be Extended +# +# The ICMP Extension Structure MAY be appended to messages of the +# following types: +# +# - ICMPv4 Destination Unreachable +# +# - ICMPv4 Time Exceeded +# +# - ICMPv4 Parameter Problem +# +# - ICMPv6 Destination Unreachable +# +# - ICMPv6 Time Exceeded +# +# The ICMP Extension Structure MUST NOT be appended to any of the other +# ICMP messages mentioned in Section 4. Extensions were not defined +# for the ICMPv6 "Packet Too Big" and "Parameter Problem" messages +# because these messages lack space for a length attribute. + +[[spec]] +level = "MAY" +quote = ''' +The ICMP Extension Structure MAY be appended to messages of the +following types: +''' + +[[spec]] +level = "MUST" +quote = ''' +The ICMP Extension Structure MUST NOT be appended to any of the other +ICMP messages mentioned in Section 4. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-4.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-4.toml new file mode 100644 index 0000000000..75f346e047 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-4.toml @@ -0,0 +1,166 @@ +target = "https://www.rfc-editor.org/rfc/rfc4884#section-4" + +# ICMP Extensibility +# +# RFC 792 defines the following ICMPv4 message types: +# +# - Destination Unreachable +# +# - Time Exceeded +# +# - Parameter Problem +# +# - Source Quench +# +# - Redirect +# +# - Echo Request/Reply +# +# - Timestamp/Timestamp Reply +# +# - Information Request/Information Reply +# +# [RFC1191] reserves bits for the "Next-Hop MTU" field in the +# Destination Unreachable message. +# +# RFC 4443 defines the following ICMPv6 message types: +# +# - Destination Unreachable +# +# - Packet Too Big +# +# - Time Exceeded +# +# - Parameter Problem +# +# - Echo Request/Reply +# +# Many ICMP messages are extensible as currently defined. Protocol +# designers can extend ICMP messages by simply appending fields or data +# structures to them. +# +# However, the following ICMP messages are not extensible as currently +# defined: +# +# - ICMPv4 Destination Unreachable (type = 3) +# +# - ICMPv4 Time Exceeded (type = 11) +# +# - ICMPv4 Parameter Problem (type = 12) +# +# - ICMPv6 Destination Unreachable (type = 1) +# +# - ICMPv6 Packet Too Big (type = 2) +# +# - ICMPv6 Time Exceeded (type = 3) +# +# - ICMPv6 Parameter Problem (type = 4) +# +# These messages contain an "original datagram" field which represents +# the leading octets of the datagram to which the ICMP message is a +# response. RFC 792 defines the "original datagram" field for ICMPv4 +# messages. In RFC 792, the "original datagram" field includes the IP +# header plus the next eight octets of the original datagram. +# [RFC1812] extends the "original datagram" field to contain as many +# octets as possible without causing the ICMP message to exceed the +# minimum IPv4 reassembly buffer size (i.e., 576 octets). RFC 4443 +# defines the "original datagram" field for ICMPv6 messages. In RFC +# 4443, the "original datagram" field always contained as many octets +# as possible without causing the ICMP message to exceed the minimum +# IPv6 MTU (i.e., 1280 octets). +# +# Unfortunately, the "original datagram" field lacks a length +# attribute. Application software infers the length of this field from +# the total length of the ICMP message. If an extension structure were +# appended to the message without adding a length attribute for the +# "original datagram" field, the message would become unparsable. +# Specifically, application software would not be able to determine +# where the "original datagram" field ends and where the extension +# structure begins. +# +# In order to solve this problem, this memo introduces an 8-bit length +# attribute to the following ICMPv4 messages. +# +# - Destination Unreachable (type = 3) +# +# - Time Exceeded (type = 11) +# +# - Parameter Problem (type = 12) +# +# It also introduces an 8-bit length attribute to the following ICMPv6 +# messages. +# +# - Destination Unreachable (type = 1) +# +# - Time Exceeded (type = 3) +# +# The length attribute MUST be specified when the ICMP Extension +# Structure is appended to the above mentioned ICMP messages. +# +# The length attribute represents the length of the "original datagram" +# field. Space for the length attribute is claimed from reserved +# octets, whose value was previously required to be zero. +# +# For ICMPv4 messages, the length attribute represents 32-bit words. +# When the length attribute is specified, the "original datagram" field +# MUST be zero padded to the nearest 32-bit boundary. Because the +# +# sixth octet of each of the impacted ICMPv4 messages was reserved for +# future use, this octet was selected as the location of the length +# attribute in ICMPv4. +# +# For ICMPv6 messages, the length attribute represents 64-bit words. +# When the length attribute is specified, the "original datagram" field +# MUST be zero padded to the nearest 64-bit boundary. Because the +# fifth octet of each of the impacted ICMPv6 messages was reserved for +# future use, this octet was selected as the location of the length +# attribute in ICMPv6. +# +# In order to achieve backwards compatibility, when the ICMP Extension +# Structure is appended to an ICMP message and that ICMP message +# contains an "original datagram" field, the "original datagram" field +# MUST contain at least 128 octets. If the original datagram did not +# contain 128 octets, the "original datagram" field MUST be zero padded +# to 128 octets. (See Section 5.1 for rationale.) +# +# The following sub-sections depict length attribute as it has been +# introduced to selected ICMP messages. + +[[spec]] +level = "MUST" +quote = ''' +The length attribute MUST be specified when the ICMP Extension +Structure is appended to the above mentioned ICMP messages. +''' + +[[spec]] +level = "MUST" +quote = ''' +When the length attribute is specified, the "original datagram" field +MUST be zero padded to the nearest 32-bit boundary. +''' + +[[spec]] +level = "MUST" +quote = ''' +When the length attribute is specified, the "original datagram" field +MUST be zero padded to the nearest 64-bit boundary. +''' + +[[spec]] +level = "MUST" +quote = ''' +In order to achieve backwards compatibility, when the ICMP Extension +Structure is appended to an ICMP message and that ICMP message +contains an "original datagram" field, the "original datagram" field +MUST contain at least 128 octets. +''' + +[[spec]] +level = "MUST" +quote = ''' +If the original datagram did not +contain 128 octets, the "original datagram" field MUST be zero padded +to 128 octets. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-5.4.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-5.4.toml new file mode 100644 index 0000000000..37a0ca7e8f --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-5.4.toml @@ -0,0 +1,16 @@ +target = "https://www.rfc-editor.org/rfc/rfc4884#section-5.4" + +# Compliant Application Receives ICMP Message with No Extensions +# +# When a compliant application receives an ICMP message, it examines +# the length attribute that is associated with the "original datagram" +# field. If the length attribute is zero, the compliant application +# MUST determine that the message contains no extensions. + +[[spec]] +level = "MUST" +quote = ''' +If the length attribute is zero, the compliant application +MUST determine that the message contains no extensions. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-5.5.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-5.5.toml new file mode 100644 index 0000000000..3faf9f4440 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-5.5.toml @@ -0,0 +1,39 @@ +target = "https://www.rfc-editor.org/rfc/rfc4884#section-5.5" + +# Compliant Application Receives ICMP Message with Non-Compliant +# +# Extensions +# +# When a compliant application receives an ICMP message, it examines +# the length attribute that is associated with the "original datagram" +# field. If the length attribute is zero, the compliant application +# MUST determine that the message contains no extensions. In this +# case, that determination is technically correct, but not backwards +# compatible with the non-compliant implementation that originated the +# ICMP message. +# +# So, to ease transition yet encourage compliant implementation, +# compliant TRACEROUTE implementations MUST include a non-default +# operation mode to also interpret non-compliant responses. +# Specifically, when a TRACEROUTE application operating in non- +# compliant mode receives a sufficiently long ICMP message that does +# not specify a length attribute, it will parse for a valid extension +# header at a fixed location, assuming a 128-octet "original datagram" +# field. If the application detects a valid version and checksum, it +# will treat the octets that follow as an extension structure. + +[[spec]] +level = "MUST" +quote = ''' +If the length attribute is zero, the compliant application +MUST determine that the message contains no extensions. +''' + +[[spec]] +level = "MUST" +quote = ''' +So, to ease transition yet encourage compliant implementation, +compliant TRACEROUTE implementations MUST include a non-default +operation mode to also interpret non-compliant responses. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-7.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-7.toml new file mode 100644 index 0000000000..6a3733dc60 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4884/section-7.toml @@ -0,0 +1,59 @@ +target = "https://www.rfc-editor.org/rfc/rfc4884#section-7" + +# The ICMP Extension Structure +# +# This memo proposes an optional ICMP Extension Structure that can be +# appended to the ICMP messages referenced in Section 4.6 of this +# document. +# +# The Extension Structure contains exactly one Extension Header +# followed by one or more objects. Having received an ICMP message +# with extensions, application software MAY process selected objects +# while ignoring others. The presence of an unrecognized object does +# not imply that an ICMP message is malformed. +# +# As stated above, the total length of the ICMP message, including +# extensions, MUST NOT exceed the minimum reassembly buffer size. +# Figure 6 depicts the ICMP Extension Header. +# +# 0 1 2 3 +# 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +# +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +# |Version| (Reserved) | Checksum | +# +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +# +# Figure 6: ICMP Extension Header +# +# The fields of the ICMP Extension Header are as follows: +# +# Version: 4 bits +# +# ICMP extension version number. This is version 2. +# +# Reserved: 12 bits +# +# Must be set to 0. +# +# Checksum: 16 bits +# +# The one's complement of the one's complement sum of the data +# structure, with the checksum field replaced by zero for the +# purpose of computing the checksum. An all-zero value means that +# no checksum was transmitted. See Section 5.2 for a description of +# how this field is used. + +[[spec]] +level = "MAY" +quote = ''' +Having received an ICMP message +with extensions, application software MAY process selected objects +while ignoring others. +''' + +[[spec]] +level = "MUST" +quote = ''' +As stated above, the total length of the ICMP message, including +extensions, MUST NOT exceed the minimum reassembly buffer size. +''' + diff --git a/.duvet/snapshot.txt b/.duvet/snapshot.txt new file mode 100644 index 0000000000..3fe31d4b2c --- /dev/null +++ b/.duvet/snapshot.txt @@ -0,0 +1,59 @@ +SPECIFICATION: https://www.rfc-editor.org/rfc/rfc4884 + SECTION: [Summary of Changes to ICMP](#section-3) + TEXT[!MAY]: An ICMP Extension Structure MAY be appended to ICMPv4 Destination + TEXT[!MAY]: Unreachable, Time Exceeded, and Parameter Problem messages. + TEXT[!MAY]: An ICMP Extension Structure MAY be appended to ICMPv6 Destination + TEXT[!MAY]: Unreachable, and Time Exceeded messages. + TEXT[!MUST,implementation,test]: When the ICMP Extension Structure is appended to an ICMP message + TEXT[!MUST,implementation,test]: and that ICMP message contains an "original datagram" field, the + TEXT[!MUST,implementation,test]: "original datagram" field MUST contain at least 128 octets. + TEXT[!MUST,implementation,test]: When the ICMP Extension Structure is appended to an ICMPv4 message + TEXT[!MUST,implementation,test]: and that ICMPv4 message contains an "original datagram" field, the + TEXT[!MUST,implementation,test]: "original datagram" field MUST be zero padded to the nearest + TEXT[!MUST,implementation,test]: 32-bit boundary. + TEXT[!MUST,implementation]: When the ICMP Extension Structure is appended to an ICMPv6 message + TEXT[!MUST,implementation]: and that ICMPv6 message contains an "original datagram" field, the + TEXT[!MUST,implementation]: "original datagram" field MUST be zero padded to the nearest + TEXT[!MUST,implementation]: 64-bit boundary. + TEXT[!SHOULD]: ICMP messages defined in the future SHOULD indicate whether or not + TEXT[!SHOULD]: they support the extension mechanism defined in this + TEXT[!SHOULD]: specification. + + SECTION: [ICMP Extensibility](#section-4) + TEXT[!MUST]: The length attribute MUST be specified when the ICMP Extension + TEXT[!MUST]: Structure is appended to the above mentioned ICMP messages. + TEXT[!MUST]: When the length attribute is specified, the "original datagram" field + TEXT[!MUST]: MUST be zero padded to the nearest 32-bit boundary. + TEXT[!MUST]: When the length attribute is specified, the "original datagram" field + TEXT[!MUST]: MUST be zero padded to the nearest 64-bit boundary. + TEXT[!MUST]: In order to achieve backwards compatibility, when the ICMP Extension + TEXT[!MUST]: Structure is appended to an ICMP message and that ICMP message + TEXT[!MUST]: contains an "original datagram" field, the "original datagram" field + TEXT[!MUST]: MUST contain at least 128 octets. + TEXT[!MUST]: If the original datagram did not + TEXT[!MUST]: contain 128 octets, the "original datagram" field MUST be zero padded + TEXT[!MUST]: to 128 octets. + + SECTION: [ICMP Messages That Can Be Extended](#section-4.6) + TEXT[!MAY]: The ICMP Extension Structure MAY be appended to messages of the + TEXT[!MAY]: following types: + TEXT[!MUST]: The ICMP Extension Structure MUST NOT be appended to any of the other + TEXT[!MUST]: ICMP messages mentioned in Section 4. + + SECTION: [Compliant Application Receives ICMP Message with No Extensions](#section-5.4) + TEXT[!MUST]: If the length attribute is zero, the compliant application + TEXT[!MUST]: MUST determine that the message contains no extensions. + + SECTION: [Compliant Application Receives ICMP Message with Non-Compliant](#section-5.5) + TEXT[!MUST]: If the length attribute is zero, the compliant application + TEXT[!MUST]: MUST determine that the message contains no extensions. + TEXT[!MUST]: So, to ease transition yet encourage compliant implementation, + TEXT[!MUST]: compliant TRACEROUTE implementations MUST include a non-default + TEXT[!MUST]: operation mode to also interpret non-compliant responses. + + SECTION: [The ICMP Extension Structure](#section-7) + TEXT[!MAY]: Having received an ICMP message + TEXT[!MAY]: with extensions, application software MAY process selected objects + TEXT[!MAY]: while ignoring others. + TEXT[!MUST]: As stated above, the total length of the ICMP message, including + TEXT[!MUST]: extensions, MUST NOT exceed the minimum reassembly buffer size. diff --git a/.duvet/specifications/www.rfc-editor.org/rfc/rfc4884.txt b/.duvet/specifications/www.rfc-editor.org/rfc/rfc4884.txt new file mode 100644 index 0000000000..85def5fce9 --- /dev/null +++ b/.duvet/specifications/www.rfc-editor.org/rfc/rfc4884.txt @@ -0,0 +1,1067 @@ + + + + + + +Network Working Group R. Bonica +Request for Comments: 4884 Juniper Networks +Updates: 792, 4443 D. Gan +Category: Standards Track Consultant + D. Tappan + Consultant + C. Pignataro + Cisco Systems, Inc. + April 2007 + + + Extended ICMP to Support Multi-Part Messages + +Status of This Memo + + This document specifies an Internet standards track protocol for the + Internet community, and requests discussion and suggestions for + improvements. Please refer to the current edition of the "Internet + Official Protocol Standards" (STD 1) for the standardization state + and status of this protocol. Distribution of this memo is unlimited. + +Copyright Notice + + Copyright (C) The IETF Trust (2007). + +Abstract + + This document redefines selected ICMP messages to support multi-part + operation. A multi-part ICMP message carries all of the information + that ICMP messages carried previously, as well as additional + information that applications may require. + + Multi-part messages are supported by an ICMP extension structure. + The extension structure is situated at the end of the ICMP message. + It includes an extension header followed by one or more extension + objects. Each extension object contains an object header and object + payload. All object headers share a common format. + + This document further redefines the above mentioned ICMP messages by + specifying a length attribute. All of the currently defined ICMP + messages to which an extension structure can be appended include an + "original datagram" field. The "original datagram" field contains + the initial octets of the datagram that elicited the ICMP error + message. Although the original datagram field is of variable length, + the ICMP message does not include a field that specifies its length. + Therefore, in order to facilitate message parsing, this document + allocates eight previously reserved bits to reflect the length of the + "original datagram" field. + + + +Bonica, et al. Standards Track [Page 1] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + + The proposed modifications change the requirements for ICMP + compliance. The impact of these changes on compliant implementations + is discussed, and new requirements for future implementations are + presented. + + This memo updates RFC 792 and RFC 4443. + +Table of Contents + + 1. Introduction ....................................................3 + 2. Conventions Used in This Document ...............................4 + 3. Summary of Changes to ICMP ......................................4 + 4. ICMP Extensibility ..............................................4 + 4.1. ICMPv4 Destination Unreachable .............................7 + 4.2. ICMPv4 Time Exceeded .......................................8 + 4.3. ICMPv4 Parameter Problem ...................................8 + 4.4. ICMPv6 Destination Unreachable .............................9 + 4.5. ICMPv6 Time Exceeded .......................................9 + 4.6. ICMP Messages That Can Be Extended ........................10 + 5. Backwards Compatibility ........................................10 + 5.1. Classic Application Receives ICMP Message with + Extensions ................................................12 + 5.2. Non-Compliant Application Receives ICMP Message + with No Extensions ........................................12 + 5.3. Non-Compliant Application Receives ICMP Message + with Compliant Extensions .................................13 + 5.4. Compliant Application Receives ICMP Message with + No Extensions .............................................14 + 5.5. Compliant Application Receives ICMP Message with + Non-Compliant Extensions ..................................14 + 6. Interaction with Network Address Translation ...................14 + 7. The ICMP Extension Structure ...................................15 + 8. ICMP Extension Objects .........................................16 + 9. Security Considerations ........................................16 + 10. IANA Considerations ...........................................17 + 11. Acknowledgments ...............................................17 + 12. References ....................................................17 + 12.1. Normative References .....................................17 + 12.2. Informative References ...................................17 + + + + + + + + + + + + +Bonica, et al. Standards Track [Page 2] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + +1. Introduction + + This document redefines selected ICMPv4 [RFC0792] and ICMPv6 + [RFC4443] messages to include an extension structure and a length + attribute. The extension structure supports multi-part ICMP + operation. Protocol designers can make an ICMP message carry + additional information by encoding that information in the extension + structure. + + This document also addresses a fundamental problem in ICMP + extensibility. All of the ICMP messages addressed by this memo + include an "original datagram" field. The "original datagram" field + contains the initial octets of the datagram that elicited the ICMP + error message. Although the "original datagram" field is of variable + length, the ICMP message does not include a field that specifies its + length. + + Application software infers the length of the "original datagram" + field from the total length of the ICMP message. If an extension + structure were appended to the message without adding a length + attribute for the "original datagram" field, the message would become + unparsable. Specifically, application software would not be able to + determine where the "original datagram" field ends and where the + extension structure begins. Therefore, this document proposes a + length attribute as well as an extension structure that is appended + to the ICMP message. + + The current memo also addresses backwards compatibility with existing + ICMP implementations that either do not implement the extensions + defined herein or implement them without adding the required length + attributes. In particular, this document addresses backwards + compatibility with certain, widely deployed, MPLS-aware ICMPv4 + implementations that send the extensions defined herein without + adding the required length attribute. + + The current memo does not define any ICMP extension objects. It + defines only the extension header and a common header that all + extension objects share. [UNNUMBERED], [ROUTING-INST], and + [MPLS-ICMP] provide sample applications of the ICMP Extension Object. + + The above mentioned memos share a common characteristic. They all + append information to the ICMP Time Expired message for consumption + by TRACEROUTE. In this case, as in many others, appending + information to the existing ICMP Time Expired Message is preferable + to defining a new message and emitting two messages whenever a packet + is dropped due to TTL expiration. + + + + + +Bonica, et al. Standards Track [Page 3] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + +2. Conventions Used in This Document + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in [RFC2119]. + +3. Summary of Changes to ICMP + + The following is a summary of changes to ICMP that are introduced by + this memo: + + An ICMP Extension Structure MAY be appended to ICMPv4 Destination + Unreachable, Time Exceeded, and Parameter Problem messages. + + An ICMP Extension Structure MAY be appended to ICMPv6 Destination + Unreachable, and Time Exceeded messages. + + The above mentioned messages include an "original datagram" field, + and the message formats are updated to specify a length attribute + for the "original datagram" field. + + When the ICMP Extension Structure is appended to an ICMP message + and that ICMP message contains an "original datagram" field, the + "original datagram" field MUST contain at least 128 octets. + + When the ICMP Extension Structure is appended to an ICMPv4 message + and that ICMPv4 message contains an "original datagram" field, the + "original datagram" field MUST be zero padded to the nearest + 32-bit boundary. + + When the ICMP Extension Structure is appended to an ICMPv6 message + and that ICMPv6 message contains an "original datagram" field, the + "original datagram" field MUST be zero padded to the nearest + 64-bit boundary. + + ICMP messages defined in the future SHOULD indicate whether or not + they support the extension mechanism defined in this + specification. It is recommended that all new messages support + extensions. + +4. ICMP Extensibility + + RFC 792 defines the following ICMPv4 message types: + + - Destination Unreachable + + - Time Exceeded + + + + +Bonica, et al. Standards Track [Page 4] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + + - Parameter Problem + + - Source Quench + + - Redirect + + - Echo Request/Reply + + - Timestamp/Timestamp Reply + + - Information Request/Information Reply + + [RFC1191] reserves bits for the "Next-Hop MTU" field in the + Destination Unreachable message. + + RFC 4443 defines the following ICMPv6 message types: + + - Destination Unreachable + + - Packet Too Big + + - Time Exceeded + + - Parameter Problem + + - Echo Request/Reply + + Many ICMP messages are extensible as currently defined. Protocol + designers can extend ICMP messages by simply appending fields or data + structures to them. + + However, the following ICMP messages are not extensible as currently + defined: + + - ICMPv4 Destination Unreachable (type = 3) + + - ICMPv4 Time Exceeded (type = 11) + + - ICMPv4 Parameter Problem (type = 12) + + - ICMPv6 Destination Unreachable (type = 1) + + - ICMPv6 Packet Too Big (type = 2) + + - ICMPv6 Time Exceeded (type = 3) + + - ICMPv6 Parameter Problem (type = 4) + + + + +Bonica, et al. Standards Track [Page 5] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + + These messages contain an "original datagram" field which represents + the leading octets of the datagram to which the ICMP message is a + response. RFC 792 defines the "original datagram" field for ICMPv4 + messages. In RFC 792, the "original datagram" field includes the IP + header plus the next eight octets of the original datagram. + [RFC1812] extends the "original datagram" field to contain as many + octets as possible without causing the ICMP message to exceed the + minimum IPv4 reassembly buffer size (i.e., 576 octets). RFC 4443 + defines the "original datagram" field for ICMPv6 messages. In RFC + 4443, the "original datagram" field always contained as many octets + as possible without causing the ICMP message to exceed the minimum + IPv6 MTU (i.e., 1280 octets). + + Unfortunately, the "original datagram" field lacks a length + attribute. Application software infers the length of this field from + the total length of the ICMP message. If an extension structure were + appended to the message without adding a length attribute for the + "original datagram" field, the message would become unparsable. + Specifically, application software would not be able to determine + where the "original datagram" field ends and where the extension + structure begins. + + In order to solve this problem, this memo introduces an 8-bit length + attribute to the following ICMPv4 messages. + + - Destination Unreachable (type = 3) + + - Time Exceeded (type = 11) + + - Parameter Problem (type = 12) + + It also introduces an 8-bit length attribute to the following ICMPv6 + messages. + + - Destination Unreachable (type = 1) + + - Time Exceeded (type = 3) + + The length attribute MUST be specified when the ICMP Extension + Structure is appended to the above mentioned ICMP messages. + + The length attribute represents the length of the "original datagram" + field. Space for the length attribute is claimed from reserved + octets, whose value was previously required to be zero. + + For ICMPv4 messages, the length attribute represents 32-bit words. + When the length attribute is specified, the "original datagram" field + MUST be zero padded to the nearest 32-bit boundary. Because the + + + +Bonica, et al. Standards Track [Page 6] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + + sixth octet of each of the impacted ICMPv4 messages was reserved for + future use, this octet was selected as the location of the length + attribute in ICMPv4. + + For ICMPv6 messages, the length attribute represents 64-bit words. + When the length attribute is specified, the "original datagram" field + MUST be zero padded to the nearest 64-bit boundary. Because the + fifth octet of each of the impacted ICMPv6 messages was reserved for + future use, this octet was selected as the location of the length + attribute in ICMPv6. + + In order to achieve backwards compatibility, when the ICMP Extension + Structure is appended to an ICMP message and that ICMP message + contains an "original datagram" field, the "original datagram" field + MUST contain at least 128 octets. If the original datagram did not + contain 128 octets, the "original datagram" field MUST be zero padded + to 128 octets. (See Section 5.1 for rationale.) + + The following sub-sections depict length attribute as it has been + introduced to selected ICMP messages. + +4.1. ICMPv4 Destination Unreachable + + Figure 1 depicts the ICMPv4 Destination Unreachable Message. + + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Type | Code | Checksum | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | unused | Length | Next-Hop MTU* | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Internet Header + leading octets of original datagram | + | | + | // | + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Figure 1: ICMPv4 Destination Unreachable + + The syntax and semantics of all fields are unchanged from RFC 792. + However, a length attribute is added to the second word. The length + attribute represents length of the padded "original datagram" field, + measured in 32-bit words. + + * The Next-Hop MTU field is not required in all cases. It is + depicted only to demonstrate that those bits are not available for + assignment in this memo. + + + +Bonica, et al. Standards Track [Page 7] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + +4.2. ICMPv4 Time Exceeded + + Figure 2 depicts the ICMPv4 Time Exceeded Message. + + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Type | Code | Checksum | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | unused | Length | unused | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Internet Header + leading octets of original datagram | + | | + | // | + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Figure 2: ICMPv4 Time Exceeded + + The syntax and semantics of all fields are unchanged from RFC 792, + except for a length attribute which is added to the second word. The + length attribute represents length of the padded "original datagram" + field, measured in 32-bit words. + +4.3. ICMPv4 Parameter Problem + + Figure 3 depicts the ICMPv4 Parameter Problem Message. + + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Type | Code | Checksum | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Pointer | Length | unused | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Internet Header + leading octets of original datagram | + | | + | // | + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Figure 3: ICMPv4 Parameter Problem + + The syntax and semantics of all fields are unchanged from RFC 792, + except for a length attribute which is added to the second word. The + length attribute represents length of the padded "original datagram" + field, measured in 32-bit words. + + + + +Bonica, et al. Standards Track [Page 8] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + +4.4. ICMPv6 Destination Unreachable + + Figure 4 depicts the ICMPv6 Destination Unreachable Message. + + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Type | Code | Checksum | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Length | Unused | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | As much of invoking packet | + + as possible without the ICMPv6 packet + + | exceeding the minimum IPv6 MTU [RFC4443] | + + Figure 4: ICMPv6 Destination Unreachable + + The syntax and semantics of all fields are unchanged from RFC 4443. + However, a length attribute is added to the second word. The length + attribute represents length of the padded "original datagram" field, + measured in 64-bit words. + +4.5. ICMPv6 Time Exceeded + + Figure 5 depicts the ICMPv6 Time Exceeded Message. + + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Type | Code | Checksum | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Length | Unused | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | As much of invoking packet | + + as possible without the ICMPv6 packet + + | exceeding the minimum IPv6 MTU [RFC4443] | + + Figure 5: ICMPv6 Time Exceeded + + The syntax and semantics of all fields are unchanged from RFC 4443, + except for a length attribute which is added to the second word. The + length attribute represents length of the padded "original datagram" + field, measured in 64-bit words. + + + + + + + + +Bonica, et al. Standards Track [Page 9] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + +4.6. ICMP Messages That Can Be Extended + + The ICMP Extension Structure MAY be appended to messages of the + following types: + + - ICMPv4 Destination Unreachable + + - ICMPv4 Time Exceeded + + - ICMPv4 Parameter Problem + + - ICMPv6 Destination Unreachable + + - ICMPv6 Time Exceeded + + The ICMP Extension Structure MUST NOT be appended to any of the other + ICMP messages mentioned in Section 4. Extensions were not defined + for the ICMPv6 "Packet Too Big" and "Parameter Problem" messages + because these messages lack space for a length attribute. + +5. Backwards Compatibility + + ICMP messages can be categorized as follows: + + - Messages that do not include any ICMP extensions + + - Messages that include non-compliant ICMP extensions + + - Messages that includes compliant ICMP extensions + + Any ICMP implementation can send a message that does not include + extensions. ICMP implementations produced prior to 1999 are not + known to send ICMP extensions. + + Some ICMP implementations, produced between 1999 and the time of this + publication, may send a non-compliant version of ICMP extensions + described in this memo. Specifically, these implementations may + append the ICMP Extension Structure to the Time Exceeded and + Destination Unreachable messages. When they do this, they send + exactly 128 octets representing the original datagram, zero padding + if required. They also calculate checksums as described in this + document. However, they do not specify a length attribute to be + associated with the "original datagram" field. + + It is assumed that ICMP implementations produced in the future will + send ICMP extensions that are compliant with this specification. + + + + + +Bonica, et al. Standards Track [Page 10] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + + Likewise, applications that consume ICMP messages can be categorized + as follows: + + - Classic applications + + - Non-compliant applications + + - Compliant applications + + Classic applications do not parse extensions defined in this memo. + They are insensitive to the length attribute that is associated with + the "original datagram" field. + + Non-compliant implementations parse the extensions defined in this + memo, but only in conjunction with the Time Expired and Destination + Unreachable messages. They require the "original datagram" field to + contain exactly 128 octets and are insensitive to the length + attribute that is associated with the "original datagram" field. + Non-compliant applications were produced between 1999 and the time of + publication of this memo. + + Compliant applications comply fully with the specifications of this + document. + + In order to demonstrate backwards compatibility, Table 1 describes + how members of each application category would parse each category of + ICMP message. + + +----------------+----------------+----------------+----------------+ + | | No Extensions | Non-compliant | Compliant | + | | | Extensions | Extensions | + +----------------+----------------+----------------+----------------+ + | Classic | - | Section 5.1 | Section 5.1 | + | Application | | | | + | | | | | + | Non-compliant | Section 5.2 | - | Section 5.3 | + | Application | | | | + | | | | | + | Compliant | Section 5.4 | Section 5.5 | - | + | Application | | | | + +----------------+----------------+----------------+----------------+ + + Table 1 + + In the table above, cells that contain a dash represent the nominal + case and require no explanation. In the following sections, we + assume that the ICMP message type is "Time Exceeded". + + + + +Bonica, et al. Standards Track [Page 11] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + +5.1. Classic Application Receives ICMP Message with Extensions + + When a classic application receives an ICMP message that includes + extensions, it will incorrectly interpret those extensions as being + part of the "original datagram" field. Fortunately, the extensions + are guaranteed to begin at least 128 octets beyond the beginning of + the "original datagram" field. So, only those ICMP applications that + process the 129th octet of the "original datagram" field will be + adversely effected. To date, only two applications falling into this + category have been identified, and the degree to which they are + effected is minimal. + + Some TCP stacks, when they receive an ICMP message, verify the + checksum in the original datagram field [ATTACKS]. If the checksum + is incorrect, the TCP stack discards the ICMP message for security + reasons. If the trailing octets of the original datagram field are + overwritten by ICMP extensions, the TCP stack will discard an ICMP + message that it would not otherwise have discarded. The impact of + this issue is considered to be minimal because many ICMP messages are + discarded for other reasons (e.g., ICMP filtering, network + congestion, checksum was incorrect because original datagram field + was truncated.) + + Another theoretically possible, but highly improbably scenario occurs + when ICMP extensions overwrite the portion of the original datagram + field that represents the TCP header, causing the TCP stack to + operate upon the wrong TCP connection. This scenario is highly + unlikely because it occurs only when the TCP header appears at or + beyond the 128th octet of the original datagram field and then only + when the extensions approximate a valid TCP header. + +5.2. Non-Compliant Application Receives ICMP Message with No Extensions + + When a non-compliant ICMPv4 application receives a message that + contains no extensions, the application examines the total length of + the ICMPv4 message. If the total ICMPv4 message length is less than + the length of its IP header plus 144 octets, the application + correctly determines that the message does not contain any + extensions. + + The 144-octet sum is derived from 8 octets for the first two words of + the ICMPv4 Time Exceeded message, 128 octets for the "original + datagram" field, 4 octets for the ICMP Extension Header, and 4 octets + for a single ICMP Object header. All of these octets would be + required if extensions were present. + + + + + + +Bonica, et al. Standards Track [Page 12] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + + If the ICMPv4 payload contains 144 octets or more, the application + must examine the 137th octet to determine whether it represents a + valid ICMPv4 Extension Header. In order to represent a valid + Extension Header, it must contain a valid version number and + checksum. If it does not contain a valid version number and + checksum, the application correctly determines that the message does + not contain any extensions. + + Non-compliant applications assume that the ICMPv4 Extension Structure + begins on the 137th octet of the Time Exceeded message, after a + 128-octet field representing the padded "original datagram" message. + + It is possible that a non-compliant application will parse an ICMPv4 + message incorrectly under the following conditions: + + - the message does not contain extensions + + - the original datagram field contains 144 octets or more + + - selected octets of the original datagram field represent the + correct values for an extension header version number and + checksum + + Although this is possible, it is very unlikely. + + A similar analysis can be performed for ICMPv6. However, the numeric + constants would change as appropriate. + +5.3. Non-Compliant Application Receives ICMP Message with Compliant + Extensions + + When a non-compliant application receives a message that contains + compliant ICMP extensions, it will parse those extensions correctly + only if the "original datagram" field contains exactly 128 octets. + This is because non-compliant applications are insensitive to the + length attribute that is associated with the "original datagram" + field. (They assume its value to be 128.) + + Provided that the entire ICMP message does not exceed the minimum + reassembly buffer size (576 octets for ICMPv4 or 1280 octets for + ICMPv6), there is no upper limit upon the length of the "original + datagram" field. However, each implementation will decide how many + octets to include. Those wishing to be backward compatible with non- + compliant TRACEROUTE implementations will include exactly 128 octets. + Those not requiring compatibility with non-compliant TRACEROUTE + applications may include more octets. + + + + + +Bonica, et al. Standards Track [Page 13] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + +5.4. Compliant Application Receives ICMP Message with No Extensions + + When a compliant application receives an ICMP message, it examines + the length attribute that is associated with the "original datagram" + field. If the length attribute is zero, the compliant application + MUST determine that the message contains no extensions. + +5.5. Compliant Application Receives ICMP Message with Non-Compliant + Extensions + + When a compliant application receives an ICMP message, it examines + the length attribute that is associated with the "original datagram" + field. If the length attribute is zero, the compliant application + MUST determine that the message contains no extensions. In this + case, that determination is technically correct, but not backwards + compatible with the non-compliant implementation that originated the + ICMP message. + + So, to ease transition yet encourage compliant implementation, + compliant TRACEROUTE implementations MUST include a non-default + operation mode to also interpret non-compliant responses. + Specifically, when a TRACEROUTE application operating in non- + compliant mode receives a sufficiently long ICMP message that does + not specify a length attribute, it will parse for a valid extension + header at a fixed location, assuming a 128-octet "original datagram" + field. If the application detects a valid version and checksum, it + will treat the octets that follow as an extension structure. + +6. Interaction with Network Address Translation + + The ICMP extensions defined in this memo do not interfere with + Network Address Translation. [RFC3022] permits traditional NAT + devices to modify selected fields within ICMP messages. These fields + include the "original datagram" field mentioned above. However, if a + NAT device modifies the "original datagram" field, it should modify + only the leading octets of that field, which represent the outermost + IP header. Because the outermost IP header is guaranteed to be + contained by the first 128 octets of the "original datagram" field, + ICMP extensions and NAT will not interfere with one another. + + It is conceivable that a NAT implementation might overstep the + restrictions of RFC 3022 and overwrite the length attribute specified + by this memo. If a NAT implementation were to overwrite the length + attribute with zeros, the resulting packet will be indistinguishable + from a packet that was generated by a non-compliant ICMP + implementation. See Section 5.5 for packet details and a discussion + of backwards compatibility. + + + + +Bonica, et al. Standards Track [Page 14] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + +7. The ICMP Extension Structure + + This memo proposes an optional ICMP Extension Structure that can be + appended to the ICMP messages referenced in Section 4.6 of this + document. + + The Extension Structure contains exactly one Extension Header + followed by one or more objects. Having received an ICMP message + with extensions, application software MAY process selected objects + while ignoring others. The presence of an unrecognized object does + not imply that an ICMP message is malformed. + + As stated above, the total length of the ICMP message, including + extensions, MUST NOT exceed the minimum reassembly buffer size. + Figure 6 depicts the ICMP Extension Header. + + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + |Version| (Reserved) | Checksum | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Figure 6: ICMP Extension Header + + The fields of the ICMP Extension Header are as follows: + + Version: 4 bits + + ICMP extension version number. This is version 2. + + Reserved: 12 bits + + Must be set to 0. + + Checksum: 16 bits + + The one's complement of the one's complement sum of the data + structure, with the checksum field replaced by zero for the + purpose of computing the checksum. An all-zero value means that + no checksum was transmitted. See Section 5.2 for a description of + how this field is used. + + + + + + + + + + +Bonica, et al. Standards Track [Page 15] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + +8. ICMP Extension Objects + + Each extension object contains one or more 32-bit words, representing + an object header and payload. All object headers share a common + format. Figure 7 depicts the object header and payload. + + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Length | Class-Num | C-Type | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + | // (Object payload) // | + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Figure 7: Object Header and Payload + + An object header has the following fields: + + Length: 16 bits + + Length of the object, measured in octets, including the object + header and object payload. + + Class-Num: 8 bits + + Identifies object class. + + C-Type: 8 bits + + Identifies object sub-type. + +9. Security Considerations + + Upon receipt of an ICMP message, application software must check it + for syntactic correctness. The extension checksum must be verified. + Improperly specified length attributes and other syntax problems may + result in buffer overruns. + + This memo does not define the conditions under which a router sends + an ICMP message. Therefore, it does not expose routers to any new + denial-of-service attacks. Routers may need to limit the rate at + which ICMP messages are sent. + + + + + + + +Bonica, et al. Standards Track [Page 16] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + +10. IANA Considerations + + The ICMP Extension Object header contains two 8-bit fields: The + Class-Num identifies the object class, and the C-Type identifies the + class sub-type. Sub-type values are defined relative to a specific + object class value, and are defined per class. + + IANA has established a registry of ICMP extension objects classes and + class sub-types. There are no values assigned within this document + to maintain. Object classes 0xF7 - 0xFF are reserved for private + use. Object class values are assignable on a first-come-first-serve + basis. The policy for assigning sub-type values should be defined in + the document defining new class values. + +11. Acknowledgments + + Thanks to Pekka Nikander, Mark Doll, Fernando Gont, Joe Touch, + Christian Voiqt, and Sharon Chrisholm for their comments regarding + this document. + +12. References + +12.1. Normative References + + [RFC0792] Postel, J., "Internet Control Message Protocol", STD + 5, RFC 792, September 1981. + + [RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", RFC + 1191, November 1990. + + [RFC1812] Baker, F., "Requirements for IP Version 4 Routers", + RFC 1812, June 1995. + + [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate + Requirement Levels", BCP 14, RFC 2119, March 1997. + + [RFC4443] Conta, A., Deering, S., and M. Gupta, Ed., "Internet + Control Message Protocol (ICMPv6) for the Internet + Protocol Version 6 (IPv6) Specification", RFC 4443, + March 2006. + +12.2. Informative References + + [UNNUMBERED] Atlas, A., Bonica, R., Rivers, JR., Shen, N., and E. + Chen, "ICMP Extensions for Unnumbered Interfaces", + Work in Progress, March 2007. + + + + + +Bonica, et al. Standards Track [Page 17] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + + [MPLS-ICMP] Bonica, R., Gan, D., Tappan, D., and C. Pignataro, + "ICMP Extensions for MultiProtocol Label Switching", + Work in Progress, January 2007. + + [ATTACKS] Gont, F., "ICMP attacks against TCP", Work in + Progress, October 2006. + + [ROUTING-INST] Shen, N. and E. Chen, "ICMP Extensions for Routing + Instances", Work in Progress, November 2006. + + [RFC3022] Srisuresh, P. and K. Egevang, "Traditional IP Network + Address Translator (Traditional NAT)", RFC 3022, + January 2001. + +Authors' Addresses + + Ronald P. Bonica + Juniper Networks + 2251 Corporate Park Drive + Herndon, VA 20171 + US + + EMail: rbonica@juniper.net + + + Der-Hwa Gan + Consultant + + EMail: derhwagan@yahoo.com + + + Daniel C. Tappan + Consultant + + EMail: Dan.Tappan@gmail.com + + + Carlos Pignataro + Cisco Systems, Inc. + 7025 Kit Creek Road + Research Triangle Park, NC 27709 + US + + EMail: cpignata@cisco.com + + + + + + + +Bonica, et al. Standards Track [Page 18] + +RFC 4884 Multi-Part ICMP Messages April 2007 + + +Full Copyright Statement + + Copyright (C) The IETF Trust (2007). + + This document is subject to the rights, licenses and restrictions + contained in BCP 78, and except as set forth therein, the authors + retain all their rights. + + This document and the information contained herein are provided on an + "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS + OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND + THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS + OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF + THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED + WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. + +Intellectual Property + + The IETF takes no position regarding the validity or scope of any + Intellectual Property Rights or other rights that might be claimed to + pertain to the implementation or use of the technology described in + this document or the extent to which any license under such rights + might or might not be available; nor does it represent that it has + made any independent effort to identify any such rights. Information + on the procedures with respect to rights in RFC documents can be + found in BCP 78 and BCP 79. + + Copies of IPR disclosures made to the IETF Secretariat and any + assurances of licenses to be made available, or the result of an + attempt made to obtain a general license or permission for the use of + such proprietary rights by implementers or users of this + specification can be obtained from the IETF on-line IPR repository at + http://www.ietf.org/ipr. + + The IETF invites any interested party to bring to its attention any + copyrights, patents or patent applications, or other proprietary + rights that may cover technology that may be required to implement + this standard. Please address the information to the IETF at + ietf-ipr@ietf.org. + +Acknowledgement + + Funding for the RFC Editor function is currently provided by the + Internet Society. + + + + + + + +Bonica, et al. Standards Track [Page 19] + diff --git a/routing/src/cli/display.rs b/routing/src/cli/display.rs index e400c9793e..38886aba5b 100644 --- a/routing/src/cli/display.rs +++ b/routing/src/cli/display.rs @@ -46,7 +46,7 @@ use std::time::Instant; use tracing::{error, warn}; -//================================= Common ==========================// +// ================================= Common ==========================// fn fmt_opt_value( f: &mut std::fmt::Formatter<'_>, name: &str, @@ -60,7 +60,7 @@ fn fmt_opt_value( if nl { writeln!(f) } else { Ok(()) } } -//========================= Encapsulations ==========================// +// ========================= Encapsulations ==========================// impl Display for VxlanEncapsulation { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { write!( @@ -83,7 +83,7 @@ impl Display for Encapsulation { } } -//=================== VRFs, routes and next-hops ====================// +// =================== VRFs, routes and next-hops ====================// impl Display for RouteOrigin { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { match self { @@ -499,7 +499,7 @@ impl Display for VrfTable { } } -//========================= Interfaces ================================// +// ========================= Interfaces ================================// impl Display for Attachment { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { @@ -602,7 +602,7 @@ impl Display for IfTable { Ok(()) } } -//========================= Interface addresses ================================// +// ========================= Interface addresses ================================// #[repr(transparent)] pub struct IfTableAddress<'a>(pub &'a IfTable); @@ -640,7 +640,7 @@ impl Display for IfTableAddress<'_> { } } -//========================= Rmac Store ================================// +// ========================= Rmac Store ================================// macro_rules! RMAC_TBL_FMT { () => { " {:<5} {:<20} {:<18} {:<8}" @@ -687,7 +687,7 @@ impl Display for RmacStore { } } -//========================= Rmac Store ================================// +// ========================= Rmac Store ================================// impl Display for Vtep { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { Heading("Local VTEP configuration").fmt(f)?; @@ -696,7 +696,7 @@ impl Display for Vtep { } } -//========================= Adjacencies ================================// +// ========================= Adjacencies ================================// macro_rules! ADJ_TBL_FMT { () => { " {:<10} {:<20} {:<18}" @@ -735,7 +735,7 @@ impl Display for AdjacencyTable { } } -//========================= Fib ================================// +// ========================= Fib ================================// impl Display for FibKey { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> { match self { @@ -930,7 +930,7 @@ impl Display for FibGroups<'_> { } } -//========================= Time utils =========================// +// ========================= Time utils =========================// use chrono::Local; pub(crate) fn fmt_time(time: &DateTime) -> String { //let fmt_iso8 = "%Y-%m-%dT%H:%M:%S%.3f%:z"; @@ -962,7 +962,7 @@ pub(crate) fn fmt_time(time: &DateTime) -> String { out } -//========================= CPI ================================// +// ========================= CPI ================================// macro_rules! STATS_ROW_FMT { () => { " {:<16} {:<12} {:<12} {:<12} {:<12} {:<12}" @@ -1046,7 +1046,7 @@ impl Display for CpiStats { } } -//========================= Frrmi ================================// +// ========================= Frrmi ================================// impl Display for FrrmiStats { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let last_conn_time = &self From a427e0cf2b14b4fd2a4779527b6b7fcbfafaa521 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 21:16:05 -0600 Subject: [PATCH 19/37] build(duvet): Track RFC 5382, and record where masquerade departs from it RFC 5382 states ten numbered requirements for how a NAT must treat TCP. It constrains values this codebase already has and chose without reference to it. Four are now cited; the other six are visible in the report as uncovered, which is the point of tracking it. ## Conformant, with a test * REQ-7, no port overloading. The allocator's bitmaps enforce it and the exclusivity property in `masquerade::fuzz` asserts it -- "two live flows never share a translation" was written before anyone read RFC 5382, and turns out to be exactly REQ-7. * REQ-10, an ICMP message must not terminate the mapping. Held by construction: no arm of `next_flow_status_icmp` yields `Closed` or `Reset`. The test that says so was written yesterday for other reasons. Both are cited rather than changed. The value is that they are now *claims* somebody can check, instead of accidents. ## Marked todo, because they are decisions rather than defects * REQ-5, idle timeouts. "Established connection idle-timeout" MUST NOT be under 2 hours 4 minutes; "transitory" MUST NOT be under 4 minutes. Ours are 5, 3 and 2 seconds for the transitory states, and the established timeout is `idle_timeout` from the masquerade config, which has no default, no bound and no validation. The short values look deliberate -- `protocol.rs` says the statuses exist "to know how much to extend the lifetime of flows for port conservation", and a gateway holding a public port for two hours per idle connection conserves nothing. But whether we are willing to state that as a deviation from a BCP is a product decision. * REQ-1, endpoint-independent mapping. Not held as stated: the allocation depends on `dst_vpcd`, so one internal endpoint reaching two destination VPCs can be given two public tuples. RFC 5382 assumes a NAT facing a single external realm; here destination VPCs are distinct address spaces behind distinct peerings, and sharing a pool across them would be the surprising choice. So this is probably an exception rather than a defect -- and "probably" is why it is `todo`. Answering it is cheap now; discovering it mattered after a peer-to-peer application fails is not. `todo` rather than `exception` in both cases deliberately. duvet has both, and an exception asserts a decision was taken. Neither of these was. Converting them needs a rationale somebody is willing to sign, and the report is where that queue lives. ## What was checked and found fine REQ-2 requires handling the TCP simultaneous-open. Traced through the state machine: the flow sits in `OneWay` while the two SYNs cross, then the peer's SYN-ACK moves it to `TwoWay` and the ACK to `Established`. It works. Not cited, because sitting in `OneWay` for two extra round trips interacts with the REQ-5 timeouts above, and citing it as conformant would overstate what was verified. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Daniel Noland (cherry picked from commit 46b4c3294363d00cc560b931586dd145f132cfa6) --- .duvet/config.toml | 6 + .../rfc/rfc5382/section-4.1.toml | 50 + .../rfc/rfc5382/section-4.2.toml | 75 ++ .../rfc/rfc5382/section-4.3.toml | 200 +++ .../rfc/rfc5382/section-5.toml | 147 ++ .../rfc/rfc5382/section-6.toml | 28 + .../rfc/rfc5382/section-7.1.toml | 37 + .../rfc/rfc5382/section-7.2.toml | 42 + .../rfc/rfc5382/section-7.3.toml | 44 + .../rfc/rfc5382/section-8.toml | 228 ++++ .duvet/snapshot.txt | 124 ++ .../www.rfc-editor.org/rfc/rfc5382.txt | 1179 +++++++++++++++++ nat/src/masquerade/apalloc/mod.rs | 21 + nat/src/masquerade/fuzz.rs | 4 + nat/src/masquerade/nf.rs | 22 + nat/src/masquerade/protocol.rs | 6 + nat/src/masquerade/state_machine.rs | 4 + 17 files changed, 2217 insertions(+) create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-4.1.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-4.2.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-4.3.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-5.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-6.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-7.1.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-7.2.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-7.3.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-8.toml create mode 100644 .duvet/specifications/www.rfc-editor.org/rfc/rfc5382.txt diff --git a/.duvet/config.toml b/.duvet/config.toml index 8b5c8ae60c..7261f4894a 100644 --- a/.duvet/config.toml +++ b/.duvet/config.toml @@ -20,6 +20,12 @@ pattern = "*/src/**/*.rs" [[specification]] source = "https://www.rfc-editor.org/rfc/rfc4884" +# RFC 5382 states 22 numbered requirements for how a NAT must treat TCP. It constrains values this +# codebase already has and chose without reference to it -- notably the idle timeouts in +# nat/src/masquerade/nf.rs. Tracked second for that reason. +[[specification]] +source = "https://www.rfc-editor.org/rfc/rfc5382" + [report.html] enabled = true diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-4.1.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-4.1.toml new file mode 100644 index 0000000000..03d4e344c8 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-4.1.toml @@ -0,0 +1,50 @@ +target = "https://www.rfc-editor.org/rfc/rfc5382#section-4.1" + +# Address and Port Mapping Behavior +# +# A NAT uses a mapping to translate packets for each TCP connection. A +# mapping is dynamically allocated for connections initiated from the +# internal side, and potentially reused for certain subsequent +# connections. NAT behavior regarding when a mapping can be reused +# differs for different NATs as described in [BEHAVE-UDP]. +# +# Consider an internal IP address and TCP port (X:x) that initiates a +# TCP connection to an external (Y1:y1) tuple. Let the mapping +# allocated by the NAT for this connection be (X1':x1'). Shortly +# thereafter, the endpoint initiates a connection from the same (X:x) +# to an external address (Y2:y2) and gets the mapping (X2':x2') on the +# NAT. As per [BEHAVE-UDP], if (X1':x1') equals (X2':x2') for all +# values of (Y2:y2), then the NAT is defined to have "Endpoint- +# Independent Mapping" behavior. If (X1':x1') equals (X2':x2') only +# when Y2 equals Y1, then the NAT is defined to have "Address-Dependent +# Mapping" behavior. If (X1':x1') equals (X2':x2') only when (Y2:y2) +# equals (Y1:y1), possible only for consecutive connections to the same +# external address shortly after the first is terminated and if the NAT +# retains state for connections in TIME_WAIT state, then the NAT is +# defined to have "Address and Port-Dependent Mapping" behavior. This +# document introduces one additional behavior where (X1':x1') never +# equals (X2':x2'), that is, for each connection a new mapping is +# allocated; in such a case, the NAT is defined to have "Connection- +# Dependent Mapping" behavior. +# +# REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior +# for TCP. +# +# Justification: REQ-1 is necessary for UNSAF methods to work. +# Endpoint-Independent Mapping behavior allows peer-to-peer +# applications to learn and advertise the external IP address and +# port allocated to an internal endpoint such that external peers +# can contact it (subject to the NAT's security policy). The +# security policy of a NAT is independent of its mapping behavior +# and is discussed later in Section 4.3. Having Endpoint- +# Independent Mapping behavior allows peer-to-peer applications to +# work consistently without compromising the security benefits of +# the NAT. + +[[spec]] +level = "MUST" +quote = ''' +REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior +for TCP. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-4.2.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-4.2.toml new file mode 100644 index 0000000000..98e84ac415 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-4.2.toml @@ -0,0 +1,75 @@ +target = "https://www.rfc-editor.org/rfc/rfc5382#section-4.2" + +# Internally Initiated Connections +# +# An internal endpoint initiates a TCP connection through a NAT by +# sending a SYN packet. The NAT allocates (or reuses) a mapping for +# the connection, as described in the previous section. The mapping +# defines the external IP address and port used for translation of all +# packets for that connection. In particular, for client-server +# +# applications where an internal client initiates the connection to an +# external server, the mapping is used to translate the outbound SYN, +# the resulting inbound SYN-ACK response, the subsequent outbound ACK, +# and other packets for the connection. This method of connection +# initiation corresponds to the 3-way handshake (defined in [RFC0793]) +# and is supported by all NATs. +# +# Peer-to-peer applications use an alternate method of connection +# initiation termed simultaneous-open (Fig. 8, [RFC0793]) to traverse +# NATs. In the simultaneous-open mode of operation, both peers send +# SYN packets for the same TCP connection. The SYN packets cross in +# the network. Upon receiving the other end's SYN packet, each end +# responds with a SYN-ACK packet, which also cross in the network. The +# connection is considered established once the SYN-ACKs are received. +# From the perspective of the NAT, the internal host's SYN packet is +# met by an inbound SYN packet for the same connection (as opposed to a +# SYN-ACK packet during a 3-way handshake). Subsequent to this +# exchange, both an outbound and an inbound SYN-ACK are seen for the +# connection. Some NATs erroneously block the inbound SYN for the +# connection in progress. Some NATs block or incorrectly translate the +# outbound SYN-ACK. Such behavior breaks TCP simultaneous-open and +# prevents peer-to-peer applications from functioning correctly behind +# a NAT. +# +# In order to provide network address translation service for TCP, it +# is necessary for a NAT to correctly receive, translate, and forward +# all packets for a connection that conform to valid transitions of the +# TCP State-Machine (Fig. 6, [RFC0793]). +# +# REQ-2: A NAT MUST support all valid sequences of TCP packets +# (defined in [RFC0793]) for connections initiated both internally +# as well as externally when the connection is permitted by the NAT. +# In particular: +# a) In addition to handling the TCP 3-way handshake mode of +# connection initiation, A NAT MUST handle the TCP simultaneous- +# open mode of connection initiation. +# +# Justification: The intent of this requirement is to allow standards +# compliant TCP stacks to traverse NATs no matter what path the +# stacks take through the TCP state-machine and no matter which end +# initiates the connection as long as the connection is permitted by +# the filtering policy of the NAT (filtering policy is described in +# the following section). +# a) In addition to TCP packets for a 3-way handshake, A NAT must be +# prepared to accept an inbound SYN and an outbound SYN-ACK for +# an internally initiated connection in order to support +# simultaneous-open. + +[[spec]] +level = "MUST" +quote = ''' +REQ-2: A NAT MUST support all valid sequences of TCP packets +(defined in [RFC0793]) for connections initiated both internally +as well as externally when the connection is permitted by the NAT. +''' + +[[spec]] +level = "MUST" +quote = ''' +In particular: +a) In addition to handling the TCP 3-way handshake mode of +connection initiation, A NAT MUST handle the TCP simultaneous- +open mode of connection initiation. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-4.3.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-4.3.toml new file mode 100644 index 0000000000..016ad29199 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-4.3.toml @@ -0,0 +1,200 @@ +target = "https://www.rfc-editor.org/rfc/rfc5382#section-4.3" + +# Externally Initiated Connections +# +# The NAT allocates a mapping for the first connection initiated by an +# internal endpoint to an external endpoint. In some scenarios, the +# NAT's policy may allow this mapping to be reused for connections +# initiated from the external side to the internal endpoint. Consider +# as before an internal IP address and port (X:x) that is assigned (or +# reuses) a mapping (X1':x1') when it initiates a connection to an +# external (Y1:y1). An external endpoint (Y2:y2) attempts to initiate +# a connection with the internal endpoint by sending a SYN to +# (X1':x1'). A NAT can choose to either allow the connection to be +# established, or to disallow the connection. If the NAT chooses to +# allow the connection, it translates the inbound SYN and routes it to +# (X:x) as per the existing mapping. It also translates the SYN-ACK +# generated by (X:x) in response and routes it to (Y2:y2), and so on. +# Alternately, the NAT can disallow the connection by filtering the +# inbound SYN. +# +# A NAT may allow an existing mapping to be reused by an externally +# initiated connection if its security policy permits. Several +# different policies are possible as described in [BEHAVE-UDP]. If a +# NAT allows the connection initiation from all (Y2:y2), then it is +# defined to have "Endpoint-Independent Filtering" behavior. If the +# NAT allows connection initiations only when Y2 equals Y1, then the +# NAT is defined to have "Address-Dependent Filtering" behavior. If +# the NAT allows connection initiations only when (Y2:y2) equals +# (Y1:y1), then the NAT is defined to have "Address and Port-Dependent +# Filtering" behavior (possible only shortly after the first connection +# has been terminated but the mapping is still active). One additional +# filtering behavior defined in this document is when the NAT does not +# allow any connection initiations from the external side; in such +# cases, the NAT is defined to have "Connection-Dependent Filtering" +# behavior. The difference between "Address and Port-Dependent +# Filtering" and "Connection-Dependent Filtering" behavior is that the +# former permits an inbound SYN during the TIME_WAIT state of the first +# connection to initiate a new connection while the latter does not. +# +# REQ-3: If application transparency is most important, it is +# RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" +# behavior for TCP. If a more stringent filtering behavior is most +# important, it is RECOMMENDED that a NAT have an "Address-Dependent +# Filtering" behavior. +# a) The filtering behavior MAY be an option configurable by the +# administrator of the NAT. +# b) The filtering behavior for TCP MAY be independent of the +# filtering behavior for UDP. +# +# Justification: The intent of this requirement is to allow peer-to- +# peer applications that do not always initiate connections from the +# internal side of the NAT to continue to work in the presence of +# NATs. This behavior also allows applications behind a BEHAVE +# compliant NAT to inter-operate with remote endpoints that are +# behind non-BEHAVE compliant (legacy) NATs. If the remote +# endpoint's NAT does not have Endpoint-Independent Mapping behavior +# but has only one external IP address, then an application can +# still traverse the combination of the two NATs if the local NAT +# has Address-Dependent Filtering. Section 9 contains a detailed +# discussion on the security implications of this requirement. +# +# If the inbound SYN packet is filtered, either because a corresponding +# mapping does not exist or because of the NAT's filtering behavior, a +# NAT has two basic choices: to ignore the packet silently, or to +# signal an error to the sender. Signaling an error through ICMP +# messages allows the sender to quickly detect that the SYN did not +# reach the intended destination. Silently dropping the packet, on the +# other hand, allows applications to perform simultaneous-open more +# reliably. +# +# Silently dropping the SYN aids simultaneous-open as follows. +# Consider that the application is attempting a simultaneous-open and +# the outbound SYN from the internal endpoint has not yet crossed the +# NAT (due to network congestion or clock skew between the two +# endpoints); this outbound SYN would otherwise have created the +# necessary mapping at the NAT to allow translation of the inbound SYN. +# Since the outbound SYN did not reach the NAT in time, the inbound SYN +# cannot be processed. If a NAT responds to the premature inbound SYN +# with an error message that forces the external endpoint to abandon +# the connection attempt, it hinders applications performing a TCP +# simultaneous-open. If instead the NAT silently ignores the inbound +# SYN, the external endpoint retransmits the SYN after a TCP timeout. +# In the meantime, the NAT creates the mapping in response to the +# (delayed) outbound SYN such that the retransmitted inbound SYN can be +# routed and simultaneous-open can succeed. The downside to this +# behavior is that in the event the inbound SYN is erroneous, the +# remote side does not learn of the error until after several TCP +# timeouts. +# +# NAT support for simultaneous-open as well as quickly signaling errors +# are both important for applications. Unfortunately, there is no way +# for a NAT to signal an error without forcing the endpoint to abort a +# potential simultaneous-open: TCP RST and ICMP Port Unreachable +# packets require the endpoint to abort the attempt while the ICMP Host +# and Network Unreachable errors may adversely affect other connections +# to the same host or network [RFC1122]. +# +# In addition, when an unsolicited SYN is received by the NAT, the NAT +# may not know whether the application is attempting a simultaneous- +# open (and that it should therefore silently drop the SYN) or whether +# the SYN is in error (and that it should notify the sender). +# +# REQ-4: A NAT MUST NOT respond to an unsolicited inbound SYN packet +# for at least 6 seconds after the packet is received. If during +# this interval the NAT receives and translates an outbound SYN for +# the connection the NAT MUST silently drop the original unsolicited +# inbound SYN packet. Otherwise, the NAT SHOULD send an ICMP Port +# Unreachable error (Type 3, Code 3) for the original SYN, unless +# REQ-4a applies. +# a) The NAT MUST silently drop the original SYN packet if sending a +# response violates the security policy of the NAT. +# +# Justification: The intent of this requirement is to allow +# simultaneous-open to work reliably in the presence of NATs as well +# as to quickly signal an error in case the unsolicited SYN is in +# error. As of writing this memo, it is not possible to achieve +# both; the requirement therefore represents a compromise. The NAT +# should tolerate some delay in the outbound SYN for a TCP +# simultaneous-open, which may be due to network congestion or loose +# synchronization between the endpoints. If the unsolicited SYN is +# not part of a simultaneous-open attempt and is in error, the NAT +# should endeavor to signal the error in accordance with [RFC1122]. +# a) There may, however, be reasons for the NAT to rate-limit or +# omit such error notifications, for example, in the case of an +# attack. Silently dropping the SYN packet when under attack +# allows simultaneous-open to work without consuming any extra +# network bandwidth or revealing the presence of the NAT to +# attackers. Section 9 mentions the security considerations for +# this requirement. +# +# For NATs that combine NAT functionality with end-host functionality +# (e.g., an end-host that also serves as a NAT for other hosts behind +# it), REQ-4 above applies only to SYNs intended for the NAT'ed hosts +# and not to SYNs intended for the NAT itself. One way to determine +# whether the inbound SYN is intended for a NAT'ed host is to allocate +# NAT mappings from one port range, and allocate ports for local +# endpoints from a different non-overlapping port range. More dynamic +# implementations can be imagined. + +[[spec]] +level = "SHOULD" +quote = ''' +REQ-3: If application transparency is most important, it is +RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" +behavior for TCP. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +If a more stringent filtering behavior is most +important, it is RECOMMENDED that a NAT have an "Address-Dependent +Filtering" behavior. +''' + +[[spec]] +level = "MAY" +quote = ''' +a) The filtering behavior MAY be an option configurable by the +administrator of the NAT. +''' + +[[spec]] +level = "MAY" +quote = ''' +b) The filtering behavior for TCP MAY be independent of the +filtering behavior for UDP. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-4: A NAT MUST NOT respond to an unsolicited inbound SYN packet +for at least 6 seconds after the packet is received. +''' + +[[spec]] +level = "MUST" +quote = ''' +If during +this interval the NAT receives and translates an outbound SYN for +the connection the NAT MUST silently drop the original unsolicited +inbound SYN packet. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +Otherwise, the NAT SHOULD send an ICMP Port +Unreachable error (Type 3, Code 3) for the original SYN, unless +REQ-4a applies. +''' + +[[spec]] +level = "MUST" +quote = ''' +a) The NAT MUST silently drop the original SYN packet if sending a +response violates the security policy of the NAT. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-5.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-5.toml new file mode 100644 index 0000000000..981ea9c711 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-5.toml @@ -0,0 +1,147 @@ +target = "https://www.rfc-editor.org/rfc/rfc5382#section-5" + +# NAT Session Refresh +# +# A NAT maintains state associated with in-progress and established +# connections. Because of this, a NAT is susceptible to a resource- +# exhaustion attack whereby an attacker (or virus) on the internal side +# attempts to cause the NAT to create more state than for which it has +# resources. To prevent such an attack, a NAT needs to abandon +# sessions in order to free the state resources. +# +# A common method that is applicable only to TCP is to preferentially +# abandon sessions for crashed endpoints, followed by closed TCP +# connections and partially open connections. A NAT can check if an +# endpoint for a session has crashed by sending a TCP keep-alive packet +# and receiving a TCP RST packet in response. If the NAT cannot +# determine whether the endpoint is active, it should not abandon the +# session until the TCP connection has been idle for some time. Note +# that an established TCP connection can stay idle (but live) +# indefinitely; hence, there is no fixed value for an idle-timeout that +# accommodates all applications. However, a large idle-timeout +# motivated by recommendations in [RFC1122] can reduce the chances of +# abandoning a live session. +# +# A TCP connection passes through three phases: partially open, +# established, and closing. During the partially open phase, endpoints +# synchronize initial sequence numbers. The phase is initiated by the +# first SYN for the connection and extends until both endpoints have +# sent a packet with the ACK flag set (TCP states: SYN_SENT and +# SYN_RCVD). ACKs in both directions mark the beginning of the +# established phase where application data can be exchanged +# indefinitely (TCP states: ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2, and +# CLOSE_WAIT). The closing phase begins when both endpoints have +# terminated their half of the connection by sending a FIN packet. +# Once FIN packets are seen in both directions, application data can no +# longer be exchanged, but the stacks still need to ensure that the FIN +# packets are received (TCP states: CLOSING and LAST_ACK). +# +# TCP connections can stay in established phase indefinitely without +# exchanging any packets. Some end-hosts can be configured to send +# keep-alive packets on such idle connections; by default, such keep- +# alive packets are sent every 2 hours if enabled [RFC1122]. +# Consequently, a NAT that waits for slightly over 2 hours can detect +# idle connections with keep-alive packets being sent at the default +# rate. TCP connections in the partially open or closing phases, on +# the other hand, can stay idle for at most 4 minutes while waiting for +# in-flight packets to be delivered [RFC1122]. +# +# The "established connection idle-timeout" for a NAT is defined as the +# minimum time a TCP connection in the established phase must remain +# idle before the NAT considers the associated session a candidate for +# removal. The "transitory connection idle-timeout" for a NAT is +# defined as the minimum time a TCP connection in the partially open or +# closing phases must remain idle before the NAT considers the +# associated session a candidate for removal. TCP connections in the +# TIME_WAIT state are not affected by the "transitory connection idle- +# timeout". +# +# REQ-5: If a NAT cannot determine whether the endpoints of a TCP +# connection are active, it MAY abandon the session if it has been +# idle for some time. In such cases, the value of the "established +# connection idle-timeout" MUST NOT be less than 2 hours 4 minutes. +# The value of the "transitory connection idle-timeout" MUST NOT be +# less than 4 minutes. +# a) The value of the NAT idle-timeouts MAY be configurable. +# +# Justification: The intent of this requirement is to minimize the +# cases where a NAT abandons session state for a live connection. +# While some NATs may choose to abandon sessions reactively in +# response to new connection initiations (allowing idle connections +# to stay up indefinitely in the absence of new initiations), other +# NATs may choose to proactively reap idle sessions. In cases where +# the NAT cannot actively determine if the connection is alive, this +# requirement ensures that applications can send keep-alive packets +# at the default rate (every 2 hours) such that the NAT can +# passively determine that the connection is alive. The additional +# 4 minutes allows time for in-flight packets to cross the NAT. +# +# NAT behavior for handling RST packets, or connections in TIME_WAIT +# state is left unspecified. A NAT MAY hold state for a connection in +# TIME_WAIT state to accommodate retransmissions of the last ACK. +# However, since the TIME_WAIT state is commonly encountered by +# internal endpoints properly closing the TCP connection, holding state +# for a closed connection may limit the throughput of connections +# through a NAT with limited resources. [RFC1337] describes hazards +# associated with TIME_WAIT assassination. +# +# The handling of non-SYN packets for connections for which there is no +# active mapping is left unspecified. Such packets may be received if +# the NAT silently abandons a live connection, or abandons a connection +# in TIME_WAIT state before the 4 minute TIME_WAIT period expires. The +# decision to either silently drop such packets or to respond with a +# TCP RST packet is left up to the implementation. +# +# NAT behavior for notifying endpoints when abandoning live connections +# is left unspecified. When a NAT abandons a live connection, for +# example due to a timeout expiring, the NAT MAY either send TCP RST +# packets to the endpoints or MAY silently abandon the connection. +# +# Sending a RST notification allows endpoint applications to recover +# more quickly; however, notifying the endpoints may not always be +# possible if, for example, session state is lost due to a power +# failure. + +[[spec]] +level = "MAY" +quote = ''' +REQ-5: If a NAT cannot determine whether the endpoints of a TCP +connection are active, it MAY abandon the session if it has been +idle for some time. +''' + +[[spec]] +level = "MUST" +quote = ''' +In such cases, the value of the "established +connection idle-timeout" MUST NOT be less than 2 hours 4 minutes. +''' + +[[spec]] +level = "MUST" +quote = ''' +The value of the "transitory connection idle-timeout" MUST NOT be +less than 4 minutes. +''' + +[[spec]] +level = "MAY" +quote = ''' +a) The value of the NAT idle-timeouts MAY be configurable. +''' + +[[spec]] +level = "MAY" +quote = ''' +A NAT MAY hold state for a connection in +TIME_WAIT state to accommodate retransmissions of the last ACK. +''' + +[[spec]] +level = "MAY" +quote = ''' +When a NAT abandons a live connection, for +example due to a timeout expiring, the NAT MAY either send TCP RST +packets to the endpoints or MAY silently abandon the connection. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-6.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-6.toml new file mode 100644 index 0000000000..6eb7cf10aa --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-6.toml @@ -0,0 +1,28 @@ +target = "https://www.rfc-editor.org/rfc/rfc5382#section-6" + +# Application Level Gateways +# +# Application Level Gateways (ALGs) in certain NATs modify IP addresses +# and TCP ports embedded inside application protocols. Such ALGs may +# interfere with UNSAF methods or protocols that try to be NAT-aware +# and must therefore be used with extreme caution. +# +# REQ-6: If a NAT includes ALGs that affect TCP, it is RECOMMENDED +# that all of those ALGs (except for FTP [RFC0959]) be disabled by +# default. +# +# Justification: The intent of this requirement is to prevent ALGs +# from interfering with UNSAF methods. The default state of an FTP +# ALG is left unspecified because of legacy concerns: as of writing +# this memo, a large fraction of legacy FTP clients do not enable +# passive (PASV) mode by default and require an ALG to traverse +# NATs. + +[[spec]] +level = "SHOULD" +quote = ''' +REQ-6: If a NAT includes ALGs that affect TCP, it is RECOMMENDED +that all of those ALGs (except for FTP [RFC0959]) be disabled by +default. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-7.1.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-7.1.toml new file mode 100644 index 0000000000..e1e74408a8 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-7.1.toml @@ -0,0 +1,37 @@ +target = "https://www.rfc-editor.org/rfc/rfc5382#section-7.1" + +# Port Assignment +# +# NATs that allow different internal endpoints to simultaneously use +# the same mapping are defined in [BEHAVE-UDP] to have a "Port +# assignment" behavior of "Port overloading". Such behavior is +# undesirable, as it prevents two internal endpoints sharing the same +# mapping from establishing simultaneous connections to a common +# external endpoint. +# +# REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port +# overloading" for TCP. +# +# Justification: This requirement allows two applications on the +# internal side of the NAT to consistently communicate with the same +# destination. +# +# NAT behavior for preserving the source TCP port range for connections +# is left unspecified. Some applications expect the source TCP port to +# be in the well-known range (TCP ports from 0 to 1023). The "r" +# series of commands (rsh, rcp, rlogin, etc.) are an example. NATs +# that preserve the range from which the source port is picked allow +# such applications to function properly through the NAT; however, by +# doing so the NAT may compromise the security of the application in +# certain situations; applications that depend only on the IP address +# and source TCP port range for security (the "r" commands, for +# example) cannot distinguish between an attacker and a legitimate user +# behind the same NAT. + +[[spec]] +level = "MUST" +quote = ''' +REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port +overloading" for TCP. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-7.2.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-7.2.toml new file mode 100644 index 0000000000..8b618f773c --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-7.2.toml @@ -0,0 +1,42 @@ +target = "https://www.rfc-editor.org/rfc/rfc5382#section-7.2" + +# Hairpinning Behavior +# +# NATs that forward packets originating from an internal address, +# destined for an external address that matches the active mapping for +# an internal address, back to that internal address are defined in +# [BEHAVE-UDP] as supporting "hairpinning". If the NAT presents the +# hairpinned packet with an external source IP address and port (i.e., +# the mapped source address and port of the originating internal +# endpoint), then it is defined to have "External source IP address and +# port" for hairpinning. Hairpinning is necessary to allow two +# internal endpoints (known to each other only by their external mapped +# addresses) to communicate with each other. "External source IP +# address and port" behavior for hairpinning avoids confusing +# implementations that expect the external source IP address and port. +# +# REQ-8: A NAT MUST support "hairpinning" for TCP. +# a) A NAT's hairpinning behavior MUST be of type "External source +# IP address and port". +# +# Justification: This requirement allows two applications behind the +# same NAT that are trying to communicate with each other using +# their external addresses. +# a) Using the external source address and port for the hairpinned +# packet is necessary for applications that do not expect to +# receive a packet from a different address than the external +# address they are trying to communicate with. + +[[spec]] +level = "MUST" +quote = ''' +REQ-8: A NAT MUST support "hairpinning" for TCP. +''' + +[[spec]] +level = "MUST" +quote = ''' +a) A NAT's hairpinning behavior MUST be of type "External source +IP address and port". +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-7.3.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-7.3.toml new file mode 100644 index 0000000000..187edd01a7 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-7.3.toml @@ -0,0 +1,44 @@ +target = "https://www.rfc-editor.org/rfc/rfc5382#section-7.3" + +# ICMP Responses to TCP Packets +# +# Several TCP mechanisms depend on the reception of ICMP error messages +# triggered by the transmission of TCP segments. One such mechanism is +# path MTU discovery [RFC1191], which is required for the correct +# +# operation of TCP. The current path MTU discovery mechanism requires +# the sender of TCP segments to be notified of ICMP "Datagram Too Big" +# responses. +# +# REQ-9: If a NAT translates TCP, it SHOULD translate ICMP Destination +# Unreachable (Type 3) messages. +# +# Justification: Translating ICMP Destination Unreachable messages, +# particularly the "Fragmentation Needed and Don't Fragment was Set" +# (Type 3, Code 4) message avoids communication failures ("black +# holes" [RFC2923]). Furthermore, TCP's connection establishment +# and maintenance mechanisms also behave much more efficiently when +# ICMP Destination Unreachable messages arrive in response to +# outgoing TCP segments. +# +# REQ-10: Receipt of any sort of ICMP message MUST NOT terminate the +# NAT mapping or TCP connection for which the ICMP was generated. +# +# Justification: This is necessary for reliably performing TCP +# simultaneous-open where a remote NAT may temporarily signal an +# ICMP error. + +[[spec]] +level = "SHOULD" +quote = ''' +REQ-9: If a NAT translates TCP, it SHOULD translate ICMP Destination +Unreachable (Type 3) messages. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-10: Receipt of any sort of ICMP message MUST NOT terminate the +NAT mapping or TCP connection for which the ICMP was generated. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-8.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-8.toml new file mode 100644 index 0000000000..9c4437acf9 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc5382/section-8.toml @@ -0,0 +1,228 @@ +target = "https://www.rfc-editor.org/rfc/rfc5382#section-8" + +# Requirements +# +# A NAT that supports all of the mandatory requirements of this +# specification (i.e., the "MUST") and is compliant with [BEHAVE-UDP], +# is "compliant with this specification". A NAT that supports all of +# the requirements of this specification (i.e., included the +# "RECOMMENDED") and is fully compliant with [BEHAVE-UDP] is "fully +# compliant with all the mandatory and recommended requirements of this +# specification". +# +# REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior +# for TCP. +# +# REQ-2: A NAT MUST support all valid sequences of TCP packets +# (defined in [RFC0793]) for connections initiated both internally +# as well as externally when the connection is permitted by the NAT. +# In particular: +# a) In addition to handling the TCP 3-way handshake mode of +# connection initiation, A NAT MUST handle the TCP simultaneous- +# open mode of connection initiation. +# +# REQ-3: If application transparency is most important, it is +# RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" +# behavior for TCP. If a more stringent filtering behavior is most +# important, it is RECOMMENDED that a NAT have an "Address-Dependent +# Filtering" behavior. +# +# a) The filtering behavior MAY be an option configurable by the +# administrator of the NAT. +# b) The filtering behavior for TCP MAY be independent of the +# filtering behavior for UDP. +# +# REQ-4: A NAT MUST NOT respond to an unsolicited inbound SYN packet +# for at least 6 seconds after the packet is received. If during +# this interval the NAT receives and translates an outbound SYN for +# the connection the NAT MUST silently drop the original unsolicited +# inbound SYN packet. Otherwise, the NAT SHOULD send an ICMP Port +# Unreachable error (Type 3, Code 3) for the original SYN, unless +# REQ-4a applies. +# a) The NAT MUST silently drop the original SYN packet if sending a +# response violates the security policy of the NAT. +# +# REQ-5: If a NAT cannot determine whether the endpoints of a TCP +# connection are active, it MAY abandon the session if it has been +# idle for some time. In such cases, the value of the "established +# connection idle-timeout" MUST NOT be less than 2 hours 4 minutes. +# The value of the "transitory connection idle-timeout" MUST NOT be +# less than 4 minutes. +# a) The value of the NAT idle-timeouts MAY be configurable. +# +# REQ-6: If a NAT includes ALGs that affect TCP, it is RECOMMENDED +# that all of those ALGs (except for FTP [RFC0959]) be disabled by +# default. +# +# The following requirements reiterate requirements from [BEHAVE-UDP] +# or [BEHAVE-ICMP] that directly affect TCP. This document does not +# relax any requirements in [BEHAVE-UDP] or [BEHAVE-ICMP]. +# +# REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port +# overloading" for TCP. +# +# REQ-8: A NAT MUST support "hairpinning" for TCP. +# a) A NAT's hairpinning behavior MUST be of type "External source +# IP address and port". +# +# REQ-9: If a NAT translates TCP, it SHOULD translate ICMP Destination +# Unreachable (Type 3) messages. +# +# REQ-10: Receipt of any sort of ICMP message MUST NOT terminate the +# NAT mapping or TCP connection for which the ICMP was generated. + +[[spec]] +level = "MUST" +quote = ''' +REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior +for TCP. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-2: A NAT MUST support all valid sequences of TCP packets +(defined in [RFC0793]) for connections initiated both internally +as well as externally when the connection is permitted by the NAT. +''' + +[[spec]] +level = "MUST" +quote = ''' +In particular: +a) In addition to handling the TCP 3-way handshake mode of +connection initiation, A NAT MUST handle the TCP simultaneous- +open mode of connection initiation. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +REQ-3: If application transparency is most important, it is +RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" +behavior for TCP. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +If a more stringent filtering behavior is most +important, it is RECOMMENDED that a NAT have an "Address-Dependent +Filtering" behavior. +''' + +[[spec]] +level = "MAY" +quote = ''' +a) The filtering behavior MAY be an option configurable by the +administrator of the NAT. +''' + +[[spec]] +level = "MAY" +quote = ''' +b) The filtering behavior for TCP MAY be independent of the +filtering behavior for UDP. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-4: A NAT MUST NOT respond to an unsolicited inbound SYN packet +for at least 6 seconds after the packet is received. +''' + +[[spec]] +level = "MUST" +quote = ''' +If during +this interval the NAT receives and translates an outbound SYN for +the connection the NAT MUST silently drop the original unsolicited +inbound SYN packet. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +Otherwise, the NAT SHOULD send an ICMP Port +Unreachable error (Type 3, Code 3) for the original SYN, unless +REQ-4a applies. +''' + +[[spec]] +level = "MUST" +quote = ''' +a) The NAT MUST silently drop the original SYN packet if sending a +response violates the security policy of the NAT. +''' + +[[spec]] +level = "MAY" +quote = ''' +REQ-5: If a NAT cannot determine whether the endpoints of a TCP +connection are active, it MAY abandon the session if it has been +idle for some time. +''' + +[[spec]] +level = "MUST" +quote = ''' +In such cases, the value of the "established +connection idle-timeout" MUST NOT be less than 2 hours 4 minutes. +''' + +[[spec]] +level = "MUST" +quote = ''' +The value of the "transitory connection idle-timeout" MUST NOT be +less than 4 minutes. +''' + +[[spec]] +level = "MAY" +quote = ''' +a) The value of the NAT idle-timeouts MAY be configurable. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +REQ-6: If a NAT includes ALGs that affect TCP, it is RECOMMENDED +that all of those ALGs (except for FTP [RFC0959]) be disabled by +default. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port +overloading" for TCP. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-8: A NAT MUST support "hairpinning" for TCP. +''' + +[[spec]] +level = "MUST" +quote = ''' +a) A NAT's hairpinning behavior MUST be of type "External source +IP address and port". +''' + +[[spec]] +level = "SHOULD" +quote = ''' +REQ-9: If a NAT translates TCP, it SHOULD translate ICMP Destination +Unreachable (Type 3) messages. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-10: Receipt of any sort of ICMP message MUST NOT terminate the +NAT mapping or TCP connection for which the ICMP was generated. +''' + diff --git a/.duvet/snapshot.txt b/.duvet/snapshot.txt index 3fe31d4b2c..3881a4c39b 100644 --- a/.duvet/snapshot.txt +++ b/.duvet/snapshot.txt @@ -57,3 +57,127 @@ SPECIFICATION: https://www.rfc-editor.org/rfc/rfc4884 TEXT[!MAY]: while ignoring others. TEXT[!MUST]: As stated above, the total length of the ICMP message, including TEXT[!MUST]: extensions, MUST NOT exceed the minimum reassembly buffer size. + +SPECIFICATION: https://www.rfc-editor.org/rfc/rfc5382 + SECTION: [Address and Port Mapping Behavior](#section-4.1) + TEXT[!MUST]: REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior + TEXT[!MUST]: for TCP. + + SECTION: [Internally Initiated Connections](#section-4.2) + TEXT[!MUST]: REQ-2: A NAT MUST support all valid sequences of TCP packets + TEXT[!MUST]: (defined in [RFC0793]) for connections initiated both internally + TEXT[!MUST]: as well as externally when the connection is permitted by the NAT. + TEXT[!MUST]: In particular: + TEXT[!MUST]: a) In addition to handling the TCP 3-way handshake mode of + TEXT[!MUST]: connection initiation, A NAT MUST handle the TCP simultaneous- + TEXT[!MUST]: open mode of connection initiation. + + SECTION: [Externally Initiated Connections](#section-4.3) + TEXT[!SHOULD]: REQ-3: If application transparency is most important, it is + TEXT[!SHOULD]: RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" + TEXT[!SHOULD]: behavior for TCP. + TEXT[!SHOULD]: If a more stringent filtering behavior is most + TEXT[!SHOULD]: important, it is RECOMMENDED that a NAT have an "Address-Dependent + TEXT[!SHOULD]: Filtering" behavior. + TEXT[!MAY]: a) The filtering behavior MAY be an option configurable by the + TEXT[!MAY]: administrator of the NAT. + TEXT[!MAY]: b) The filtering behavior for TCP MAY be independent of the + TEXT[!MAY]: filtering behavior for UDP. + TEXT[!MUST]: REQ-4: A NAT MUST NOT respond to an unsolicited inbound SYN packet + TEXT[!MUST]: for at least 6 seconds after the packet is received. + TEXT[!MUST]: If during + TEXT[!MUST]: this interval the NAT receives and translates an outbound SYN for + TEXT[!MUST]: the connection the NAT MUST silently drop the original unsolicited + TEXT[!MUST]: inbound SYN packet. + TEXT[!SHOULD]: Otherwise, the NAT SHOULD send an ICMP Port + TEXT[!SHOULD]: Unreachable error (Type 3, Code 3) for the original SYN, unless + TEXT[!SHOULD]: REQ-4a applies. + TEXT[!MUST]: a) The NAT MUST silently drop the original SYN packet if sending a + TEXT[!MUST]: response violates the security policy of the NAT. + + SECTION: [NAT Session Refresh](#section-5) + TEXT[!MAY]: REQ-5: If a NAT cannot determine whether the endpoints of a TCP + TEXT[!MAY]: connection are active, it MAY abandon the session if it has been + TEXT[!MAY]: idle for some time. + TEXT[!MUST]: In such cases, the value of the "established + TEXT[!MUST]: connection idle-timeout" MUST NOT be less than 2 hours 4 minutes. + TEXT[!MUST]: The value of the "transitory connection idle-timeout" MUST NOT be + TEXT[!MUST]: less than 4 minutes. + TEXT[!MAY]: a) The value of the NAT idle-timeouts MAY be configurable. + TEXT[!MAY]: A NAT MAY hold state for a connection in + TEXT[!MAY]: TIME_WAIT state to accommodate retransmissions of the last ACK. + TEXT[!MAY]: When a NAT abandons a live connection, for + TEXT[!MAY]: example due to a timeout expiring, the NAT MAY either send TCP RST + TEXT[!MAY]: packets to the endpoints or MAY silently abandon the connection. + + SECTION: [Application Level Gateways](#section-6) + TEXT[!SHOULD]: REQ-6: If a NAT includes ALGs that affect TCP, it is RECOMMENDED + TEXT[!SHOULD]: that all of those ALGs (except for FTP [RFC0959]) be disabled by + TEXT[!SHOULD]: default. + + SECTION: [Port Assignment](#section-7.1) + TEXT[!MUST]: REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port + TEXT[!MUST]: overloading" for TCP. + + SECTION: [Hairpinning Behavior](#section-7.2) + TEXT[!MUST]: REQ-8: A NAT MUST support "hairpinning" for TCP. + TEXT[!MUST]: a) A NAT's hairpinning behavior MUST be of type "External source + TEXT[!MUST]: IP address and port". + + SECTION: [ICMP Responses to TCP Packets](#section-7.3) + TEXT[!SHOULD]: REQ-9: If a NAT translates TCP, it SHOULD translate ICMP Destination + TEXT[!SHOULD]: Unreachable (Type 3) messages. + TEXT[!MUST]: REQ-10: Receipt of any sort of ICMP message MUST NOT terminate the + TEXT[!MUST]: NAT mapping or TCP connection for which the ICMP was generated. + + SECTION: [Requirements](#section-8) + TEXT[!MUST,todo]: REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior + TEXT[!MUST,todo]: for TCP. + TEXT[!MUST]: REQ-2: A NAT MUST support all valid sequences of TCP packets + TEXT[!MUST]: (defined in [RFC0793]) for connections initiated both internally + TEXT[!MUST]: as well as externally when the connection is permitted by the NAT. + TEXT[!MUST]: In particular: + TEXT[!MUST]: a) In addition to handling the TCP 3-way handshake mode of + TEXT[!MUST]: connection initiation, A NAT MUST handle the TCP simultaneous- + TEXT[!MUST]: open mode of connection initiation. + TEXT[!SHOULD]: REQ-3: If application transparency is most important, it is + TEXT[!SHOULD]: RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" + TEXT[!SHOULD]: behavior for TCP. + TEXT[!SHOULD]: If a more stringent filtering behavior is most + TEXT[!SHOULD]: important, it is RECOMMENDED that a NAT have an "Address-Dependent + TEXT[!SHOULD]: Filtering" behavior. + TEXT[!MAY]: a) The filtering behavior MAY be an option configurable by the + TEXT[!MAY]: administrator of the NAT. + TEXT[!MAY]: b) The filtering behavior for TCP MAY be independent of the + TEXT[!MAY]: filtering behavior for UDP. + TEXT[!MUST]: REQ-4: A NAT MUST NOT respond to an unsolicited inbound SYN packet + TEXT[!MUST]: for at least 6 seconds after the packet is received. + TEXT[!MUST]: If during + TEXT[!MUST]: this interval the NAT receives and translates an outbound SYN for + TEXT[!MUST]: the connection the NAT MUST silently drop the original unsolicited + TEXT[!MUST]: inbound SYN packet. + TEXT[!SHOULD]: Otherwise, the NAT SHOULD send an ICMP Port + TEXT[!SHOULD]: Unreachable error (Type 3, Code 3) for the original SYN, unless + TEXT[!SHOULD]: REQ-4a applies. + TEXT[!MUST]: a) The NAT MUST silently drop the original SYN packet if sending a + TEXT[!MUST]: response violates the security policy of the NAT. + TEXT[!MAY,todo]: REQ-5: If a NAT cannot determine whether the endpoints of a TCP + TEXT[!MAY,todo]: connection are active, it MAY abandon the session if it has been + TEXT[!MAY,todo]: idle for some time. + TEXT[!MUST,todo]: In such cases, the value of the "established + TEXT[!MUST,todo]: connection idle-timeout" MUST NOT be less than 2 hours 4 minutes. + TEXT[!MUST,todo]: The value of the "transitory connection idle-timeout" MUST NOT be + TEXT[!MUST,todo]: less than 4 minutes. + TEXT[!MAY]: a) The value of the NAT idle-timeouts MAY be configurable. + TEXT[!SHOULD]: REQ-6: If a NAT includes ALGs that affect TCP, it is RECOMMENDED + TEXT[!SHOULD]: that all of those ALGs (except for FTP [RFC0959]) be disabled by + TEXT[!SHOULD]: default. + TEXT[!MUST,implementation,test]: REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port + TEXT[!MUST,implementation,test]: overloading" for TCP. + TEXT[!MUST]: REQ-8: A NAT MUST support "hairpinning" for TCP. + TEXT[!MUST]: a) A NAT's hairpinning behavior MUST be of type "External source + TEXT[!MUST]: IP address and port". + TEXT[!SHOULD]: REQ-9: If a NAT translates TCP, it SHOULD translate ICMP Destination + TEXT[!SHOULD]: Unreachable (Type 3) messages. + TEXT[!MUST,implementation,test]: REQ-10: Receipt of any sort of ICMP message MUST NOT terminate the + TEXT[!MUST,implementation,test]: NAT mapping or TCP connection for which the ICMP was generated. diff --git a/.duvet/specifications/www.rfc-editor.org/rfc/rfc5382.txt b/.duvet/specifications/www.rfc-editor.org/rfc/rfc5382.txt new file mode 100644 index 0000000000..995986c3ce --- /dev/null +++ b/.duvet/specifications/www.rfc-editor.org/rfc/rfc5382.txt @@ -0,0 +1,1179 @@ + + + + + + +Network Working Group S. Guha, Ed. +Request for Comments: 5382 Cornell U. +BCP: 142 K. Biswas +Category: Best Current Practice Cisco Systems + B. Ford + MPI-SWS + S. Sivakumar + Cisco Systems + P. Srisuresh + Kazeon Systems + October 2008 + + + NAT Behavioral Requirements for TCP + +Status of This Memo + + This document specifies an Internet Best Current Practices for the + Internet Community, and requests discussion and suggestions for + improvements. Distribution of this memo is unlimited. + +Abstract + + This document defines a set of requirements for NATs that handle TCP + that would allow many applications, such as peer-to-peer applications + and online games to work consistently. Developing NATs that meet + this set of requirements will greatly increase the likelihood that + these applications will function properly. + + + + + + + + + + + + + + + + + + + + + + + +Guha, et al. Best Current Practice [Page 1] + +RFC 5382 NAT TCP Requirements October 2008 + + +Table of Contents + + 1. Applicability Statement . . . . . . . . . . . . . . . . . . . 3 + 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 + 3. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 + 4. TCP Connection Initiation . . . . . . . . . . . . . . . . . . 4 + 4.1. Address and Port Mapping Behavior . . . . . . . . . . . . 5 + 4.2. Internally Initiated Connections . . . . . . . . . . . . . 5 + 4.3. Externally Initiated Connections . . . . . . . . . . . . . 7 + 5. NAT Session Refresh . . . . . . . . . . . . . . . . . . . . . 10 + 6. Application Level Gateways . . . . . . . . . . . . . . . . . . 12 + 7. Other Requirements Applicable to TCP . . . . . . . . . . . . . 12 + 7.1. Port Assignment . . . . . . . . . . . . . . . . . . . . . 12 + 7.2. Hairpinning Behavior . . . . . . . . . . . . . . . . . . . 13 + 7.3. ICMP Responses to TCP Packets . . . . . . . . . . . . . . 13 + 8. Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 14 + 9. Security Considerations . . . . . . . . . . . . . . . . . . . 16 + 10. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 17 + 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 18 + 11.1. Normative References . . . . . . . . . . . . . . . . . . . 18 + 11.2. Informational References . . . . . . . . . . . . . . . . . 18 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Guha, et al. Best Current Practice [Page 2] + +RFC 5382 NAT TCP Requirements October 2008 + + +1. Applicability Statement + + This document is adjunct to [BEHAVE-UDP], which defines many terms + relating to NATs, lays out general requirements for all NATs, and + sets requirements for NATs that handle IP and unicast UDP traffic. + The purpose of this document is to set requirements for NATs that + handle TCP traffic. + + The requirements of this specification apply to traditional NATs as + described in [RFC2663]. + + This document only covers the TCP aspects of NAT traversal. + Middlebox behavior that is not necessary for network address + translation of TCP is out of scope. Packet inspection above the TCP + layer and firewalls are out of scope except for Application Level + Gateway (ALG) behavior that may interfere with NAT traversal. + Application and OS aspects of TCP NAT traversal are out of scope. + Signaling-based approaches to NAT traversal, such as Middlebox + Communication (MIDCOM) and Universal Plug and Play (UPnP), that + directly control the NAT are out of scope. Finally, TCP connections + intended for the NAT (e.g., an HTTP or Secure Shell Protocol (SSH) + management interface) and TCP connections initiated by the NAT (e.g., + reliable syslog client) are out of scope. + +2. Introduction + + Network Address Translators (NATs) hinder connectivity in + applications where sessions may be initiated to internal hosts. + Readers may refer to [RFC3022] for detailed information on + traditional NATs. [BEHAVE-UDP] lays out the terminology and + requirements for NATs in the context of IP and UDP. This document + supplements these by setting requirements for NATs that handle TCP + traffic. All definitions and requirements in [BEHAVE-UDP] are + inherited here. + + [RFC4614] chronicles the evolution of TCP from the original + definition [RFC0793] to present-day implementations. While much has + changed in TCP with regards to congestion control and flow control, + security, and support for high-bandwidth networks, the process of + initiating a connection (i.e., the 3-way handshake or simultaneous- + open) has changed little. It is the process of connection initiation + that NATs affect the most. Experimental approaches such as T/TCP + [RFC1644] have proposed alternate connection initiation approaches, + but have been found to be complex and susceptible to denial-of- + service attacks. Modern operating systems and NATs consequently + primarily support the 3-way handshake and simultaneous-open modes of + connection initiation as described in [RFC0793]. + + + + +Guha, et al. Best Current Practice [Page 3] + +RFC 5382 NAT TCP Requirements October 2008 + + + Recently, many techniques have been devised to make peer-to-peer TCP + applications work across NATs. [STUNT], [NATBLASTER], and [P2PNAT] + describe Unilateral Self-Address Fixing (UNSAF) mechanisms that allow + peer-to-peer applications to establish TCP through NATs. These + approaches require only endpoint applications to be modified and work + with standards compliant OS stacks. The approaches, however, depend + on specific NAT behavior that is usually, but not always, supported + by NATs (see [TCPTRAV] and [P2PNAT] for details). Consequently, a + complete TCP NAT traversal solution is sometimes forced to rely on + public TCP relays to traverse NATs that do not cooperate. This + document defines requirements that ensure that TCP NAT traversal + approaches are not forced to use data relays. + +3. Terminology + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in [RFC2119]. + + "NAT" in this specification includes both "Basic NAT" and "Network + Address/Port Translator (NAPT)" [RFC2663]. The term "NAT Session" is + adapted from [NAT-MIB] and is defined as follows. + + NAT Session - A NAT session is an association between a TCP session + as seen in the internal realm and a TCP session as seen in the + external realm, by virtue of NAT translation. The NAT session will + provide the translation glue between the two session representations. + + This document uses the term "TCP connection" (or just "connection") + to refer to individual TCP flows identified by the 4-tuple (source + and destination IP address and TCP port) and the initial sequence + numbers (ISN). + + This document uses the term "address and port mapping" (or just + "mapping") as defined in [BEHAVE-UDP] to refer to state at the NAT + necessary for network address and port translation of TCP + connections. This document also uses the terms "Endpoint-Independent + Mapping", "Address-Dependent Mapping", "Address and Port-Dependent + Mapping", "filtering behavior", "Endpoint-Independent Filtering", + "Address-Dependent Filtering", "Address and Port-Dependent + Filtering", "Port assignment", "Port overloading", "hairpinning", and + "External source IP address and port" as defined in [BEHAVE-UDP]. + +4. TCP Connection Initiation + + This section describes various NAT behaviors applicable to TCP + connection initiation. + + + + +Guha, et al. Best Current Practice [Page 4] + +RFC 5382 NAT TCP Requirements October 2008 + + +4.1. Address and Port Mapping Behavior + + A NAT uses a mapping to translate packets for each TCP connection. A + mapping is dynamically allocated for connections initiated from the + internal side, and potentially reused for certain subsequent + connections. NAT behavior regarding when a mapping can be reused + differs for different NATs as described in [BEHAVE-UDP]. + + Consider an internal IP address and TCP port (X:x) that initiates a + TCP connection to an external (Y1:y1) tuple. Let the mapping + allocated by the NAT for this connection be (X1':x1'). Shortly + thereafter, the endpoint initiates a connection from the same (X:x) + to an external address (Y2:y2) and gets the mapping (X2':x2') on the + NAT. As per [BEHAVE-UDP], if (X1':x1') equals (X2':x2') for all + values of (Y2:y2), then the NAT is defined to have "Endpoint- + Independent Mapping" behavior. If (X1':x1') equals (X2':x2') only + when Y2 equals Y1, then the NAT is defined to have "Address-Dependent + Mapping" behavior. If (X1':x1') equals (X2':x2') only when (Y2:y2) + equals (Y1:y1), possible only for consecutive connections to the same + external address shortly after the first is terminated and if the NAT + retains state for connections in TIME_WAIT state, then the NAT is + defined to have "Address and Port-Dependent Mapping" behavior. This + document introduces one additional behavior where (X1':x1') never + equals (X2':x2'), that is, for each connection a new mapping is + allocated; in such a case, the NAT is defined to have "Connection- + Dependent Mapping" behavior. + + REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior + for TCP. + + Justification: REQ-1 is necessary for UNSAF methods to work. + Endpoint-Independent Mapping behavior allows peer-to-peer + applications to learn and advertise the external IP address and + port allocated to an internal endpoint such that external peers + can contact it (subject to the NAT's security policy). The + security policy of a NAT is independent of its mapping behavior + and is discussed later in Section 4.3. Having Endpoint- + Independent Mapping behavior allows peer-to-peer applications to + work consistently without compromising the security benefits of + the NAT. + +4.2. Internally Initiated Connections + + An internal endpoint initiates a TCP connection through a NAT by + sending a SYN packet. The NAT allocates (or reuses) a mapping for + the connection, as described in the previous section. The mapping + defines the external IP address and port used for translation of all + packets for that connection. In particular, for client-server + + + +Guha, et al. Best Current Practice [Page 5] + +RFC 5382 NAT TCP Requirements October 2008 + + + applications where an internal client initiates the connection to an + external server, the mapping is used to translate the outbound SYN, + the resulting inbound SYN-ACK response, the subsequent outbound ACK, + and other packets for the connection. This method of connection + initiation corresponds to the 3-way handshake (defined in [RFC0793]) + and is supported by all NATs. + + Peer-to-peer applications use an alternate method of connection + initiation termed simultaneous-open (Fig. 8, [RFC0793]) to traverse + NATs. In the simultaneous-open mode of operation, both peers send + SYN packets for the same TCP connection. The SYN packets cross in + the network. Upon receiving the other end's SYN packet, each end + responds with a SYN-ACK packet, which also cross in the network. The + connection is considered established once the SYN-ACKs are received. + From the perspective of the NAT, the internal host's SYN packet is + met by an inbound SYN packet for the same connection (as opposed to a + SYN-ACK packet during a 3-way handshake). Subsequent to this + exchange, both an outbound and an inbound SYN-ACK are seen for the + connection. Some NATs erroneously block the inbound SYN for the + connection in progress. Some NATs block or incorrectly translate the + outbound SYN-ACK. Such behavior breaks TCP simultaneous-open and + prevents peer-to-peer applications from functioning correctly behind + a NAT. + + In order to provide network address translation service for TCP, it + is necessary for a NAT to correctly receive, translate, and forward + all packets for a connection that conform to valid transitions of the + TCP State-Machine (Fig. 6, [RFC0793]). + + REQ-2: A NAT MUST support all valid sequences of TCP packets + (defined in [RFC0793]) for connections initiated both internally + as well as externally when the connection is permitted by the NAT. + In particular: + a) In addition to handling the TCP 3-way handshake mode of + connection initiation, A NAT MUST handle the TCP simultaneous- + open mode of connection initiation. + + Justification: The intent of this requirement is to allow standards + compliant TCP stacks to traverse NATs no matter what path the + stacks take through the TCP state-machine and no matter which end + initiates the connection as long as the connection is permitted by + the filtering policy of the NAT (filtering policy is described in + the following section). + a) In addition to TCP packets for a 3-way handshake, A NAT must be + prepared to accept an inbound SYN and an outbound SYN-ACK for + an internally initiated connection in order to support + simultaneous-open. + + + + +Guha, et al. Best Current Practice [Page 6] + +RFC 5382 NAT TCP Requirements October 2008 + + +4.3. Externally Initiated Connections + + The NAT allocates a mapping for the first connection initiated by an + internal endpoint to an external endpoint. In some scenarios, the + NAT's policy may allow this mapping to be reused for connections + initiated from the external side to the internal endpoint. Consider + as before an internal IP address and port (X:x) that is assigned (or + reuses) a mapping (X1':x1') when it initiates a connection to an + external (Y1:y1). An external endpoint (Y2:y2) attempts to initiate + a connection with the internal endpoint by sending a SYN to + (X1':x1'). A NAT can choose to either allow the connection to be + established, or to disallow the connection. If the NAT chooses to + allow the connection, it translates the inbound SYN and routes it to + (X:x) as per the existing mapping. It also translates the SYN-ACK + generated by (X:x) in response and routes it to (Y2:y2), and so on. + Alternately, the NAT can disallow the connection by filtering the + inbound SYN. + + A NAT may allow an existing mapping to be reused by an externally + initiated connection if its security policy permits. Several + different policies are possible as described in [BEHAVE-UDP]. If a + NAT allows the connection initiation from all (Y2:y2), then it is + defined to have "Endpoint-Independent Filtering" behavior. If the + NAT allows connection initiations only when Y2 equals Y1, then the + NAT is defined to have "Address-Dependent Filtering" behavior. If + the NAT allows connection initiations only when (Y2:y2) equals + (Y1:y1), then the NAT is defined to have "Address and Port-Dependent + Filtering" behavior (possible only shortly after the first connection + has been terminated but the mapping is still active). One additional + filtering behavior defined in this document is when the NAT does not + allow any connection initiations from the external side; in such + cases, the NAT is defined to have "Connection-Dependent Filtering" + behavior. The difference between "Address and Port-Dependent + Filtering" and "Connection-Dependent Filtering" behavior is that the + former permits an inbound SYN during the TIME_WAIT state of the first + connection to initiate a new connection while the latter does not. + + REQ-3: If application transparency is most important, it is + RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" + behavior for TCP. If a more stringent filtering behavior is most + important, it is RECOMMENDED that a NAT have an "Address-Dependent + Filtering" behavior. + a) The filtering behavior MAY be an option configurable by the + administrator of the NAT. + b) The filtering behavior for TCP MAY be independent of the + filtering behavior for UDP. + + + + + +Guha, et al. Best Current Practice [Page 7] + +RFC 5382 NAT TCP Requirements October 2008 + + + Justification: The intent of this requirement is to allow peer-to- + peer applications that do not always initiate connections from the + internal side of the NAT to continue to work in the presence of + NATs. This behavior also allows applications behind a BEHAVE + compliant NAT to inter-operate with remote endpoints that are + behind non-BEHAVE compliant (legacy) NATs. If the remote + endpoint's NAT does not have Endpoint-Independent Mapping behavior + but has only one external IP address, then an application can + still traverse the combination of the two NATs if the local NAT + has Address-Dependent Filtering. Section 9 contains a detailed + discussion on the security implications of this requirement. + + If the inbound SYN packet is filtered, either because a corresponding + mapping does not exist or because of the NAT's filtering behavior, a + NAT has two basic choices: to ignore the packet silently, or to + signal an error to the sender. Signaling an error through ICMP + messages allows the sender to quickly detect that the SYN did not + reach the intended destination. Silently dropping the packet, on the + other hand, allows applications to perform simultaneous-open more + reliably. + + Silently dropping the SYN aids simultaneous-open as follows. + Consider that the application is attempting a simultaneous-open and + the outbound SYN from the internal endpoint has not yet crossed the + NAT (due to network congestion or clock skew between the two + endpoints); this outbound SYN would otherwise have created the + necessary mapping at the NAT to allow translation of the inbound SYN. + Since the outbound SYN did not reach the NAT in time, the inbound SYN + cannot be processed. If a NAT responds to the premature inbound SYN + with an error message that forces the external endpoint to abandon + the connection attempt, it hinders applications performing a TCP + simultaneous-open. If instead the NAT silently ignores the inbound + SYN, the external endpoint retransmits the SYN after a TCP timeout. + In the meantime, the NAT creates the mapping in response to the + (delayed) outbound SYN such that the retransmitted inbound SYN can be + routed and simultaneous-open can succeed. The downside to this + behavior is that in the event the inbound SYN is erroneous, the + remote side does not learn of the error until after several TCP + timeouts. + + NAT support for simultaneous-open as well as quickly signaling errors + are both important for applications. Unfortunately, there is no way + for a NAT to signal an error without forcing the endpoint to abort a + potential simultaneous-open: TCP RST and ICMP Port Unreachable + packets require the endpoint to abort the attempt while the ICMP Host + and Network Unreachable errors may adversely affect other connections + to the same host or network [RFC1122]. + + + + +Guha, et al. Best Current Practice [Page 8] + +RFC 5382 NAT TCP Requirements October 2008 + + + In addition, when an unsolicited SYN is received by the NAT, the NAT + may not know whether the application is attempting a simultaneous- + open (and that it should therefore silently drop the SYN) or whether + the SYN is in error (and that it should notify the sender). + + REQ-4: A NAT MUST NOT respond to an unsolicited inbound SYN packet + for at least 6 seconds after the packet is received. If during + this interval the NAT receives and translates an outbound SYN for + the connection the NAT MUST silently drop the original unsolicited + inbound SYN packet. Otherwise, the NAT SHOULD send an ICMP Port + Unreachable error (Type 3, Code 3) for the original SYN, unless + REQ-4a applies. + a) The NAT MUST silently drop the original SYN packet if sending a + response violates the security policy of the NAT. + + Justification: The intent of this requirement is to allow + simultaneous-open to work reliably in the presence of NATs as well + as to quickly signal an error in case the unsolicited SYN is in + error. As of writing this memo, it is not possible to achieve + both; the requirement therefore represents a compromise. The NAT + should tolerate some delay in the outbound SYN for a TCP + simultaneous-open, which may be due to network congestion or loose + synchronization between the endpoints. If the unsolicited SYN is + not part of a simultaneous-open attempt and is in error, the NAT + should endeavor to signal the error in accordance with [RFC1122]. + a) There may, however, be reasons for the NAT to rate-limit or + omit such error notifications, for example, in the case of an + attack. Silently dropping the SYN packet when under attack + allows simultaneous-open to work without consuming any extra + network bandwidth or revealing the presence of the NAT to + attackers. Section 9 mentions the security considerations for + this requirement. + + For NATs that combine NAT functionality with end-host functionality + (e.g., an end-host that also serves as a NAT for other hosts behind + it), REQ-4 above applies only to SYNs intended for the NAT'ed hosts + and not to SYNs intended for the NAT itself. One way to determine + whether the inbound SYN is intended for a NAT'ed host is to allocate + NAT mappings from one port range, and allocate ports for local + endpoints from a different non-overlapping port range. More dynamic + implementations can be imagined. + + + + + + + + + + +Guha, et al. Best Current Practice [Page 9] + +RFC 5382 NAT TCP Requirements October 2008 + + +5. NAT Session Refresh + + A NAT maintains state associated with in-progress and established + connections. Because of this, a NAT is susceptible to a resource- + exhaustion attack whereby an attacker (or virus) on the internal side + attempts to cause the NAT to create more state than for which it has + resources. To prevent such an attack, a NAT needs to abandon + sessions in order to free the state resources. + + A common method that is applicable only to TCP is to preferentially + abandon sessions for crashed endpoints, followed by closed TCP + connections and partially open connections. A NAT can check if an + endpoint for a session has crashed by sending a TCP keep-alive packet + and receiving a TCP RST packet in response. If the NAT cannot + determine whether the endpoint is active, it should not abandon the + session until the TCP connection has been idle for some time. Note + that an established TCP connection can stay idle (but live) + indefinitely; hence, there is no fixed value for an idle-timeout that + accommodates all applications. However, a large idle-timeout + motivated by recommendations in [RFC1122] can reduce the chances of + abandoning a live session. + + A TCP connection passes through three phases: partially open, + established, and closing. During the partially open phase, endpoints + synchronize initial sequence numbers. The phase is initiated by the + first SYN for the connection and extends until both endpoints have + sent a packet with the ACK flag set (TCP states: SYN_SENT and + SYN_RCVD). ACKs in both directions mark the beginning of the + established phase where application data can be exchanged + indefinitely (TCP states: ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2, and + CLOSE_WAIT). The closing phase begins when both endpoints have + terminated their half of the connection by sending a FIN packet. + Once FIN packets are seen in both directions, application data can no + longer be exchanged, but the stacks still need to ensure that the FIN + packets are received (TCP states: CLOSING and LAST_ACK). + + TCP connections can stay in established phase indefinitely without + exchanging any packets. Some end-hosts can be configured to send + keep-alive packets on such idle connections; by default, such keep- + alive packets are sent every 2 hours if enabled [RFC1122]. + Consequently, a NAT that waits for slightly over 2 hours can detect + idle connections with keep-alive packets being sent at the default + rate. TCP connections in the partially open or closing phases, on + the other hand, can stay idle for at most 4 minutes while waiting for + in-flight packets to be delivered [RFC1122]. + + + + + + +Guha, et al. Best Current Practice [Page 10] + +RFC 5382 NAT TCP Requirements October 2008 + + + The "established connection idle-timeout" for a NAT is defined as the + minimum time a TCP connection in the established phase must remain + idle before the NAT considers the associated session a candidate for + removal. The "transitory connection idle-timeout" for a NAT is + defined as the minimum time a TCP connection in the partially open or + closing phases must remain idle before the NAT considers the + associated session a candidate for removal. TCP connections in the + TIME_WAIT state are not affected by the "transitory connection idle- + timeout". + + REQ-5: If a NAT cannot determine whether the endpoints of a TCP + connection are active, it MAY abandon the session if it has been + idle for some time. In such cases, the value of the "established + connection idle-timeout" MUST NOT be less than 2 hours 4 minutes. + The value of the "transitory connection idle-timeout" MUST NOT be + less than 4 minutes. + a) The value of the NAT idle-timeouts MAY be configurable. + + Justification: The intent of this requirement is to minimize the + cases where a NAT abandons session state for a live connection. + While some NATs may choose to abandon sessions reactively in + response to new connection initiations (allowing idle connections + to stay up indefinitely in the absence of new initiations), other + NATs may choose to proactively reap idle sessions. In cases where + the NAT cannot actively determine if the connection is alive, this + requirement ensures that applications can send keep-alive packets + at the default rate (every 2 hours) such that the NAT can + passively determine that the connection is alive. The additional + 4 minutes allows time for in-flight packets to cross the NAT. + + NAT behavior for handling RST packets, or connections in TIME_WAIT + state is left unspecified. A NAT MAY hold state for a connection in + TIME_WAIT state to accommodate retransmissions of the last ACK. + However, since the TIME_WAIT state is commonly encountered by + internal endpoints properly closing the TCP connection, holding state + for a closed connection may limit the throughput of connections + through a NAT with limited resources. [RFC1337] describes hazards + associated with TIME_WAIT assassination. + + The handling of non-SYN packets for connections for which there is no + active mapping is left unspecified. Such packets may be received if + the NAT silently abandons a live connection, or abandons a connection + in TIME_WAIT state before the 4 minute TIME_WAIT period expires. The + decision to either silently drop such packets or to respond with a + TCP RST packet is left up to the implementation. + + + + + + +Guha, et al. Best Current Practice [Page 11] + +RFC 5382 NAT TCP Requirements October 2008 + + + NAT behavior for notifying endpoints when abandoning live connections + is left unspecified. When a NAT abandons a live connection, for + example due to a timeout expiring, the NAT MAY either send TCP RST + packets to the endpoints or MAY silently abandon the connection. + + Sending a RST notification allows endpoint applications to recover + more quickly; however, notifying the endpoints may not always be + possible if, for example, session state is lost due to a power + failure. + +6. Application Level Gateways + + Application Level Gateways (ALGs) in certain NATs modify IP addresses + and TCP ports embedded inside application protocols. Such ALGs may + interfere with UNSAF methods or protocols that try to be NAT-aware + and must therefore be used with extreme caution. + + REQ-6: If a NAT includes ALGs that affect TCP, it is RECOMMENDED + that all of those ALGs (except for FTP [RFC0959]) be disabled by + default. + + Justification: The intent of this requirement is to prevent ALGs + from interfering with UNSAF methods. The default state of an FTP + ALG is left unspecified because of legacy concerns: as of writing + this memo, a large fraction of legacy FTP clients do not enable + passive (PASV) mode by default and require an ALG to traverse + NATs. + +7. Other Requirements Applicable to TCP + + A list of general and UDP-specific NAT behavioral requirements are + described in [BEHAVE-UDP]. A list of ICMP-specific NAT behavioral + requirements are described in [BEHAVE-ICMP]. The requirements listed + below reiterate the requirements from these two documents that + directly affect TCP. The following requirements do not relax any + requirements in [BEHAVE-UDP] or [BEHAVE-ICMP]. + +7.1. Port Assignment + + NATs that allow different internal endpoints to simultaneously use + the same mapping are defined in [BEHAVE-UDP] to have a "Port + assignment" behavior of "Port overloading". Such behavior is + undesirable, as it prevents two internal endpoints sharing the same + mapping from establishing simultaneous connections to a common + external endpoint. + + REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port + overloading" for TCP. + + + +Guha, et al. Best Current Practice [Page 12] + +RFC 5382 NAT TCP Requirements October 2008 + + + Justification: This requirement allows two applications on the + internal side of the NAT to consistently communicate with the same + destination. + + NAT behavior for preserving the source TCP port range for connections + is left unspecified. Some applications expect the source TCP port to + be in the well-known range (TCP ports from 0 to 1023). The "r" + series of commands (rsh, rcp, rlogin, etc.) are an example. NATs + that preserve the range from which the source port is picked allow + such applications to function properly through the NAT; however, by + doing so the NAT may compromise the security of the application in + certain situations; applications that depend only on the IP address + and source TCP port range for security (the "r" commands, for + example) cannot distinguish between an attacker and a legitimate user + behind the same NAT. + +7.2. Hairpinning Behavior + + NATs that forward packets originating from an internal address, + destined for an external address that matches the active mapping for + an internal address, back to that internal address are defined in + [BEHAVE-UDP] as supporting "hairpinning". If the NAT presents the + hairpinned packet with an external source IP address and port (i.e., + the mapped source address and port of the originating internal + endpoint), then it is defined to have "External source IP address and + port" for hairpinning. Hairpinning is necessary to allow two + internal endpoints (known to each other only by their external mapped + addresses) to communicate with each other. "External source IP + address and port" behavior for hairpinning avoids confusing + implementations that expect the external source IP address and port. + + REQ-8: A NAT MUST support "hairpinning" for TCP. + a) A NAT's hairpinning behavior MUST be of type "External source + IP address and port". + + Justification: This requirement allows two applications behind the + same NAT that are trying to communicate with each other using + their external addresses. + a) Using the external source address and port for the hairpinned + packet is necessary for applications that do not expect to + receive a packet from a different address than the external + address they are trying to communicate with. + +7.3. ICMP Responses to TCP Packets + + Several TCP mechanisms depend on the reception of ICMP error messages + triggered by the transmission of TCP segments. One such mechanism is + path MTU discovery [RFC1191], which is required for the correct + + + +Guha, et al. Best Current Practice [Page 13] + +RFC 5382 NAT TCP Requirements October 2008 + + + operation of TCP. The current path MTU discovery mechanism requires + the sender of TCP segments to be notified of ICMP "Datagram Too Big" + responses. + + REQ-9: If a NAT translates TCP, it SHOULD translate ICMP Destination + Unreachable (Type 3) messages. + + Justification: Translating ICMP Destination Unreachable messages, + particularly the "Fragmentation Needed and Don't Fragment was Set" + (Type 3, Code 4) message avoids communication failures ("black + holes" [RFC2923]). Furthermore, TCP's connection establishment + and maintenance mechanisms also behave much more efficiently when + ICMP Destination Unreachable messages arrive in response to + outgoing TCP segments. + + REQ-10: Receipt of any sort of ICMP message MUST NOT terminate the + NAT mapping or TCP connection for which the ICMP was generated. + + Justification: This is necessary for reliably performing TCP + simultaneous-open where a remote NAT may temporarily signal an + ICMP error. + +8. Requirements + + A NAT that supports all of the mandatory requirements of this + specification (i.e., the "MUST") and is compliant with [BEHAVE-UDP], + is "compliant with this specification". A NAT that supports all of + the requirements of this specification (i.e., included the + "RECOMMENDED") and is fully compliant with [BEHAVE-UDP] is "fully + compliant with all the mandatory and recommended requirements of this + specification". + + REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior + for TCP. + + REQ-2: A NAT MUST support all valid sequences of TCP packets + (defined in [RFC0793]) for connections initiated both internally + as well as externally when the connection is permitted by the NAT. + In particular: + a) In addition to handling the TCP 3-way handshake mode of + connection initiation, A NAT MUST handle the TCP simultaneous- + open mode of connection initiation. + + REQ-3: If application transparency is most important, it is + RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" + behavior for TCP. If a more stringent filtering behavior is most + important, it is RECOMMENDED that a NAT have an "Address-Dependent + Filtering" behavior. + + + +Guha, et al. Best Current Practice [Page 14] + +RFC 5382 NAT TCP Requirements October 2008 + + + a) The filtering behavior MAY be an option configurable by the + administrator of the NAT. + b) The filtering behavior for TCP MAY be independent of the + filtering behavior for UDP. + + REQ-4: A NAT MUST NOT respond to an unsolicited inbound SYN packet + for at least 6 seconds after the packet is received. If during + this interval the NAT receives and translates an outbound SYN for + the connection the NAT MUST silently drop the original unsolicited + inbound SYN packet. Otherwise, the NAT SHOULD send an ICMP Port + Unreachable error (Type 3, Code 3) for the original SYN, unless + REQ-4a applies. + a) The NAT MUST silently drop the original SYN packet if sending a + response violates the security policy of the NAT. + + REQ-5: If a NAT cannot determine whether the endpoints of a TCP + connection are active, it MAY abandon the session if it has been + idle for some time. In such cases, the value of the "established + connection idle-timeout" MUST NOT be less than 2 hours 4 minutes. + The value of the "transitory connection idle-timeout" MUST NOT be + less than 4 minutes. + a) The value of the NAT idle-timeouts MAY be configurable. + + REQ-6: If a NAT includes ALGs that affect TCP, it is RECOMMENDED + that all of those ALGs (except for FTP [RFC0959]) be disabled by + default. + + The following requirements reiterate requirements from [BEHAVE-UDP] + or [BEHAVE-ICMP] that directly affect TCP. This document does not + relax any requirements in [BEHAVE-UDP] or [BEHAVE-ICMP]. + + REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port + overloading" for TCP. + + REQ-8: A NAT MUST support "hairpinning" for TCP. + a) A NAT's hairpinning behavior MUST be of type "External source + IP address and port". + + REQ-9: If a NAT translates TCP, it SHOULD translate ICMP Destination + Unreachable (Type 3) messages. + + REQ-10: Receipt of any sort of ICMP message MUST NOT terminate the + NAT mapping or TCP connection for which the ICMP was generated. + + + + + + + + +Guha, et al. Best Current Practice [Page 15] + +RFC 5382 NAT TCP Requirements October 2008 + + +9. Security Considerations + + [BEHAVE-UDP] discusses security considerations for NATs that handle + IP and unicast UDP traffic. Security concerns specific to handling + TCP packets are discussed in this section. + + Security considerations for REQ-1: This requirement does not + introduce any TCP-specific security concerns. + + Security considerations for REQ-2: This requirement does not + introduce any TCP-specific security concerns. Simultaneous-open + and other transitions in the TCP state machine are by-design and + necessary for TCP to work correctly in all scenarios. Further, + this requirement only affects connections already in progress as + authorized by the NAT in accordance with its policy. + + Security considerations for REQ-3: The security provided by the NAT + is governed by its filtering behavior as addressed in + [BEHAVE-UDP]. Connection-Dependent Filtering behavior is most + secure from a firewall perspective, but severely restricts + connection initiations through a NAT. Endpoint-Independent + Filtering behavior, which is most transparent to applications, + requires an attacker to guess the IP address and port of an active + mapping in order to get his packet to an internal host. Address- + Dependent Filtering, on the other hand, is less transparent than + Endpoint-Independent Filtering but more transparent than + Connection-Dependent Filtering; it is more secure than Endpoint- + Independent Filtering as it requires an attacker to additionally + guess the address of the external endpoint for a NAT session + associated with the mapping and be able to receive packets + addressed to the same. While this protects against most attackers + on the Internet, it does not necessarily protect against attacks + that originate from behind a remote NAT with a single IP address + that is also translating a legitimate connection to the victim. + + Security considerations for REQ-4: This document recommends that a + NAT respond to unsolicited inbound SYN packets with an ICMP error + delayed by a few seconds. Doing so may reveal the presence of a + NAT to an external attacker. Silently dropping the SYN makes it + harder to diagnose network problems and forces applications to + wait for the TCP stack to finish several retransmissions before + reporting an error. An implementer must therefore understand and + carefully weigh the effects of not sending an ICMP error or rate- + limiting such ICMP errors to a very small number. + + + + + + + +Guha, et al. Best Current Practice [Page 16] + +RFC 5382 NAT TCP Requirements October 2008 + + + Security considerations for REQ-5: This document recommends that a + NAT that passively monitors TCP state keep idle sessions alive for + at least 2 hours 4 minutes or 4 minutes depending on the state of + the connection. If a NAT is under attack, it may attempt to + actively determine the liveliness of a TCP connection or let the + NAT administrator configure more conservative timeouts. + + Security considerations for REQ-6: This requirement does not + introduce any TCP-specific security concerns. + + Security considerations for REQ-7: This requirement does not + introduce any TCP-specific security concerns. + + Security considerations for REQ-8: This requirement does not + introduce any TCP-specific security concerns. + + Security considerations for REQ-9: This requirement does not + introduce any TCP-specific security concerns. + + Security considerations for REQ-10: This requirement does not + introduce any TCP-specific security concerns. + + NAT implementations that modify TCP sequence numbers (e.g., for + privacy reasons or for ALG support) must ensure that TCP packets with + Selective Acknowledgement (SACK) notifications [RFC2018] are properly + handled. + + NAT implementations that modify local state based on TCP flags in + packets must ensure that out-of-window TCP packets are properly + handled. [RFC4953] summarizes and discusses a variety of solutions + designed to prevent attackers from affecting TCP connections. + +10. Acknowledgments + + Joe Touch contributed the mechanism for handling unsolicited inbound + SYNs. Thanks to Mark Allman, Francois Audet, Lars Eggert, Paul + Francis, Fernando Gont, Sam Hartman, Paul Hoffman, Dave Hudson, + Cullen Jennings, Philip Matthews, Tom Petch, Magnus Westerlund, and + Dan Wing for their many contributions, comments, and suggestions. + + + + + + + + + + + + +Guha, et al. Best Current Practice [Page 17] + +RFC 5382 NAT TCP Requirements October 2008 + + +11. References + +11.1. Normative References + + [BEHAVE-UDP] Audet, F. and C. Jennings, "Network Address + Translation (NAT) Behavioral Requirements for Unicast + UDP", BCP 127, RFC 4787, January 2007. + + [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, + RFC 793, September 1981. + + [RFC0959] Postel, J. and J. Reynolds, "File Transfer Protocol", + STD 9, RFC 959, October 1985. + + [RFC1122] Braden, R., "Requirements for Internet Hosts - + Communication Layers", STD 3, RFC 1122, October 1989. + + [RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", + RFC 1191, November 1990. + + [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate + Requirement Levels", BCP 14, RFC 2119, March 1997. + +11.2. Informational References + + [BEHAVE-ICMP] Srisuresh, P., Ford, B., Sivakumar, S., and S. Guha, + "NAT Behavioral Requirements for ICMP protocol", Work + in Progress, June 2008. + + [NAT-MIB] Rohit, R., Srisuresh, P., Raghunarayan, R., Pai, N., + and C. Wang, "Definitions of Managed Objects for + Network Address Translators (NAT)", RFC 4008, + March 2005. + + [NATBLASTER] Biggadike, A., Ferullo, D., Wilson, G., and A. Perrig, + "NATBLASTER: Establishing TCP connections between + hosts behind NATs", Proceedings of the ACM SIGCOMM + Asia Workshop (Beijing, China), April 2005. + + [P2PNAT] Ford, B., Srisuresh, P., and D. Kegel, "Peer-to-peer + communication across network address translators", + Proceedings of the USENIX Annual Technical + Conference (Anaheim, CA), April 2005. + + [RFC1337] Braden, B., "TIME-WAIT Assassination Hazards in TCP", + RFC 1337, May 1992. + + + + + +Guha, et al. Best Current Practice [Page 18] + +RFC 5382 NAT TCP Requirements October 2008 + + + [RFC1644] Braden, B., "T/TCP -- TCP Extensions for Transactions + Functional Specification", RFC 1644, July 1994. + + [RFC2018] Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, + "TCP Selective Acknowledgment Options", RFC 2018, + October 1996. + + [RFC2663] Srisuresh, P. and M. Holdrege, "IP Network Address + Translator (NAT) Terminology and Considerations", + RFC 2663, August 1999. + + [RFC2923] Lahey, K., "TCP Problems with Path MTU Discovery", + RFC 2923, September 2000. + + [RFC3022] Srisuresh, P. and K. Egevang, "Traditional IP Network + Address Translator (Traditional NAT)", RFC 3022, + January 2001. + + [RFC4614] Duke, M., Braden, R., Eddy, W., and E. Blanton, "A + Roadmap for Transmission Control Protocol (TCP) + Specification Documents", RFC 4614, September 2006. + + [RFC4953] Touch, J., "Defending TCP Against Spoofing Attacks", + RFC 4953, July 2007. + + [STUNT] Guha, S. and P. Francis, "NUTSS: A SIP based approach + to UDP and TCP connectivity", Proceedings of the ACM + SIGCOMM Workshop on Future Directions in Network + Architecture (Portland, OR), August 2004. + + [TCPTRAV] Guha, S. and P. Francis, "Characterization and + Measurement of TCP Traversal through NATs and + Firewalls", Proceedings of the Internet Measurement + Conference (Berkeley, CA), October 2005. + + + + + + + + + + + + + + + + + +Guha, et al. Best Current Practice [Page 19] + +RFC 5382 NAT TCP Requirements October 2008 + + +Authors' Addresses + + Saikat Guha (editor) + Cornell University + 331 Upson Hall + Ithaca, NY 14853 + US + Phone: +1 607 255 1008 + EMail: saikat@cs.cornell.edu + + Kaushik Biswas + Cisco Systems, Inc. + 170 West Tasman Dr. + San Jose, CA 95134 + US + Phone: +1 408 525 5134 + EMail: kbiswas@cisco.com + + Bryan Ford + Max Planck Institute for Software Systems + Campus Building E1 4 + D-66123 Saarbruecken + Germany + Phone: +49-681-9325657 + EMail: baford@mpi-sws.org + + Senthil Sivakumar + Cisco Systems, Inc. + 7100-8 Kit Creek Road + PO Box 14987 + Research Triangle Park, NC 27709-4987 + US + Phone: +1 919 392 5158 + EMail: ssenthil@cisco.com + + Pyda Srisuresh + Kazeon Systems, Inc. + 1161 San Antonio Rd. + Mountain View, CA 94043 + US + Phone: +1 408 836 4773 + EMail: srisuresh@yahoo.com + + + + + + + + + +Guha, et al. Best Current Practice [Page 20] + +RFC 5382 NAT TCP Requirements October 2008 + + +Full Copyright Statement + + Copyright (C) The IETF Trust (2008). + + This document is subject to the rights, licenses and restrictions + contained in BCP 78, and except as set forth therein, the authors + retain all their rights. + + This document and the information contained herein are provided on an + "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS + OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND + THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS + OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF + THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED + WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. + +Intellectual Property + + The IETF takes no position regarding the validity or scope of any + Intellectual Property Rights or other rights that might be claimed to + pertain to the implementation or use of the technology described in + this document or the extent to which any license under such rights + might or might not be available; nor does it represent that it has + made any independent effort to identify any such rights. Information + on the procedures with respect to rights in RFC documents can be + found in BCP 78 and BCP 79. + + Copies of IPR disclosures made to the IETF Secretariat and any + assurances of licenses to be made available, or the result of an + attempt made to obtain a general license or permission for the use of + such proprietary rights by implementers or users of this + specification can be obtained from the IETF on-line IPR repository at + http://www.ietf.org/ipr. + + The IETF invites any interested party to bring to its attention any + copyrights, patents or patent applications, or other proprietary + rights that may cover technology that may be required to implement + this standard. Please address the information to the IETF at + ietf-ipr@ietf.org. + + + + + + + + + + + + +Guha, et al. Best Current Practice [Page 21] + diff --git a/nat/src/masquerade/apalloc/mod.rs b/nat/src/masquerade/apalloc/mod.rs index 05b9204473..90204c9c1d 100644 --- a/nat/src/masquerade/apalloc/mod.rs +++ b/nat/src/masquerade/apalloc/mod.rs @@ -281,6 +281,27 @@ impl NatAllocator { self.genid.store(genid, Ordering::Relaxed); } + //= https://www.rfc-editor.org/rfc/rfc5382#section-8 + //= type=todo + //# REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior + //# for TCP. + // + // Not held as stated, and the deviation is architectural rather than accidental: the + // allocation depends on `dst_vpcd`, so one internal endpoint talking to two destination VPCs + // can be given two different public tuples. RFC 5382 was written for a NAT facing a single + // external realm, where "endpoint" means a destination address and port; here distinct + // destination VPCs are distinct address spaces reached through distinct peerings, and sharing + // a pool across them would be the surprising choice. + // + // So this is probably an exception rather than a defect -- but "probably" is the reason it is + // marked `todo`. Whether a destination VPC counts as an endpoint for REQ-1 is a question about + // the product, and nobody has answered it. Answering it is cheap; discovering the answer + // mattered after a peer-to-peer application fails is not. + //= https://www.rfc-editor.org/rfc/rfc5382#section-8 + //# REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port + //# overloading" for TCP. + // + // No live allocation is ever handed out twice; the port bitmaps below are what enforce it. fn allocate_v4( &self, src_vpcd: VpcDiscriminant, diff --git a/nat/src/masquerade/fuzz.rs b/nat/src/masquerade/fuzz.rs index d35e7c447c..238eab8d54 100644 --- a/nat/src/masquerade/fuzz.rs +++ b/nat/src/masquerade/fuzz.rs @@ -272,6 +272,10 @@ fn out_unchanged(out: &[Packet], before: (IpAddr, u16)) -> bool { out[0].is_done() || source_of(&out[0]) == before } +//= https://www.rfc-editor.org/rfc/rfc5382#section-8 +//= type=test +//# REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port +//# overloading" for TCP. /// Two live flows never share a translation. /// /// The exclusivity claim the allocator exists to keep, stated where it matters: at the stage, over diff --git a/nat/src/masquerade/nf.rs b/nat/src/masquerade/nf.rs index e84c02caff..b62de137b0 100644 --- a/nat/src/masquerade/nf.rs +++ b/nat/src/masquerade/nf.rs @@ -83,6 +83,28 @@ impl Masquerade { }; // Internal flow timeouts for masquerading + // + //= https://www.rfc-editor.org/rfc/rfc5382#section-8 + //= type=todo + //# REQ-5: If a NAT cannot determine whether the endpoints of a TCP + //# connection are active, it MAY abandon the session if it has been + //# idle for some time. In such cases, the value of the "established + //# connection idle-timeout" MUST NOT be less than 2 hours 4 minutes. + //# The value of the "transitory connection idle-timeout" MUST NOT be + //# less than 4 minutes. + // + // We are far under both floors and this has not been ruled on. The three constants below are + // transitory timeouts in RFC 5382's sense -- the connection is opening or closing -- and they + // are seconds against a four-minute floor. The established timeout is `idle_timeout` from the + // masquerade configuration, which has no default, no bound and no validation, so a deployment + // can set it anywhere including well under two hours four minutes. + // + // The short values are deliberate in intent: this file's own comment says the statuses exist + // "to know how much to extend the lifetime of flows for port conservation", and a gateway + // holding a public port for two hours per idle connection conserves nothing. Whether that + // trade is one we are willing to state as a deviation from a BCP is a product decision, not a + // code one -- hence `todo` rather than `exception`. Converting it needs a rationale somebody + // is willing to sign. pub const MASQUERADE_ONEWAY_TIMEOUT: Duration = Duration::from_secs(5 * Self::TIMEOUT_SCALE); pub const MASQUERADE_TWOWAY_TIMEOUT: Duration = Duration::from_secs(3 * Self::TIMEOUT_SCALE); pub const MASQUERADE_CLOSING_TIMEOUT: Duration = Duration::from_secs(2 * Self::TIMEOUT_SCALE); diff --git a/nat/src/masquerade/protocol.rs b/nat/src/masquerade/protocol.rs index 1e101a3ee9..f4207f867c 100644 --- a/nat/src/masquerade/protocol.rs +++ b/nat/src/masquerade/protocol.rs @@ -50,6 +50,12 @@ fn next_flow_status_udp(action: NatAction, status: NatFlowStatus) -> NatFlowStat } } +//= https://www.rfc-editor.org/rfc/rfc5382#section-8 +//# REQ-10: Receipt of any sort of ICMP message MUST NOT terminate the +//# NAT mapping or TCP connection for which the ICMP was generated. +// +// Held by construction: no arm below yields `Closed` or `Reset`, so no ICMP message can end a +// mapping. The only transition available is the one that records that traffic came back. #[allow(clippy::match_single_binding)] fn next_flow_status_icmp(action: NatAction, status: NatFlowStatus) -> NatFlowStatus { match action { diff --git a/nat/src/masquerade/state_machine.rs b/nat/src/masquerade/state_machine.rs index 9ef582b45a..d526f34841 100644 --- a/nat/src/masquerade/state_machine.rs +++ b/nat/src/masquerade/state_machine.rs @@ -267,6 +267,10 @@ fn ordinary_udp_opens_and_settles() { } } +//= https://www.rfc-editor.org/rfc/rfc5382#section-8 +//= type=test +//# REQ-10: Receipt of any sort of ICMP message MUST NOT terminate the +//# NAT mapping or TCP connection for which the ICMP was generated. /// An ICMP echo reply makes a one-way flow two-way, and nothing else moves. /// /// ICMP has no flags to read and no close sequence, so the only evidence available is that a packet From ab5d0863fb80bd36f14ba9671025054be01aa4e8 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 22:40:30 -0600 Subject: [PATCH 20/37] docs(testing): Record what duvet can and cannot parse, and what is still open Two specifications are tracked and the tool has already paid for itself, but nothing recorded where the method stops working. That boundary is not obvious and is expensive to rediscover, so this writes it down before more specifications are added. The audit behind it ran duvet's extractor over the entire RFC series, 9,827 documents in 23 seconds, twice: - It is deterministic. Two sweeps produced 68,257 emitted files that are byte-identical, and single-threaded output matches parallel, so snapshot regression gating is sound. - 35 documents fail loudly, all invalid UTF-8, nearly all pre-1990. - It is blind to lowercase normative language. RFC 8200 and RFC 3022 contain zero RFC 2119 keywords and never cite RFC 2119, so the two specifications closest to what this dataplane is cannot be tracked directly. This is the real boundary of the method. - Composite BCP files silently lose most of their content. BCP 127 is RFC 4787 + 6888 + 7857 concatenated; duvet keys requirements by section anchor, every member has its own section 5, and the last one wins. 42 requirements extracted where the three members separately yield 129, with no warning and exit code 0. RFC 4787 loses section 5, NAT Session Refresh, which is where the UDP timeout requirements live. The note also records the intended route around the lowercase problem -- duvet accepts a Markdown specification, so the obligations can be restated in RFC 2119 form in-repo -- together with the hazard that goes with it. That is the one place the method can certify itself: once we author the specification we control both sides of the match, and the cheapest thing to write is the requirement the code already satisfies. It is the entrenchment failure mutation testing has, moved up a level and much harder to see, so the rules for synthesis are written down next to the mechanism. The open-questions list is deliberately a list rather than a plan, and is expected to grow. It is in the repository so that it grows in one place. Also corrects the RFC 5382 requirement count in .duvet/config.toml. It states ten numbered requirements, REQ-1 through REQ-10; 22 is RFC 5508's clause count. (cherry picked from commit bec3879e1452866d76b0f314cc1809042916d796) --- .duvet/config.toml | 6 +- development/code/README.md | 3 + development/code/spec-compliance.md | 177 ++++++++++++++++++++++++++++ 3 files changed, 183 insertions(+), 3 deletions(-) create mode 100644 development/code/spec-compliance.md diff --git a/.duvet/config.toml b/.duvet/config.toml index 7261f4894a..29eac3c732 100644 --- a/.duvet/config.toml +++ b/.duvet/config.toml @@ -20,9 +20,9 @@ pattern = "*/src/**/*.rs" [[specification]] source = "https://www.rfc-editor.org/rfc/rfc4884" -# RFC 5382 states 22 numbered requirements for how a NAT must treat TCP. It constrains values this -# codebase already has and chose without reference to it -- notably the idle timeouts in -# nat/src/masquerade/nf.rs. Tracked second for that reason. +# RFC 5382 states 10 numbered requirements (REQ-1 .. REQ-10) for how a NAT must treat TCP. It +# constrains values this codebase already has and chose without reference to it -- notably the +# idle timeouts in nat/src/masquerade/nf.rs. Tracked second for that reason. [[specification]] source = "https://www.rfc-editor.org/rfc/rfc5382" diff --git a/development/code/README.md b/development/code/README.md index 3617e64bce..82b4e54033 100644 --- a/development/code/README.md +++ b/development/code/README.md @@ -14,6 +14,8 @@ If you need to write a test, prefer [property-based tests] over simple unit tests. To find out what your tests are _not_ saying, see the [mutation testing note][mutants]; it is a report we read, not a gate that blocks anything. +To find out whether they are saying what the specification asked for, see the +[specification compliance note][duvet]. For inputs too large to generate directly -- a whole configuration, say -- build them from an algebra of valid operations and derive the oracles from that same algebra; see the [config algebra note][config-algebra]. If you need to handle errors, prefer `Result` types over panics in general, but see the @@ -31,6 +33,7 @@ If you need to [handle an error][error], follow the guidelines. [property-based tests]: ./property-testing.md [config-algebra]: ./config-algebra-testing.md [mutants]: ./mutation-testing.md +[duvet]: ./spec-compliance.md [clock]: ../../clock/src/lib.rs [error]: ./error-handling.md diff --git a/development/code/spec-compliance.md b/development/code/spec-compliance.md new file mode 100644 index 0000000000..6b5b8f02a8 --- /dev/null +++ b/development/code/spec-compliance.md @@ -0,0 +1,177 @@ +# Specification compliance with duvet + +Status: **two specifications tracked, the RFC corpus audited, the procedure itself not yet proven.** +The open-questions list below is expected to grow; it is written down so that it grows in one place +rather than in four people's heads. + +## What it is for + +[duvet] matches citations in the source -- `//= ` followed by the requirement text, quoted +verbatim -- against requirements it extracts from a specification. A requirement nothing implements, +or an implementation nothing tests, becomes visible. + +It is the third leg of a stool. [bolero](./property-testing.md) asks whether a property holds. +[cargo-mutants](./mutation-testing.md) asks whether there are enough properties. Neither can ask +whether they are the properties the specification called for. + +That third question is not decoration. Mutation testing can actively **entrench a deviation**: +the cheapest way to kill a surviving mutant is to assert the behaviour that was observed, which +cements whatever the code already did. A suite can converge on a perfect score against the wrong +specification, and nothing inside the suite can notice. + +## What it found, first time out + +Two specifications, roughly one afternoon. + +| | outcome | +| --- | --- | +| RFC 4884 length validation | **real defect**, fixed in `net/src/headers/embedded.rs` | +| RFC 5382 REQ-7, REQ-10 | already held, already tested, never named | +| RFC 5382 REQ-5, REQ-1 | conformance gaps, recorded as `todo` | + +The defect is the strongest argument for the tool: `is_full_payload()` checked the RFC 4884 length +attribute in **bits** where the RFC counts 32-bit **words**, so it rejected seven of eight +conforming lengths and admitted sub-128-octet fields the RFC forbids. It had property tests. They +passed. + +The two already-held requirements are the second-strongest argument, for the opposite reason. The +masquerade exclusivity property -- "two live flows never share a translation" -- was written before +anyone read RFC 5382, and turns out to _be_ REQ-7 verbatim. Citing it converts an accident into a +claim a reviewer can check and a refactor cannot quietly undo. + +### `todo` versus `exception` + +Both are ways of saying "not implemented". They are not interchangeable. An `exception` asserts that +somebody weighed the requirement and declined it; a `todo` asserts only that nobody has yet. Using +`exception` for an undecided requirement is self-granted absolution, and it is invisible afterwards. +REQ-1 and REQ-5 are `todo` for exactly this reason. + +## What the tool will and will not parse + +Measured by running the extractor over the entire RFC series -- 9,827 documents, 23 seconds. + +**It is deterministic.** Two full sweeps produced 68,257 emitted files that are byte-identical, and +single-threaded output matches parallel. Snapshot regression gating is safe. + +**It fails loudly on 35 documents**, all `invalid utf-8`, nearly all pre-1990 documents carrying +Latin-1 bytes. The only ones a networking project might want are RFC 1305 (NTPv3) and RFC 2557. + +**It is blind to lowercase normative language.** This is the important one: + +| | RFC 2119 keywords | lowercase must/should | cites RFC 2119 | +| --- | --- | --- | --- | +| RFC 8200 (IPv6, STD 86) | 0 | 79 | no | +| RFC 3022 (traditional NAT) | 0 | 24 | no | + +RFC 8200 says "It must obey the protocol requirements for routers when receiving (forwarding) +interfaces." That is a real obligation with no uppercase token to key on. duvet is not +malfunctioning -- there is nothing to grip -- but the effect is that **the two specifications +closest to what this dataplane is cannot be tracked directly.** This is the boundary of the method: +it covers BCP-style documents with numbered `REQ-` clauses very well and foundational standards-track +documents not at all. + +Of the 3,767 documents that extract nothing, almost all are legitimately requirement-free. The large +cluster showing exactly ten keywords is the boilerplate "The key words MUST, MUST NOT, ..." +paragraph, which duvet correctly declines to treat as normative. + +**Modern format is fine.** There is no cliff at RFC 8650; xml2rfc v3 output parses (RFC 9000: 522 +requirements, RFC 9110: 412, RFC 8446: 431). + +## Do not cite a composite BCP + +The worst failure found, because it exits 0 and reports a plausible number. + +209 of 239 BCP entries in the mirror are symlinks to a single RFC and are harmless. The other 27 are +concatenations, and **BCP 127 is one of them**: RFC 4787 + RFC 6888 + RFC 7857 in one file. + +| | requirements | +| --- | --- | +| `bcp127.txt` | **42** | +| RFC 4787 + 6888 + 7857, extracted separately | **129** | + +duvet keys requirements by section anchor, and each member document has its own `section-5`, so the +last document in the concatenation wins. RFC 6888 loses all three of its sections; RFC 4787 loses +eight of thirteen, including section 5, _NAT Session Refresh_, where the UDP timeout requirements +live. No warning is emitted. + +Always cite the individual RFC. The composites remain useful as a **membership oracle** -- "BCP 127 +now contains an RFC we do not track" is the cheapest available drift alarm, and it is a `grep` over +27 files rather than something duvet has to parse. + +## Synthesizing requirements for a non-conforming specification + +duvet accepts a Markdown specification (`-f markdown`), and this is the intended route for RFC 8200 +and RFC 3022: restate their lowercase obligations in RFC 2119 form, in-repo, as a separate +`[[specification]]`. + +The mechanism cooperates. Section anchors are heading slugs rather than numbers, so the composite +collision cannot occur, and the prose around a requirement is carried into the emitted TOML as a +comment, so a derivation note travels with it. + +**The hazard is that this is the one place the method can certify itself.** duvet's value is that +the quoted unit is a sentence somebody else wrote. Once we author the specification we control both +sides of the match, and the path of least resistance is to write the requirement the code already +satisfies -- the same entrenchment failure as mutation testing, moved up a level and much harder to +see, because the result looks like compliance with RFC 8200. + +Rules, therefore: + +- Every synthesized requirement quotes its **source sentence verbatim**, adjacent to it. +- A synthesized requirement may **never be more specific** than the sentence it derives from. +- Where the original is genuinely ambiguous, that ambiguity **is the finding**. Record both readings + for a human; do not resolve it into one confident restatement. +- Review is by somebody who reads the original. A reviewer looking only at our Markdown cannot catch + the failure this is guarding against. + +## Open questions + +Expected to expand. Nothing here is scheduled. + +1. **Which RFCs apply to us at all.** Prior to everything else, and never yet enumerated. RFC 4787 + (59 requirements), RFC 5508 (92), RFC 6888 (41) and RFC 7857 (29) are duvet-friendly, directly + on-topic and untracked -- 221 requirements one config edit away, no synthesis needed. +2. **Which of those are not RFC 2119 conforming**, and so need synthesis per the section above. +3. **Is a citation true?** duvet checks that a `type=test` citation _exists_, not that the test + exercises the requirement. This is the vacuity problem that the llvm-cov execution counters + caught twice. The cross-check uses artifacts we already produce: mutate the region cited + `type=implementation` and see whether the test cited `type=test` fails. If it does not, the + citation is decorative. This is the only item on this list that makes the three tools check each + other rather than merely coexist. +4. **Errata.** The rsync corpus carries no errata bodies -- `inline-errata/` holds stylesheets only. + It reports that 2,613 RFCs have errata and never what they say. A second fetch leg is needed + regardless of how the corpus is pinned. Outstanding and concrete: **RFC 4884 has errata, and the + `MIN_ORIGINAL_DATAGRAM_OCTETS` fix was written without reading them.** +5. **How the corpus is pinned** -- a git mirror, or `oras` into ghcr.io behind `npins`. Sizing: the + metadata that drives every drift alarm (indexes, `bcp/`, `std/`) is 5MB; the 232MB is RFC bodies, + of which we cite perhaps ten. Text gzips about 4:1. +6. **Two alarms, not one.** RFC bodies are immutable, so a diff in `rfc5382.txt` means a rerender or + a corrupt mirror: rare and loud. A new erratum, a new `updated_by`, a new BCP member is expected + churn: a routine pull request. Collapsing both into "the pin moved" trains people to rubber-stamp + it. The corpus has a `rerendered/` directory precisely because the first case is real. +7. **A scoping policy, before the first large specification lands.** Adding STUN drops 200 uncited + requirements into the snapshot in one commit; QUIC would add 522. Without a rule for scoping + _within_ a specification the report becomes wallpaper on the day it gets interesting -- the same + lesson as "do not test printers" and "classify, do not eliminate". +8. **Should the snapshot be a blocking gate?** Unlike cargo-mutants it can be: `duvet report` takes + 4ms and is bit-for-bit deterministic. It would be the cheapest correctness gate we have. +9. **A coverage report analogous to the existing ones**, so that specification coverage is read the + same way as line and mutant coverage. + +## Operational notes + +- **`//=` is duvet's citation marker.** Banner comments of the form `//======== Fib ========//` are + parsed as citations and produce errors; `routing/src/cli/display.rs` needed a space inserted after + the slashes. +- **Everything under `.duvet/` is committed except `reports/`.** `duvet report` reads the + specification from `.duvet/specifications/` and only reaches the network when it is missing, so + vendoring the text is what lets the report run in a nix build sandbox. Verified offline under + `unshare -rn`. +- **The mirror is a drop-in for the network fetch.** Text pulled by rsync is byte-identical to what + duvet fetches from rfc-editor.org, and re-extraction produces identical requirements apart from + the `target` line. +- **`see_also` is empty** in the per-RFC JSON, so the RFC-to-BCP mapping has to come from the `bcp/` + symlinks or the index files, not the metadata. +- **`-n.json`** in the corpus is an all-null artifact of the mirror. It parses, so it will not crash + a loader, but anything iterating the JSON should skip a null `doc_id`. + +[duvet]: https://awslabs.github.io/duvet/ From 16bd80c8f001f36eb0b5fd2e0139df5ecbe04b19 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 22:49:54 -0600 Subject: [PATCH 21/37] build(duvet): Track RFC 4787, and record where masquerade departs from it A trial run, chosen to answer two questions: whether a second specification is cheaper than the first, and where the procedure hurts. RFC 4787 is the UDP counterpart of RFC 5382, so it lands on code that already carries citations. Answer to the first question: much cheaper, because the specifications overlap. Four of the nine requirements cited here are restatements of RFC 5382 clauses already cited on the same lines -- REQ-1 is RFC 5382 REQ-1 without the "for TCP" qualifier, REQ-3 is REQ-7, REQ-12 is REQ-10. Those took minutes and mostly consist of noting that one decision settles two citations. Answer to the second: the requirements that do not overlap are where the findings are. REQ-5, the UDP mapping timer, is a materially worse deviation than its TCP sibling and the reason is structural. RFC 5382 distinguishes "established" from "transitory" connections, which is what makes our five- and three-second constants arguable -- they govern states where a connection is opening or closing. RFC 4787 draws no such distinction. A UDP mapping is a UDP mapping and the timer is "the time a mapping will stay active without packets traversing the NAT", against a floor of two minutes. Traced through `next_flow_status_udp`, a plain request/response exchange creates the flow at `OneWay` (five seconds), the reply moves it to `TwoWay` (three), and only a second outbound packet reaches `Established` and the two-minute `idle_timeout`. One round trip and a four-second pause loses the mapping. That is short of the floor by a factor of forty, for all UDP that is not a resolver exchange, and REQ-5a does not cover it: that exemption is for timers specific to one IANA-registered application on one well-known port, not a blanket rule. The resolver fast-close in protocol.rs turns out to be exactly what REQ-5a describes, and is cited as such -- with the caveat that 8853 sits above 1023, so the exemption reaches 53 and 853 but not it. REQ-14 is cited onto an existing bare `TODO: Check whether the packet is fragmented`. The TODO was already right; it now says which BCP it is a TODO about, and records that REQ-14a wants out-of-order fragment handling that cannot become a denial of service vector. REQ-9, hairpinning, is a MUST with no implementation anywhere -- "hairpin" does not appear in the workspace. There is a real argument that it does not apply under masquerade, where a public tuple exists only for the lifetime of an outbound flow and is not something a peer can learn and dial. The argument is recorded next to the citation and the citation is still `todo`, because nobody who owns that decision has made it. REQ-3a is the first `exception` in the tree, and is what an exception is for: `setup.rs` excludes well-known ports for TCP and UDP alike, so a host sourcing from a port below 1024 is always translated above it and the requirement can never be met. That was decided -- the range is named, a flag carries it, tests hold it -- so it is recorded as a decision rather than as an omission. Also corrects a claim in the RFC 5382 REQ-5 note: the masquerade `idle_timeout` does have a default, `DEFAULT_MASQUERADE_IDLE_TIMEOUT`, of two minutes. It has no lower bound and no validation, which is the part that stands. Cited as the individual RFC. BCP 127 concatenates RFC 4787, 6888 and 7857, whose section numbers collide, and duvet silently keeps only the last. (cherry picked from commit 1994193a683b5e8547e9f2e617eac7b752c0838c) --- .duvet/config.toml | 10 + .../rfc/rfc4787/section-10.toml | 44 + .../rfc/rfc4787/section-11.toml | 51 + .../rfc/rfc4787/section-12.toml | 326 ++++ .../rfc/rfc4787/section-4.1.toml | 140 ++ .../rfc/rfc4787/section-4.2.1.toml | 115 ++ .../rfc/rfc4787/section-4.2.2.toml | 38 + .../rfc/rfc4787/section-4.3.toml | 114 ++ .../rfc/rfc4787/section-4.4.toml | 93 + .../rfc/rfc4787/section-5.toml | 104 ++ .../rfc/rfc4787/section-6.toml | 66 + .../rfc/rfc4787/section-7.toml | 46 + .../rfc/rfc4787/section-8.toml | 65 + .../rfc/rfc4787/section-9.toml | 80 + .duvet/snapshot.txt | 173 ++ .../www.rfc-editor.org/rfc/rfc4787.txt | 1627 +++++++++++++++++ nat/src/masquerade/apalloc/mod.rs | 13 + nat/src/masquerade/apalloc/port_alloc.rs | 18 + nat/src/masquerade/fuzz.rs | 4 + nat/src/masquerade/mod.rs | 17 + nat/src/masquerade/nf.rs | 50 +- nat/src/masquerade/protocol.rs | 23 + 22 files changed, 3215 insertions(+), 2 deletions(-) create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-10.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-11.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-12.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.1.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.2.1.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.2.2.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.3.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.4.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-5.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-6.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-7.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-8.toml create mode 100644 .duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-9.toml create mode 100644 .duvet/specifications/www.rfc-editor.org/rfc/rfc4787.txt diff --git a/.duvet/config.toml b/.duvet/config.toml index 29eac3c732..0bd4b1d3d8 100644 --- a/.duvet/config.toml +++ b/.duvet/config.toml @@ -34,3 +34,13 @@ enabled = true # is reformatted. [report.snapshot] enabled = true + +# RFC 4787 is the UDP counterpart of RFC 5382 and states 14 numbered requirements. Tracked third +# because masquerade translates UDP on the same path it translates TCP, so its mapping and timeout +# requirements land on code that already carries RFC 5382 citations. +# +# Cited as the individual RFC, never as BCP 127. That composite concatenates RFC 4787, RFC 6888 and +# RFC 7857, whose section numbers collide, and duvet silently keeps only the last -- 42 requirements +# where the three separately yield 129. See development/code/spec-compliance.md. +[[specification]] +source = "https://www.rfc-editor.org/rfc/rfc4787" diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-10.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-10.toml new file mode 100644 index 0000000000..d1175894d7 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-10.toml @@ -0,0 +1,44 @@ +target = "https://www.rfc-editor.org/rfc/rfc4787#section-10" + +# Fragmentation of Outgoing Packets +# +# When the MTU of the adjacent link is too small, fragmentation of +# packets going from the internal side to the external side of the NAT +# may occur. This can occur if the NAT is doing Point-to-Point over +# Ethernet (PPPoE), or if the NAT has been configured with a small MTU +# to reduce serialization delay when sending large packets and small +# higher-priority packets, or for other reasons. +# +# It is worth noting that many IP stacks do not use Path MTU Discovery +# with UDP packets. +# +# The packet could have its Don't Fragment bit set to 1 (DF=1) or 0 +# (DF=0). +# +# REQ-13: If the packet received on an internal IP address has DF=1, +# the NAT MUST send back an ICMP message "Fragmentation needed and +# DF set" to the host, as described in [RFC0792]. +# +# a) If the packet has DF=0, the NAT MUST fragment the packet and +# SHOULD send the fragments in order. +# +# Justification: This is as per RFC 792. +# +# a) This is the same function a router performs in a similar +# situation [RFC1812]. + +[[spec]] +level = "MUST" +quote = ''' +REQ-13: If the packet received on an internal IP address has DF=1, +the NAT MUST send back an ICMP message "Fragmentation needed and +DF set" to the host, as described in [RFC0792]. +''' + +[[spec]] +level = "MUST" +quote = ''' +a) If the packet has DF=0, the NAT MUST fragment the packet and +SHOULD send the fragments in order. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-11.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-11.toml new file mode 100644 index 0000000000..e8364092d3 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-11.toml @@ -0,0 +1,51 @@ +target = "https://www.rfc-editor.org/rfc/rfc4787#section-11" + +# Receiving Fragmented Packets +# +# For a variety of reasons, a NAT may receive a fragmented packet. The +# IP packet containing the header could arrive in any fragment, +# depending on network conditions, packet ordering, and the +# implementation of the IP stack that generated the fragments. +# +# A NAT that is capable only of receiving fragments in order (that is, +# with the header in the first packet) and forwarding each of the +# fragments to the internal host is described as "Received Fragments +# Ordered". +# +# A NAT that is capable of receiving fragments in or out of order and +# forwarding the individual fragments (or a reassembled packet) to the +# internal host is referred to as "Receive Fragments Out of Order". +# See the Security Considerations section of this document for a +# discussion of this behavior. +# +# A NAT that is neither of these is referred to as "Receive Fragments +# None". +# +# REQ-14: A NAT MUST support receiving in-order and out-of-order +# fragments, so it MUST have "Received Fragment Out of Order" +# behavior. +# +# a) A NAT's out-of-order fragment processing mechanism MUST be +# designed so that fragmentation-based DoS attacks do not +# compromise the NAT's ability to process in-order and +# unfragmented IP packets. +# +# Justification: See Security Considerations. + +[[spec]] +level = "MUST" +quote = ''' +REQ-14: A NAT MUST support receiving in-order and out-of-order +fragments, so it MUST have "Received Fragment Out of Order" +behavior. +''' + +[[spec]] +level = "MUST" +quote = ''' +a) A NAT's out-of-order fragment processing mechanism MUST be +designed so that fragmentation-based DoS attacks do not +compromise the NAT's ability to process in-order and +unfragmented IP packets. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-12.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-12.toml new file mode 100644 index 0000000000..98be36cf25 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-12.toml @@ -0,0 +1,326 @@ +target = "https://www.rfc-editor.org/rfc/rfc4787#section-12" + +# Requirements +# +# The requirements in this section are aimed at minimizing the +# complications caused by NATs to applications, such as realtime +# communications and online gaming. The requirements listed earlier in +# the document are consolidated here into a single section. +# +# It should be understood, however, that applications normally do not +# know in advance if the NAT conforms to the recommendations defined in +# this section. Peer-to-peer media applications still need to use +# normal procedures, such as ICE [ICE]. +# +# A NAT that supports all the mandatory requirements of this +# specification (i.e., the "MUST"), is "compliant with this +# specification". A NAT that supports all the requirements of this +# specification (i.e., including the "RECOMMENDED") is "fully compliant +# with all the mandatory and recommended requirements of this +# specification". +# +# REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior. +# +# REQ-2: It is RECOMMENDED that a NAT have an "IP address pooling" +# behavior of "Paired". Note that this requirement is not +# applicable to NATs that do not support IP address pooling. +# +# REQ-3: A NAT MUST NOT have a "Port assignment" behavior of "Port +# overloading". +# +# a) If the host's source port was in the range 0-1023, it is +# RECOMMENDED the NAT's source port be in the same range. If the +# host's source port was in the range 1024-65535, it is +# RECOMMENDED that the NAT's source port be in that range. +# +# REQ-4: It is RECOMMENDED that a NAT have a "Port parity +# preservation" behavior of "Yes". +# +# REQ-5: A NAT UDP mapping timer MUST NOT expire in less than two +# minutes, unless REQ-5a applies. +# +# a) For specific destination ports in the well-known port range +# (ports 0-1023), a NAT MAY have shorter UDP mapping timers that +# are specific to the IANA-registered application running over +# that specific destination port. +# +# b) The value of the NAT UDP mapping timer MAY be configurable. +# +# c) A default value of five minutes or more for the NAT UDP mapping +# timer is RECOMMENDED. +# +# REQ-6: The NAT mapping Refresh Direction MUST have a "NAT Outbound +# refresh behavior" of "True". +# +# a) The NAT mapping Refresh Direction MAY have a "NAT Inbound +# refresh behavior" of "True". +# +# REQ-7 A NAT device whose external IP interface can be configured +# dynamically MUST either (1) Automatically ensure that its internal +# network uses IP addresses that do not conflict with its external +# network, or (2) Be able to translate and forward traffic between +# all internal nodes and all external nodes whose IP addresses +# numerically conflict with the internal network. +# +# REQ-8: If application transparency is most important, it is +# RECOMMENDED that a NAT have "Endpoint-Independent Filtering" +# behavior. If a more stringent filtering behavior is most +# important, it is RECOMMENDED that a NAT have "Address-Dependent +# Filtering" behavior. +# +# a) The filtering behavior MAY be an option configurable by the +# administrator of the NAT. +# +# REQ-9: A NAT MUST support "Hairpinning". +# +# a) A NAT Hairpinning behavior MUST be "External source IP address +# and port". +# +# REQ-10: To eliminate interference with UNSAF NAT traversal +# mechanisms and allow integrity protection of UDP communications, +# NAT ALGs for UDP-based protocols SHOULD be turned off. Future +# standards track specifications that define an ALG can update this +# to recommend the ALGs on which they define default. +# +# a) If a NAT includes ALGs, it is RECOMMENDED that the NAT allow +# the NAT administrator to enable or disable each ALG separately. +# +# REQ-11: A NAT MUST have deterministic behavior, i.e., it MUST NOT +# change the NAT translation (Section 4) or the Filtering +# (Section 5) Behavior at any point in time, or under any particular +# conditions. +# +# REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the +# NAT mapping. +# +# a) The NAT's default configuration SHOULD NOT filter ICMP messages +# based on their source IP address. +# +# b) It is RECOMMENDED that a NAT support ICMP Destination +# Unreachable messages. +# +# REQ-13 If the packet received on an internal IP address has DF=1, +# the NAT MUST send back an ICMP message "Fragmentation needed and +# DF set" to the host, as described in [RFC0792]. +# +# a) If the packet has DF=0, the NAT MUST fragment the packet and +# SHOULD send the fragments in order. +# +# REQ-14: A NAT MUST support receiving in-order and out-of-order +# fragments, so it MUST have "Received Fragment Out of Order" +# behavior. +# +# a) A NAT's out-of-order fragment processing mechanism MUST be +# designed so that fragmentation-based DoS attacks do not +# compromise the NAT's ability to process in-order and +# unfragmented IP packets. + +[[spec]] +level = "MUST" +quote = ''' +REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +REQ-2: It is RECOMMENDED that a NAT have an "IP address pooling" +behavior of "Paired". +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-3: A NAT MUST NOT have a "Port assignment" behavior of "Port +overloading". +''' + +[[spec]] +level = "SHOULD" +quote = ''' +a) If the host's source port was in the range 0-1023, it is +RECOMMENDED the NAT's source port be in the same range. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +If the +host's source port was in the range 1024-65535, it is +RECOMMENDED that the NAT's source port be in that range. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +REQ-4: It is RECOMMENDED that a NAT have a "Port parity +preservation" behavior of "Yes". +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-5: A NAT UDP mapping timer MUST NOT expire in less than two +minutes, unless REQ-5a applies. +''' + +[[spec]] +level = "MAY" +quote = ''' +a) For specific destination ports in the well-known port range +(ports 0-1023), a NAT MAY have shorter UDP mapping timers that +are specific to the IANA-registered application running over +that specific destination port. +''' + +[[spec]] +level = "MAY" +quote = ''' +b) The value of the NAT UDP mapping timer MAY be configurable. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +c) A default value of five minutes or more for the NAT UDP mapping +timer is RECOMMENDED. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-6: The NAT mapping Refresh Direction MUST have a "NAT Outbound +refresh behavior" of "True". +''' + +[[spec]] +level = "MAY" +quote = ''' +a) The NAT mapping Refresh Direction MAY have a "NAT Inbound +refresh behavior" of "True". +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-7 A NAT device whose external IP interface can be configured +dynamically MUST either (1) Automatically ensure that its internal +network uses IP addresses that do not conflict with its external +network, or (2) Be able to translate and forward traffic between +all internal nodes and all external nodes whose IP addresses +numerically conflict with the internal network. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +REQ-8: If application transparency is most important, it is +RECOMMENDED that a NAT have "Endpoint-Independent Filtering" +behavior. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +If a more stringent filtering behavior is most +important, it is RECOMMENDED that a NAT have "Address-Dependent +Filtering" behavior. +''' + +[[spec]] +level = "MAY" +quote = ''' +a) The filtering behavior MAY be an option configurable by the +administrator of the NAT. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-9: A NAT MUST support "Hairpinning". +''' + +[[spec]] +level = "MUST" +quote = ''' +a) A NAT Hairpinning behavior MUST be "External source IP address +and port". +''' + +[[spec]] +level = "SHOULD" +quote = ''' +REQ-10: To eliminate interference with UNSAF NAT traversal +mechanisms and allow integrity protection of UDP communications, +NAT ALGs for UDP-based protocols SHOULD be turned off. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +a) If a NAT includes ALGs, it is RECOMMENDED that the NAT allow +the NAT administrator to enable or disable each ALG separately. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-11: A NAT MUST have deterministic behavior, i.e., it MUST NOT +change the NAT translation (Section 4) or the Filtering +(Section 5) Behavior at any point in time, or under any particular +conditions. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the +NAT mapping. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +a) The NAT's default configuration SHOULD NOT filter ICMP messages +based on their source IP address. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +b) It is RECOMMENDED that a NAT support ICMP Destination +Unreachable messages. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-13 If the packet received on an internal IP address has DF=1, +the NAT MUST send back an ICMP message "Fragmentation needed and +DF set" to the host, as described in [RFC0792]. +''' + +[[spec]] +level = "MUST" +quote = ''' +a) If the packet has DF=0, the NAT MUST fragment the packet and +SHOULD send the fragments in order. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-14: A NAT MUST support receiving in-order and out-of-order +fragments, so it MUST have "Received Fragment Out of Order" +behavior. +''' + +[[spec]] +level = "MUST" +quote = ''' +a) A NAT's out-of-order fragment processing mechanism MUST be +designed so that fragmentation-based DoS attacks do not +compromise the NAT's ability to process in-order and +unfragmented IP packets. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.1.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.1.toml new file mode 100644 index 0000000000..9497e22ce8 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.1.toml @@ -0,0 +1,140 @@ +target = "https://www.rfc-editor.org/rfc/rfc4787#section-4.1" + +# Address and Port Mapping +# +# When an internal endpoint opens an outgoing session through a NAT, +# the NAT assigns the session an external IP address and port number so +# that subsequent response packets from the external endpoint can be +# received by the NAT, translated, and forwarded to the internal +# endpoint. This is a mapping between an internal IP address and port +# IP:port and external IP:port tuple. It establishes the translation +# +# that will be performed by the NAT for the duration of the session. +# For many applications, it is important to distinguish the behavior of +# the NAT when there are multiple simultaneous sessions established to +# different external endpoints. +# +# The key behavior to describe is the criteria for reuse of a mapping +# for new sessions to external endpoints, after establishing a first +# mapping between an internal X:x address and port and an external +# Y1:y1 address tuple. Let's assume that the internal IP address and +# port X:x are mapped to X1':x1' for this first session. The endpoint +# then sends from X:x to an external address Y2:y2 and gets a mapping +# of X2':x2' on the NAT. The relationship between X1':x1' and X2':x2' +# for various combinations of the relationship between Y1:y1 and Y2:y2 +# is critical for describing the NAT behavior. This arrangement is +# illustrated in the following diagram: +# +# E +# +------+ +------+ x +# | Y1 | | Y2 | t +# +--+---+ +---+--+ e +# | Y1:y1 Y2:y2 | r +# +----------+ +----------+ n +# | | a +# X1':x1' | | X2':x2' l +# +--+---+-+ +# ...........| NAT |............... +# +--+---+-+ I +# | | n +# X:x | | X:x t +# ++---++ e +# | X | r +# +-----+ n +# a +# l +# +# Address and Port Mapping +# +# The following address and port mapping behavior are defined: +# +# Endpoint-Independent Mapping: +# +# The NAT reuses the port mapping for subsequent packets sent +# from the same internal IP address and port (X:x) to any +# external IP address and port. Specifically, X1':x1' equals +# X2':x2' for all values of Y2:y2. +# +# Address-Dependent Mapping: +# +# The NAT reuses the port mapping for subsequent packets sent +# from the same internal IP address and port (X:x) to the same +# external IP address, regardless of the external port. +# Specifically, X1':x1' equals X2':x2' if and only if, Y2 equals +# Y1. +# +# Address and Port-Dependent Mapping: +# +# The NAT reuses the port mapping for subsequent packets sent +# from the same internal IP address and port (X:x) to the same +# external IP address and port while the mapping is still active. +# Specifically, X1':x1' equals X2':x2' if and only if, Y2:y2 +# equals Y1:y1. +# +# It is important to note that these three possible choices make no +# difference to the security properties of the NAT. The security +# properties are fully determined by which packets the NAT allows in +# and which it does not. This is determined by the filtering behavior +# in the filtering portions of the NAT. +# +# REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior. +# +# Justification: In order for UNSAF methods to work, REQ-1 needs to be +# met. Failure to meet REQ-1 will force the use of a UDP relay, +# which is very often impractical. +# +# Some NATs are capable of assigning IP addresses from a pool of IP +# addresses on the external side of the NAT, as opposed to just a +# single IP address. This is especially common with larger NATs. Some +# NATs use the external IP address mapping in an arbitrary fashion +# (i.e., randomly): one internal IP address could have multiple +# external IP address mappings active at the same time for different +# sessions. These NATs have an "IP address pooling" behavior of +# "Arbitrary". Some large Enterprise NATs use an IP address pooling +# behavior of "Arbitrary" as a means of hiding the IP address assigned +# to specific endpoints by making their assignment less predictable. +# Other NATs use the same external IP address mapping for all sessions +# associated with the same internal IP address. These NATs have an "IP +# address pooling" behavior of "Paired". NATs that use an "IP address +# pooling" behavior of "Arbitrary" can cause issues for applications +# that use multiple ports from the same endpoint, but that do not +# negotiate IP addresses individually (e.g., some applications using +# RTP and RTCP). +# +# REQ-2: It is RECOMMENDED that a NAT have an "IP address pooling" +# behavior of "Paired". Note that this requirement is not +# applicable to NATs that do not support IP address pooling. +# +# Justification: This will allow applications that use multiple ports +# originating from the same internal IP address to also have the +# same external IP address. This is to avoid breaking peer-to-peer +# applications that are not capable of negotiating the IP address +# for RTP and the IP address for RTCP separately. As such it is +# envisioned that this requirement will become less important as +# applications become NAT-friendlier with time. The main reason why +# this requirement is here is that in a peer-to-peer application, +# you are subject to the other peer's mistake. In particular, in +# the context of SIP, if my application supports the extensions +# defined in [RFC3605] for indicating RTP and RTCP addresses and +# ports separately, but the other peer does not, there may still be +# breakage in the form of the stream losing RTCP packets. This +# requirement will avoid the loss of RTP in this context, although +# the loss of RTCP may be inevitable in this particular example. It +# is also worth noting that RFC 3605 is unfortunately not a +# mandatory part of SIP [RFC3261]. Therefore, this requirement will +# address a particularly nasty problem that will prevail for a +# significant period of time. + +[[spec]] +level = "MUST" +quote = ''' +REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +REQ-2: It is RECOMMENDED that a NAT have an "IP address pooling" +behavior of "Paired". +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.2.1.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.2.1.toml new file mode 100644 index 0000000000..b773183c5a --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.2.1.toml @@ -0,0 +1,115 @@ +target = "https://www.rfc-editor.org/rfc/rfc4787#section-4.2.1" + +# Port Assignment Behavior +# +# This section uses the following diagram for reference. +# +# E +# +-------+ +-------+ x +# | Y1 | | Y2 | t +# +---+---+ +---+---+ e +# | Y1:y1 Y2:y2 | r +# +---------+ +---------+ n +# | | a +# X1':x1' | | X2':x2' l +# +--+---+--+ +# ...........| NAT |............... +# +--+---+--+ I +# | | n +# +---------+ +---------+ t +# | X1:x1 X2:x2 | e +# +---+---+ +---+---+ r +# | X1 | | X2 | n +# +-------+ +-------+ a +# l +# +# Port Assignment +# +# Some NATs attempt to preserve the port number used internally when +# assigning a mapping to an external IP address and port (e.g., x1=x1', +# x2=x2'). This port assignment behavior is referred to as "port +# preservation". In case of port collision, these NATs attempt a +# variety of techniques for coping. For example, some NATs will +# overridden the previous mapping to preserve the same port. Other +# NATs will assign a different IP address from a pool of external IP +# addresses; this is only possible as long as the NAT has enough +# external IP addresses; if the port is already in use on all available +# external IP addresses, then these NATs will pick a different port +# (i.e., they don't do port preservation anymore). +# +# Some NATs use "Port overloading", i.e., they always use port +# preservation even in the case of collision (i.e., X1'=X2' and +# x1=x2=x1'=x2'). Most applications will fail if the NAT uses "Port +# overloading". +# +# A NAT that does not attempt to make the external port numbers match +# the internal port numbers in any case is referred to as "no port +# preservation". +# +# When NATs do allocate a new source port, there is the issue of which +# IANA-defined range of port to choose. The ranges are "well-known" +# from 0 to 1023, "registered" from 1024 to 49151, and "dynamic/ +# private" from 49152 through 65535. For most protocols, these are +# destination ports and not source ports, so mapping a source port to a +# source port that is already registered is unlikely to have any bad +# effects. Some NATs may choose to use only the ports in the dynamic +# range; the only downside of this practice is that it limits the +# number of ports available. Other NAT devices may use everything but +# the well-known range and may prefer to use the dynamic range first, +# or possibly avoid the actual registered ports in the registered +# range. Other NATs preserve the port range if it is in the well-known +# range. [RFC0768] specifies that the source port is set to zero if no +# reply packets are expected. In this case, it does not matter what +# the NAT maps it to, as the source port will not be used. However, +# many common OS APIs do not allow a user to send from port zero, +# applications do not use port zero, and the behavior of various +# existing NATs with regards to a packet with a source of port zero is +# unknown. This document does not specify any normative behavior for a +# NAT when handling a packet with a source port of zero which means +# that applications cannot count on any sort of deterministic behavior +# for these packets. +# +# REQ-3: A NAT MUST NOT have a "Port assignment" behavior of "Port +# overloading". +# +# a) If the host's source port was in the range 0-1023, it is +# RECOMMENDED the NAT's source port be in the same range. If the +# host's source port was in the range 1024-65535, it is +# RECOMMENDED that the NAT's source port be in that range. +# +# Justification: This requirement must be met in order to enable two +# applications on the internal side of the NAT both to use the same +# port to try to communicate with the same destination. NATs that +# implement port preservation have to deal with conflicts on ports, +# and the multiple code paths this introduces often result in +# nondeterministic behavior. However, it should be understood that +# when a port is randomly assigned, it may just randomly happen to +# be assigned the same port. Applications must, therefore, be able +# to deal with both port preservation and no port preservation. +# +# a) Certain applications expect the source UDP port to be in the +# well-known range. See the discussion of Network File System +# port expectations in [RFC2623] for an example. + +[[spec]] +level = "MUST" +quote = ''' +REQ-3: A NAT MUST NOT have a "Port assignment" behavior of "Port +overloading". +''' + +[[spec]] +level = "SHOULD" +quote = ''' +a) If the host's source port was in the range 0-1023, it is +RECOMMENDED the NAT's source port be in the same range. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +If the +host's source port was in the range 1024-65535, it is +RECOMMENDED that the NAT's source port be in that range. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.2.2.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.2.2.toml new file mode 100644 index 0000000000..c3b614d3f1 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.2.2.toml @@ -0,0 +1,38 @@ +target = "https://www.rfc-editor.org/rfc/rfc4787#section-4.2.2" + +# Port Parity +# +# Some NATs preserve the parity of the UDP port, i.e., an even port +# will be mapped to an even port, and an odd port will be mapped to an +# odd port. This behavior respects the [RFC3550] rule that RTP use +# even ports, and RTCP use odd ports. RFC 3550 allows any port numbers +# to be used for RTP and RTCP if the two numbers are specified +# separately; for example, using [RFC3605]. However, some +# implementations do not include RFC 3605, and do not recognize when +# the peer has specified the RTCP port separately using RFC 3605. If +# such an implementation receives an odd RTP port number from the peer +# (perhaps after having been translated by a NAT), and then follows the +# RFC 3550 rule to change the RTP port to the next lower even number, +# this would obviously result in the loss of RTP. NAT-friendly +# application aspects are outside the scope of this document. It is +# expected that this issue will fade away with time, as implementations +# improve. Preserving the port parity allows for supporting +# communication with peers that do not support explicit specification +# of both RTP and RTCP port numbers. +# +# REQ-4: It is RECOMMENDED that a NAT have a "Port parity +# preservation" behavior of "Yes". +# +# Justification: This is to avoid breaking peer-to-peer applications +# that do not explicitly and separately specify RTP and RTCP port +# numbers and that follow the RFC 3550 rule to decrement an odd RTP +# port to make it even. The same considerations apply, as per the +# IP address pooling requirement. + +[[spec]] +level = "SHOULD" +quote = ''' +REQ-4: It is RECOMMENDED that a NAT have a "Port parity +preservation" behavior of "Yes". +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.3.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.3.toml new file mode 100644 index 0000000000..88b2fb29ec --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.3.toml @@ -0,0 +1,114 @@ +target = "https://www.rfc-editor.org/rfc/rfc4787#section-4.3" + +# Mapping Refresh +# +# NAT mapping timeout implementations vary, but include the timer's +# value and the way the mapping timer is refreshed to keep the mapping +# alive. +# +# The mapping timer is defined as the time a mapping will stay active +# without packets traversing the NAT. There is great variation in the +# values used by different NATs. +# +# REQ-5: A NAT UDP mapping timer MUST NOT expire in less than two +# minutes, unless REQ-5a applies. +# +# a) For specific destination ports in the well-known port range +# (ports 0-1023), a NAT MAY have shorter UDP mapping timers that +# are specific to the IANA-registered application running over +# that specific destination port. +# +# b) The value of the NAT UDP mapping timer MAY be configurable. +# +# c) A default value of five minutes or more for the NAT UDP mapping +# timer is RECOMMENDED. +# +# Justification: This requirement is to ensure that the timeout is +# long enough to avoid too-frequent timer refresh packets. +# +# a) Some UDP protocols using UDP use very short-lived connections. +# There can be very many such connections; keeping them all in a +# connections table could cause considerable load on the NAT. +# Having shorter timers for these specific applications is, +# therefore, an optimization technique. It is important that the +# shorter timers applied to specific protocols be used sparingly, +# and only for protocols using well-known destination ports that +# are known to have a shorter timer, and that are known not to be +# used by any applications for other purposes. +# +# b) Configuration is desirable for adapting to specific networks +# and troubleshooting. +# +# c) This default is to avoid too-frequent timer refresh packets. +# +# Some NATs keep the mapping active (i.e., refresh the timer value) +# when a packet goes from the internal side of the NAT to the external +# side of the NAT. This is referred to as having a NAT Outbound +# refresh behavior of "True". +# +# Some NATs keep the mapping active when a packet goes from the +# external side of the NAT to the internal side of the NAT. This is +# referred to as having a NAT Inbound Refresh Behavior of "True". +# +# Some NATs keep the mapping active on both, in which case, both +# properties are "True". +# +# REQ-6: The NAT mapping Refresh Direction MUST have a "NAT Outbound +# refresh behavior" of "True". +# +# a) The NAT mapping Refresh Direction MAY have a "NAT Inbound +# refresh behavior" of "True". +# +# Justification: Outbound refresh is necessary for allowing the client +# to keep the mapping alive. +# +# a) Inbound refresh may be useful for applications with no outgoing +# UDP traffic. However, allowing inbound refresh may allow an +# external attacker or misbehaving application to keep a mapping +# alive indefinitely. This may be a security risk. Also, if the +# process is repeated with different ports, over time, it could +# use up all the ports on the NAT. + +[[spec]] +level = "MUST" +quote = ''' +REQ-5: A NAT UDP mapping timer MUST NOT expire in less than two +minutes, unless REQ-5a applies. +''' + +[[spec]] +level = "MAY" +quote = ''' +a) For specific destination ports in the well-known port range +(ports 0-1023), a NAT MAY have shorter UDP mapping timers that +are specific to the IANA-registered application running over +that specific destination port. +''' + +[[spec]] +level = "MAY" +quote = ''' +b) The value of the NAT UDP mapping timer MAY be configurable. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +c) A default value of five minutes or more for the NAT UDP mapping +timer is RECOMMENDED. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-6: The NAT mapping Refresh Direction MUST have a "NAT Outbound +refresh behavior" of "True". +''' + +[[spec]] +level = "MAY" +quote = ''' +a) The NAT mapping Refresh Direction MAY have a "NAT Inbound +refresh behavior" of "True". +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.4.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.4.toml new file mode 100644 index 0000000000..e35199e317 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-4.4.toml @@ -0,0 +1,93 @@ +target = "https://www.rfc-editor.org/rfc/rfc4787#section-4.4" + +# Conflicting Internal and External IP Address Spaces +# +# Many NATs, particularly consumer-level devices designed to be +# deployed by nontechnical users, routinely obtain their external IP +# address, default router, and other IP configuration information for +# their external interface dynamically from an external network, such +# as an upstream ISP. The NAT, in turn, automatically sets up its own +# internal subnet in one of the private IP address spaces assigned to +# this purpose in [RFC1918], typically providing dynamic IP +# configuration services for hosts on this internal network. +# +# Auto-configuration of NATs and private networks can be problematic, +# however, if the NAT's external network is also in RFC 1918 private +# address space. In a common scenario, an ISP places its customers +# behind a NAT and hands out private RFC 1918 addresses to them. Some +# of these customers, in turn, deploy consumer-level NATs, which, in +# effect, act as "second-level" NATs, multiplexing their own private +# RFC 1918 IP subnets onto the single RFC 1918 IP address provided by +# the ISP. There is no inherent guarantee, in this case, that the +# ISP's "intermediate" privately-addressed network and the customer's +# internal privately-addressed network will not use numerically +# identical or overlapping RFC 1918 IP subnets. Furthermore, customers +# of consumer-level NATs cannot be expected to have the technical +# +# knowledge to prevent this scenario from occurring by manually +# configuring their internal network with non-conflicting RFC 1918 +# subnets. +# +# NAT vendors need to design their NATs to ensure that they function +# correctly and robustly even in such problematic scenarios. One +# possible solution is for the NAT to ensure that whenever its external +# link is configured with an RFC 1918 private IP address, the NAT +# automatically selects a different, non-conflicting RFC 1918 IP subnet +# for its internal network. A disadvantage of this solution is that, +# if the NAT's external interface is dynamically configured or re- +# configured after its internal network is already in use, then the NAT +# may have to renumber its entire internal network dynamically if it +# detects a conflict. +# +# An alternative solution is for the NAT to be designed so that it can +# translate and forward traffic correctly, even when its external and +# internal interfaces are configured with numerically overlapping IP +# subnets. In this scenario, for example, if the NAT's external +# interface has been assigned an IP address P in RFC 1918 space, then +# there might also be an internal node I having the same RFC 1918 +# private IP address P. An IP packet with destination address P on the +# external network is directed at the NAT, whereas an IP packet with +# the same destination address P on the internal network is directed at +# node I. The NAT therefore needs to maintain a clear operational +# distinction between "external IP addresses" and "internal IP +# addresses" to avoid confusing internal node I with its own external +# interface. In general, the NAT needs to allow all internal nodes +# (including I) to communicate with all external nodes having public +# (non-RFC 1918) IP addresses, or having private IP addresses that do +# not conflict with the addresses used by its internal network. +# +# REQ-7: A NAT device whose external IP interface can be configured +# dynamically MUST either (1) automatically ensure that its internal +# network uses IP addresses that do not conflict with its external +# network, or (2) be able to translate and forward traffic between +# all internal nodes and all external nodes whose IP addresses +# numerically conflict with the internal network. +# +# Justification: If a NAT's external and internal interfaces are +# configured with overlapping IP subnets, then there is, of course, +# no way for an internal host with RFC 1918 IP address Q to initiate +# a direct communication session to an external node having the same +# RFC 1918 address Q, or to other external nodes with IP addresses +# that numerically conflict with the internal subnet. Such nodes +# can still open communication sessions indirectly via NAT traversal +# techniques, however, with the help of a third-party server, such +# as a STUN server having a public, non-RFC 1918 IP address. In +# +# this case, nodes with conflicting private RFC 1918 addresses on +# opposite sides of the second-level NAT can communicate with each +# other via their respective temporary public endpoints on the main +# Internet, as long as their common, first-level NAT (e.g., the +# upstream ISP's NAT) supports hairpinning behavior, as described in +# Section 6. + +[[spec]] +level = "MUST" +quote = ''' +REQ-7: A NAT device whose external IP interface can be configured +dynamically MUST either (1) automatically ensure that its internal +network uses IP addresses that do not conflict with its external +network, or (2) be able to translate and forward traffic between +all internal nodes and all external nodes whose IP addresses +numerically conflict with the internal network. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-5.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-5.toml new file mode 100644 index 0000000000..1f67749d2e --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-5.toml @@ -0,0 +1,104 @@ +target = "https://www.rfc-editor.org/rfc/rfc4787#section-5" + +# Filtering Behavior +# +# This section describes various filtering behaviors observed in NATs. +# +# When an internal endpoint opens an outgoing session through a NAT, +# the NAT assigns a filtering rule for the mapping between an internal +# IP:port (X:x) and external IP:port (Y:y) tuple. +# +# The key behavior to describe is what criteria are used by the NAT to +# filter packets originating from specific external endpoints. +# +# Endpoint-Independent Filtering: +# +# The NAT filters out only packets not destined to the internal +# address and port X:x, regardless of the external IP address and +# port source (Z:z). The NAT forwards any packets destined to +# X:x. In other words, sending packets from the internal side of +# the NAT to any external IP address is sufficient to allow any +# packets back to the internal endpoint. +# +# Address-Dependent Filtering: +# +# The NAT filters out packets not destined to the internal +# address X:x. Additionally, the NAT will filter out packets +# from Y:y destined for the internal endpoint X:x if X:x has not +# sent packets to Y:any previously (independently of the port +# used by Y). In other words, for receiving packets from a +# specific external endpoint, it is necessary for the internal +# endpoint to send packets first to that specific external +# endpoint's IP address. +# +# Address and Port-Dependent Filtering: +# +# This is similar to the previous behavior, except that the +# external port is also relevant. The NAT filters out packets +# not destined for the internal address X:x. Additionally, the +# NAT will filter out packets from Y:y destined for the internal +# endpoint X:x if X:x has not sent packets to Y:y previously. In +# other words, for receiving packets from a specific external +# endpoint, it is necessary for the internal endpoint to send +# packets first to that external endpoint's IP address and port. +# +# REQ-8: If application transparency is most important, it is +# RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" +# behavior. If a more stringent filtering behavior is most +# important, it is RECOMMENDED that a NAT have an "Address-Dependent +# Filtering" behavior. +# +# a) The filtering behavior MAY be an option configurable by the +# administrator of the NAT. +# +# Justification: The recommendation to use Endpoint-Independent +# Filtering is aimed at maximizing application transparency; in +# particular, for applications that receive media simultaneously +# from multiple locations (e.g., gaming), or applications that use +# rendezvous techniques. However, it is also possible that, in some +# circumstances, it may be preferable to have a more stringent +# filtering behavior. Filtering independently of the external +# endpoint is not as secure: An unauthorized packet could get +# through a specific port while the port was kept open if it was +# lucky enough to find the port open. In theory, filtering based on +# both IP address and port is more secure than filtering based only +# on the IP address (because the external endpoint could, in +# reality, be two endpoints behind another NAT, where one of the two +# endpoints is an attacker). However, such a policy could interfere +# with applications that expect to receive UDP packets on more than +# one UDP port. Using Endpoint-Independent Filtering or Address- +# Dependent Filtering instead of Address and Port-Dependent +# Filtering on a NAT (say, NAT-A) also has benefits when the other +# endpoint is behind a non-BEHAVE compliant NAT (say, NAT-B) that +# does not support REQ-1. When the endpoints use ICE, if NAT-A uses +# Address and Port-Dependent Filtering, connectivity will require a +# UDP relay. However, if NAT-A uses Endpoint-Independent Filtering +# or Address-Dependent Filtering, ICE will ultimately find +# connectivity without requiring a UDP relay. Having the filtering +# behavior being an option configurable by the administrator of the +# NAT ensures that a NAT can be used in the widest variety of +# deployment scenarios. + +[[spec]] +level = "SHOULD" +quote = ''' +REQ-8: If application transparency is most important, it is +RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" +behavior. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +If a more stringent filtering behavior is most +important, it is RECOMMENDED that a NAT have an "Address-Dependent +Filtering" behavior. +''' + +[[spec]] +level = "MAY" +quote = ''' +a) The filtering behavior MAY be an option configurable by the +administrator of the NAT. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-6.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-6.toml new file mode 100644 index 0000000000..4a59770de9 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-6.toml @@ -0,0 +1,66 @@ +target = "https://www.rfc-editor.org/rfc/rfc4787#section-6" + +# Hairpinning Behavior +# +# If two hosts (called X1 and X2) are behind the same NAT and +# exchanging traffic, the NAT may allocate an address on the outside of +# the NAT for X2, called X2':x2'. If X1 sends traffic to X2':x2', it +# goes to the NAT, which must relay the traffic from X1 to X2. This is +# referred to as hairpinning and is illustrated below. +# +# NAT +# +----+ from X1:x1 to X2':x2' +-----+ X1':x1' +# | X1 |>>>>>>>>>>>>>>>>>>>>>>>>>>>>>--+--- +# +----+ | v | +# | v | +# | v | +# | v | +# +----+ from X1':x1' to X2:x2 | v | X2':x2' +# | X2 |<<<<<<<<<<<<<<<<<<<<<<<<<<<<<--+--- +# +----+ +-----+ +# +# Hairpinning Behavior +# +# Hairpinning allows two endpoints on the internal side of the NAT to +# communicate even if they only use each other's external IP addresses +# and ports. +# +# More formally, a NAT that supports hairpinning forwards packets +# originating from an internal address, X1:x1, destined for an external +# address X2':x2' that has an active mapping to an internal address +# X2:x2, back to that internal address, X2:x2. Note that typically X1' +# is the same as X2'. +# +# Furthermore, the NAT may present the hairpinned packet with either an +# internal (X1:x1) or an external (X1':x1') source IP address and port. +# Therefore, the hairpinning NAT behavior can be either "External +# source IP address and port" or "Internal source IP address and port". +# "Internal source IP address and port" may cause problems by confusing +# implementations that expect an external IP address and port. +# +# REQ-9: A NAT MUST support "Hairpinning". +# +# a) A NAT Hairpinning behavior MUST be "External source IP address +# and port". +# +# Justification: This requirement is to allow communications between +# two endpoints behind the same NAT when they are trying each +# other's external IP addresses. +# +# a) Using the external source IP address is necessary for +# applications with a restrictive policy of not accepting packets +# from IP addresses that differ from what is expected. + +[[spec]] +level = "MUST" +quote = ''' +REQ-9: A NAT MUST support "Hairpinning". +''' + +[[spec]] +level = "MUST" +quote = ''' +a) A NAT Hairpinning behavior MUST be "External source IP address +and port". +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-7.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-7.toml new file mode 100644 index 0000000000..4d53ac5ade --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-7.toml @@ -0,0 +1,46 @@ +target = "https://www.rfc-editor.org/rfc/rfc4787#section-7" + +# Application Level Gateways +# +# Certain NATs have implemented Application Level Gateways (ALGs) for +# various protocols, including protocols for negotiating peer-to-peer +# sessions, such as SIP. +# +# Certain NATs have these ALGs turned on permanently, others have them +# turned on by default but allow them to be turned off, and others have +# them turned off by default but allow them be turned on. +# +# NAT ALGs may interfere with UNSAF methods or protocols that try to be +# NAT-aware and therefore must be used with extreme caution. +# +# REQ-10: To eliminate interference with UNSAF NAT traversal +# mechanisms and allow integrity protection of UDP communications, +# NAT ALGs for UDP-based protocols SHOULD be turned off. Future +# standards track specifications that define ALGs can update this to +# recommend the defaults for the ALGs that they define. +# +# a) If a NAT includes ALGs, it is RECOMMENDED that the NAT allow +# the NAT administrator to enable or disable each ALG separately. +# +# Justification: NAT ALGs may interfere with UNSAF methods. +# +# a) This requirement allows the user to enable those ALGs that are +# necessary to aid in the operation of some applications without +# enabling ALGs, which interfere with the operation of other +# applications. + +[[spec]] +level = "SHOULD" +quote = ''' +REQ-10: To eliminate interference with UNSAF NAT traversal +mechanisms and allow integrity protection of UDP communications, +NAT ALGs for UDP-based protocols SHOULD be turned off. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +a) If a NAT includes ALGs, it is RECOMMENDED that the NAT allow +the NAT administrator to enable or disable each ALG separately. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-8.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-8.toml new file mode 100644 index 0000000000..ead654edd6 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-8.toml @@ -0,0 +1,65 @@ +target = "https://www.rfc-editor.org/rfc/rfc4787#section-8" + +# Deterministic Properties +# +# The classification of NATs is further complicated by the fact that, +# under some conditions, the same NAT will exhibit different behaviors. +# This has been seen on NATs that preserve ports or have specific +# algorithms for selecting a port other than a free one. If the +# external port that the NAT wishes to use is already in use by another +# session, the NAT must select a different port. This results in +# different code paths for this conflict case, which results in +# different behavior. +# +# For example, if three hosts X1, X2, and X3 all send from the same +# port x, through a port preserving NAT with only one external IP +# address, called X1', the first one to send (i.e., X1) will get an +# external port of x, but the next two will get x2' and x3' (where +# these are not equal to x). There are NATs where the External NAT +# mapping characteristics and the External Filter characteristics +# change between the X1:x and the X2:x mapping. To make matters worse, +# there are NATs where the behavior may be the same on the X1:x and +# X2:x mappings, but different on the third X3:x mapping. +# +# Another example is that some NATs have an "Endpoint-Independent +# Mapping", combined with "Port Overloading", as long as two endpoints +# are not establishing sessions to the same external direction, but +# then switch their behavior to "Address and Port-Dependent Mapping" +# +# without "Port Preservation" upon detection of these conflicting +# sessions establishments. +# +# Any NAT that changes the NAT Mapping or the Filtering behavior +# without configuration changes, at any point in time, under any +# particular conditions, is referred to as a "non-deterministic" NAT. +# NATs that don't are called "deterministic". +# +# Non-deterministic NATs generally change behavior when a conflict of +# some sort happens, i.e., when the port that would normally be used is +# already in use by another mapping. The NAT mapping and External +# Filtering in the absence of conflict is referred to as the Primary +# behavior. The behavior after the first conflict is referred to as +# Secondary and after the second conflict is referred to as Tertiary. +# No NATs have been observed that change on further conflicts, but it +# is certainly possible that they exist. +# +# REQ-11: A NAT MUST have deterministic behavior, i.e., it MUST NOT +# change the NAT translation (Section 4) or the Filtering +# (Section 5) Behavior at any point in time, or under any particular +# conditions. +# +# Justification: Non-deterministic NATs are very difficult to +# troubleshoot because they require more intensive testing. This +# non-deterministic behavior is the root cause of much of the +# uncertainty that NATs introduce about whether or not applications +# will work. + +[[spec]] +level = "MUST" +quote = ''' +REQ-11: A NAT MUST have deterministic behavior, i.e., it MUST NOT +change the NAT translation (Section 4) or the Filtering +(Section 5) Behavior at any point in time, or under any particular +conditions. +''' + diff --git a/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-9.toml b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-9.toml new file mode 100644 index 0000000000..81b9e9e248 --- /dev/null +++ b/.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-9.toml @@ -0,0 +1,80 @@ +target = "https://www.rfc-editor.org/rfc/rfc4787#section-9" + +# ICMP Destination Unreachable Behavior +# +# When a NAT sends a packet toward a host on the other side of the NAT, +# an ICMP message may be sent in response to that packet. That ICMP +# message may be sent by the destination host or by any router along +# the network path. The NAT's default configuration SHOULD NOT filter +# ICMP messages based on their source IP address. Such ICMP messages +# SHOULD be rewritten by the NAT (specifically, the IP headers and the +# ICMP payload) and forwarded to the appropriate internal or external +# host. The NAT needs to perform this function for as long as the UDP +# mapping is active. Receipt of any sort of ICMP message MUST NOT +# destroy the NAT mapping. A NAT that performs the functions described +# in the paragraph above is referred to as "support ICMP Processing". +# +# There is no significant security advantage to blocking ICMP +# Destination Unreachable packets. Additionally, blocking ICMP +# Destination Unreachable packets can interfere with application +# failover, UDP Path MTU Discovery (see [RFC1191] and [RFC1435]), and +# traceroute. Blocking any ICMP message is discouraged, and blocking +# ICMP Destination Unreachable is strongly discouraged. +# +# REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the +# NAT mapping. +# +# a) The NAT's default configuration SHOULD NOT filter ICMP messages +# based on their source IP address. +# +# b) It is RECOMMENDED that a NAT support ICMP Destination +# Unreachable messages. +# +# Justification: This is easy to do and is used for many things +# including MTU discovery and rapid detection of error conditions, +# and has no negative consequences. + +[[spec]] +level = "SHOULD" +quote = ''' +The NAT's default configuration SHOULD NOT filter +ICMP messages based on their source IP address. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +Such ICMP messages +SHOULD be rewritten by the NAT (specifically, the IP headers and the +ICMP payload) and forwarded to the appropriate internal or external +host. +''' + +[[spec]] +level = "MUST" +quote = ''' +Receipt of any sort of ICMP message MUST NOT +destroy the NAT mapping. +''' + +[[spec]] +level = "MUST" +quote = ''' +REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the +NAT mapping. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +a) The NAT's default configuration SHOULD NOT filter ICMP messages +based on their source IP address. +''' + +[[spec]] +level = "SHOULD" +quote = ''' +b) It is RECOMMENDED that a NAT support ICMP Destination +Unreachable messages. +''' + diff --git a/.duvet/snapshot.txt b/.duvet/snapshot.txt index 3881a4c39b..6e05273b0f 100644 --- a/.duvet/snapshot.txt +++ b/.duvet/snapshot.txt @@ -1,3 +1,176 @@ +SPECIFICATION: https://www.rfc-editor.org/rfc/rfc4787 + SECTION: [Address and Port Mapping](#section-4.1) + TEXT[!MUST,todo]: REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior. + TEXT[!SHOULD]: REQ-2: It is RECOMMENDED that a NAT have an "IP address pooling" + TEXT[!SHOULD]: behavior of "Paired". + + SECTION: [Port Assignment Behavior](#section-4.2.1) + TEXT[!MUST,implementation,test]: REQ-3: A NAT MUST NOT have a "Port assignment" behavior of "Port + TEXT[!MUST,implementation,test]: overloading". + TEXT[!SHOULD,exception]: a) If the host's source port was in the range 0-1023, it is + TEXT[!SHOULD,exception]: RECOMMENDED the NAT's source port be in the same range. + TEXT[!SHOULD]: If the + TEXT[!SHOULD]: host's source port was in the range 1024-65535, it is + TEXT[!SHOULD]: RECOMMENDED that the NAT's source port be in that range. + + SECTION: [Port Parity](#section-4.2.2) + TEXT[!SHOULD]: REQ-4: It is RECOMMENDED that a NAT have a "Port parity + TEXT[!SHOULD]: preservation" behavior of "Yes". + + SECTION: [Mapping Refresh](#section-4.3) + TEXT[!MUST,todo]: REQ-5: A NAT UDP mapping timer MUST NOT expire in less than two + TEXT[!MUST,todo]: minutes, unless REQ-5a applies. + TEXT[!MAY,implementation]: a) For specific destination ports in the well-known port range + TEXT[!MAY,implementation]: (ports 0-1023), a NAT MAY have shorter UDP mapping timers that + TEXT[!MAY,implementation]: are specific to the IANA-registered application running over + TEXT[!MAY,implementation]: that specific destination port. + TEXT[!MAY]: b) The value of the NAT UDP mapping timer MAY be configurable. + TEXT[!SHOULD,todo]: c) A default value of five minutes or more for the NAT UDP mapping + TEXT[!SHOULD,todo]: timer is RECOMMENDED. + TEXT[!MUST]: REQ-6: The NAT mapping Refresh Direction MUST have a "NAT Outbound + TEXT[!MUST]: refresh behavior" of "True". + TEXT[!MAY]: a) The NAT mapping Refresh Direction MAY have a "NAT Inbound + TEXT[!MAY]: refresh behavior" of "True". + + SECTION: [Conflicting Internal and External IP Address Spaces](#section-4.4) + TEXT[!MUST]: REQ-7: A NAT device whose external IP interface can be configured + TEXT[!MUST]: dynamically MUST either (1) automatically ensure that its internal + TEXT[!MUST]: network uses IP addresses that do not conflict with its external + TEXT[!MUST]: network, or (2) be able to translate and forward traffic between + TEXT[!MUST]: all internal nodes and all external nodes whose IP addresses + TEXT[!MUST]: numerically conflict with the internal network. + + SECTION: [Filtering Behavior](#section-5) + TEXT[!SHOULD]: REQ-8: If application transparency is most important, it is + TEXT[!SHOULD]: RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" + TEXT[!SHOULD]: behavior. + TEXT[!SHOULD]: If a more stringent filtering behavior is most + TEXT[!SHOULD]: important, it is RECOMMENDED that a NAT have an "Address-Dependent + TEXT[!SHOULD]: Filtering" behavior. + TEXT[!MAY]: a) The filtering behavior MAY be an option configurable by the + TEXT[!MAY]: administrator of the NAT. + + SECTION: [Hairpinning Behavior](#section-6) + TEXT[!MUST,todo]: REQ-9: A NAT MUST support "Hairpinning". + TEXT[!MUST]: a) A NAT Hairpinning behavior MUST be "External source IP address + TEXT[!MUST]: and port". + + SECTION: [Application Level Gateways](#section-7) + TEXT[!SHOULD]: REQ-10: To eliminate interference with UNSAF NAT traversal + TEXT[!SHOULD]: mechanisms and allow integrity protection of UDP communications, + TEXT[!SHOULD]: NAT ALGs for UDP-based protocols SHOULD be turned off. + TEXT[!SHOULD]: a) If a NAT includes ALGs, it is RECOMMENDED that the NAT allow + TEXT[!SHOULD]: the NAT administrator to enable or disable each ALG separately. + + SECTION: [Deterministic Properties](#section-8) + TEXT[!MUST]: REQ-11: A NAT MUST have deterministic behavior, i.e., it MUST NOT + TEXT[!MUST]: change the NAT translation (Section 4) or the Filtering + TEXT[!MUST]: (Section 5) Behavior at any point in time, or under any particular + TEXT[!MUST]: conditions. + + SECTION: [ICMP Destination Unreachable Behavior](#section-9) + TEXT[!SHOULD]: The NAT's default configuration SHOULD NOT filter + TEXT[!SHOULD]: ICMP messages based on their source IP address. + TEXT[!SHOULD]: Such ICMP messages + TEXT[!SHOULD]: SHOULD be rewritten by the NAT (specifically, the IP headers and the + TEXT[!SHOULD]: ICMP payload) and forwarded to the appropriate internal or external + TEXT[!SHOULD]: host. + TEXT[!MUST]: Receipt of any sort of ICMP message MUST NOT + TEXT[!MUST]: destroy the NAT mapping. + TEXT[!MUST,implementation]: REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the + TEXT[!MUST,implementation]: NAT mapping. + TEXT[!SHOULD]: a) The NAT's default configuration SHOULD NOT filter ICMP messages + TEXT[!SHOULD]: based on their source IP address. + TEXT[!SHOULD]: b) It is RECOMMENDED that a NAT support ICMP Destination + TEXT[!SHOULD]: Unreachable messages. + + SECTION: [Fragmentation of Outgoing Packets](#section-10) + TEXT[!MUST]: REQ-13: If the packet received on an internal IP address has DF=1, + TEXT[!MUST]: the NAT MUST send back an ICMP message "Fragmentation needed and + TEXT[!MUST]: DF set" to the host, as described in [RFC0792]. + TEXT[!MUST]: a) If the packet has DF=0, the NAT MUST fragment the packet and + TEXT[!MUST]: SHOULD send the fragments in order. + + SECTION: [Receiving Fragmented Packets](#section-11) + TEXT[!MUST,todo]: REQ-14: A NAT MUST support receiving in-order and out-of-order + TEXT[!MUST,todo]: fragments, so it MUST have "Received Fragment Out of Order" + TEXT[!MUST,todo]: behavior. + TEXT[!MUST]: a) A NAT's out-of-order fragment processing mechanism MUST be + TEXT[!MUST]: designed so that fragmentation-based DoS attacks do not + TEXT[!MUST]: compromise the NAT's ability to process in-order and + TEXT[!MUST]: unfragmented IP packets. + + SECTION: [Requirements](#section-12) + TEXT[!MUST]: REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior. + TEXT[!SHOULD]: REQ-2: It is RECOMMENDED that a NAT have an "IP address pooling" + TEXT[!SHOULD]: behavior of "Paired". + TEXT[!MUST]: REQ-3: A NAT MUST NOT have a "Port assignment" behavior of "Port + TEXT[!MUST]: overloading". + TEXT[!SHOULD]: a) If the host's source port was in the range 0-1023, it is + TEXT[!SHOULD]: RECOMMENDED the NAT's source port be in the same range. + TEXT[!SHOULD]: If the + TEXT[!SHOULD]: host's source port was in the range 1024-65535, it is + TEXT[!SHOULD]: RECOMMENDED that the NAT's source port be in that range. + TEXT[!SHOULD]: REQ-4: It is RECOMMENDED that a NAT have a "Port parity + TEXT[!SHOULD]: preservation" behavior of "Yes". + TEXT[!MUST]: REQ-5: A NAT UDP mapping timer MUST NOT expire in less than two + TEXT[!MUST]: minutes, unless REQ-5a applies. + TEXT[!MAY]: a) For specific destination ports in the well-known port range + TEXT[!MAY]: (ports 0-1023), a NAT MAY have shorter UDP mapping timers that + TEXT[!MAY]: are specific to the IANA-registered application running over + TEXT[!MAY]: that specific destination port. + TEXT[!MAY]: b) The value of the NAT UDP mapping timer MAY be configurable. + TEXT[!SHOULD]: c) A default value of five minutes or more for the NAT UDP mapping + TEXT[!SHOULD]: timer is RECOMMENDED. + TEXT[!MUST]: REQ-6: The NAT mapping Refresh Direction MUST have a "NAT Outbound + TEXT[!MUST]: refresh behavior" of "True". + TEXT[!MAY]: a) The NAT mapping Refresh Direction MAY have a "NAT Inbound + TEXT[!MAY]: refresh behavior" of "True". + TEXT[!MUST]: REQ-7 A NAT device whose external IP interface can be configured + TEXT[!MUST]: dynamically MUST either (1) Automatically ensure that its internal + TEXT[!MUST]: network uses IP addresses that do not conflict with its external + TEXT[!MUST]: network, or (2) Be able to translate and forward traffic between + TEXT[!MUST]: all internal nodes and all external nodes whose IP addresses + TEXT[!MUST]: numerically conflict with the internal network. + TEXT[!SHOULD]: REQ-8: If application transparency is most important, it is + TEXT[!SHOULD]: RECOMMENDED that a NAT have "Endpoint-Independent Filtering" + TEXT[!SHOULD]: behavior. + TEXT[!SHOULD]: If a more stringent filtering behavior is most + TEXT[!SHOULD]: important, it is RECOMMENDED that a NAT have "Address-Dependent + TEXT[!SHOULD]: Filtering" behavior. + TEXT[!MAY]: a) The filtering behavior MAY be an option configurable by the + TEXT[!MAY]: administrator of the NAT. + TEXT[!MUST]: REQ-9: A NAT MUST support "Hairpinning". + TEXT[!MUST]: a) A NAT Hairpinning behavior MUST be "External source IP address + TEXT[!MUST]: and port". + TEXT[!SHOULD]: REQ-10: To eliminate interference with UNSAF NAT traversal + TEXT[!SHOULD]: mechanisms and allow integrity protection of UDP communications, + TEXT[!SHOULD]: NAT ALGs for UDP-based protocols SHOULD be turned off. + TEXT[!SHOULD]: a) If a NAT includes ALGs, it is RECOMMENDED that the NAT allow + TEXT[!SHOULD]: the NAT administrator to enable or disable each ALG separately. + TEXT[!MUST]: REQ-11: A NAT MUST have deterministic behavior, i.e., it MUST NOT + TEXT[!MUST]: change the NAT translation (Section 4) or the Filtering + TEXT[!MUST]: (Section 5) Behavior at any point in time, or under any particular + TEXT[!MUST]: conditions. + TEXT[!MUST]: REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the + TEXT[!MUST]: NAT mapping. + TEXT[!SHOULD]: a) The NAT's default configuration SHOULD NOT filter ICMP messages + TEXT[!SHOULD]: based on their source IP address. + TEXT[!SHOULD]: b) It is RECOMMENDED that a NAT support ICMP Destination + TEXT[!SHOULD]: Unreachable messages. + TEXT[!MUST]: REQ-13 If the packet received on an internal IP address has DF=1, + TEXT[!MUST]: the NAT MUST send back an ICMP message "Fragmentation needed and + TEXT[!MUST]: DF set" to the host, as described in [RFC0792]. + TEXT[!MUST]: a) If the packet has DF=0, the NAT MUST fragment the packet and + TEXT[!MUST]: SHOULD send the fragments in order. + TEXT[!MUST]: REQ-14: A NAT MUST support receiving in-order and out-of-order + TEXT[!MUST]: fragments, so it MUST have "Received Fragment Out of Order" + TEXT[!MUST]: behavior. + TEXT[!MUST]: a) A NAT's out-of-order fragment processing mechanism MUST be + TEXT[!MUST]: designed so that fragmentation-based DoS attacks do not + TEXT[!MUST]: compromise the NAT's ability to process in-order and + TEXT[!MUST]: unfragmented IP packets. + SPECIFICATION: https://www.rfc-editor.org/rfc/rfc4884 SECTION: [Summary of Changes to ICMP](#section-3) TEXT[!MAY]: An ICMP Extension Structure MAY be appended to ICMPv4 Destination diff --git a/.duvet/specifications/www.rfc-editor.org/rfc/rfc4787.txt b/.duvet/specifications/www.rfc-editor.org/rfc/rfc4787.txt new file mode 100644 index 0000000000..00219c79bf --- /dev/null +++ b/.duvet/specifications/www.rfc-editor.org/rfc/rfc4787.txt @@ -0,0 +1,1627 @@ + + + + + + +Network Working Group F. Audet, Ed. +Request for Comments: 4787 Nortel Networks +BCP: 127 C. Jennings +Category: Best Current Practice Cisco Systems + January 2007 + + + Network Address Translation (NAT) Behavioral Requirements + for Unicast UDP + +Status of This Memo + + This document specifies an Internet Best Current Practices for the + Internet Community, and requests discussion and suggestions for + improvements. Distribution of this memo is unlimited. + +Copyright Notice + + Copyright (C) The IETF Trust (2007). + +Abstract + + This document defines basic terminology for describing different + types of Network Address Translation (NAT) behavior when handling + Unicast UDP and also defines a set of requirements that would allow + many applications, such as multimedia communications or online + gaming, to work consistently. Developing NATs that meet this set of + requirements will greatly increase the likelihood that these + applications will function properly. + + + + + + + + + + + + + + + + + + + + + + +Audet & Jennings Best Current Practice [Page 1] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + +Table of Contents + + 1. Applicability Statement . . . . . . . . . . . . . . . . . . . 3 + 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 + 3. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 + 4. Network Address and Port Translation Behavior . . . . . . . . 5 + 4.1. Address and Port Mapping . . . . . . . . . . . . . . . . . 5 + 4.2. Port Assignment . . . . . . . . . . . . . . . . . . . . . 9 + 4.2.1. Port Assignment Behavior . . . . . . . . . . . . . . . 9 + 4.2.2. Port Parity . . . . . . . . . . . . . . . . . . . . . 11 + 4.2.3. Port Contiguity . . . . . . . . . . . . . . . . . . . 11 + 4.3. Mapping Refresh . . . . . . . . . . . . . . . . . . . . . 12 + 4.4. Conflicting Internal and External IP Address Spaces . . . 13 + 5. Filtering Behavior . . . . . . . . . . . . . . . . . . . . . . 15 + 6. Hairpinning Behavior . . . . . . . . . . . . . . . . . . . . . 16 + 7. Application Level Gateways . . . . . . . . . . . . . . . . . . 17 + 8. Deterministic Properties . . . . . . . . . . . . . . . . . . . 18 + 9. ICMP Destination Unreachable Behavior . . . . . . . . . . . . 19 + 10. Fragmentation of Outgoing Packets . . . . . . . . . . . . . . 20 + 11. Receiving Fragmented Packets . . . . . . . . . . . . . . . . . 20 + 12. Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 21 + 13. Security Considerations . . . . . . . . . . . . . . . . . . . 24 + 14. IAB Considerations . . . . . . . . . . . . . . . . . . . . . . 25 + 15. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 26 + 16. References . . . . . . . . . . . . . . . . . . . . . . . . . . 26 + 16.1. Normative References . . . . . . . . . . . . . . . . . . . 26 + 16.2. Informative References . . . . . . . . . . . . . . . . . . 26 + + + + + + + + + + + + + + + + + + + + + + + + +Audet & Jennings Best Current Practice [Page 2] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + +1. Applicability Statement + + The purpose of this specification is to define a set of requirements + for NATs that would allow many applications, such as multimedia + communications or online gaming, to work consistently. Developing + NATs that meet this set of requirements will greatly increase the + likelihood that these applications will function properly. + + The requirements of this specification apply to Traditional NATs as + described in [RFC2663]. + + This document is meant to cover NATs of any size, from small + residential NATs to large Enterprise NATs. However, it should be + understood that Enterprise NATs normally provide much more than just + NAT capabilities; for example, they typically provide firewall + functionalities. A comprehensive description of firewall behaviors + and associated requirements is specifically out-of-scope for this + specification. However, this specification does cover basic firewall + aspects present in NATs (see Section 5). + + Approaches using directly signaled control of middle boxes are out of + scope. + + UDP Relays (e.g., Traversal Using Relay NAT [TURN]) are out of scope. + + Application aspects are out of scope, as the focus here is strictly + on the NAT itself. + + This document only covers aspects of NAT traversal related to Unicast + UDP [RFC0768] over IP [RFC0791] and their dependencies on other + protocols. + +2. Introduction + + Network Address Translators (NATs) are well known to cause very + significant problems with applications that carry IP addresses in the + payload (see [RFC3027]). Applications that suffer from this problem + include Voice Over IP and Multimedia Over IP (e.g., SIP [RFC3261] and + H.323 [ITU.H323]), as well as online gaming. + + Many techniques are used to attempt to make realtime multimedia + applications, online games, and other applications work across NATs. + Application Level Gateways [RFC2663] are one such mechanism. STUN + [RFC3489bis] describes a UNilateral Self-Address Fixing (UNSAF) + mechanism [RFC3424]. Teredo [RFC4380] describes an UNSAF mechanism + consisting of tunnelling IPv6 [RFC2460] over UDP/IPv4. UDP Relays + have also been used to enable applications across NATs, but these are + generally seen as a solution of last resort. Interactive + + + +Audet & Jennings Best Current Practice [Page 3] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + Connectivity Establishment [ICE] describes a methodology for using + many of these techniques and avoiding a UDP relay, unless the type of + NAT is such that it forces the use of such a UDP relay. This + specification defines requirements for improving NATs. Meeting these + requirements ensures that applications will not be forced to use UDP + relay. + + As pointed out in UNSAF [RFC3424], "From observations of deployed + networks, it is clear that different NAT box implementations vary + widely in terms of how they handle different traffic and addressing + cases". This wide degree of variability is one factor in the overall + brittleness introduced by NATs and makes it extremely difficult to + predict how any given protocol will behave on a network traversing + NAT. Discussions with many of the major NAT vendors have made it + clear that they would prefer to deploy NATs that were deterministic + and caused the least harm to applications while still meeting the + requirements that caused their customers to deploy NATs in the first + place. The problem NAT vendors face is that they are not sure how + best to do that or how to document their NATs' behavior. + + The goals of this document are to define a set of common terminology + for describing the behavior of NATs and to produce a set of + requirements on a specific set of behaviors for NATs. + + This document forms a common set of requirements that are simple and + useful for voice, video, and games, which can be implemented by NAT + vendors. This document will simplify the analysis of protocols for + deciding whether or not they work in this environment and will allow + providers of services that have NAT traversal issues to make + statements about where their applications will work and where they + will not, as well as to specify their own NAT requirements. + +3. Terminology + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in [RFC2119]. + + Readers are urged to refer to [RFC2663] for information on NAT + taxonomy and terminology. Traditional NAT is the most common type of + NAT device deployed. Readers may refer to [RFC3022] for detailed + information on traditional NAT. Traditional NAT has two main + varieties -- Basic NAT and Network Address/Port Translator (NAPT). + + NAPT is by far the most commonly deployed NAT device. NAPT allows + multiple internal hosts to share a single public IP address + simultaneously. When an internal host opens an outgoing TCP or UDP + session through a NAPT, the NAPT assigns the session a public IP + + + +Audet & Jennings Best Current Practice [Page 4] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + address and port number, so that subsequent response packets from the + external endpoint can be received by the NAPT, translated, and + forwarded to the internal host. The effect is that the NAPT + establishes a NAT session to translate the (private IP address, + private port number) tuple to a (public IP address, public port + number) tuple, and vice versa, for the duration of the session. An + issue of relevance to peer-to-peer applications is how the NAT + behaves when an internal host initiates multiple simultaneous + sessions from a single (private IP, private port) endpoint to + multiple distinct endpoints on the external network. In this + specification, the term "NAT" refers to both "Basic NAT" and "Network + Address/Port Translator (NAPT)". + + This document uses the term "session" as defined in RFC 2663: "TCP/ + UDP sessions are uniquely identified by the tuple of (source IP + address, source TCP/UDP ports, target IP address, target TCP/UDP + Port)". + + This document uses the term "address and port mapping" as the + translation between an external address and port and an internal + address and port. Note that this is not the same as an "address + binding" as defined in RFC 2663. + + This document uses IANA terminology for port ranges, i.e., "Well + Known Ports" is 0-1023, "Registered" is 1024-49151, and "Dynamic + and/or Private" is 49152-65535, as defined in + http://www.iana.org/assignments/port-numbers. + + STUN [RFC3489] used the terms "Full Cone", "Restricted Cone", "Port + Restricted Cone", and "Symmetric" to refer to different variations of + NATs applicable to UDP only. Unfortunately, this terminology has + been the source of much confusion, as it has proven inadequate at + describing real-life NAT behavior. This specification therefore + refers to specific individual NAT behaviors instead of using the + Cone/Symmetric terminology. + +4. Network Address and Port Translation Behavior + + This section describes the various NAT behaviors applicable to NATs. + +4.1. Address and Port Mapping + + When an internal endpoint opens an outgoing session through a NAT, + the NAT assigns the session an external IP address and port number so + that subsequent response packets from the external endpoint can be + received by the NAT, translated, and forwarded to the internal + endpoint. This is a mapping between an internal IP address and port + IP:port and external IP:port tuple. It establishes the translation + + + +Audet & Jennings Best Current Practice [Page 5] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + that will be performed by the NAT for the duration of the session. + For many applications, it is important to distinguish the behavior of + the NAT when there are multiple simultaneous sessions established to + different external endpoints. + + The key behavior to describe is the criteria for reuse of a mapping + for new sessions to external endpoints, after establishing a first + mapping between an internal X:x address and port and an external + Y1:y1 address tuple. Let's assume that the internal IP address and + port X:x are mapped to X1':x1' for this first session. The endpoint + then sends from X:x to an external address Y2:y2 and gets a mapping + of X2':x2' on the NAT. The relationship between X1':x1' and X2':x2' + for various combinations of the relationship between Y1:y1 and Y2:y2 + is critical for describing the NAT behavior. This arrangement is + illustrated in the following diagram: + + E + +------+ +------+ x + | Y1 | | Y2 | t + +--+---+ +---+--+ e + | Y1:y1 Y2:y2 | r + +----------+ +----------+ n + | | a + X1':x1' | | X2':x2' l + +--+---+-+ + ...........| NAT |............... + +--+---+-+ I + | | n + X:x | | X:x t + ++---++ e + | X | r + +-----+ n + a + l + + Address and Port Mapping + + The following address and port mapping behavior are defined: + + Endpoint-Independent Mapping: + + The NAT reuses the port mapping for subsequent packets sent + from the same internal IP address and port (X:x) to any + external IP address and port. Specifically, X1':x1' equals + X2':x2' for all values of Y2:y2. + + + + + + +Audet & Jennings Best Current Practice [Page 6] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + Address-Dependent Mapping: + + The NAT reuses the port mapping for subsequent packets sent + from the same internal IP address and port (X:x) to the same + external IP address, regardless of the external port. + Specifically, X1':x1' equals X2':x2' if and only if, Y2 equals + Y1. + + Address and Port-Dependent Mapping: + + The NAT reuses the port mapping for subsequent packets sent + from the same internal IP address and port (X:x) to the same + external IP address and port while the mapping is still active. + Specifically, X1':x1' equals X2':x2' if and only if, Y2:y2 + equals Y1:y1. + + It is important to note that these three possible choices make no + difference to the security properties of the NAT. The security + properties are fully determined by which packets the NAT allows in + and which it does not. This is determined by the filtering behavior + in the filtering portions of the NAT. + + REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior. + + Justification: In order for UNSAF methods to work, REQ-1 needs to be + met. Failure to meet REQ-1 will force the use of a UDP relay, + which is very often impractical. + + Some NATs are capable of assigning IP addresses from a pool of IP + addresses on the external side of the NAT, as opposed to just a + single IP address. This is especially common with larger NATs. Some + NATs use the external IP address mapping in an arbitrary fashion + (i.e., randomly): one internal IP address could have multiple + external IP address mappings active at the same time for different + sessions. These NATs have an "IP address pooling" behavior of + "Arbitrary". Some large Enterprise NATs use an IP address pooling + behavior of "Arbitrary" as a means of hiding the IP address assigned + to specific endpoints by making their assignment less predictable. + Other NATs use the same external IP address mapping for all sessions + associated with the same internal IP address. These NATs have an "IP + address pooling" behavior of "Paired". NATs that use an "IP address + pooling" behavior of "Arbitrary" can cause issues for applications + that use multiple ports from the same endpoint, but that do not + negotiate IP addresses individually (e.g., some applications using + RTP and RTCP). + + + + + + +Audet & Jennings Best Current Practice [Page 7] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + REQ-2: It is RECOMMENDED that a NAT have an "IP address pooling" + behavior of "Paired". Note that this requirement is not + applicable to NATs that do not support IP address pooling. + + Justification: This will allow applications that use multiple ports + originating from the same internal IP address to also have the + same external IP address. This is to avoid breaking peer-to-peer + applications that are not capable of negotiating the IP address + for RTP and the IP address for RTCP separately. As such it is + envisioned that this requirement will become less important as + applications become NAT-friendlier with time. The main reason why + this requirement is here is that in a peer-to-peer application, + you are subject to the other peer's mistake. In particular, in + the context of SIP, if my application supports the extensions + defined in [RFC3605] for indicating RTP and RTCP addresses and + ports separately, but the other peer does not, there may still be + breakage in the form of the stream losing RTCP packets. This + requirement will avoid the loss of RTP in this context, although + the loss of RTCP may be inevitable in this particular example. It + is also worth noting that RFC 3605 is unfortunately not a + mandatory part of SIP [RFC3261]. Therefore, this requirement will + address a particularly nasty problem that will prevail for a + significant period of time. + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Audet & Jennings Best Current Practice [Page 8] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + +4.2. Port Assignment + +4.2.1. Port Assignment Behavior + + This section uses the following diagram for reference. + + E + +-------+ +-------+ x + | Y1 | | Y2 | t + +---+---+ +---+---+ e + | Y1:y1 Y2:y2 | r + +---------+ +---------+ n + | | a + X1':x1' | | X2':x2' l + +--+---+--+ + ...........| NAT |............... + +--+---+--+ I + | | n + +---------+ +---------+ t + | X1:x1 X2:x2 | e + +---+---+ +---+---+ r + | X1 | | X2 | n + +-------+ +-------+ a + l + + Port Assignment + + Some NATs attempt to preserve the port number used internally when + assigning a mapping to an external IP address and port (e.g., x1=x1', + x2=x2'). This port assignment behavior is referred to as "port + preservation". In case of port collision, these NATs attempt a + variety of techniques for coping. For example, some NATs will + overridden the previous mapping to preserve the same port. Other + NATs will assign a different IP address from a pool of external IP + addresses; this is only possible as long as the NAT has enough + external IP addresses; if the port is already in use on all available + external IP addresses, then these NATs will pick a different port + (i.e., they don't do port preservation anymore). + + Some NATs use "Port overloading", i.e., they always use port + preservation even in the case of collision (i.e., X1'=X2' and + x1=x2=x1'=x2'). Most applications will fail if the NAT uses "Port + overloading". + + A NAT that does not attempt to make the external port numbers match + the internal port numbers in any case is referred to as "no port + preservation". + + + + +Audet & Jennings Best Current Practice [Page 9] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + When NATs do allocate a new source port, there is the issue of which + IANA-defined range of port to choose. The ranges are "well-known" + from 0 to 1023, "registered" from 1024 to 49151, and "dynamic/ + private" from 49152 through 65535. For most protocols, these are + destination ports and not source ports, so mapping a source port to a + source port that is already registered is unlikely to have any bad + effects. Some NATs may choose to use only the ports in the dynamic + range; the only downside of this practice is that it limits the + number of ports available. Other NAT devices may use everything but + the well-known range and may prefer to use the dynamic range first, + or possibly avoid the actual registered ports in the registered + range. Other NATs preserve the port range if it is in the well-known + range. [RFC0768] specifies that the source port is set to zero if no + reply packets are expected. In this case, it does not matter what + the NAT maps it to, as the source port will not be used. However, + many common OS APIs do not allow a user to send from port zero, + applications do not use port zero, and the behavior of various + existing NATs with regards to a packet with a source of port zero is + unknown. This document does not specify any normative behavior for a + NAT when handling a packet with a source port of zero which means + that applications cannot count on any sort of deterministic behavior + for these packets. + + REQ-3: A NAT MUST NOT have a "Port assignment" behavior of "Port + overloading". + + a) If the host's source port was in the range 0-1023, it is + RECOMMENDED the NAT's source port be in the same range. If the + host's source port was in the range 1024-65535, it is + RECOMMENDED that the NAT's source port be in that range. + + Justification: This requirement must be met in order to enable two + applications on the internal side of the NAT both to use the same + port to try to communicate with the same destination. NATs that + implement port preservation have to deal with conflicts on ports, + and the multiple code paths this introduces often result in + nondeterministic behavior. However, it should be understood that + when a port is randomly assigned, it may just randomly happen to + be assigned the same port. Applications must, therefore, be able + to deal with both port preservation and no port preservation. + + a) Certain applications expect the source UDP port to be in the + well-known range. See the discussion of Network File System + port expectations in [RFC2623] for an example. + + + + + + + +Audet & Jennings Best Current Practice [Page 10] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + +4.2.2. Port Parity + + Some NATs preserve the parity of the UDP port, i.e., an even port + will be mapped to an even port, and an odd port will be mapped to an + odd port. This behavior respects the [RFC3550] rule that RTP use + even ports, and RTCP use odd ports. RFC 3550 allows any port numbers + to be used for RTP and RTCP if the two numbers are specified + separately; for example, using [RFC3605]. However, some + implementations do not include RFC 3605, and do not recognize when + the peer has specified the RTCP port separately using RFC 3605. If + such an implementation receives an odd RTP port number from the peer + (perhaps after having been translated by a NAT), and then follows the + RFC 3550 rule to change the RTP port to the next lower even number, + this would obviously result in the loss of RTP. NAT-friendly + application aspects are outside the scope of this document. It is + expected that this issue will fade away with time, as implementations + improve. Preserving the port parity allows for supporting + communication with peers that do not support explicit specification + of both RTP and RTCP port numbers. + + REQ-4: It is RECOMMENDED that a NAT have a "Port parity + preservation" behavior of "Yes". + + Justification: This is to avoid breaking peer-to-peer applications + that do not explicitly and separately specify RTP and RTCP port + numbers and that follow the RFC 3550 rule to decrement an odd RTP + port to make it even. The same considerations apply, as per the + IP address pooling requirement. + +4.2.3. Port Contiguity + + Some NATs attempt to preserve the port contiguity rule of RTCP=RTP+1. + These NATs do things like sequential assignment or port reservation. + Sequential port assignment assumes that the application will open a + mapping for RTP first and then open a mapping for RTCP. It is not + practical to enforce this requirement on all applications. + Furthermore, there is a problem with glare if many applications (or + endpoints) are trying to open mappings simultaneously. Port + preservation is also problematic since it is wasteful, especially + considering that a NAT cannot reliably distinguish between RTP over + UDP and other UDP packets where there is no contiguity rule. For + those reasons, it would be too complex to attempt to preserve the + contiguity rule by suggesting specific NAT behavior, and it would + certainly break the deterministic behavior rule. + + In order to support both RTP and RTCP, it will therefore be necessary + that applications follow rules to negotiate RTP and RTCP separately, + and account for the very real possibility that the RTCP=RTP+1 rule + + + +Audet & Jennings Best Current Practice [Page 11] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + will be broken. As this is an application requirement, it is outside + the scope of this document. + +4.3. Mapping Refresh + + NAT mapping timeout implementations vary, but include the timer's + value and the way the mapping timer is refreshed to keep the mapping + alive. + + The mapping timer is defined as the time a mapping will stay active + without packets traversing the NAT. There is great variation in the + values used by different NATs. + + REQ-5: A NAT UDP mapping timer MUST NOT expire in less than two + minutes, unless REQ-5a applies. + + a) For specific destination ports in the well-known port range + (ports 0-1023), a NAT MAY have shorter UDP mapping timers that + are specific to the IANA-registered application running over + that specific destination port. + + b) The value of the NAT UDP mapping timer MAY be configurable. + + c) A default value of five minutes or more for the NAT UDP mapping + timer is RECOMMENDED. + + Justification: This requirement is to ensure that the timeout is + long enough to avoid too-frequent timer refresh packets. + + a) Some UDP protocols using UDP use very short-lived connections. + There can be very many such connections; keeping them all in a + connections table could cause considerable load on the NAT. + Having shorter timers for these specific applications is, + therefore, an optimization technique. It is important that the + shorter timers applied to specific protocols be used sparingly, + and only for protocols using well-known destination ports that + are known to have a shorter timer, and that are known not to be + used by any applications for other purposes. + + b) Configuration is desirable for adapting to specific networks + and troubleshooting. + + c) This default is to avoid too-frequent timer refresh packets. + + Some NATs keep the mapping active (i.e., refresh the timer value) + when a packet goes from the internal side of the NAT to the external + side of the NAT. This is referred to as having a NAT Outbound + refresh behavior of "True". + + + +Audet & Jennings Best Current Practice [Page 12] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + Some NATs keep the mapping active when a packet goes from the + external side of the NAT to the internal side of the NAT. This is + referred to as having a NAT Inbound Refresh Behavior of "True". + + Some NATs keep the mapping active on both, in which case, both + properties are "True". + + REQ-6: The NAT mapping Refresh Direction MUST have a "NAT Outbound + refresh behavior" of "True". + + a) The NAT mapping Refresh Direction MAY have a "NAT Inbound + refresh behavior" of "True". + + Justification: Outbound refresh is necessary for allowing the client + to keep the mapping alive. + + a) Inbound refresh may be useful for applications with no outgoing + UDP traffic. However, allowing inbound refresh may allow an + external attacker or misbehaving application to keep a mapping + alive indefinitely. This may be a security risk. Also, if the + process is repeated with different ports, over time, it could + use up all the ports on the NAT. + +4.4. Conflicting Internal and External IP Address Spaces + + Many NATs, particularly consumer-level devices designed to be + deployed by nontechnical users, routinely obtain their external IP + address, default router, and other IP configuration information for + their external interface dynamically from an external network, such + as an upstream ISP. The NAT, in turn, automatically sets up its own + internal subnet in one of the private IP address spaces assigned to + this purpose in [RFC1918], typically providing dynamic IP + configuration services for hosts on this internal network. + + Auto-configuration of NATs and private networks can be problematic, + however, if the NAT's external network is also in RFC 1918 private + address space. In a common scenario, an ISP places its customers + behind a NAT and hands out private RFC 1918 addresses to them. Some + of these customers, in turn, deploy consumer-level NATs, which, in + effect, act as "second-level" NATs, multiplexing their own private + RFC 1918 IP subnets onto the single RFC 1918 IP address provided by + the ISP. There is no inherent guarantee, in this case, that the + ISP's "intermediate" privately-addressed network and the customer's + internal privately-addressed network will not use numerically + identical or overlapping RFC 1918 IP subnets. Furthermore, customers + of consumer-level NATs cannot be expected to have the technical + + + + + +Audet & Jennings Best Current Practice [Page 13] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + knowledge to prevent this scenario from occurring by manually + configuring their internal network with non-conflicting RFC 1918 + subnets. + + NAT vendors need to design their NATs to ensure that they function + correctly and robustly even in such problematic scenarios. One + possible solution is for the NAT to ensure that whenever its external + link is configured with an RFC 1918 private IP address, the NAT + automatically selects a different, non-conflicting RFC 1918 IP subnet + for its internal network. A disadvantage of this solution is that, + if the NAT's external interface is dynamically configured or re- + configured after its internal network is already in use, then the NAT + may have to renumber its entire internal network dynamically if it + detects a conflict. + + An alternative solution is for the NAT to be designed so that it can + translate and forward traffic correctly, even when its external and + internal interfaces are configured with numerically overlapping IP + subnets. In this scenario, for example, if the NAT's external + interface has been assigned an IP address P in RFC 1918 space, then + there might also be an internal node I having the same RFC 1918 + private IP address P. An IP packet with destination address P on the + external network is directed at the NAT, whereas an IP packet with + the same destination address P on the internal network is directed at + node I. The NAT therefore needs to maintain a clear operational + distinction between "external IP addresses" and "internal IP + addresses" to avoid confusing internal node I with its own external + interface. In general, the NAT needs to allow all internal nodes + (including I) to communicate with all external nodes having public + (non-RFC 1918) IP addresses, or having private IP addresses that do + not conflict with the addresses used by its internal network. + + REQ-7: A NAT device whose external IP interface can be configured + dynamically MUST either (1) automatically ensure that its internal + network uses IP addresses that do not conflict with its external + network, or (2) be able to translate and forward traffic between + all internal nodes and all external nodes whose IP addresses + numerically conflict with the internal network. + + Justification: If a NAT's external and internal interfaces are + configured with overlapping IP subnets, then there is, of course, + no way for an internal host with RFC 1918 IP address Q to initiate + a direct communication session to an external node having the same + RFC 1918 address Q, or to other external nodes with IP addresses + that numerically conflict with the internal subnet. Such nodes + can still open communication sessions indirectly via NAT traversal + techniques, however, with the help of a third-party server, such + as a STUN server having a public, non-RFC 1918 IP address. In + + + +Audet & Jennings Best Current Practice [Page 14] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + this case, nodes with conflicting private RFC 1918 addresses on + opposite sides of the second-level NAT can communicate with each + other via their respective temporary public endpoints on the main + Internet, as long as their common, first-level NAT (e.g., the + upstream ISP's NAT) supports hairpinning behavior, as described in + Section 6. + +5. Filtering Behavior + + This section describes various filtering behaviors observed in NATs. + + When an internal endpoint opens an outgoing session through a NAT, + the NAT assigns a filtering rule for the mapping between an internal + IP:port (X:x) and external IP:port (Y:y) tuple. + + The key behavior to describe is what criteria are used by the NAT to + filter packets originating from specific external endpoints. + + Endpoint-Independent Filtering: + + The NAT filters out only packets not destined to the internal + address and port X:x, regardless of the external IP address and + port source (Z:z). The NAT forwards any packets destined to + X:x. In other words, sending packets from the internal side of + the NAT to any external IP address is sufficient to allow any + packets back to the internal endpoint. + + Address-Dependent Filtering: + + The NAT filters out packets not destined to the internal + address X:x. Additionally, the NAT will filter out packets + from Y:y destined for the internal endpoint X:x if X:x has not + sent packets to Y:any previously (independently of the port + used by Y). In other words, for receiving packets from a + specific external endpoint, it is necessary for the internal + endpoint to send packets first to that specific external + endpoint's IP address. + + Address and Port-Dependent Filtering: + + This is similar to the previous behavior, except that the + external port is also relevant. The NAT filters out packets + not destined for the internal address X:x. Additionally, the + NAT will filter out packets from Y:y destined for the internal + endpoint X:x if X:x has not sent packets to Y:y previously. In + other words, for receiving packets from a specific external + endpoint, it is necessary for the internal endpoint to send + packets first to that external endpoint's IP address and port. + + + +Audet & Jennings Best Current Practice [Page 15] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + REQ-8: If application transparency is most important, it is + RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" + behavior. If a more stringent filtering behavior is most + important, it is RECOMMENDED that a NAT have an "Address-Dependent + Filtering" behavior. + + a) The filtering behavior MAY be an option configurable by the + administrator of the NAT. + + Justification: The recommendation to use Endpoint-Independent + Filtering is aimed at maximizing application transparency; in + particular, for applications that receive media simultaneously + from multiple locations (e.g., gaming), or applications that use + rendezvous techniques. However, it is also possible that, in some + circumstances, it may be preferable to have a more stringent + filtering behavior. Filtering independently of the external + endpoint is not as secure: An unauthorized packet could get + through a specific port while the port was kept open if it was + lucky enough to find the port open. In theory, filtering based on + both IP address and port is more secure than filtering based only + on the IP address (because the external endpoint could, in + reality, be two endpoints behind another NAT, where one of the two + endpoints is an attacker). However, such a policy could interfere + with applications that expect to receive UDP packets on more than + one UDP port. Using Endpoint-Independent Filtering or Address- + Dependent Filtering instead of Address and Port-Dependent + Filtering on a NAT (say, NAT-A) also has benefits when the other + endpoint is behind a non-BEHAVE compliant NAT (say, NAT-B) that + does not support REQ-1. When the endpoints use ICE, if NAT-A uses + Address and Port-Dependent Filtering, connectivity will require a + UDP relay. However, if NAT-A uses Endpoint-Independent Filtering + or Address-Dependent Filtering, ICE will ultimately find + connectivity without requiring a UDP relay. Having the filtering + behavior being an option configurable by the administrator of the + NAT ensures that a NAT can be used in the widest variety of + deployment scenarios. + +6. Hairpinning Behavior + + If two hosts (called X1 and X2) are behind the same NAT and + exchanging traffic, the NAT may allocate an address on the outside of + the NAT for X2, called X2':x2'. If X1 sends traffic to X2':x2', it + goes to the NAT, which must relay the traffic from X1 to X2. This is + referred to as hairpinning and is illustrated below. + + + + + + + +Audet & Jennings Best Current Practice [Page 16] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + NAT + +----+ from X1:x1 to X2':x2' +-----+ X1':x1' + | X1 |>>>>>>>>>>>>>>>>>>>>>>>>>>>>>--+--- + +----+ | v | + | v | + | v | + | v | + +----+ from X1':x1' to X2:x2 | v | X2':x2' + | X2 |<<<<<<<<<<<<<<<<<<<<<<<<<<<<<--+--- + +----+ +-----+ + + Hairpinning Behavior + + Hairpinning allows two endpoints on the internal side of the NAT to + communicate even if they only use each other's external IP addresses + and ports. + + More formally, a NAT that supports hairpinning forwards packets + originating from an internal address, X1:x1, destined for an external + address X2':x2' that has an active mapping to an internal address + X2:x2, back to that internal address, X2:x2. Note that typically X1' + is the same as X2'. + + Furthermore, the NAT may present the hairpinned packet with either an + internal (X1:x1) or an external (X1':x1') source IP address and port. + Therefore, the hairpinning NAT behavior can be either "External + source IP address and port" or "Internal source IP address and port". + "Internal source IP address and port" may cause problems by confusing + implementations that expect an external IP address and port. + + REQ-9: A NAT MUST support "Hairpinning". + + a) A NAT Hairpinning behavior MUST be "External source IP address + and port". + + Justification: This requirement is to allow communications between + two endpoints behind the same NAT when they are trying each + other's external IP addresses. + + a) Using the external source IP address is necessary for + applications with a restrictive policy of not accepting packets + from IP addresses that differ from what is expected. + +7. Application Level Gateways + + Certain NATs have implemented Application Level Gateways (ALGs) for + various protocols, including protocols for negotiating peer-to-peer + sessions, such as SIP. + + + +Audet & Jennings Best Current Practice [Page 17] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + Certain NATs have these ALGs turned on permanently, others have them + turned on by default but allow them to be turned off, and others have + them turned off by default but allow them be turned on. + + NAT ALGs may interfere with UNSAF methods or protocols that try to be + NAT-aware and therefore must be used with extreme caution. + + REQ-10: To eliminate interference with UNSAF NAT traversal + mechanisms and allow integrity protection of UDP communications, + NAT ALGs for UDP-based protocols SHOULD be turned off. Future + standards track specifications that define ALGs can update this to + recommend the defaults for the ALGs that they define. + + a) If a NAT includes ALGs, it is RECOMMENDED that the NAT allow + the NAT administrator to enable or disable each ALG separately. + + Justification: NAT ALGs may interfere with UNSAF methods. + + a) This requirement allows the user to enable those ALGs that are + necessary to aid in the operation of some applications without + enabling ALGs, which interfere with the operation of other + applications. + +8. Deterministic Properties + + The classification of NATs is further complicated by the fact that, + under some conditions, the same NAT will exhibit different behaviors. + This has been seen on NATs that preserve ports or have specific + algorithms for selecting a port other than a free one. If the + external port that the NAT wishes to use is already in use by another + session, the NAT must select a different port. This results in + different code paths for this conflict case, which results in + different behavior. + + For example, if three hosts X1, X2, and X3 all send from the same + port x, through a port preserving NAT with only one external IP + address, called X1', the first one to send (i.e., X1) will get an + external port of x, but the next two will get x2' and x3' (where + these are not equal to x). There are NATs where the External NAT + mapping characteristics and the External Filter characteristics + change between the X1:x and the X2:x mapping. To make matters worse, + there are NATs where the behavior may be the same on the X1:x and + X2:x mappings, but different on the third X3:x mapping. + + Another example is that some NATs have an "Endpoint-Independent + Mapping", combined with "Port Overloading", as long as two endpoints + are not establishing sessions to the same external direction, but + then switch their behavior to "Address and Port-Dependent Mapping" + + + +Audet & Jennings Best Current Practice [Page 18] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + without "Port Preservation" upon detection of these conflicting + sessions establishments. + + Any NAT that changes the NAT Mapping or the Filtering behavior + without configuration changes, at any point in time, under any + particular conditions, is referred to as a "non-deterministic" NAT. + NATs that don't are called "deterministic". + + Non-deterministic NATs generally change behavior when a conflict of + some sort happens, i.e., when the port that would normally be used is + already in use by another mapping. The NAT mapping and External + Filtering in the absence of conflict is referred to as the Primary + behavior. The behavior after the first conflict is referred to as + Secondary and after the second conflict is referred to as Tertiary. + No NATs have been observed that change on further conflicts, but it + is certainly possible that they exist. + + REQ-11: A NAT MUST have deterministic behavior, i.e., it MUST NOT + change the NAT translation (Section 4) or the Filtering + (Section 5) Behavior at any point in time, or under any particular + conditions. + + Justification: Non-deterministic NATs are very difficult to + troubleshoot because they require more intensive testing. This + non-deterministic behavior is the root cause of much of the + uncertainty that NATs introduce about whether or not applications + will work. + +9. ICMP Destination Unreachable Behavior + + When a NAT sends a packet toward a host on the other side of the NAT, + an ICMP message may be sent in response to that packet. That ICMP + message may be sent by the destination host or by any router along + the network path. The NAT's default configuration SHOULD NOT filter + ICMP messages based on their source IP address. Such ICMP messages + SHOULD be rewritten by the NAT (specifically, the IP headers and the + ICMP payload) and forwarded to the appropriate internal or external + host. The NAT needs to perform this function for as long as the UDP + mapping is active. Receipt of any sort of ICMP message MUST NOT + destroy the NAT mapping. A NAT that performs the functions described + in the paragraph above is referred to as "support ICMP Processing". + + There is no significant security advantage to blocking ICMP + Destination Unreachable packets. Additionally, blocking ICMP + Destination Unreachable packets can interfere with application + failover, UDP Path MTU Discovery (see [RFC1191] and [RFC1435]), and + traceroute. Blocking any ICMP message is discouraged, and blocking + ICMP Destination Unreachable is strongly discouraged. + + + +Audet & Jennings Best Current Practice [Page 19] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the + NAT mapping. + + a) The NAT's default configuration SHOULD NOT filter ICMP messages + based on their source IP address. + + b) It is RECOMMENDED that a NAT support ICMP Destination + Unreachable messages. + + Justification: This is easy to do and is used for many things + including MTU discovery and rapid detection of error conditions, + and has no negative consequences. + +10. Fragmentation of Outgoing Packets + + When the MTU of the adjacent link is too small, fragmentation of + packets going from the internal side to the external side of the NAT + may occur. This can occur if the NAT is doing Point-to-Point over + Ethernet (PPPoE), or if the NAT has been configured with a small MTU + to reduce serialization delay when sending large packets and small + higher-priority packets, or for other reasons. + + It is worth noting that many IP stacks do not use Path MTU Discovery + with UDP packets. + + The packet could have its Don't Fragment bit set to 1 (DF=1) or 0 + (DF=0). + + REQ-13: If the packet received on an internal IP address has DF=1, + the NAT MUST send back an ICMP message "Fragmentation needed and + DF set" to the host, as described in [RFC0792]. + + a) If the packet has DF=0, the NAT MUST fragment the packet and + SHOULD send the fragments in order. + + Justification: This is as per RFC 792. + + a) This is the same function a router performs in a similar + situation [RFC1812]. + +11. Receiving Fragmented Packets + + For a variety of reasons, a NAT may receive a fragmented packet. The + IP packet containing the header could arrive in any fragment, + depending on network conditions, packet ordering, and the + implementation of the IP stack that generated the fragments. + + + + + +Audet & Jennings Best Current Practice [Page 20] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + A NAT that is capable only of receiving fragments in order (that is, + with the header in the first packet) and forwarding each of the + fragments to the internal host is described as "Received Fragments + Ordered". + + A NAT that is capable of receiving fragments in or out of order and + forwarding the individual fragments (or a reassembled packet) to the + internal host is referred to as "Receive Fragments Out of Order". + See the Security Considerations section of this document for a + discussion of this behavior. + + A NAT that is neither of these is referred to as "Receive Fragments + None". + + REQ-14: A NAT MUST support receiving in-order and out-of-order + fragments, so it MUST have "Received Fragment Out of Order" + behavior. + + a) A NAT's out-of-order fragment processing mechanism MUST be + designed so that fragmentation-based DoS attacks do not + compromise the NAT's ability to process in-order and + unfragmented IP packets. + + Justification: See Security Considerations. + +12. Requirements + + The requirements in this section are aimed at minimizing the + complications caused by NATs to applications, such as realtime + communications and online gaming. The requirements listed earlier in + the document are consolidated here into a single section. + + It should be understood, however, that applications normally do not + know in advance if the NAT conforms to the recommendations defined in + this section. Peer-to-peer media applications still need to use + normal procedures, such as ICE [ICE]. + + A NAT that supports all the mandatory requirements of this + specification (i.e., the "MUST"), is "compliant with this + specification". A NAT that supports all the requirements of this + specification (i.e., including the "RECOMMENDED") is "fully compliant + with all the mandatory and recommended requirements of this + specification". + + + + + + + + +Audet & Jennings Best Current Practice [Page 21] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior. + + REQ-2: It is RECOMMENDED that a NAT have an "IP address pooling" + behavior of "Paired". Note that this requirement is not + applicable to NATs that do not support IP address pooling. + + REQ-3: A NAT MUST NOT have a "Port assignment" behavior of "Port + overloading". + + a) If the host's source port was in the range 0-1023, it is + RECOMMENDED the NAT's source port be in the same range. If the + host's source port was in the range 1024-65535, it is + RECOMMENDED that the NAT's source port be in that range. + + REQ-4: It is RECOMMENDED that a NAT have a "Port parity + preservation" behavior of "Yes". + + REQ-5: A NAT UDP mapping timer MUST NOT expire in less than two + minutes, unless REQ-5a applies. + + a) For specific destination ports in the well-known port range + (ports 0-1023), a NAT MAY have shorter UDP mapping timers that + are specific to the IANA-registered application running over + that specific destination port. + + b) The value of the NAT UDP mapping timer MAY be configurable. + + c) A default value of five minutes or more for the NAT UDP mapping + timer is RECOMMENDED. + + REQ-6: The NAT mapping Refresh Direction MUST have a "NAT Outbound + refresh behavior" of "True". + + a) The NAT mapping Refresh Direction MAY have a "NAT Inbound + refresh behavior" of "True". + + REQ-7 A NAT device whose external IP interface can be configured + dynamically MUST either (1) Automatically ensure that its internal + network uses IP addresses that do not conflict with its external + network, or (2) Be able to translate and forward traffic between + all internal nodes and all external nodes whose IP addresses + numerically conflict with the internal network. + + REQ-8: If application transparency is most important, it is + RECOMMENDED that a NAT have "Endpoint-Independent Filtering" + behavior. If a more stringent filtering behavior is most + important, it is RECOMMENDED that a NAT have "Address-Dependent + Filtering" behavior. + + + +Audet & Jennings Best Current Practice [Page 22] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + a) The filtering behavior MAY be an option configurable by the + administrator of the NAT. + + REQ-9: A NAT MUST support "Hairpinning". + + a) A NAT Hairpinning behavior MUST be "External source IP address + and port". + + REQ-10: To eliminate interference with UNSAF NAT traversal + mechanisms and allow integrity protection of UDP communications, + NAT ALGs for UDP-based protocols SHOULD be turned off. Future + standards track specifications that define an ALG can update this + to recommend the ALGs on which they define default. + + a) If a NAT includes ALGs, it is RECOMMENDED that the NAT allow + the NAT administrator to enable or disable each ALG separately. + + REQ-11: A NAT MUST have deterministic behavior, i.e., it MUST NOT + change the NAT translation (Section 4) or the Filtering + (Section 5) Behavior at any point in time, or under any particular + conditions. + + REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the + NAT mapping. + + a) The NAT's default configuration SHOULD NOT filter ICMP messages + based on their source IP address. + + b) It is RECOMMENDED that a NAT support ICMP Destination + Unreachable messages. + + REQ-13 If the packet received on an internal IP address has DF=1, + the NAT MUST send back an ICMP message "Fragmentation needed and + DF set" to the host, as described in [RFC0792]. + + a) If the packet has DF=0, the NAT MUST fragment the packet and + SHOULD send the fragments in order. + + REQ-14: A NAT MUST support receiving in-order and out-of-order + fragments, so it MUST have "Received Fragment Out of Order" + behavior. + + a) A NAT's out-of-order fragment processing mechanism MUST be + designed so that fragmentation-based DoS attacks do not + compromise the NAT's ability to process in-order and + unfragmented IP packets. + + + + + +Audet & Jennings Best Current Practice [Page 23] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + +13. Security Considerations + + NATs are often deployed to achieve security goals. Most of the + recommendations and requirements in this document do not affect the + security properties of these devices, but a few of them do have + security implications and are discussed in this section. + + This document recommends that the timers for mapping be refreshed on + outgoing packets (see REQ-6) and does not make recommendations about + whether or not inbound packets should update the timers. If inbound + packets update the timers, an external attacker can keep the mapping + alive forever and attack future devices that may end up with the same + internal address. A device that was also the DHCP server for the + private address space could mitigate this by cleaning any mappings + when a DHCP lease expired. For unicast UDP traffic (the scope of + this document), it may not seem relevant to support inbound timer + refresh; however, for multicast UDP, the question is harder. It is + expected that future documents discussing NAT behavior with multicast + traffic will refine the requirements around handling of the inbound + refresh timer. Some devices today do update the timers on inbound + packets. + + This document recommends that the NAT filters be specific to the + external IP address only (see REQ-8) and not to the external IP + address and UDP port. It can be argued that this is less secure than + using the IP and port. Devices that wish to filter on IP and port do + still comply with these requirements. + + Non-deterministic NATs are risky from a security point of view. They + are very difficult to test because they are, well, non-deterministic. + Testing by a person configuring one may result in the person thinking + it is behaving as desired, yet under different conditions, which an + attacker can create, the NAT may behave differently. These + requirements recommend that devices be deterministic. + + This document requires that NATs have an "external NAT mapping is + endpoint independent" behavior. This does not reduce the security of + devices. Which packets are allowed to flow across the device is + determined by the external filtering behavior, which is independent + of the mapping behavior. + + When a fragmented packet is received from the external side, and the + packets are out of order so that the initial fragment does not arrive + first, many systems simply discard the out-of-order packets. + Moreover, since some networks deliver small packets ahead of large + ones, there can be many out-of-order fragments. NATs that are + capable of delivering these out-of-order packets are possible, but + they need to store the out-of-order fragments, which can open up a + + + +Audet & Jennings Best Current Practice [Page 24] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + Denial-of-Service (DoS) opportunity, if done incorrectly. + Fragmentation has been a tool used in many attacks, some involving + passing fragmented packets through NATs, and others involving DoS + attacks based on the state needed to reassemble the fragments. NAT + implementers should be aware of [RFC3128] and [RFC1858]. + +14. IAB Considerations + + The IAB has studied the problem of "Unilateral Self Address Fixing", + which is the general process by which a client attempts to determine + its address in another realm on the other side of a NAT through a + collaborative protocol reflection mechanism [RFC3424]. + + This specification does not, in itself, constitute an UNSAF + application. It consists of a series of requirements for NATs aimed + at minimizing the negative impact that those devices have on peer-to- + peer media applications, especially when those applications are using + UNSAF methods. + + Section 3 of UNSAF lists several practical issues with solutions to + NAT problems. This document makes recommendations to reduce the + uncertainty and problems introduced by these practical issues with + NATs. In addition, UNSAF lists five architectural considerations. + Although this is not an UNSAF proposal, it is interesting to consider + the impact of this work on these architectural considerations. + + Arch-1: The scope of this is limited to UDP packets in NATs like the + ones widely deployed today. The "fix" helps constrain the + variability of NATs for true UNSAF solutions such as STUN. + + Arch-2: This will exit at the same rate that NATs exit. It does not + imply any protocol machinery that would continue to live + after NATs were gone, or make it more difficult to remove + them. + + Arch-3: This does not reduce the overall brittleness of NATs, but + will hopefully reduce some of the more outrageous NAT + behaviors and make it easer to discuss and predict NAT + behavior in given situations. + + Arch-4: This work and the results [RESULTS] of various NATs + represent the most comprehensive work at IETF on what the + real issues are with NATs for applications like VoIP. This + work and STUN have pointed out, more than anything else, the + brittleness NATs introduce and the difficulty of addressing + these issues. + + + + + +Audet & Jennings Best Current Practice [Page 25] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + Arch-5: This work and the test results [RESULTS] provide a reference + model for what any UNSAF proposal might encounter in + deployed NATs. + +15. Acknowledgments + + The editor would like to acknowledge Bryan Ford, Pyda Srisuresh, and + Dan Kegel for their multiple contributions on peer-to-peer + communications across a NAT. Dan Wing contributed substantial text + on IP fragmentation and ICMP behavior. Thanks to Rohan Mahy, + Jonathan Rosenberg, Mary Barnes, Melinda Shore, Lyndsay Campbell, + Geoff Huston, Jiri Kuthan, Harald Welte, Steve Casner, Robert + Sanders, Spencer Dawkins, Saikat Guha, Christian Huitema, Yutaka + Takeda, Paul Hoffman, Lisa Dusseault, Pekka Savola, Peter Koch, Jari + Arkko, and Alfred Hoenes for their contributions. + +16. References + +16.1. Normative References + + [RFC0768] Postel, J., "User Datagram Protocol", STD 6, RFC 768, + August 1980. + + [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, + September 1981. + + [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate + Requirement Levels", BCP 14, RFC 2119, March 1997. + +16.2. Informative References + + [RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, + RFC 792, September 1981. + + [RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", + RFC 1191, November 1990. + + [RFC1435] Knowles, S., "IESG Advice from Experience with Path MTU + Discovery", RFC 1435, March 1993. + + [RFC1812] Baker, F., "Requirements for IP Version 4 Routers", + RFC 1812, June 1995. + + [RFC1858] Ziemba, G., Reed, D., and P. Traina, "Security + Considerations for IP Fragment Filtering", RFC 1858, + October 1995. + + + + + +Audet & Jennings Best Current Practice [Page 26] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + [RFC1918] Rekhter, Y., Moskowitz, R., Karrenberg, D., Groot, G., + and E. Lear, "Address Allocation for Private + Internets", BCP 5, RFC 1918, February 1996. + + [RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version + 6 (IPv6) Specification", RFC 2460, December 1998. + + [RFC2623] Eisler, M., "NFS Version 2 and Version 3 Security + Issues and the NFS Protocol's Use of RPCSEC_GSS and + Kerberos V5", RFC 2623, June 1999. + + [RFC2663] Srisuresh, P. and M. Holdrege, "IP Network Address + Translator (NAT) Terminology and Considerations", + RFC 2663, August 1999. + + [RFC3022] Srisuresh, P. and K. Egevang, "Traditional IP Network + Address Translator (Traditional NAT)", RFC 3022, + January 2001. + + [RFC3027] Holdrege, M. and P. Srisuresh, "Protocol Complications + with the IP Network Address Translator", RFC 3027, + January 2001. + + [RFC3128] Miller, I., "Protection Against a Variant of the Tiny + Fragment Attack (RFC 1858)", RFC 3128, June 2001. + + [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., + Johnston, A., Peterson, J., Sparks, R., Handley, M., + and E. Schooler, "SIP: Session Initiation Protocol", + RFC 3261, June 2002. + + [RFC3424] Daigle, L. and IAB, "IAB Considerations for UNilateral + Self-Address Fixing (UNSAF) Across Network Address + Translation", RFC 3424, November 2002. + + [RFC3489] Rosenberg, J., Weinberger, J., Huitema, C., and R. + Mahy, "STUN - Simple Traversal of User Datagram + Protocol (UDP) Through Network Address Translators + (NATs)", RFC 3489, March 2003. + + [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. + Jacobson, "RTP: A Transport Protocol for Real-Time + Applications", STD 64, RFC 3550, July 2003. + + [RFC3605] Huitema, C., "Real Time Control Protocol (RTCP) + attribute in Session Description Protocol (SDP)", + RFC 3605, October 2003. + + + + +Audet & Jennings Best Current Practice [Page 27] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + + [RFC4380] Huitema, C., "Teredo: Tunneling IPv6 over UDP through + Network Address Translations (NATs)", RFC 4380, + February 2006. + + [RFC3489bis] Rosenberg, J., "Simple Traversal Underneath Network + Address Translators (NAT) (STUN)", Work in Progress, + October 2006. + + [ICE] Rosenberg, J., "Interactive Connectivity Establishment + (ICE): A Methodology for Network Address Translator + (NAT) Traversal for Offer/Answer Protocols", Work + in Progress, October 2006. + + [RESULTS] Jennings, C., "NAT Classification Test Results", Work + in Progress, October 2006. + + [TURN] Rosenberg, J., "Obtaining Relay Addresses from Simple + Traversal Underneath NAT (STUN)", Work in Progress, + October 2006. + + [ITU.H323] "Packet-based Multimedia Communications Systems", ITU- + T Recommendation H.323, July 2003. + +Authors' Addresses + + Francois Audet (editor) + Nortel Networks + 4655 Great America Parkway + Santa Clara, CA 95054 + US + + Phone: +1 408 495 2456 + EMail: audet@nortel.com + + + Cullen Jennings + Cisco Systems + 170 West Tasman Drive + MS: SJC-21/2 + San Jose, CA 95134 + US + + Phone: +1 408 902 3341 + EMail: fluffy@cisco.com + + + + + + + +Audet & Jennings Best Current Practice [Page 28] + +RFC 4787 NAT UDP Unicast Requirements January 2007 + + +Full Copyright Statement + + Copyright (C) The IETF Trust (2007). + + This document is subject to the rights, licenses and restrictions + contained in BCP 78, and except as set forth therein, the authors + retain all their rights. + + This document and the information contained herein are provided on an + "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS + OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND + THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS + OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF + THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED + WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. + +Intellectual Property + + The IETF takes no position regarding the validity or scope of any + Intellectual Property Rights or other rights that might be claimed to + pertain to the implementation or use of the technology described in + this document or the extent to which any license under such rights + might or might not be available; nor does it represent that it has + made any independent effort to identify any such rights. Information + on the procedures with respect to rights in RFC documents can be + found in BCP 78 and BCP 79. + + Copies of IPR disclosures made to the IETF Secretariat and any + assurances of licenses to be made available, or the result of an + attempt made to obtain a general license or permission for the use of + such proprietary rights by implementers or users of this + specification can be obtained from the IETF on-line IPR repository at + http://www.ietf.org/ipr. + + The IETF invites any interested party to bring to its attention any + copyrights, patents or patent applications, or other proprietary + rights that may cover technology that may be required to implement + this standard. Please address the information to the IETF at + ietf-ipr@ietf.org. + +Acknowledgement + + Funding for the RFC Editor function is currently provided by the + Internet Society. + + + + + + + +Audet & Jennings Best Current Practice [Page 29] + diff --git a/nat/src/masquerade/apalloc/mod.rs b/nat/src/masquerade/apalloc/mod.rs index 90204c9c1d..9deddbdf1c 100644 --- a/nat/src/masquerade/apalloc/mod.rs +++ b/nat/src/masquerade/apalloc/mod.rs @@ -297,11 +297,24 @@ impl NatAllocator { // marked `todo`. Whether a destination VPC counts as an endpoint for REQ-1 is a question about // the product, and nobody has answered it. Answering it is cheap; discovering the answer // mattered after a peer-to-peer application fails is not. + // + //= https://www.rfc-editor.org/rfc/rfc4787#section-4.1 + //= type=todo + //# REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior. + // + // The same deviation, for UDP, on the same code. RFC 4787 states it without the "for TCP" + // qualifier, so if the destination-VPC reading above is wrong then it is wrong for both + // protocols at once. One decision settles both citations. //= https://www.rfc-editor.org/rfc/rfc5382#section-8 //# REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port //# overloading" for TCP. // + //= https://www.rfc-editor.org/rfc/rfc4787#section-4.2.1 + //# REQ-3: A NAT MUST NOT have a "Port assignment" behavior of "Port + //# overloading". + // // No live allocation is ever handed out twice; the port bitmaps below are what enforce it. + // The allocator is protocol-agnostic, so the UDP and TCP requirements are one implementation. fn allocate_v4( &self, src_vpcd: VpcDiscriminant, diff --git a/nat/src/masquerade/apalloc/port_alloc.rs b/nat/src/masquerade/apalloc/port_alloc.rs index c951899dc1..53fc237169 100644 --- a/nat/src/masquerade/apalloc/port_alloc.rs +++ b/nat/src/masquerade/apalloc/port_alloc.rs @@ -92,6 +92,24 @@ pub(crate) struct PortAllocator { exclude_wellknown_ports: bool, } +//= https://www.rfc-editor.org/rfc/rfc4787#section-4.2.1 +//= type=exception +//# a) If the host's source port was in the range 0-1023, it is +//# RECOMMENDED the NAT's source port be in the same range. +// +// Declined, deliberately and unconditionally. `setup.rs` sets `exclude_wellknown_ports` for TCP +// and UDP alike, so an internal host sourcing from a well-known port is always translated to a +// port at or above 1024 and this requirement can never be met. +// +// This is the one requirement in RFC 4787 that is marked `exception` rather than `todo`, because +// unlike the timeout and mapping deviations it was actually decided: the range is named, the +// policy is stated on the constant below, a flag carries it, and tests hold it. Handing a tenant +// a public source port below 1024 would let it originate traffic that peers and middleboxes read +// as a privileged service, which is a worse trade than breaking the NFS-style clients RFC 4787 +// cites as the beneficiaries. +// +// The second half of REQ-3a -- a source port in 1024-65535 mapping to that same range -- is held, +// and held by the same line. /// Ports 0..=1023 cover the IANA system/well-known range and should not be /// allocated by masquerade NAT for TCP or UDP. pub(super) const IANA_WELLKNOWN_PORT_LIMIT: u16 = 1024; diff --git a/nat/src/masquerade/fuzz.rs b/nat/src/masquerade/fuzz.rs index 238eab8d54..b62027d638 100644 --- a/nat/src/masquerade/fuzz.rs +++ b/nat/src/masquerade/fuzz.rs @@ -276,6 +276,10 @@ fn out_unchanged(out: &[Packet], before: (IpAddr, u16)) -> bool { //= type=test //# REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port //# overloading" for TCP. +//= https://www.rfc-editor.org/rfc/rfc4787#section-4.2.1 +//= type=test +//# REQ-3: A NAT MUST NOT have a "Port assignment" behavior of "Port +//# overloading". /// Two live flows never share a translation. /// /// The exclusivity claim the allocator exists to keep, stated where it matters: at the stage, over diff --git a/nat/src/masquerade/mod.rs b/nat/src/masquerade/mod.rs index 1e708dd93d..99d4d7106c 100644 --- a/nat/src/masquerade/mod.rs +++ b/nat/src/masquerade/mod.rs @@ -17,6 +17,23 @@ mod state; mod state_machine; mod test; +//= https://www.rfc-editor.org/rfc/rfc4787#section-6 +//= type=todo +//# REQ-9: A NAT MUST support "Hairpinning". +// +// Not implemented, and not implemented anywhere: "hairpin" does not appear in this workspace. Two +// hosts inside one VPC that address each other by a public masqueraded address are not turned +// back at the gateway. +// +// There is a real argument that the requirement does not apply as written. RFC 4787 assumes hosts +// behind the NAT have discoverable external addresses to aim at; under masquerade a public tuple +// exists only for the lifetime of an outbound flow and is not something a peer can learn and dial. +// Stable inbound addresses are port forwarding and static NAT, which are different code. +// +// That argument may well be right, which is exactly why this is `todo`. It has not been made by +// anyone who owns the decision, and REQ-9 is a MUST. If it is correct, this becomes an +// `exception` with the reasoning above; if it is not, hairpinning is missing from a NAT that +// claims to be one. // re exports pub use allocator_writer::MasqueradeConfig; pub use allocator_writer::NatAllocatorWriter; diff --git a/nat/src/masquerade/nf.rs b/nat/src/masquerade/nf.rs index b62de137b0..3805776fcc 100644 --- a/nat/src/masquerade/nf.rs +++ b/nat/src/masquerade/nf.rs @@ -96,8 +96,9 @@ impl Masquerade { // We are far under both floors and this has not been ruled on. The three constants below are // transitory timeouts in RFC 5382's sense -- the connection is opening or closing -- and they // are seconds against a four-minute floor. The established timeout is `idle_timeout` from the - // masquerade configuration, which has no default, no bound and no validation, so a deployment - // can set it anywhere including well under two hours four minutes. + // masquerade configuration, which defaults to two minutes + // (`apalloc::setup::DEFAULT_MASQUERADE_IDLE_TIMEOUT`) but has no lower bound and no + // validation, so a deployment can set it anywhere, including zero. // // The short values are deliberate in intent: this file's own comment says the statuses exist // "to know how much to extend the lifetime of flows for port conservation", and a gateway @@ -105,6 +106,37 @@ impl Masquerade { // trade is one we are willing to state as a deviation from a BCP is a product decision, not a // code one -- hence `todo` rather than `exception`. Converting it needs a rationale somebody // is willing to sign. + // + //= https://www.rfc-editor.org/rfc/rfc4787#section-4.3 + //= type=todo + //# REQ-5: A NAT UDP mapping timer MUST NOT expire in less than two + //# minutes, unless REQ-5a applies. + // + //= https://www.rfc-editor.org/rfc/rfc4787#section-4.3 + //= type=todo + //# c) A default value of five minutes or more for the NAT UDP mapping + //# timer is RECOMMENDED. + // + // The UDP case is worse than the TCP one above, and for a reason worth stating plainly: + // RFC 4787 has no "transitory" category. A UDP mapping is a UDP mapping, and the timer is + // defined as "the time a mapping will stay active without packets traversing the NAT". The + // argument that rescues the TCP numbers -- that these are opening and closing states -- has + // nothing to attach to here. + // + // Trace a plain request/response exchange through `next_flow_status_udp`. The first outbound + // packet creates the flow at `OneWay`, so five seconds. The reply moves it to `TwoWay`, so + // three. Only a *second* outbound packet reaches `Established` and the two-minute + // `idle_timeout`. A single round trip followed by a four-second pause therefore loses its + // mapping, against a floor of two minutes -- short by a factor of forty. + // + // REQ-5a does not cover this. It permits shorter timers only for specific well-known + // destination ports and only where the shorter timer is specific to the IANA-registered + // application on that port; a blanket three-second timer for all UDP is not that. The one + // place we do apply a port-specific timer -- the resolver fast-close in `protocol.rs` -- is + // the thing REQ-5a describes, and is cited there. + // + // Even the settled case misses REQ-5c: the default is two minutes where five or more is + // RECOMMENDED. pub const MASQUERADE_ONEWAY_TIMEOUT: Duration = Duration::from_secs(5 * Self::TIMEOUT_SCALE); pub const MASQUERADE_TWOWAY_TIMEOUT: Duration = Duration::from_secs(3 * Self::TIMEOUT_SCALE); pub const MASQUERADE_CLOSING_TIMEOUT: Duration = Duration::from_secs(2 * Self::TIMEOUT_SCALE); @@ -566,6 +598,20 @@ impl Masquerade { return; } + //= https://www.rfc-editor.org/rfc/rfc4787#section-11 + //= type=todo + //# REQ-14: A NAT MUST support receiving in-order and out-of-order + //# fragments, so it MUST have "Received Fragment Out of Order" + //# behavior. + // + // The TODO below predates the citation; RFC 4787 is what it is a TODO about. Nothing on + // this path inspects the fragment offset or the more-fragments flag, so a translated flow + // whose packets arrive fragmented is handled by whatever the transport-header parse makes + // of a fragment that does not carry one. + // + // REQ-14a additionally requires that out-of-order fragment handling not become a denial of + // service vector, which is a constraint on the design that does not exist yet rather than + // on the code that does. // TODO: Check whether the packet is fragmented if let Err(error) = self.masquerade_packet(packet) { packet.done((&error).into()); diff --git a/nat/src/masquerade/protocol.rs b/nat/src/masquerade/protocol.rs index f4207f867c..df0fca1162 100644 --- a/nat/src/masquerade/protocol.rs +++ b/nat/src/masquerade/protocol.rs @@ -14,6 +14,21 @@ use net::packet::Packet; use net::tcp::Tcp; impl NatFlowStatus { + //= https://www.rfc-editor.org/rfc/rfc4787#section-4.3 + //# a) For specific destination ports in the well-known port range + //# (ports 0-1023), a NAT MAY have shorter UDP mapping timers that + //# are specific to the IANA-registered application running over + //# that specific destination port. + // + // This is the exemption REQ-5a describes, and it is the only port-specific timer we have. The + // match is on the reply's *source* port, which is the destination port of the session that + // asked -- the thing REQ-5a is written about. + // + // Two of the three ports qualify. 53 and 853 are inside the well-known range and are the + // IANA registrations for DNS and DNS-over-TLS. 8853 is not: it is above 1023, so REQ-5a does + // not reach it and closing that flow immediately is a plain deviation from REQ-5 rather than a + // permitted optimisation. It is a small one -- the flow is a resolver exchange either way -- + // but it is not covered by the exemption the other two sit under. fn udp_status_patch_dnat(self, packet: &Packet) -> NatFlowStatus { match packet.headers().pat().eth().net().udp().done() { Some((_, _, udp)) => match udp.source().as_u16() { @@ -54,8 +69,16 @@ fn next_flow_status_udp(action: NatAction, status: NatFlowStatus) -> NatFlowStat //# REQ-10: Receipt of any sort of ICMP message MUST NOT terminate the //# NAT mapping or TCP connection for which the ICMP was generated. // +//= https://www.rfc-editor.org/rfc/rfc4787#section-9 +//# REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the +//# NAT mapping. +// // Held by construction: no arm below yields `Closed` or `Reset`, so no ICMP message can end a // mapping. The only transition available is the one that records that traffic came back. +// +// Both specifications state this requirement, and this function is where both are kept. It runs +// for every ICMP packet regardless of the protocol of the flow it belongs to, so the guarantee +// does not depend on which specification you read it under. #[allow(clippy::match_single_binding)] fn next_flow_status_icmp(action: NatAction, status: NatFlowStatus) -> NatFlowStatus { match action { From 8287e76e08a18c006f1bd4292b3c9ae38fde0ced Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 23:33:31 -0600 Subject: [PATCH 22/37] test(masquerade): State RFC 4787 endpoint independence as a property, and measure it The overarching goal is properties at the abstraction an RFC is written at: configure the NF, feed it generated packets, assert something implementation independent. RFC 4787 REQ-1 is a good test of whether that lines up, because it is a statement about the NF that no unit can make. It does line up, and the property immediately refuted a comment written two commits ago. That comment said the mapping was endpoint-dependent only across destination VPCs, and reasoned from `allocate_v4`'s signature, which takes no destination address and no destination port. That looked like grounds to call the deviation architectural and probably defensible. `an_internal_endpoint_keeps_one_public_address` holds the internal endpoint fixed and moves the destination. Written first as the full REQ-1 claim, it failed on the first input drawn: 10.0.0.0:1 gets public port 1024 talking to 3.3.3.1:1 and 1025 talking to 3.3.3.1:2. Same destination address, same VPC, only the destination *port* changed. That is "Address and Port-Dependent Mapping" in RFC 4787 section 4.1 -- the most restrictive of the three classes and the one REQ-1 forbids. The cost is UNSAF traversal, which is the entire justification the RFC gives for the requirement. The dependence was never in the allocator's arguments. It is in being called again for each new flow, which is invisible at the allocator and visible at the stage. That is the argument for this level of testing in one sentence. What the committed property asserts is the half that holds. The public *address* is stable across destinations even when the port is not, which is REQ-2, "IP address pooling behavior of Paired". So the same two lines in `Pool::allocate` satisfy REQ-2 and miss REQ-1, and they are now cited as both -- the partial conformance case in its clearest available form. Verified against a break: swapping `reuse_allocated_ip` and `allocate_from_new_ip` so a fresh address is drawn per flow moves the public address from 172.16.0.0 to 172.16.0.1 and the property fails. 20/20 configurations built, 160 flows reached the assertion. REQ-1 stays `todo` rather than becoming an `exception`. An exception asserts somebody weighed the requirement and accepted the cost; that has not happened, and now that the cost is stated precisely it is worth asking for. Also carries the first use of `reason=` on a citation, which duvet permits on exception, implication, implementation and test but not on todo, and which must fit on one line -- a bare continuation is parsed as a second source. (cherry picked from commit 794605d1c0f09fa34ed5b2e237a0a2da8f323476) --- .duvet/snapshot.txt | 4 +- nat/src/masquerade/apalloc/alloc.rs | 10 ++++ nat/src/masquerade/apalloc/mod.rs | 43 ++++++++++----- nat/src/masquerade/fuzz.rs | 81 +++++++++++++++++++++++++++++ 4 files changed, 123 insertions(+), 15 deletions(-) diff --git a/.duvet/snapshot.txt b/.duvet/snapshot.txt index 6e05273b0f..c2a2510859 100644 --- a/.duvet/snapshot.txt +++ b/.duvet/snapshot.txt @@ -1,8 +1,8 @@ SPECIFICATION: https://www.rfc-editor.org/rfc/rfc4787 SECTION: [Address and Port Mapping](#section-4.1) TEXT[!MUST,todo]: REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior. - TEXT[!SHOULD]: REQ-2: It is RECOMMENDED that a NAT have an "IP address pooling" - TEXT[!SHOULD]: behavior of "Paired". + TEXT[!SHOULD,implementation,test]: REQ-2: It is RECOMMENDED that a NAT have an "IP address pooling" + TEXT[!SHOULD,implementation,test]: behavior of "Paired". SECTION: [Port Assignment Behavior](#section-4.2.1) TEXT[!MUST,implementation,test]: REQ-3: A NAT MUST NOT have a "Port assignment" behavior of "Port diff --git a/nat/src/masquerade/apalloc/alloc.rs b/nat/src/masquerade/apalloc/alloc.rs index 51cf9caf5b..07ad4c1699 100644 --- a/nat/src/masquerade/apalloc/alloc.rs +++ b/nat/src/masquerade/apalloc/alloc.rs @@ -117,6 +117,16 @@ impl IpAllocator { // FIXME: Should we clean up every time?? self.cleanup_used_ips(); + //= https://www.rfc-editor.org/rfc/rfc4787#section-4.1 + //= reason=held: reuse before draw is what makes the pooling behaviour "Paired" + //# REQ-2: It is RECOMMENDED that a NAT have an "IP address pooling" + //# behavior of "Paired". + // + // These two lines are the whole of REQ-2. Reusing an address already in use before drawing + // a new one is what makes one internal address keep one public address across all its + // sessions; drawing first would give the same host a different public address per flow and + // break peers that negotiate media addresses once. + // // Draw a fresh address only when the addresses already in use are exhausted. Other errors // describe allocator failure and must be preserved. match self.reuse_allocated_ip(allow_null) { diff --git a/nat/src/masquerade/apalloc/mod.rs b/nat/src/masquerade/apalloc/mod.rs index 9deddbdf1c..4e29f76652 100644 --- a/nat/src/masquerade/apalloc/mod.rs +++ b/nat/src/masquerade/apalloc/mod.rs @@ -286,25 +286,42 @@ impl NatAllocator { //# REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior //# for TCP. // - // Not held as stated, and the deviation is architectural rather than accidental: the - // allocation depends on `dst_vpcd`, so one internal endpoint talking to two destination VPCs - // can be given two different public tuples. RFC 5382 was written for a NAT facing a single - // external realm, where "endpoint" means a destination address and port; here distinct - // destination VPCs are distinct address spaces reached through distinct peerings, and sharing - // a pool across them would be the surprising choice. + // Not held, and measured rather than reasoned about. An earlier reading of this code blamed + // `dst_vpcd` and concluded the deviation only showed up across destination VPCs, which would + // have made it defensible. `fuzz::an_internal_endpoint_keeps_one_public_address` was written + // to check that and refuted it on the first input it drew. // - // So this is probably an exception rather than a defect -- but "probably" is the reason it is - // marked `todo`. Whether a destination VPC counts as an endpoint for REQ-1 is a question about - // the product, and nobody has answered it. Answering it is cheap; discovering the answer - // mattered after a peer-to-peer application fails is not. + // Holding the internal endpoint at 10.0.0.0:1 and moving only the *destination port*, from + // 3.3.3.1:1 to 3.3.3.1:2, moves the public port from 1024 to 1025. Same destination address, + // same VPC, different mapping. In RFC 4787's taxonomy (section 4.1) that is + // "Address and Port-Dependent Mapping" -- the most restrictive of the three classes, and the + // one REQ-1 exists to forbid. + // + // The signature above is what made the earlier reading plausible: `allocate_v4` takes no + // destination address and no destination port, so it looks endpoint-independent. The + // dependence is not in its arguments, it is in being called again for every new flow. Nothing + // at the allocator level can see that, which is why the property lives at the stage. + // + // What this costs is UNSAF traversal, which is the entire justification RFC 4787 gives for + // REQ-1: a peer learns its public tuple by talking to a third party, and here that tuple is + // worth nothing for talking to anybody else. Whether this gateway intends to carry + // peer-to-peer traffic is a product question, and it is a much sharper one than the + // destination-VPC question it replaces. + // + // Still `todo` rather than `exception`, on the same grounds as before: an exception asserts + // somebody weighed this and accepted it. Now that the cost is stated precisely, that decision + // is worth asking for. // //= https://www.rfc-editor.org/rfc/rfc4787#section-4.1 //= type=todo //# REQ-1: A NAT MUST have an "Endpoint-Independent Mapping" behavior. // - // The same deviation, for UDP, on the same code. RFC 4787 states it without the "for TCP" - // qualifier, so if the destination-VPC reading above is wrong then it is wrong for both - // protocols at once. One decision settles both citations. + // The same deviation, for UDP, on the same code, and the measurement above was taken over UDP + // probes. RFC 4787 states it without the "for TCP" qualifier. One decision settles both. + // + // The half that *is* held is REQ-2, cited in `alloc.rs`: the public address is stable across + // destinations even though the port is not. That is the partial-conformance case in its + // clearest form -- one requirement met, its neighbour missed, by the same two lines of code. //= https://www.rfc-editor.org/rfc/rfc5382#section-8 //# REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port //# overloading" for TCP. diff --git a/nat/src/masquerade/fuzz.rs b/nat/src/masquerade/fuzz.rs index b62027d638..b8f4ad9c4e 100644 --- a/nat/src/masquerade/fuzz.rs +++ b/nat/src/masquerade/fuzz.rs @@ -272,6 +272,87 @@ fn out_unchanged(out: &[Packet], before: (IpAddr, u16)) -> bool { out[0].is_done() || source_of(&out[0]) == before } +//= https://www.rfc-editor.org/rfc/rfc4787#section-4.1 +//= type=test +//# REQ-2: It is RECOMMENDED that a NAT have an "IP address pooling" +//# behavior of "Paired". +/// One internal endpoint keeps one public address, whatever it is talking to. +/// +/// RFC 4787 REQ-1 states this as "Endpoint-Independent Mapping": the same internal address and +/// port must be given the same external address and port regardless of the external endpoint it +/// is sending to. It is the requirement UNSAF traversal is built on -- a peer learns your public +/// tuple by talking to a third party, and that is worth nothing if talking to the peer produces a +/// different one. +/// +/// Stated at the stage rather than at the allocator, because the allocator cannot answer it. Its +/// signature takes `src_ip` and no destination address or port, which looks endpoint-independent +/// until you ask who calls it and how often. The question is what a *second flow* from the same +/// internal endpoint receives, and only the stage, with the flow table in the loop, knows that. +/// +/// The probe holds source and source port fixed and moves the destination -- a different peer +/// address where the fabric offers one, and a different destination port either way. +#[test] +fn an_internal_endpoint_keeps_one_public_address() { + let tally = Tally::default(); + + with_runtime(|| { + bolero::check!() + .with_generator(Scenario { strays: false }) + .cloned() + .for_each(|(exposes, probes): (Vec, Vec)| { + tally.seen.fetch_add(1, Ordering::Relaxed); + let Some(fabric) = fabric(&exposes) else { + return; + }; + tally.built.fetch_add(1, Ordering::Relaxed); + let (mut lookup, mut masq) = fabric.stages(); + + for spec in &probes { + let probe = (*spec).resolve(&fabric); + let before = (probe.source, probe.sport); + let first = run(&mut lookup, &mut masq, vec![probe.packet()], probe.arrival.dst_vpcd); + if out_unchanged(&first, before) { + continue; + } + + // The same internal endpoint, a different external one. + let mut elsewhere = (*spec).resolve(&fabric); + elsewhere.dport = elsewhere.dport.wrapping_add(1).max(1); + if let Some(other) = fabric.peer.iter().find(|a| **a != probe.destination) { + elsewhere.destination = *other; + } + if (elsewhere.destination, elsewhere.dport) == (probe.destination, probe.dport) { + continue; + } + + let second = run( + &mut lookup, + &mut masq, + vec![elsewhere.packet()], + elsewhere.arrival.dst_vpcd, + ); + if out_unchanged(&second, before) { + continue; + } + + assert_eq!( + source_of(&second[0]).0, + source_of(&first[0]).0, + "{before:?} was given {:?} talking to {:?} and {:?} talking to {:?}, so the \ + public address it is given depends on who it is addressing", + source_of(&first[0]), + (probe.destination, probe.dport), + source_of(&second[0]), + (elsewhere.destination, elsewhere.dport) + ); + tally.reached.fetch_add(1, Ordering::Relaxed); + } + }); + }); + + tally.report("address pairing"); +} + //= https://www.rfc-editor.org/rfc/rfc5382#section-8 //= type=test //# REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port From da33f3ada436a0daa4030147d2ea6a300e508e29 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Tue, 18 Aug 2026 23:49:03 -0600 Subject: [PATCH 23/37] test(masquerade): Assert RFC 4787 outbound refresh, and record where it is not held REQ-6 is a MUST: an outbound packet must keep a mapping alive. The module already had `traffic_extends_a_flow_past_its_first_deadline`, which refreshes with replies -- that is *inbound* refresh, REQ-6a, and only a MAY. The permitted behaviour was covered and the required one was not. Writing the missing test turned up a defect. Measured on a paused clock, control against treatment, with a five-second `OneWay` lifetime. Silent for eight seconds: the reply to the mapping is dropped, as expected. An outbound packet at four seconds, then the same probe at eight seconds: also dropped. The packet changed nothing. `refresh_masquerade_state` is where it comes from. Its `OneWay` arm yields `None` for the extension, so no outbound packet ever moves the deadline of a flow that has not yet had a reply. The comment there reasons about the reverse direction and treats `OneWay` as a corner case, which is what makes returning `None` look harmless. It is not a corner: it is the steady state of every outbound-only flow. Syslog, netflow, telemetry, a resolver query nobody answers -- each has its mapping torn down five seconds after its first packet however much it sends, and rebuilt on the next one. Once a reply arrives the flow reaches `Established` and outbound refresh does work, so REQ-6 is met for connections and missed for one-way traffic. That half is now asserted: three outbound packets a hundred seconds apart against a hundred-and-twenty second idle timeout, five minutes with nothing arriving from outside. Two details make the assertion mean what it says. The step is near the timeout, because a step comfortably inside the lifetime the previous packet already bought would pass with refresh deleted. And the probe comes after a delay longer than a `OneWay` lifetime, so a mapping that had been silently torn down and rebuilt by the last outbound packet is already dead when it is checked -- reissuing the identical tuple cannot fake a pass. Verified by breaking the `Established` arm to `None`, which fails the test on exactly that assertion. The `OneWay` gap is recorded as `todo` rather than `exception`, and deliberately not written as a test. A test pinning the current behaviour would make the deviation permanent, which is the entrenchment failure this whole exercise exists to avoid. (cherry picked from commit 8a262fa6b589f58b3b4cce08018ad3d73ed82238) --- .duvet/snapshot.txt | 4 +- nat/src/masquerade/expiry.rs | 85 ++++++++++++++++++++++++++++++++++++ nat/src/masquerade/nf.rs | 26 +++++++++++ 3 files changed, 113 insertions(+), 2 deletions(-) diff --git a/.duvet/snapshot.txt b/.duvet/snapshot.txt index c2a2510859..fe89aaf19a 100644 --- a/.duvet/snapshot.txt +++ b/.duvet/snapshot.txt @@ -27,8 +27,8 @@ SPECIFICATION: https://www.rfc-editor.org/rfc/rfc4787 TEXT[!MAY]: b) The value of the NAT UDP mapping timer MAY be configurable. TEXT[!SHOULD,todo]: c) A default value of five minutes or more for the NAT UDP mapping TEXT[!SHOULD,todo]: timer is RECOMMENDED. - TEXT[!MUST]: REQ-6: The NAT mapping Refresh Direction MUST have a "NAT Outbound - TEXT[!MUST]: refresh behavior" of "True". + TEXT[!MUST,test,todo]: REQ-6: The NAT mapping Refresh Direction MUST have a "NAT Outbound + TEXT[!MUST,test,todo]: refresh behavior" of "True". TEXT[!MAY]: a) The NAT mapping Refresh Direction MAY have a "NAT Inbound TEXT[!MAY]: refresh behavior" of "True". diff --git a/nat/src/masquerade/expiry.rs b/nat/src/masquerade/expiry.rs index d3f028808a..e7f2b1e03b 100644 --- a/nat/src/masquerade/expiry.rs +++ b/nat/src/masquerade/expiry.rs @@ -46,6 +46,13 @@ const PAST_EXPIRY: Duration = Duration::from_secs(30); /// Comfortably inside it. const WITHIN_LIFETIME: Duration = Duration::from_secs(1); +/// Inside the two-minute established idle timeout, but most of the way through it. +/// +/// The step has to be near the timeout or the test proves nothing: a step well inside the lifetime +/// the previous packet already bought would hold with refresh deleted entirely. At 100 seconds +/// against 120, each packet is the only reason the mapping survives to see the next one. +const NEARLY_ESTABLISHED: Duration = Duration::from_secs(100); + fn vni(raw: u32) -> Vni { Vni::new_checked(raw).unwrap_or_else(|_| unreachable!()) } @@ -218,6 +225,84 @@ fn traffic_extends_a_flow_past_its_first_deadline() { }); } +//= https://www.rfc-editor.org/rfc/rfc4787#section-4.3 +//= type=test +//= reason=held: for established flows; see the OneWay gap recorded in nf.rs +//# REQ-6: The NAT mapping Refresh Direction MUST have a "NAT Outbound +//# refresh behavior" of "True". +/// Outbound traffic alone keeps an established mapping alive. +/// +/// The sibling above refreshes with replies, which is *inbound* refresh -- RFC 4787 REQ-6a, and only +/// a MAY. REQ-6 is a MUST about the outbound direction, and nothing asserted it until this test: the +/// permitted behaviour was covered and the required one was not. +/// +/// Three steps of a hundred seconds against a hundred-and-twenty second idle timeout, so each +/// outbound packet is the only reason the mapping survives to see the next. Five minutes elapse in +/// total, with nothing arriving from outside after the single reply that opens the connection. +/// +/// The tail is what keeps it honest: having shown traffic holds the mapping open, it stops and shows +/// the mapping does then expire. A flow table that expired nothing would pass the first half and +/// mean nothing by it. +/// +/// **This covers established flows only, and that is the whole of what we hold.** A flow that has +/// never received a reply stays in `OneWay`, where outbound packets do not refresh at all -- measured +/// and recorded at `Masquerade::refresh_masquerade_state`. It is deliberately not asserted here, +/// because a test that pinned the current behaviour would make the deviation permanent. +#[test] +fn outbound_traffic_keeps_an_established_mapping_alive() { + with_paused_clock(|| async { + let (fabric, _) = fabric(); + let (mut lookup, mut masq) = fabric.stages(); + let peer = fabric.peer[0]; + let source: IpAddr = "10.0.0.7".parse().unwrap_or_else(|_| unreachable!()); + + // Out, in, out: the shortest path to `Established` and the two-minute timer. + let translated = open_flow(&mut lookup, &mut masq, source, peer, 1234) + .unwrap_or_else(|| unreachable!("a fixed private source is masqueraded")); + assert_eq!( + reply_to(&mut lookup, &mut masq, peer, translated), + Some(source), + "the reply that establishes the connection was not delivered" + ); + assert_eq!( + open_flow(&mut lookup, &mut masq, source, peer, 1234), + Some(translated), + "the packet that establishes the connection changed its translation" + ); + + // Nothing arrives from outside from here on. Outbound packets only. + for step in 1..=3 { + advance(NEARLY_ESTABLISHED).await; + assert_eq!( + open_flow(&mut lookup, &mut masq, source, peer, 1234), + Some(translated), + "at {}s an outbound packet no longer found the mapping", + step * NEARLY_ESTABLISHED.as_secs() + ); + } + + // Probe after longer than a `OneWay` lifetime but well inside an established one. This is + // what makes the assertion mean "refreshed" rather than "silently torn down and rebuilt": + // a flow rebuilt by the last outbound packet would be in `OneWay` and already dead here, + // even if the allocator handed back the identical tuple. + advance(PAST_EXPIRY).await; + assert_eq!( + reply_to(&mut lookup, &mut masq, peer, translated), + Some(source), + "the mapping did not survive five minutes of outbound traffic, so outbound packets \ + are not refreshing it" + ); + + // Traffic stops. The mapping must not be immortal. + advance(Duration::from_mins(5)).await; + assert_eq!( + reply_to(&mut lookup, &mut masq, peer, translated), + None, + "a mapping held open by outbound traffic never expired once that traffic stopped" + ); + }); +} + /// An expired flow does not come back, and its public tuple may be handed to someone else. /// /// The **never resurrected** disposition, and the property that needs the whole facade: three diff --git a/nat/src/masquerade/nf.rs b/nat/src/masquerade/nf.rs index 3805776fcc..3ce7cd4667 100644 --- a/nat/src/masquerade/nf.rs +++ b/nat/src/masquerade/nf.rs @@ -226,6 +226,32 @@ impl Masquerade { | NatFlowStatus::CHalfClose | NatFlowStatus::SHalfClose | NatFlowStatus::LastAck => Some(Self::MASQUERADE_CLOSING_TIMEOUT), + //= https://www.rfc-editor.org/rfc/rfc4787#section-4.3 + //= type=todo + //# REQ-6: The NAT mapping Refresh Direction MUST have a "NAT Outbound + //# refresh behavior" of "True". + // + // Not held in this state, and measured rather than inferred. `OneWay` is not only the + // odd transient the comment below describes: it is the *steady* state of any flow that + // has never had a reply, and returning `None` here means no amount of outbound traffic + // moves its deadline. + // + // A control and a treatment, on a paused clock, five seconds of `OneWay` lifetime: + // silent for eight seconds, the reply to the mapping is dropped; an outbound packet at + // four seconds and then the same probe at eight, and it is *also* dropped. The packet + // changed nothing. An outbound-only flow -- syslog, netflow, telemetry, a resolver + // query nobody answers -- is therefore torn down five seconds after its first packet + // however much it sends, and rebuilt from scratch on the next one. + // + // Once a reply arrives the flow reaches `Established` and outbound refresh does work; + // `expiry::outbound_traffic_keeps_an_established_mapping_alive` holds that, verified + // over five minutes against a two-minute timer. So REQ-6 is met for connections and + // missed for one-way traffic. + // + // Marked `todo` rather than `exception` because this reads like an oversight rather + // than a decision: the comment below reasons about the *reverse* direction and treats + // `OneWay` as a corner, which is what makes returning `None` look harmless. Nothing + // here weighs one-way outbound traffic and declines to support it. NatFlowStatus::OneWay => { // this could happen if a burst of packets are sent before any state is there (snat), // or if we got a TCP segment back without expected flags. This should never happen for From 265365215e48b29f7b722ff5702ccf151ba69366 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Wed, 19 Aug 2026 00:16:42 -0600 Subject: [PATCH 24/37] test(masquerade): Classify masquerade's filtering, and check RFC 4787 determinism A deliberate move to a different species of requirement, to find where the method strains. Everything cited so far has been first order and unconditional -- do this, never do that, not less than this many seconds. RFC 4787 has two that are neither. REQ-11 is second order. It is not a requirement about a packet; it requires that the answers to the *other* requirements stay the same "at any point in time, or under any particular conditions". Citing it correctly needs two readings settled first. "Behavior" means the class from section 4, not the values -- section 4.2.1 explicitly permits random port assignment, so the allocator shuffling port blocks is not a violation, and a naive citation would have called it one. And the conflict the RFC is aimed at, in section 8, is port preservation with a fallback path, which does not exist here because nothing ever tries to preserve a source port. That leaves address pooling as the thing that could still change under pressure, and it does not. 254 hosts, 256 flows each, 65,024 flows against a public /24: the pool spills to a second public address and no internal host is ever given more than one. Pairing before the spill is pairing after it. Two structural facts fell out of measuring it. A single internal host can never break pairing, because its own source port space is exactly the size of one public address's port space -- 64,512 flows from one host stayed on one address with nothing denied. So the transition only exists under contention between hosts, which is why the probe needs 254 of them. It costs fourteen seconds against a suite that runs in four, so it is `#[ignore]`d as a characterization probe, following the precedent in acl/src/dpdk/dyn_table.rs. REQ-8 is conditional, and is the one that actually gives the method trouble. It does not state a behaviour; it states two and picks between them on a priority nobody has written down -- Endpoint-Independent Filtering "if application transparency is most important", Address-Dependent Filtering "if a more stringent filtering behavior is most important". Three probes against one flow classify what we do exactly: same address and port delivered, same address different port dropped, different address dropped. That is Address and Port-Dependent Filtering, the most restrictive of section 5's three classes, and it satisfies neither branch of REQ-8 -- we are stricter than the stringent option, which would let the second probe through. Stricter than a SHOULD asks is still a departure from it. Recorded as `todo` rather than `exception` because this reads as a consequence of keying the flow table on the whole five-tuple rather than a filtering policy anyone chose; what is missing is a recorded priority, not code. The filtering test is committed unignored regardless of how REQ-8 is resolved. An unsolicited packet reaching a tenant because it guessed a live public tuple is a security failure, and the second and third probes are what rule it out. It carries its own positive control: the first probe is delivered through the same path the other two are dropped by, so it cannot pass vacuously. Also confirms a FIXME in apalloc/setup.rs is unreachable rather than latent. It warns that a public range restricted to a port range is not modelled by the pools, which reads as a silent misconfiguration; in fact validation refuses such a config outright with "Port ranges are not supported with masquerade". (cherry picked from commit 7e5aa334ea4a6caac5f8afc6d482d977e19d99c1) --- .duvet/snapshot.txt | 20 ++--- nat/src/masquerade/expiry.rs | 141 +++++++++++++++++++++++++++++++++++ 2 files changed, 151 insertions(+), 10 deletions(-) diff --git a/.duvet/snapshot.txt b/.duvet/snapshot.txt index fe89aaf19a..0d495baebf 100644 --- a/.duvet/snapshot.txt +++ b/.duvet/snapshot.txt @@ -41,12 +41,12 @@ SPECIFICATION: https://www.rfc-editor.org/rfc/rfc4787 TEXT[!MUST]: numerically conflict with the internal network. SECTION: [Filtering Behavior](#section-5) - TEXT[!SHOULD]: REQ-8: If application transparency is most important, it is - TEXT[!SHOULD]: RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" - TEXT[!SHOULD]: behavior. - TEXT[!SHOULD]: If a more stringent filtering behavior is most - TEXT[!SHOULD]: important, it is RECOMMENDED that a NAT have an "Address-Dependent - TEXT[!SHOULD]: Filtering" behavior. + TEXT[!SHOULD,todo]: REQ-8: If application transparency is most important, it is + TEXT[!SHOULD,todo]: RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" + TEXT[!SHOULD,todo]: behavior. + TEXT[!SHOULD,todo]: If a more stringent filtering behavior is most + TEXT[!SHOULD,todo]: important, it is RECOMMENDED that a NAT have an "Address-Dependent + TEXT[!SHOULD,todo]: Filtering" behavior. TEXT[!MAY]: a) The filtering behavior MAY be an option configurable by the TEXT[!MAY]: administrator of the NAT. @@ -63,10 +63,10 @@ SPECIFICATION: https://www.rfc-editor.org/rfc/rfc4787 TEXT[!SHOULD]: the NAT administrator to enable or disable each ALG separately. SECTION: [Deterministic Properties](#section-8) - TEXT[!MUST]: REQ-11: A NAT MUST have deterministic behavior, i.e., it MUST NOT - TEXT[!MUST]: change the NAT translation (Section 4) or the Filtering - TEXT[!MUST]: (Section 5) Behavior at any point in time, or under any particular - TEXT[!MUST]: conditions. + TEXT[!MUST,test]: REQ-11: A NAT MUST have deterministic behavior, i.e., it MUST NOT + TEXT[!MUST,test]: change the NAT translation (Section 4) or the Filtering + TEXT[!MUST,test]: (Section 5) Behavior at any point in time, or under any particular + TEXT[!MUST,test]: conditions. SECTION: [ICMP Destination Unreachable Behavior](#section-9) TEXT[!SHOULD]: The NAT's default configuration SHOULD NOT filter diff --git a/nat/src/masquerade/expiry.rs b/nat/src/masquerade/expiry.rs index e7f2b1e03b..b204659a72 100644 --- a/nat/src/masquerade/expiry.rs +++ b/nat/src/masquerade/expiry.rs @@ -137,6 +137,147 @@ fn reply_to( .flatten() } +//= https://www.rfc-editor.org/rfc/rfc4787#section-8 +//= type=test +//= reason=held for the mapping dimension: pairing is unchanged by exhaustion, measured below +//# REQ-11: A NAT MUST have deterministic behavior, i.e., it MUST NOT +//# change the NAT translation (Section 4) or the Filtering +//# (Section 5) Behavior at any point in time, or under any particular +//# conditions. +/// Address pairing does not change when the pool is under pressure. +/// +/// REQ-11 is second order: it is not a requirement about a packet, it is a requirement that the +/// answers to the *other* requirements stay the same. RFC 4787 section 8 says what it is aimed at -- +/// NATs that take a different code path once the port they wanted is taken, so their behaviour has a +/// "Primary" form before the first conflict and a "Secondary" form after it. +/// +/// Two readings have to be got right before this can be cited at all. "Behavior" means the class +/// from section 4, not the values: section 4.2.1 explicitly permits random port assignment, so the +/// allocator shuffling port blocks is not a violation. And the conflict class section 8 describes -- +/// port preservation with a fallback path -- does not exist here, because nothing ever tries to +/// preserve a source port. +/// +/// What can still change is address pooling, so that is what this measures: 254 internal hosts, +/// 256 flows each, 65,024 flows against a public /24. The pool does spill -- two public addresses +/// end up in use -- and **no internal host is ever given more than one of them**. Pairing before +/// the spill and pairing after it are the same behaviour, which is REQ-11 for the mapping +/// dimension. +/// +/// Ignored by default because it costs about fourteen seconds against a suite that otherwise runs +/// in four. It is a measurement to re-take when the allocator changes, not a guard on every commit. +#[test] +#[ignore = "characterization probe; run with --ignored --nocapture"] +fn pairing_is_unchanged_by_pool_exhaustion() { + use std::collections::{BTreeMap, BTreeSet}; + with_paused_clock(|| async { + let (fabric, _) = fabric(); + let (mut lookup, mut masq) = fabric.stages(); + let peer = fabric.peer[0]; + let mut given: BTreeMap> = BTreeMap::new(); + + for host in 1..=254u16 { + let source: IpAddr = format!("10.0.0.{host}") + .parse() + .unwrap_or_else(|_| unreachable!()); + for sport in 1024..1024 + 256u16 { + if let Some((public, _)) = open_flow(&mut lookup, &mut masq, source, peer, sport) { + given.entry(source).or_default().insert(public); + } + } + } + + let publics: BTreeSet<_> = given.values().flatten().copied().collect(); + println!( + "{} hosts, {} public addresses in use", + given.len(), + publics.len() + ); + assert!( + publics.len() > 1, + "the pool never spilled to a second address, so this measured nothing about conflict" + ); + let split: Vec<_> = given.iter().filter(|(_, a)| a.len() > 1).collect(); + assert!( + split.is_empty(), + "pooling changed under pressure: {} hosts were given more than one public address, \ + so the behaviour before the spill is not the behaviour after it", + split.len() + ); + }); +} + +/// Send an inbound packet to a translated tuple from an arbitrary external endpoint. +fn inbound_from( + lookup: &mut FlowLookup, + masq: &mut Masquerade, + from: IpAddr, + sport: u16, + translated: (IpAddr, u16), +) -> bool { + let mut packet = build(from, translated.0, false, sport, translated.1); + Arrival::inbound().stamp(&mut packet); + let out: Vec> = run(lookup, masq, vec![packet], Some(vni(LOCAL_VNI))); + !out[0].is_done() +} + +//= https://www.rfc-editor.org/rfc/rfc4787#section-5 +//= type=todo +//# REQ-8: If application transparency is most important, it is +//# RECOMMENDED that a NAT have an "Endpoint-Independent Filtering" +//# behavior. If a more stringent filtering behavior is most +//# important, it is RECOMMENDED that a NAT have an "Address-Dependent +//# Filtering" behavior. +/// Only the endpoint a flow addressed can answer it. +/// +/// Three probes against one flow to `3.3.3.1:80` classify the filtering behaviour exactly, in +/// RFC 4787 section 5's terms: the same address and port is delivered, the same address on a +/// different port is dropped, and a different address is dropped. That is +/// **"Address and Port-Dependent Filtering"**, the most restrictive of the three classes. +/// +/// REQ-8 is a different species from the requirements around it, and worth reading carefully. It +/// does not state one behaviour; it states two, and picks between them on a priority nobody has +/// written down -- transparency or stringency. We satisfy neither branch, because we are stricter +/// than the stringent one: REQ-8's "more stringent" option is Address-*Dependent* filtering, which +/// would let `3.3.3.1:81` through. +/// +/// Stricter than a `SHOULD` asks is still a departure from it, and the honest reading is that this +/// is a consequence of keying the flow table on the whole five-tuple rather than a filtering policy +/// anybody chose. Hence `todo`: what is missing is not code, it is a recorded priority. +/// +/// The behaviour itself is worth pinning regardless of how REQ-8 is resolved -- an unsolicited +/// packet reaching a tenant because it guessed a live public tuple is a security failure, and the +/// second and third probes are what rule it out. +#[test] +fn only_the_endpoint_a_flow_addressed_can_reply() { + with_paused_clock(|| async { + let (fabric, _) = fabric(); + let (mut lookup, mut masq) = fabric.stages(); + let peer = fabric.peer[0]; + let elsewhere = *fabric + .peer + .iter() + .find(|a| **a != peer) + .unwrap_or_else(|| unreachable!("the fixture offers two peer addresses")); + let source: IpAddr = "10.0.0.7".parse().unwrap_or_else(|_| unreachable!()); + + let translated = open_flow(&mut lookup, &mut masq, source, peer, 1234) + .unwrap_or_else(|| unreachable!("a fixed private source is masqueraded")); + + assert!( + inbound_from(&mut lookup, &mut masq, peer, 80, translated), + "the endpoint the flow addressed could not answer it" + ); + assert!( + !inbound_from(&mut lookup, &mut masq, peer, 81, translated), + "a packet from the right address on the wrong port reached the tenant" + ); + assert!( + !inbound_from(&mut lookup, &mut masq, elsewhere, 80, translated), + "a packet from an address the flow never addressed reached the tenant" + ); + }); +} + /// A flow inside its lifetime is unaffected by the passage of time. /// /// The **preserved** disposition. Stated as a property rather than assumed, because the cheap way to From 74f81a457052fccf32139eac241ad206f6c90f89 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Wed, 19 Aug 2026 11:49:16 -0600 Subject: [PATCH 25/37] docs(testing): Record what the RFC errata say, and where the corpus lies The errata question was open with one concrete item attached to it: RFC 4884 has errata, and MIN_ORIGINAL_DATAGRAM_OCTETS was chosen without reading them. It resolves to nothing. EID 3 corrects Section 7's description of the Extension Header checksum, carries no RFC 2119 keyword, and so was never extracted as a requirement; our citations are all in Section 3 and Section 5, on the length attribute. RFC 4787, 5382, 5508, 6888 and 7857 have no errata of any status, so the 221 untracked requirements are erratum-free. Two traps in the corpus are worth more than the answer. RFCs_for_errata.txt misses ten RFCs that do have verified errata, including RFC 1191 -- every one verified after the index was generated, so the index is stale forward and the rendering is authoritative. And 524 of the 1,750 renderings carry an erratum as an endnote with nothing spliced into the body, because its original text is not a locatable quote, so diffing a rendering against the base text under-reports. RFC 2663 EID 400 turns out to matter without being a defect. Corrected, it warns that a NAT cannot assume a FIN or RST is the last packet -- which is what masquerade assumes when it invalidates the pair on Reset or Closed. RFC 5382 leaves that behavior unspecified and names the throughput argument for taking it, so this is a decision rather than an oversight, and now a recorded one. Co-Authored-By: Claude Opus 5 (1M context) (cherry picked from commit 73a199eeaca4a7e1f3a061d0978ac255a1e83864) --- development/code/spec-compliance.md | 69 ++++++++++++++++++++++++++--- 1 file changed, 64 insertions(+), 5 deletions(-) diff --git a/development/code/spec-compliance.md b/development/code/spec-compliance.md index 6b5b8f02a8..bfa9ad8472 100644 --- a/development/code/spec-compliance.md +++ b/development/code/spec-compliance.md @@ -1,6 +1,7 @@ # Specification compliance with duvet -Status: **two specifications tracked, the RFC corpus audited, the procedure itself not yet proven.** +Status: **three specifications tracked, the RFC corpus and its errata audited, the procedure +itself not yet proven.** The open-questions list below is expected to grow; it is written down so that it grows in one place rather than in four people's heads. @@ -123,6 +124,65 @@ Rules, therefore: - Review is by somebody who reads the original. A reviewer looking only at our Markdown cannot catch the failure this is guarding against. +## Errata + +An RFC body is immutable, so a correction to one lives only in its errata. The corpus carries them +in `inline-errata/`: 1,750 RFCs rendered as HTML with their **Verified** errata spliced into the +text, the corrected passage wrapped in `` and an endnote block +giving the EID, section, original text, corrected text and notes. + +Only Verified errata are inlined -- all 1,750 renderings say so in their header, and no other status +appears. `RFCs_for_errata.txt` names 2,584 RFCs, so **847 have errata that this corpus never shows +you**: Reported, Held for Document Update, or Rejected. That is the residual fetch leg, and it is a +much smaller one than it looked. + +**Use the file, not the index.** Ten RFCs have renderings and are absent from +`RFCs_for_errata.txt` -- among them RFC 1191, Path MTU Discovery. Every one was verified after the +index's own timestamp, so the index is stale in one direction only: it never lists a spurious RFC, +it just misses recent ones. The presence or absence of `inline-errata/rfcNNNN.html` is authoritative +for "has Verified errata"; the index is authoritative for nothing. + +What that says about the specifications in play: + +| | errata | +| --- | --- | +| RFC 4787, RFC 5382, RFC 5508, RFC 6888, RFC 7857 | none of any status | +| RFC 4884 | one, EID 3 | + +**EID 3 does not touch us.** It corrects Section 7's description of the ICMP Extension Header +checksum from "the one's complement sum of the data structure" to "...of the ICMP Extension +Structure" -- naming what was already unambiguous from context. The sentence carries no RFC 2119 +keyword, so duvet never extracted it: the two requirements it does extract from Section 7 are a +`MAY` about ignoring unrecognised objects and a `MUST` about the reassembly buffer size, and neither +is edited. Nothing in the tree parses an extension structure or verifies its checksum; every RFC +4884 citation we hold is in Section 3 or Section 5, on the length attribute. So +`MIN_ORIGINAL_DATAGRAM_OCTETS` was decided against text the erratum leaves alone. + +**Read the endnotes, do not diff the body.** In 524 of the 1,750 renderings an erratum has an +endnote but no spliced-in span, because its "Original Text" is not a verbatim quote the renderer +could locate -- it is a `GLOBAL` scope, or a prose commentary rather than a passage. RFC 8200 is one +of them. Diffing a rendering against the base text therefore under-reports; the endnote list is the +complete one. + +Two errata on specifications we have discussed but do not track are worth having read: + +- **RFC 2663 EID 400** corrects Section 2.6 from "segments containing FINs or SYNs will be the last + packets of the session" to "FINs or **RSTs**" -- the original sentence was nonsense, since a SYN + never ends a session. Corrected, it is a warning against exactly what masquerade does: we + invalidate the pair the moment `next_flow_status` returns `Reset` or `Closed`, and RFC 2663 says a + NAT cannot assume no retransmission follows. **This is not a defect.** RFC 5382 revisits the same + question and leaves it to us -- "NAT behavior for handling RST packets, or connections in + TIME_WAIT state is left unspecified", with an explicit `MAY` to hold state and an explicit note + that holding it "may limit the throughput of connections through a NAT with limited resources". + We took the throughput side. The erratum is what makes that a decision rather than an oversight. +- **RFC 8200 EID 5945** rewrites Section 4.5 from the three-part fragmentation model back to the + two-part model of RFC 2460, dropping the "Extension & Upper-Layer Headers" division. It concerns + source fragmentation, which this dataplane does not perform, so it is recorded and not acted on. + +**The vendored specification stays the base text.** `.duvet/specifications/` must hold what duvet +would fetch, or the quotes stop matching and the vendoring stops being a drop-in for the network. +Errata are checked alongside it, not merged into it. + ## Open questions Expected to expand. Nothing here is scheduled. @@ -137,10 +197,9 @@ Expected to expand. Nothing here is scheduled. `type=implementation` and see whether the test cited `type=test` fails. If it does not, the citation is decorative. This is the only item on this list that makes the three tools check each other rather than merely coexist. -4. **Errata.** The rsync corpus carries no errata bodies -- `inline-errata/` holds stylesheets only. - It reports that 2,613 RFCs have errata and never what they say. A second fetch leg is needed - regardless of how the corpus is pinned. Outstanding and concrete: **RFC 4884 has errata, and the - `MIN_ORIGINAL_DATAGRAM_OCTETS` fix was written without reading them.** +4. **Errata.** Mostly settled; see [Errata](#errata) below. What remains is the 847 RFCs whose + errata exist but are not Verified, for which the corpus holds no body. None of them is tracked + today; RFC 4443 and RFC 3022 are both in that set and both plausible future targets. 5. **How the corpus is pinned** -- a git mirror, or `oras` into ghcr.io behind `npins`. Sizing: the metadata that drives every drift alarm (indexes, `bcp/`, `std/`) is 5MB; the 232MB is RFC bodies, of which we cite perhaps ten. Text gzips about 4:1. From 29a7861c8c0d9b4de75b94acc856e21ea61dc5e2 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Wed, 19 Aug 2026 12:09:18 -0600 Subject: [PATCH 26/37] test(masquerade): State RFC 4787 REQ-12 as an executable contract A duvet citation records that somebody read a requirement. It cannot record that the code still does what the sentence says, and it cannot record that the test cited `type=test` checks the same thing as the code cited `type=implementation`. Those are two independent readings of one sentence, and a refactor can separate them without either comment changing. `contract::rfc4787::Req12` is the predicate written once. The implementation calls it from a `debug_assert!`; the exhaustive state machine test calls it directly. Injecting a mutant that sends an established flow to `Closed` on an ICMP packet now panics at the implementation site with the two states named, rather than at whichever assertion happened to notice. This closes a live gap rather than only demonstrating the pattern. RFC 4787 REQ-12 and RFC 5382 REQ-10 are the same sentence, kept by the same function and proved by the same test, but only RFC 5382 carried a `type=test` citation, so REQ-12 stood at `[!MUST,implementation]` -- implemented, untested -- while the test that establishes it sat six lines away. Manual bookkeeping across two specifications is exactly what a shared predicate removes. Only local predicates belong in `contract`. REQ-1 relates two mappings made at different times, REQ-6 relates a packet to a timer, REQ-11 is a statement about the answers to the other requirements; none is decidable at a point and all stay in `fuzz`. Where a requirement can be encoded more strongly it should be, per development/code/avoid-global-reasoning.md, and then it does not belong here at all. An `rfc5382::Req10` alias was written and deleted. Nothing called it, the compiler said so, and an uncalled contract is the decoration this replaces. The snapshot in .duvet/ is not regenerated here: duvet is not on PATH outside the dev shell, and hand-editing the regression gate would defeat it. Co-Authored-By: Claude Opus 5 (1M context) (cherry picked from commit d1a850c9dc45f38b5a3490371d28d1158c73e4f1) --- nat/src/masquerade/contract.rs | 155 ++++++++++++++++++++++++++++ nat/src/masquerade/mod.rs | 1 + nat/src/masquerade/protocol.rs | 21 ++-- nat/src/masquerade/state_machine.rs | 32 ++++-- 4 files changed, 192 insertions(+), 17 deletions(-) create mode 100644 nat/src/masquerade/contract.rs diff --git a/nat/src/masquerade/contract.rs b/nat/src/masquerade/contract.rs new file mode 100644 index 0000000000..873cae6b35 --- /dev/null +++ b/nat/src/masquerade/contract.rs @@ -0,0 +1,155 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright Open Network Fabric Authors + +//! Specification requirements, stated as things the program can execute. +//! +//! A duvet citation is a quoted sentence in a comment. It records that somebody read the +//! requirement; it cannot record that the code still does what the sentence says, and it cannot +//! record that the test cited `type=test` checks the same thing as the code cited +//! `type=implementation`. Those two citations are independent readings of one sentence, and a +//! refactor can separate them without either comment changing. +//! +//! A type in this module is the predicate itself, written once. The implementation calls it from a +//! [`debug_assert!`]; the test that carries the `type=test` citation calls it directly. The two +//! citations are then provably about the same predicate, because there is only one. +//! +//! This does not make a citation non-vacuous on its own -- a test can still feed inputs that never +//! reach the interesting case, which is what the `MIN_REACHED` counters in +//! [`fuzz`](super::fuzz) and `cargo-mutants` are for. It rules out the other failure: a test that +//! checks something unrelated to the requirement it names. +//! +//! # What belongs here +//! +//! Only requirements that are **local predicates** -- decidable from values available at one point +//! in the program. Most are not. RFC 4787 REQ-1 relates two mappings made at different times, +//! REQ-6 relates a packet to a timer, REQ-11 is a statement about the answers to the other +//! requirements. Those can only be properties, and they stay in [`fuzz`](super::fuzz). +//! +//! Where a requirement can be encoded more strongly, it should be, and then it does not belong +//! here either. `development/code/avoid-global-reasoning.md` ranks the options: make the illegal +//! state unrepresentable, then encode it in a type, and only then check it at runtime. A +//! constraint on a `const` is a `const` assertion and fails the build, which beats any of this. +//! +//! # Naming +//! +//! `rfc::`. The section number is deliberately absent: +//! it is already in the `//=` URL of the citation, and a second copy is a second thing to keep +//! correct. Where a specification numbers nothing -- RFC 4884 has no `REQ-` clauses, its unit is a +//! sentence in a section -- name the contract for what it says rather than inventing an index the +//! specification does not have. + +use crate::common::NatFlowStatus; + +/// A requirement did not hold. +/// +/// Carries the values that broke it, because a `debug_assert!` that says only `false` costs more +/// time than it saves. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct Violation { + /// The requirement, as `rfc4787#section-9 REQ-12`. + pub(crate) requirement: &'static str, + /// What went wrong, in the specification's own terms. + pub(crate) detail: &'static str, +} + +impl std::fmt::Display for Violation { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}: {}", self.requirement, self.detail) + } +} + +/// [RFC 4787](https://www.rfc-editor.org/rfc/rfc4787), NAT behavioural requirements for UDP. +pub(crate) mod rfc4787 { + use super::{NatFlowStatus, Violation}; + + /// > REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the NAT mapping. + /// + /// Terminating a mapping is what releases the public address and port it holds, so this is + /// read as: an ICMP packet may not move a live flow into a state that ends it. A flow already + /// in a terminal state stays there -- the requirement forbids ICMP *causing* termination, not + /// the flow having been terminated by something else. + /// + /// RFC 5382 REQ-10 is the same sentence for TCP, and is kept by this same check: masquerade's + /// ICMP transition runs for every ICMP packet regardless of the protocol of the flow it + /// belongs to. RFC 5382 states the stronger obligation -- the mapping *or the TCP connection* + /// -- but the second half is about not injecting a reset, which this stage never does. + /// + /// A `rfc5382::Req10` alias was written and deleted: nothing called it, and an uncalled + /// contract is the decoration this module exists to replace. + #[derive(Debug, Clone, Copy)] + pub(crate) struct Req12 { + before: NatFlowStatus, + after: NatFlowStatus, + } + + impl Req12 { + /// State the requirement over one ICMP-driven transition. + pub(crate) const fn new(before: NatFlowStatus, after: NatFlowStatus) -> Self { + Self { before, after } + } + + /// Whether an ICMP packet has ended a mapping that was live. + pub(crate) const fn check(self) -> Result<(), Violation> { + if Self::terminal(self.after) && !Self::terminal(self.before) { + return Err(Violation { + requirement: "rfc4787#section-9 REQ-12 / rfc5382#section-8 REQ-10", + detail: "an ICMP packet moved a live flow into a terminal state", + }); + } + Ok(()) + } + + /// The states in which the mapping is over and the tuple is released. + const fn terminal(status: NatFlowStatus) -> bool { + matches!(status, NatFlowStatus::Closed | NatFlowStatus::Reset) + } + } +} + +#[cfg(test)] +mod test { + use super::rfc4787::Req12; + use crate::common::NatFlowStatus; + + /// Every status the machine can be in, terminal ones last. + const STATUSES: [NatFlowStatus; 10] = [ + NatFlowStatus::OneWay, + NatFlowStatus::TwoWay, + NatFlowStatus::Established, + NatFlowStatus::CClosing, + NatFlowStatus::SClosing, + NatFlowStatus::CHalfClose, + NatFlowStatus::SHalfClose, + NatFlowStatus::LastAck, + NatFlowStatus::Reset, + NatFlowStatus::Closed, + ]; + + /// The contract is only worth calling if it can fail, and only correct if it fails on exactly + /// the transitions the requirement forbids. + /// + /// This tests the *statement*, not the implementation that satisfies it. Without it a contract + /// that returned `Ok` unconditionally would make every caller pass, including the + /// `debug_assert!` and the state machine test, which is the failure mode this whole pattern + /// exists to prevent. + #[test] + fn the_contract_rejects_exactly_the_forbidden_transitions() { + let terminal = |s| matches!(s, NatFlowStatus::Closed | NatFlowStatus::Reset); + let mut rejected = 0; + for before in STATUSES { + for after in STATUSES { + let forbidden = terminal(after) && !terminal(before); + assert_eq!( + Req12::new(before, after).check().is_err(), + forbidden, + "{before:?} -> {after:?}" + ); + rejected += usize::from(forbidden); + } + } + assert_eq!( + rejected, 16, + "8 live statuses times 2 terminal ones must be the whole forbidden set" + ); + } +} diff --git a/nat/src/masquerade/mod.rs b/nat/src/masquerade/mod.rs index 99d4d7106c..7dafbb2b47 100644 --- a/nat/src/masquerade/mod.rs +++ b/nat/src/masquerade/mod.rs @@ -4,6 +4,7 @@ pub(crate) mod allocation; mod allocator_writer; pub mod apalloc; +mod contract; mod expiry; pub(crate) mod flows; mod fuzz; diff --git a/nat/src/masquerade/protocol.rs b/nat/src/masquerade/protocol.rs index df0fca1162..85a4620482 100644 --- a/nat/src/masquerade/protocol.rs +++ b/nat/src/masquerade/protocol.rs @@ -6,6 +6,7 @@ //! for port conservation. use crate::common::{NatAction, NatFlowStatus}; +use crate::masquerade::contract::rfc4787::Req12; use net::buffer::PacketBufferMut; use net::headers::{TryHeaders, TryIp, TryTcp}; @@ -66,22 +67,21 @@ fn next_flow_status_udp(action: NatAction, status: NatFlowStatus) -> NatFlowStat } //= https://www.rfc-editor.org/rfc/rfc5382#section-8 +//= type=implementation //# REQ-10: Receipt of any sort of ICMP message MUST NOT terminate the //# NAT mapping or TCP connection for which the ICMP was generated. // //= https://www.rfc-editor.org/rfc/rfc4787#section-9 +//= type=implementation //# REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the //# NAT mapping. // -// Held by construction: no arm below yields `Closed` or `Reset`, so no ICMP message can end a -// mapping. The only transition available is the one that records that traffic came back. -// -// Both specifications state this requirement, and this function is where both are kept. It runs -// for every ICMP packet regardless of the protocol of the flow it belongs to, so the guarantee -// does not depend on which specification you read it under. +// Held by construction: no arm below yields `Closed` or `Reset`. The `debug_assert!` is what keeps +// it that way, and it is the same predicate the test cited `type=test` calls -- see +// `contract::rfc4787::Req12`. #[allow(clippy::match_single_binding)] fn next_flow_status_icmp(action: NatAction, status: NatFlowStatus) -> NatFlowStatus { - match action { + let next = match action { NatAction::SrcNat => match status { _ => status, }, @@ -89,7 +89,12 @@ fn next_flow_status_icmp(action: NatAction, status: NatFlowStatus) -> NatFlowSta NatFlowStatus::OneWay => NatFlowStatus::TwoWay, _ => status, }, - } + }; + debug_assert!( + Req12::new(status, next).check().is_ok(), + "{action} {status:?} -> {next:?}" + ); + next } fn next_flow_status_tcp(action: NatAction, status: NatFlowStatus, tcp: &Tcp) -> NatFlowStatus { diff --git a/nat/src/masquerade/state_machine.rs b/nat/src/masquerade/state_machine.rs index d526f34841..aa45b9747a 100644 --- a/nat/src/masquerade/state_machine.rs +++ b/nat/src/masquerade/state_machine.rs @@ -37,6 +37,7 @@ #![cfg(test)] use crate::common::{NatAction, NatFlowStatus}; +use crate::masquerade::contract::rfc4787::Req12; use crate::masquerade::protocol::next_flow_status; use net::buffer::TestBuffer; use net::headers::TryTcpMut; @@ -271,6 +272,11 @@ fn ordinary_udp_opens_and_settles() { //= type=test //# REQ-10: Receipt of any sort of ICMP message MUST NOT terminate the //# NAT mapping or TCP connection for which the ICMP was generated. +// +//= https://www.rfc-editor.org/rfc/rfc4787#section-9 +//= type=test +//# REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the +//# NAT mapping. /// An ICMP echo reply makes a one-way flow two-way, and nothing else moves. /// /// ICMP has no flags to read and no close sequence, so the only evidence available is that a packet @@ -294,17 +300,25 @@ fn an_icmp_reply_makes_a_flow_two_way_and_nothing_more() { // Every other status, in both directions, is left exactly where it was: there is no further // evidence an icmp exchange can offer. + // + // The `Req12` call is the requirement; the `assert_eq!` around it is the stronger local claim + // that nothing moves at all. Both are wanted -- the requirement is what a reviewer checks + // against the RFC, and it is the same predicate `next_flow_status_icmp` asserts, so neither + // can drift from the other without this test failing. for status in STATUSES { - assert_eq!( - next_flow_status(&packet, NatAction::SrcNat, status), - status, - "an outbound icmp packet moved a flow in {status:?}" - ); - if status != NatFlowStatus::OneWay { + for action in [NatAction::SrcNat, NatAction::DstNat] { + let next = next_flow_status(&packet, action, status); assert_eq!( - next_flow_status(&packet, NatAction::DstNat, status), - status, - "an inbound icmp packet moved a flow in {status:?}" + Req12::new(status, next).check(), + Ok(()), + "{action} icmp packet terminated a flow in {status:?}" + ); + if action == NatAction::DstNat && status == NatFlowStatus::OneWay { + continue; + } + assert_eq!( + next, status, + "an {action} icmp packet moved a flow in {status:?}" ); } } From e7d05200d9316072f5f0e0687e2b82a28b786bae Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Wed, 19 Aug 2026 13:02:41 -0600 Subject: [PATCH 27/37] test(masquerade): Make a stale citation a build failure `Requirement` replaces the inherent `check` method, so the convention cannot be got wrong by the next contract: a trait has one shape, an inherent method has as many as there are authors. `Violation` and its two `&'static str` fields are gone, replaced by a `thiserror` type per requirement carrying the values that broke it -- development/code/error-handling.md calls string errors "actively hostile", and nothing forced one here, since `before` and `after` were already in hand. The part worth having is `SPEC` and `ID` being `const`. duvet emits one TOML per specification section under `.duvet/requirements/`, so the tree already holds a machine-readable copy of every requirement tracked, and a `const` assertion over `include_str!` checks that the section a contract names really states the requirement it claims. Changing `REQ-12` to `REQ-42` now fails the build with E0080 rather than passing review. Dropping RFC 4787 from .duvet/config.toml would fail it too, and because `include_str!` is recorded in rustc's dependency information, re-extracting a specification rebuilds the check instead of leaving it stale. This is what the citation axis could not do on its own. duvet verifies that a quoted sentence matches the specification; it cannot verify that the code naming that sentence still exists, and nothing verified the reverse direction at all. `unreachable!` rather than `panic!` at the call site, per development/code/error-handling.md: reaching it is programmer error. It is guarded on `cfg!(debug_assertions)` first so the check does not run in release, and it names the specification URL, the requirement id and both states, so a failure is readable without opening the file. The trait method cannot be `const fn` on stable, which settles where the build-time tier lives: constraints over `const`s -- RFC 4787 REQ-5 bounds timers that are `const`s -- stay plain `const` assertions and are deliberately not `Requirement`s. Co-Authored-By: Claude Opus 5 (1M context) (cherry picked from commit 86cd54b7071c22108f46ee0e5389bfa18c221f63) --- nat/src/masquerade/contract.rs | 182 +++++++++++++++++++++------- nat/src/masquerade/protocol.rs | 21 ++-- nat/src/masquerade/state_machine.rs | 1 + 3 files changed, 156 insertions(+), 48 deletions(-) diff --git a/nat/src/masquerade/contract.rs b/nat/src/masquerade/contract.rs index 873cae6b35..3e4b8835c5 100644 --- a/nat/src/masquerade/contract.rs +++ b/nat/src/masquerade/contract.rs @@ -9,14 +9,26 @@ //! `type=implementation`. Those two citations are independent readings of one sentence, and a //! refactor can separate them without either comment changing. //! -//! A type in this module is the predicate itself, written once. The implementation calls it from a +//! A [`Requirement`] here is the predicate itself, written once. The implementation calls it from a //! [`debug_assert!`]; the test that carries the `type=test` citation calls it directly. The two //! citations are then provably about the same predicate, because there is only one. //! //! This does not make a citation non-vacuous on its own -- a test can still feed inputs that never -//! reach the interesting case, which is what the `MIN_REACHED` counters in -//! [`fuzz`](super::fuzz) and `cargo-mutants` are for. It rules out the other failure: a test that -//! checks something unrelated to the requirement it names. +//! reach the interesting case, which is what the `MIN_REACHED` counters in [`fuzz`](super::fuzz) +//! and `cargo-mutants` are for. It rules out the other failure: a test that checks something +//! unrelated to the requirement it names. +//! +//! # The citation is checked at compile time +//! +//! [`Requirement::SPEC`] and [`Requirement::ID`] are not decoration. duvet emits one TOML file per +//! specification section under `.duvet/requirements/`, so the tree already holds a machine-readable +//! copy of every requirement we track. Each contract asserts in a `const` block that the section it +//! names really does state the requirement it claims. A contract citing a requirement its +//! specification does not contain -- or citing a specification since dropped from +//! `.duvet/config.toml` -- fails the build rather than the review. +//! +//! `include_str!` is recorded in rustc's dependency information, so re-extracting a specification +//! rebuilds these checks rather than leaving them stale. //! //! # What belongs here //! @@ -25,51 +37,88 @@ //! REQ-6 relates a packet to a timer, REQ-11 is a statement about the answers to the other //! requirements. Those can only be properties, and they stay in [`fuzz`](super::fuzz). //! -//! Where a requirement can be encoded more strongly, it should be, and then it does not belong -//! here either. `development/code/avoid-global-reasoning.md` ranks the options: make the illegal -//! state unrepresentable, then encode it in a type, and only then check it at runtime. A -//! constraint on a `const` is a `const` assertion and fails the build, which beats any of this. +//! Where a requirement can be encoded more strongly it should be, and then it does not belong here +//! either. `development/code/avoid-global-reasoning.md` ranks the options: make the illegal state +//! unrepresentable, then encode it in a type, and only then check it at runtime. A constraint on a +//! `const` -- RFC 4787 REQ-5 bounds timers that are `const`s -- is a `const` assertion and fails +//! the build, which beats anything this trait can do. Such requirements are deliberately not +//! [`Requirement`]s: a trait method cannot be `const fn` on stable. //! //! # Naming //! -//! `rfc::`. The section number is deliberately absent: -//! it is already in the `//=` URL of the citation, and a second copy is a second thing to keep -//! correct. Where a specification numbers nothing -- RFC 4884 has no `REQ-` clauses, its unit is a -//! sentence in a section -- name the contract for what it says rather than inventing an index the -//! specification does not have. +//! `rfc::`. The section number is deliberately absent +//! from the module path: it is already in [`Requirement::SPEC`], and a second copy is a second +//! thing to keep correct. Where a specification numbers nothing -- RFC 4884 has no `REQ-` clauses, +//! its unit is a sentence in a section -- name the contract for what it says rather than inventing +//! an index the specification does not have. use crate::common::NatFlowStatus; -/// A requirement did not hold. +/// A specification requirement that can be decided from values available at one point. /// -/// Carries the values that broke it, because a `debug_assert!` that says only `false` costs more -/// time than it saves. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub(crate) struct Violation { - /// The requirement, as `rfc4787#section-9 REQ-12`. - pub(crate) requirement: &'static str, - /// What went wrong, in the specification's own terms. - pub(crate) detail: &'static str, +/// Implementors are constructed where the requirement applies, carrying exactly the values it is +/// about, and asked whether it holds. +pub(crate) trait Requirement { + /// What a violation carries. One type per requirement, because the evidence differs and + /// `development/code/error-handling.md` asks for a dedicated error type rather than a string. + type Error: core::error::Error; + + /// The citation target, in the same form as duvet's `//=` marker. + const SPEC: &'static str; + + /// The specification's own identifier, such as `REQ-12`. + const ID: &'static str; + + /// Whether the requirement holds for these values. + /// + /// # Errors + /// + /// Returns the evidence that it does not. + fn check(&self) -> Result<(), Self::Error>; } -impl std::fmt::Display for Violation { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(f, "{}: {}", self.requirement, self.detail) +/// Whether `haystack` contains `needle`, in a `const` context. +/// +/// Exists because the cross-check against duvet's extracted requirements has to run before the +/// program does, and `str::contains` is not `const` on stable. +const fn contains(haystack: &str, needle: &str) -> bool { + let (h, n) = (haystack.as_bytes(), needle.as_bytes()); + if n.is_empty() { + return true; } + if h.len() < n.len() { + return false; + } + let mut i = 0; + while i <= h.len() - n.len() { + let mut j = 0; + while j < n.len() && h[i + j] == n[j] { + j += 1; + } + if j == n.len() { + return true; + } + i += 1; + } + false } /// [RFC 4787](https://www.rfc-editor.org/rfc/rfc4787), NAT behavioural requirements for UDP. pub(crate) mod rfc4787 { - use super::{NatFlowStatus, Violation}; + use super::{NatFlowStatus, Requirement, contains}; + + /// What duvet extracted from the section [`Req12`] cites. + const SECTION_9: &str = + include_str!("../../../.duvet/requirements/www.rfc-editor.org/rfc/rfc4787/section-9.toml"); /// > REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the NAT mapping. /// - /// Terminating a mapping is what releases the public address and port it holds, so this is - /// read as: an ICMP packet may not move a live flow into a state that ends it. A flow already - /// in a terminal state stays there -- the requirement forbids ICMP *causing* termination, not - /// the flow having been terminated by something else. + /// Terminating a mapping is what releases the public address and port it holds, so this reads + /// as: an ICMP packet may not move a live flow into a state that ends it. A flow already in a + /// terminal state stays there -- the requirement forbids ICMP *causing* termination, not the + /// flow having been terminated by something else. /// - /// RFC 5382 REQ-10 is the same sentence for TCP, and is kept by this same check: masquerade's + /// RFC 5382 REQ-10 is the same sentence for TCP and is kept by this same check: masquerade's /// ICMP transition runs for every ICMP packet regardless of the protocol of the flow it /// belongs to. RFC 5382 states the stronger obligation -- the mapping *or the TCP connection* /// -- but the second half is about not injecting a reset, which this stage never does. @@ -82,33 +131,57 @@ pub(crate) mod rfc4787 { after: NatFlowStatus, } + /// An ICMP packet ended a mapping that was live. + /// + /// Carries both states because "the requirement broke" is not actionable and + /// `Established -> Closed` is. + #[derive(Debug, Clone, Copy, PartialEq, thiserror::Error)] + #[error("an ICMP packet moved a live flow from {before:?} to {after:?}")] + pub(crate) struct Req12Violated { + before: NatFlowStatus, + after: NatFlowStatus, + } + impl Req12 { /// State the requirement over one ICMP-driven transition. pub(crate) const fn new(before: NatFlowStatus, after: NatFlowStatus) -> Self { Self { before, after } } - /// Whether an ICMP packet has ended a mapping that was live. - pub(crate) const fn check(self) -> Result<(), Violation> { + /// The states in which the mapping is over and the tuple is released. + const fn terminal(status: NatFlowStatus) -> bool { + matches!(status, NatFlowStatus::Closed | NatFlowStatus::Reset) + } + } + + impl Requirement for Req12 { + type Error = Req12Violated; + const SPEC: &'static str = "https://www.rfc-editor.org/rfc/rfc4787#section-9"; + const ID: &'static str = "REQ-12"; + + fn check(&self) -> Result<(), Self::Error> { if Self::terminal(self.after) && !Self::terminal(self.before) { - return Err(Violation { - requirement: "rfc4787#section-9 REQ-12 / rfc5382#section-8 REQ-10", - detail: "an ICMP packet moved a live flow into a terminal state", + return Err(Req12Violated { + before: self.before, + after: self.after, }); } Ok(()) } - - /// The states in which the mapping is over and the tuple is released. - const fn terminal(status: NatFlowStatus) -> bool { - matches!(status, NatFlowStatus::Closed | NatFlowStatus::Reset) - } } + + // The specification we cite must state the requirement we claim. This is what `SPEC` and `ID` + // are `const` for: a citation that has gone stale stops the build. + const _: () = assert!( + contains(SECTION_9, ::ID), + "rfc4787#section-9 does not state REQ-12" + ); } #[cfg(test)] mod test { use super::rfc4787::Req12; + use super::{Requirement, contains}; use crate::common::NatFlowStatus; /// Every status the machine can be in, terminal ones last. @@ -125,6 +198,21 @@ mod test { NatFlowStatus::Closed, ]; + /// The compile-time cross-check is only worth having if it can fail. + /// + /// A `const` assertion cannot demonstrate its own negative case -- getting it wrong stops the + /// build rather than reporting -- so the search it relies on is exercised here instead. + #[test] + fn the_specification_search_can_fail() { + assert!(contains("REQ-12: Receipt of any", "REQ-12")); + assert!(!contains("REQ-12: Receipt of any", "REQ-42")); + assert!(!contains("REQ-1", "REQ-12"), "a prefix is not a match"); + assert!( + contains("anything", ""), + "the empty needle is always present" + ); + } + /// The contract is only worth calling if it can fail, and only correct if it fails on exactly /// the transitions the requirement forbids. /// @@ -152,4 +240,16 @@ mod test { "8 live statuses times 2 terminal ones must be the whole forbidden set" ); } + + /// A violation names both states, because that is what makes it actionable. + #[test] + fn a_violation_reports_the_transition_that_caused_it() { + let err = Req12::new(NatFlowStatus::Established, NatFlowStatus::Closed) + .check() + .expect_err("an established flow moved to closed must violate REQ-12"); + assert_eq!( + err.to_string(), + "an ICMP packet moved a live flow from Established to Closed" + ); + } } diff --git a/nat/src/masquerade/protocol.rs b/nat/src/masquerade/protocol.rs index 85a4620482..a9727f5d8e 100644 --- a/nat/src/masquerade/protocol.rs +++ b/nat/src/masquerade/protocol.rs @@ -6,6 +6,7 @@ //! for port conservation. use crate::common::{NatAction, NatFlowStatus}; +use crate::masquerade::contract::Requirement; use crate::masquerade::contract::rfc4787::Req12; use net::buffer::PacketBufferMut; use net::headers::{TryHeaders, TryIp, TryTcp}; @@ -76,9 +77,8 @@ fn next_flow_status_udp(action: NatAction, status: NatFlowStatus) -> NatFlowStat //# REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the //# NAT mapping. // -// Held by construction: no arm below yields `Closed` or `Reset`. The `debug_assert!` is what keeps -// it that way, and it is the same predicate the test cited `type=test` calls -- see -// `contract::rfc4787::Req12`. +// Held by construction: no arm below yields `Closed` or `Reset`. `contract::rfc4787::Req12` is what +// keeps it that way, and it is the same predicate the test cited `type=test` calls. #[allow(clippy::match_single_binding)] fn next_flow_status_icmp(action: NatAction, status: NatFlowStatus) -> NatFlowStatus { let next = match action { @@ -90,10 +90,17 @@ fn next_flow_status_icmp(action: NatAction, status: NatFlowStatus) -> NatFlowSta _ => status, }, }; - debug_assert!( - Req12::new(status, next).check().is_ok(), - "{action} {status:?} -> {next:?}" - ); + // `unreachable!` rather than `panic!`, per development/code/error-handling.md: reaching this is + // programmer error, not a runtime condition. The whole block folds away in release. + if cfg!(debug_assertions) + && let Err(violation) = Req12::new(status, next).check() + { + unreachable!( + "{spec} {id}: {violation} ({action})", + spec = Req12::SPEC, + id = Req12::ID + ); + } next } diff --git a/nat/src/masquerade/state_machine.rs b/nat/src/masquerade/state_machine.rs index aa45b9747a..d25dffc938 100644 --- a/nat/src/masquerade/state_machine.rs +++ b/nat/src/masquerade/state_machine.rs @@ -37,6 +37,7 @@ #![cfg(test)] use crate::common::{NatAction, NatFlowStatus}; +use crate::masquerade::contract::Requirement; use crate::masquerade::contract::rfc4787::Req12; use crate::masquerade::protocol::next_flow_status; use net::buffer::TestBuffer; From faf1071f86162fbd67fb9272b8f06df0edf30ff3 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Wed, 19 Aug 2026 20:05:23 -0600 Subject: [PATCH 28/37] docs(forwarding): Record RFC 4787 REQ-13, and why it is one decision not three Both clauses are uncited and unheld, and the reason is structural rather than a missing branch: there is no MTU anywhere on this datapath. net::interface::Mtu appears in ten files, all of them control plane -- config, the FRR renderer and interface-manager, which push it to the kernel over netlink. It reaches neither dataplane, pipeline nor nat. Cited at vxlan_encap because that is the one place the dataplane makes a packet larger, which is the condition RFC 4787 section 10 governs. Its only failure modes on size are mbuf headroom and the 2^16 ceiling of the IP length field, and neither is a link MTU. The finding worth taking to the team is the scope. This stage originates no ICMP error at all: TTL expiry drops on DoneReason::HopLimitExceeded where RFC 1812 asks a router for ICMP Time Exceeded, and nat::icmp_handler only translates errors that arrive. REQ-13, REQ-13a and the TTL case are one question -- does this gateway originate ICMP errors? -- whose answer needs an egress MTU first. Co-Authored-By: Claude Opus 5 (1M context) (cherry picked from commit 017b8819d9f00eab1a3701d7c1f287e7cf3f18ec) --- dataplane/src/packet_processor/ipforward.rs | 30 +++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/dataplane/src/packet_processor/ipforward.rs b/dataplane/src/packet_processor/ipforward.rs index 719d471f22..d130fa96b6 100644 --- a/dataplane/src/packet_processor/ipforward.rs +++ b/dataplane/src/packet_processor/ipforward.rs @@ -219,6 +219,36 @@ impl IpForwarder { } /// Encapsulate a packet in Vxlan with the provided [`VxlanEncapsulation`] params + //= https://www.rfc-editor.org/rfc/rfc4787#section-10 + //= type=todo + //# REQ-13: If the packet received on an internal IP address has DF=1, + //# the NAT MUST send back an ICMP message "Fragmentation needed and + //# DF set" to the host, as described in [RFC0792]. + // + //= https://www.rfc-editor.org/rfc/rfc4787#section-10 + //= type=todo + //# a) If the packet has DF=0, the NAT MUST fragment the packet and + //# SHOULD send the fragments in order. + // + // Neither is held, and not for want of a branch: there is no MTU on this datapath to compare a + // packet against. `net::interface::Mtu` lives entirely in the control plane -- `config`, the + // FRR renderer and `interface-manager`, which push it to the kernel over netlink -- and reaches + // neither `dataplane`, `pipeline` nor `nat`. + // + // The requirement lands here because this is the one place the dataplane makes a packet + // *larger*: VxLAN encapsulation prepends an outer Ethernet, IP, UDP and VxLAN header to a frame + // already sized for the tenant's link. The only size limits it can fail on are the mbuf's own + // headroom and the 2^16 ceiling of the IP length field, and neither is a link MTU, so a frame + // too large for the underlay leaves here intact and is dropped without notice further on. + // + // The wider fact is that this stage originates no ICMP error at all. `decrement_ttl` below has + // the same shape: expiry sets `DoneReason::HopLimitExceeded` and drops, where RFC 1812 section + // 4.3.2.3 asks a router for ICMP Time Exceeded. `nat::icmp_handler` translates and forwards + // ICMP errors that arrive; nothing in the tree builds one. + // + // So REQ-13, REQ-13a and the TTL case are one decision rather than three findings -- does this + // gateway originate ICMP errors? -- and answering it needs an egress MTU first, which is the + // larger half of the work. `todo` rather than `exception` because nobody has ruled. fn vxlan_encap( &self, packet: &mut Packet, From 631488c4eef2b63b714b0c6fe8483ac0697f3c2f Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Thu, 20 Aug 2026 15:11:15 -0600 Subject: [PATCH 29/37] fix(duvet): Regenerate the snapshot the REQ-12 and REQ-13 commits left stale Nothing regenerated it, so the regression gate had drifted two commits after being introduced. `just duvet-check` now catches this. (cherry picked from commit 02860ace769a4a961b424a6082222eb0414e6280) --- .duvet/snapshot.txt | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/.duvet/snapshot.txt b/.duvet/snapshot.txt index 0d495baebf..f79ef0e675 100644 --- a/.duvet/snapshot.txt +++ b/.duvet/snapshot.txt @@ -77,19 +77,19 @@ SPECIFICATION: https://www.rfc-editor.org/rfc/rfc4787 TEXT[!SHOULD]: host. TEXT[!MUST]: Receipt of any sort of ICMP message MUST NOT TEXT[!MUST]: destroy the NAT mapping. - TEXT[!MUST,implementation]: REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the - TEXT[!MUST,implementation]: NAT mapping. + TEXT[!MUST,implementation,test]: REQ-12: Receipt of any sort of ICMP message MUST NOT terminate the + TEXT[!MUST,implementation,test]: NAT mapping. TEXT[!SHOULD]: a) The NAT's default configuration SHOULD NOT filter ICMP messages TEXT[!SHOULD]: based on their source IP address. TEXT[!SHOULD]: b) It is RECOMMENDED that a NAT support ICMP Destination TEXT[!SHOULD]: Unreachable messages. SECTION: [Fragmentation of Outgoing Packets](#section-10) - TEXT[!MUST]: REQ-13: If the packet received on an internal IP address has DF=1, - TEXT[!MUST]: the NAT MUST send back an ICMP message "Fragmentation needed and - TEXT[!MUST]: DF set" to the host, as described in [RFC0792]. - TEXT[!MUST]: a) If the packet has DF=0, the NAT MUST fragment the packet and - TEXT[!MUST]: SHOULD send the fragments in order. + TEXT[!MUST,todo]: REQ-13: If the packet received on an internal IP address has DF=1, + TEXT[!MUST,todo]: the NAT MUST send back an ICMP message "Fragmentation needed and + TEXT[!MUST,todo]: DF set" to the host, as described in [RFC0792]. + TEXT[!MUST,todo]: a) If the packet has DF=0, the NAT MUST fragment the packet and + TEXT[!MUST,todo]: SHOULD send the fragments in order. SECTION: [Receiving Fragmented Packets](#section-11) TEXT[!MUST,todo]: REQ-14: A NAT MUST support receiving in-order and out-of-order From 13e981b9977cf14a127c08ce965e02021d3587bf Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Thu, 20 Aug 2026 15:11:26 -0600 Subject: [PATCH 30/37] docs(testing): Record what the interlock found, and which specifications apply The RFC enumeration answers the first open question and changes the plan for extending past NAT: the largest specification surface in the tree, RFC 8200 in `net`, is the one duvet cannot parse at all. (cherry picked from commit c1bb982e7414ff31c18af7b06a9eb9dfedd94382) --- development/code/mutation-testing.md | 25 ++++++ development/code/spec-compliance.md | 124 ++++++++++++++++++++++++--- 2 files changed, 136 insertions(+), 13 deletions(-) diff --git a/development/code/mutation-testing.md b/development/code/mutation-testing.md index 4a8e84a7b2..3197b22a06 100644 --- a/development/code/mutation-testing.md +++ b/development/code/mutation-testing.md @@ -73,6 +73,31 @@ Test scaffolding compiled into a library -- `net/src/buffer/test_buffer.rs` is t with thirty survivors -- belongs here too. Files behind a module-level `#![cfg(test)]` are skipped automatically and need no exclusion. +## `StructField` mutants ignore every filter + +Measured, and it silently voids part of the configuration above. + +`cargo mutants` does not apply `--re`, `--exclude-re`, or `.cargo/mutants.toml` to mutants of +the `StructField` genre -- the `delete field X from struct Y expression` ones. A pattern that +matches nothing still returns them: + +```console +$ cargo mutants --list -p dataplane-net --re 'zzz_no_such_mutant_zzz' +net/src/flows/flow_info.rs:478:13: delete field status from struct Self expression in ... +net/src/headers/embedded.rs:1677:21: delete field net from struct EmbeddedHeaders expression in contract::... +... 13 in total +``` + +Two consequences: + +- **The `contract::` exclusion is not doing what this document says it does.** Four of those + thirteen are in a `contract` module and are excluded by name in `.cargo/mutants.toml`. They are + generated and run regardless. The exclusion works for every other genre, which is why it looked + effective. +- **A filtered run is not scoped the way it appears to be.** Anything reading `missed.txt` and + assuming it holds only what was asked for will over-report. `scripts/spec-interlock.ts` filters + the result set itself for this reason rather than trusting the flags. + ## Cost Measured on `dataplane-net`: 2,271 mutants, about 4.5s to build and 7.6s to test each, four at a diff --git a/development/code/spec-compliance.md b/development/code/spec-compliance.md index bfa9ad8472..02594f4f96 100644 --- a/development/code/spec-compliance.md +++ b/development/code/spec-compliance.md @@ -1,7 +1,7 @@ # Specification compliance with duvet -Status: **three specifications tracked, the RFC corpus and its errata audited, the procedure -itself not yet proven.** +Status: **three specifications tracked, the RFC corpus and its errata audited, the citation +interlock built and returning findings.** The open-questions list below is expected to grow; it is written down so that it grows in one place rather than in four people's heads. @@ -78,6 +78,67 @@ paragraph, which duvet correctly declines to treat as normative. **Modern format is fine.** There is no cliff at RFC 8650; xml2rfc v3 output parses (RFC 9000: 522 requirements, RFC 9110: 412, RFC 8446: 431). +## Is a citation true? The interlock + +`just spec-interlock`, implemented by `scripts/spec-interlock.ts`. + +duvet checks that a `type=test` citation _exists_. It cannot check that the test named by one +says anything about the code named by the other: both are comments, and a refactor can separate +them without either changing. A requirement can therefore show implementation **and** test -- +the fully-green state -- while nothing tests it. + +The check is to make the tools check each other. For each requirement duvet has matched to both +an implementation and a test, mutate **only** the cited implementation region and run **only** +the cited tests. A mutant that survives is a change to the code that claims to implement the +requirement which the test that claims to check it does not notice. + +The unit is a (requirement, implementation, test) triple, not a file. `cargo mutants -f ` +answers a weaker question -- "is this file tested" -- and buries the signal: the RFC 4884 finding +below is four mutants among the 125 that `embedded.rs` generates. + +### Outcomes + +Four, and the last two matter as much as the first. + +| | meaning | +| --- | --- | +| **held** | every mutant in the cited region was caught by the cited tests | +| **decorative** | a mutant survived; the citation does not carry the weight it claims | +| **no-mutants** | the region produced nothing testable -- usually every mutant unviable | +| **stale** | the cited test name matches no test; the citation has rotted | + +`no-mutants` is not a pass. Reporting it as one would credit a citation for a check that never +ran, which is the failure the tool exists to catch. `stale` exists because it is the tool's own +worst failure mode: a renamed test makes the filter match nothing, nextest exits 0 having run +nothing, every mutant survives, and a citation that is merely out of date is reported as +decorative. The test names are checked against `cargo nextest list` before any mutant runs. + +### What it found, first time out + +Seven requirements carry both an implementation and a test. Three hold. Six minutes. + +**RFC 4884, "at least 128 octets" -- decorative, and on the path where the original defect was.** +Four mutants survive `a_field_shorter_than_128_octets_is_refused`: + +- The test iterates `[120, 124]`. It never tests 128, so `<` versus `<=` at the boundary is + invisible -- the same boundary class the `flow_info.rs` measurement above found. +- The citation is on **both** the ICMPv4 and the ICMPv6 branch, and there is no v6 test helper at + all. All three of the v6 branch's mutants survive. The RFC 4884 fix was applied to both + branches and tested on one. + +**RFC 4787 REQ-2, "IP address pooling: Paired" -- decorative.** `replace match guard +e.is_exhaustion() with true` survives. The guard is what separates allocator exhaustion from +every other allocator error, and the cited test never produces a non-exhaustion error. This is +the `protocol.rs` failure mode again: a test that walks a path rather than discriminating one. + +**RFC 4787 REQ-3 and RFC 5382 REQ-7, "no port overloading" -- not checkable as cited.** Every +mutant of the cited region is unviable. The citation sits on `allocate_v4`, which only forwards +to `allocate_from_tables`; the code that enforces the requirement is the one it delegates to. +The citation names the wrong region, and no amount of testing would have revealed that. + +That last category is the one to keep in mind when extending this. A citation can be wrong about +_where_ the requirement lives, and neither duvet nor a coverage report can see it. + ## Do not cite a composite BCP The worst failure found, because it exits 0 and reports a plausible number. @@ -183,20 +244,56 @@ Two errata on specifications we have discussed but do not track are worth having would fetch, or the quotes stop matching and the vendoring stops being a drop-in for the network. Errata are checked alongside it, not merged into it. +## Which specifications apply to us + +A first enumeration, from what the tree already names: every `RFC ####` mention in a `.rs` file, +mapped to the crate that makes it. It is a lower bound -- it finds specifications somebody has +already thought about, not ones nobody has -- but it is evidence rather than recollection, and it +is a `grep` to regenerate. + +Twenty-eight RFCs, of which three are tracked. + +| RFC | crates | note | +| --- | --- | --- | +| 8200 (IPv6) | `net` | **46 mentions, the most in the tree, and duvet cannot parse it** -- 0 uppercase keywords, 79 lowercase. Synthesis or nothing. | +| 4787 (NAT/UDP) | `nat`, `dataplane` | tracked | +| 4884 (ICMP extension) | `nat`, `net` | tracked | +| 7348 (VXLAN) | `net`, `dpdk` | 15 mentions, untracked | +| 4302 (IP AH) | `net` | 11 mentions, untracked | +| 5382 (NAT/TCP) | `nat` | tracked | +| 792, 1812, 1122, 1191 | `net`, `dataplane` | foundational; expect the same lowercase problem as 8200 | +| 9293 (TCP), 2018, 7323, 3168, 6946, 3540, 2675 | `net` | TCP option and fragmentation behaviour | +| 5508 (NAT/ICMP) | `nat`, `net` | duvet-friendly, on-topic, untracked -- 92 requirements | +| 6437, 4861, 6918 | `net` | IPv6 flow label, ND | +| 1624 | `net` | checksum update | +| 7854, 8671 | `routing` | BMP | +| 3339, 9562, 7637 | `config`, `mgmt`, `id`, `dpdk` | formats, not behaviour | + +What this changes about the plan: + +- **The largest specification surface is the one duvet handles worst.** `net` is the biggest crate + and its obligations are concentrated in RFC 8200, RFC 791, RFC 792 and RFC 1812 -- the + lowercase-normative, foundational documents. Extending compliance tracking to the whole codebase + is therefore mostly a _synthesis_ problem, not a configuration problem, and synthesis is the part + of this method with a hazard attached. +- **The cheap wins are still in NAT.** RFC 5508 is duvet-friendly, on-topic, and one config edit + from being tracked. +- **Most crates have no external specification at all.** `lpm`, `acl`, `flow-filter`, `config`, + `mgmt` and roughly thirty others implement internal semantics. duvet has nothing to say about + them; bolero and cargo-mutants have everything to say. Whether to synthesize in-repo + specifications for them is deliberately still open -- see the scoping question below. + ## Open questions Expected to expand. Nothing here is scheduled. -1. **Which RFCs apply to us at all.** Prior to everything else, and never yet enumerated. RFC 4787 - (59 requirements), RFC 5508 (92), RFC 6888 (41) and RFC 7857 (29) are duvet-friendly, directly - on-topic and untracked -- 221 requirements one config edit away, no synthesis needed. +1. ~~**Which RFCs apply to us at all.**~~ A first pass is above. What remains is the harder half: + the specifications that constrain us and that nothing in the tree mentions. 2. **Which of those are not RFC 2119 conforming**, and so need synthesis per the section above. -3. **Is a citation true?** duvet checks that a `type=test` citation _exists_, not that the test - exercises the requirement. This is the vacuity problem that the llvm-cov execution counters - caught twice. The cross-check uses artifacts we already produce: mutate the region cited - `type=implementation` and see whether the test cited `type=test` fails. If it does not, the - citation is decorative. This is the only item on this list that makes the three tools check each - other rather than merely coexist. + RFC 8200 is the known case and the most consequential one. +3. ~~**Is a citation true?**~~ Built; see [the interlock](#is-a-citation-true-the-interlock). What + remains is deciding whether it becomes a gate. It is far too slow to run per pull request over + everything -- six minutes for seven requirements -- but `--only` on a changed citation is cheap. 4. **Errata.** Mostly settled; see [Errata](#errata) below. What remains is the 847 RFCs whose errata exist but are not Verified, for which the corpus holds no body. None of them is tracked today; RFC 4443 and RFC 3022 are both in that set and both plausible future targets. @@ -211,8 +308,9 @@ Expected to expand. Nothing here is scheduled. requirements into the snapshot in one commit; QUIC would add 522. Without a rule for scoping _within_ a specification the report becomes wallpaper on the day it gets interesting -- the same lesson as "do not test printers" and "classify, do not eliminate". -8. **Should the snapshot be a blocking gate?** Unlike cargo-mutants it can be: `duvet report` takes - 4ms and is bit-for-bit deterministic. It would be the cheapest correctness gate we have. +8. ~~**Should the snapshot be a blocking gate?**~~ Yes, and it is: `just duvet-check`. The snapshot + had drifted two commits after being introduced, which settled the argument -- a + regenerate-by-hand rule is one nobody runs. What remains is wiring it into CI. 9. **A coverage report analogous to the existing ones**, so that specification coverage is read the same way as line and mutant coverage. From bbf5690ef29a2c3d4408ef185bbe12534e0d4a35 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Thu, 20 Aug 2026 15:59:03 -0600 Subject: [PATCH 31/37] docs(testing): Record where a citation goes when the code is abstract Measured, because the worry that this method penalises delegation, generics and macros is reasonable and was half-right. Macros are the real blind spot: cargo-mutants emits nothing for macro-generated code, so a normative comparison written inside a macro body cannot be checked. Generics and trait defaults cost nothing. The REQ-3 citation was diagnosed here earlier as a delegation problem; the wrong region and the unviable mutants turn out to have separate causes. (cherry picked from commit 6236cb239fd570ee771ca6a7c307368315255b0f) --- development/code/spec-compliance.md | 69 ++++++++++++++++++++++++++++- 1 file changed, 68 insertions(+), 1 deletion(-) diff --git a/development/code/spec-compliance.md b/development/code/spec-compliance.md index 02594f4f96..d8b4eebc75 100644 --- a/development/code/spec-compliance.md +++ b/development/code/spec-compliance.md @@ -134,11 +134,78 @@ the `protocol.rs` failure mode again: a test that walks a path rather than discr **RFC 4787 REQ-3 and RFC 5382 REQ-7, "no port overloading" -- not checkable as cited.** Every mutant of the cited region is unviable. The citation sits on `allocate_v4`, which only forwards to `allocate_from_tables`; the code that enforces the requirement is the one it delegates to. -The citation names the wrong region, and no amount of testing would have revealed that. +The citation names the wrong region, and no amount of testing would have revealed that. See +[Where a citation goes](#where-a-citation-goes-and-what-that-costs-an-abstraction) -- the wrong +region and the unviable mutants have separate causes, and only the first is a citation error. That last category is the one to keep in mind when extending this. A citation can be wrong about _where_ the requirement lives, and neither duvet nor a coverage report can see it. +## Where a citation goes, and what that costs an abstraction + +The rule is one sentence: **cite the narrowest region whose mutation would violate the +requirement.** + +It is worth stating because the obvious alternative -- cite the function whose name matches the +requirement -- is what produced the RFC 4787 REQ-3 result above, and because there is a live +worry that a citation model tied to source regions will end up penalising delegation, generics +and macros. Measured, it mostly does not, and where it does the constraint is narrow. + +### Generics and traits: no cost + +Trait default bodies and generic functions mutate normally, and one citation on them covers +every instantiation. `net/src/checksum.rs` is the worked example: the `Checksum` trait's provided +methods carry the RFC 1624 incremental-update arithmetic, and the mutants land exactly on it +(`delete !`, `replace >> with <<`). Its low mutant density -- 3.1 per 100 lines against 12.0 for +`ipv4/mod.rs` -- is signatures without bodies, not logic that got away. + +This is an argument _for_ abstraction. A requirement implemented once behind a generic needs one +citation and one test. The RFC 4884 finding above is what the alternative costs: lines 259 and +290 of `embedded.rs` are the same check written twice, once for ICMPv4 and once for ICMPv6, and +the v6 copy went untested and uncaught. Duplication is what hid it. + +### Delegation: cite the delegate, and mind the return type + +Two separate things, which the REQ-3 result ran together. + +A thin forwarding function is the wrong place to cite because it decides nothing -- the +requirement lives in what it forwards to. That is the real error in the REQ-3 citation: +`allocate_v4` forwards to `allocate_from_tables`, which is shared by v4 and v6 and is where one +citation would cover both. + +Its mutants being _unviable_ is a different problem with a different cause. cargo-mutants +replaces a function body with a synthesised return value, so whether it can mutate a function at +all depends on how hard that type is to fabricate. +`Result>, AllocatorError>` defeats it; a delegating +function returning `bool` mutates fine. So an unviable region is not evidence of over-abstraction +-- it is evidence that nothing there was checkable, whatever the reason. + +### Macros: a real blind spot, and the one rule worth keeping + +cargo-mutants generates **nothing** for macro-generated code. Probed directly: + +```rust +macro_rules! bounded { ($name:ident, $min:expr) => { + pub fn $name(len: usize) -> bool { len >= $min } +}; } +bounded!(at_least_128, 128); // zero mutants -- the boundary cannot be broken +pub fn concrete(len: usize) -> bool { len >= 128 } // three mutants +``` + +The `>=` in the macro is exactly the boundary class that RFC 4884 got wrong, and it is +unreachable by mutation and therefore by the interlock. + +That is narrower than "macros are a problem". In this tree the separation already holds: the +protocol logic a specification constrains is written directly and mutates well -- `tcp/mod.rs` +15.3 mutants per 100 lines, `icmp6/mod.rs` 14.6, `ipv4/mod.rs` 12.0 -- while the macro-heavy +files are the combinator and accessor layer, `headers/view.rs` at 1.5 and `headers/pat.rs` at +2.1, which no RFC has an opinion about. No requirement cited today sits in a macro body. + +So the rule is not "avoid macros". It is: **do not put a normative decision inside a macro +body.** Generate the plumbing; write the comparison the specification names. If that is ever too +expensive, the fallback is the `contract::` pattern -- lift the decision into a `Requirement` +predicate the macro calls, which is ordinary mutable code with a citation on it. + ## Do not cite a composite BCP The worst failure found, because it exits 0 and reports a plausible number. From d75aaa7b35526f10d408ce66e903d6f49438d480 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Fri, 21 Aug 2026 11:40:55 -0600 Subject: [PATCH 32/37] test(net): Take the RFC 4884 minimum from both sides, in both families The citation claimed more than the test checked, twice over. Refusing 124 does not state "at least 128" -- a check written `<=` refuses a conforming 128-octet field and passes that test -- and the minimum is implemented once per address family, so the ICMPv6 copy had no test at all. Found by `just spec-interlock`, which flips this requirement from decorative to held: 4 surviving mutants to 6 caught. (cherry picked from commit 187a8d55b46d1126635d4754d27ee40401442d19) --- net/src/headers/embedded.rs | 77 +++++++++++++++++++++++++++++++++---- 1 file changed, 70 insertions(+), 7 deletions(-) diff --git a/net/src/headers/embedded.rs b/net/src/headers/embedded.rs index 81492687e6..4bf5383421 100644 --- a/net/src/headers/embedded.rs +++ b/net/src/headers/embedded.rs @@ -1003,6 +1003,33 @@ mod tests { buf } + // Create IPv6 + full TCP header + 60 bytes payload. + // + // 120 octets in total, matching `create_full_ipv4_tcp_packet_with_payload`, so that the two + // families can be driven through the same length cases. The IPv6 header is 40 octets against + // IPv4's 20, so the payload is 60 rather than 80 to keep the totals equal. + fn create_full_ipv6_tcp_packet_with_payload() -> Vec { + let ipv6_header = Ipv6Header { + traffic_class: 0, + flow_label: 0.try_into().unwrap(), + payload_length: 80, // 20 bytes TCP + 60 bytes payload + next_header: IpNumber::TCP, + hop_limit: 64, + source: [0x20, 0x01, 0x0d, 0xb8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], + destination: [0x20, 0x01, 0x0d, 0xb8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2], + }; + + let mut buf = Vec::new(); + ipv6_header.write(&mut buf).unwrap(); + + let tcp_header = etherparse::TcpHeader::new(80, 443, 1000, 0); + tcp_header.write(&mut buf).unwrap(); + + buf.extend_from_slice(&[1u8; 60]); + + buf + } + // Basic parsing, deparsing checks #[test] @@ -1361,6 +1388,23 @@ mod tests { (headers, consumed.get() as usize, buf) } + /// As `v4_with_field_of`, for `ICMPv6`. + /// + /// `check_full_payload` implements the 128-octet minimum twice, once per address family, and + /// the two copies are not reachable by the same fixture. Without this one the ICMPv6 arm has + /// no test at all. + fn v6_with_field_of(field_len: usize, padding_byte: u8) -> (EmbeddedHeaders, usize, Vec) { + let mut buf = create_full_ipv6_tcp_packet_with_payload(); + assert_eq!(buf.len(), 120, "the embedded packet is 120 octets"); + buf.extend(std::iter::repeat_n(padding_byte, field_len - buf.len())); + assert_eq!(buf.len(), field_len); + // Extension structure, which is not part of the field. + buf.extend_from_slice(&[0x55u8; 32]); + let (headers, consumed) = + EmbeddedHeaders::parse_with(EmbeddedIpVersion::Ipv6, &buf).unwrap(); + (headers, consumed.get() as usize, buf) + } + /// A field is accepted at any 32-bit-aligned length, not only at multiples of 32 octets. /// /// The length attribute counts 32-bit words, so every value it can express is already aligned. @@ -1386,23 +1430,42 @@ mod tests { } } - /// A field shorter than 128 octets is refused. + /// The 128-octet minimum is exact, and it holds for both address families. /// /// This is the requirement the octet/bit confusion displaced: the old check let a 32-octet /// field through and rejected a 132-octet one, which is exactly backwards. + /// + /// "At least 128" is a boundary, so refusing 124 does not state it -- that is equally true of + /// a check written `<=`, which would refuse a conforming 128-octet field. The requirement is + /// only pinned by taking the boundary from both sides. Both families are driven because + /// `check_full_payload` implements the minimum once per family and carries this citation on + /// each; testing one leaves the other's copy free to say anything. //= https://www.rfc-editor.org/rfc/rfc4884#section-3 //= type=test //# When the ICMP Extension Structure is appended to an ICMP message //# and that ICMP message contains an "original datagram" field, the //# "original datagram" field MUST contain at least 128 octets. #[test] - fn a_field_shorter_than_128_octets_is_refused() { - for field_len in [120usize, 124] { - let (mut headers, consumed, buf) = v4_with_field_of(field_len, 0); - headers.check_full_payload(&buf, buf.len(), consumed, field_len); + fn the_128_octet_minimum_is_exact() { + type Fixture = fn(usize, u8) -> (EmbeddedHeaders, usize, Vec); + for (family, build) in [ + ("ICMPv4", v4_with_field_of as Fixture), + ("ICMPv6", v6_with_field_of as Fixture), + ] { + for field_len in [120usize, 124] { + let (mut headers, consumed, buf) = build(field_len, 0); + headers.check_full_payload(&buf, buf.len(), consumed, field_len); + assert!( + !headers.is_full_payload(), + "{family}: a {field_len}-octet field is below the 128-octet minimum" + ); + } + + let (mut headers, consumed, buf) = build(128, 0); + headers.check_full_payload(&buf, buf.len(), consumed, 128); assert!( - !headers.is_full_payload(), - "a {field_len}-octet field is below the 128-octet minimum" + headers.is_full_payload(), + "{family}: 128 octets is the minimum, so a 128-octet field must be accepted" ); } } From 56ced014801fc90b545c8ff89dc17085acf15ec5 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Fri, 21 Aug 2026 11:55:19 -0600 Subject: [PATCH 33/37] test(nat): Cite port overloading on the code that could commit it REQ-3 and REQ-7 sat on `allocate_v4`, which forwards to `allocate_from_tables` and decides nothing. Every mutant of it was unviable, so the interlock could not check the citation at all -- and a citation that nothing can break is not a claim. Moving it to the bitmap makes the claim checkable, and it immediately fails: of fourteen mutants the cited property catches four. The seven in the second-half path are unreached because no test exhausts 128 ports from one block, and one of those -- `|=` to `&=` -- is port overloading itself. The remaining three divert allocation to the second half but still yield unique ports, so they do not bear on this requirement. Recorded as a decorative citation rather than fixed here: closing it needs the property to drive a half-block dry, which is a change to what the generator produces, not to what it asserts. (cherry picked from commit 24b5b770a57f0d5e82607bb741e6e9eb613fee74) --- nat/src/masquerade/apalloc/mod.rs | 14 +++++--------- nat/src/masquerade/apalloc/port_alloc.rs | 16 ++++++++++++++++ 2 files changed, 21 insertions(+), 9 deletions(-) diff --git a/nat/src/masquerade/apalloc/mod.rs b/nat/src/masquerade/apalloc/mod.rs index 4e29f76652..61eef74b60 100644 --- a/nat/src/masquerade/apalloc/mod.rs +++ b/nat/src/masquerade/apalloc/mod.rs @@ -322,16 +322,12 @@ impl NatAllocator { // The half that *is* held is REQ-2, cited in `alloc.rs`: the public address is stable across // destinations even though the port is not. That is the partial-conformance case in its // clearest form -- one requirement met, its neighbour missed, by the same two lines of code. - //= https://www.rfc-editor.org/rfc/rfc5382#section-8 - //# REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port - //# overloading" for TCP. - // - //= https://www.rfc-editor.org/rfc/rfc4787#section-4.2.1 - //# REQ-3: A NAT MUST NOT have a "Port assignment" behavior of "Port - //# overloading". // - // No live allocation is ever handed out twice; the port bitmaps below are what enforce it. - // The allocator is protocol-agnostic, so the UDP and TCP requirements are one implementation. + // RFC 4787 REQ-3 and RFC 5382 REQ-7 -- "MUST NOT have a Port assignment behavior of Port + // overloading" -- were cited here, and now sit on `Bitmap256::allocate_port_from_bitmap`, + // which is what would have to hand a port out twice for either to break. This function only + // forwards to `allocate_from_tables`; there is nothing here to get wrong, which is why + // `just spec-interlock` could not check the citation while it was here. fn allocate_v4( &self, src_vpcd: VpcDiscriminant, diff --git a/nat/src/masquerade/apalloc/port_alloc.rs b/nat/src/masquerade/apalloc/port_alloc.rs index 53fc237169..87b08a260f 100644 --- a/nat/src/masquerade/apalloc/port_alloc.rs +++ b/nat/src/masquerade/apalloc/port_alloc.rs @@ -848,6 +848,22 @@ impl Bitmap256 { // // In the last example above, we have three trailing ones in the first half, telling us that // port at 1 << 3 (port number 3) is free. + //= https://www.rfc-editor.org/rfc/rfc5382#section-8 + //# REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port + //# overloading" for TCP. + // + //= https://www.rfc-editor.org/rfc/rfc4787#section-4.2.1 + //# REQ-3: A NAT MUST NOT have a "Port assignment" behavior of "Port + //# overloading". + // + // Port overloading is handing the same public port to two live flows, and this is the + // function that would have to do it. Every port the allocator returns is claimed here, by + // the bit set below, and nothing else marks one used -- so this is the narrowest region + // whose mutation would violate either requirement. Both were cited on `allocate_v4` until + // the interlock found nothing there it could break. + // + // The allocator is protocol-agnostic, so the UDP and TCP requirements are one + // implementation, and one test discharges both. fn allocate_port_from_bitmap(&mut self) -> Result { #[allow(clippy::cast_possible_truncation)] // max value is 128 let ones = self.first_half.trailing_ones() as u16; From 1799836b96d132b9ae6c3e1efeafb0e02bc0c10e Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Fri, 21 Aug 2026 13:16:23 -0600 Subject: [PATCH 34/37] test(nat): Cite the exhaustion walk alongside the stage property One requirement, two tests, because they reach different code. The stage property states port overloading where it is observable -- two flows, one reply path -- but it draws a handful of ports, so it never fills a 256-port block and never enters the second half of the bitmap. Walking a region dry does. Nine of the ten mutants the interlock reported now die, including the one that replaced the bit marking a port used. The test is unchanged: it already asserted this. Only the citation was incomplete, which is a failure mode duvet cannot see -- a requirement can be fully tested and still name the wrong test. (cherry picked from commit ad57787181301cf4bca56d6504855735960c86af) --- nat/src/masquerade/apalloc/pool_fuzz.rs | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/nat/src/masquerade/apalloc/pool_fuzz.rs b/nat/src/masquerade/apalloc/pool_fuzz.rs index 9f7ee7cc08..c4992414c6 100644 --- a/nat/src/masquerade/apalloc/pool_fuzz.rs +++ b/nat/src/masquerade/apalloc/pool_fuzz.rs @@ -259,6 +259,26 @@ fn re_reservation_after_a_config_change_is_honoured() { }); } +/// Exhausting a region hands out every port exactly once. +/// +/// This carries the port-overloading citations as well as +/// `distinct_flows_do_not_share_a_translation`, because the two reach different code and neither +/// is sufficient alone. The stage-level property states the claim where it is observable -- two +/// flows, one reply path -- but it draws a handful of ports, so it never exhausts a 256-port +/// block and never reaches the second half of [`Bitmap256`]. Walking a region dry does, and +/// `seen` is what turns "handed out twice" into a failure. +/// +/// The interlock is what made the gap visible: with only the stage property cited, seven mutants +/// in the bitmap's second half survived, one of them replacing the bit that marks a port used -- +/// port overloading itself. See `development/code/spec-compliance.md`. +//= https://www.rfc-editor.org/rfc/rfc5382#section-8 +//= type=test +//# REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port +//# overloading" for TCP. +//= https://www.rfc-editor.org/rfc/rfc4787#section-4.2.1 +//= type=test +//# REQ-3: A NAT MUST NOT have a "Port assignment" behavior of "Port +//# overloading". #[test] #[cfg_attr(miri, ignore = "exhaustive allocator walk is too slow under miri")] fn a_region_can_be_allocated_dry() { From 88b7b6413c8cec6a4867a02148caf0feb58c7988 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Fri, 21 Aug 2026 13:24:32 -0600 Subject: [PATCH 35/37] docs(testing): Record the three tiers, and the two ways a citation can lie The interlock is no longer one check but an ordered three: coverage, then mutation, then a person. The order is the point -- coverage can fail a citation outright without building a mutant, which is what makes the expensive tier affordable. What the RFC 4787 REQ-3 case taught is worth more than the finding. A citation can sit on the wrong code, and it can sit on the right code while naming the wrong test -- the requirement was already fully tested by a test nobody had cited. duvet cannot see either, and only `no-mutants` catches the first, which is why it is not a pass. (cherry picked from commit 433253491396e3d666b73262856309a6c57957a7) --- development/code/README.md | 5 +- development/code/mutation-testing.md | 22 +++ development/code/spec-compliance.md | 204 ++++++++++++++++++--------- 3 files changed, 166 insertions(+), 65 deletions(-) diff --git a/development/code/README.md b/development/code/README.md index 82b4e54033..625fccbca1 100644 --- a/development/code/README.md +++ b/development/code/README.md @@ -15,7 +15,10 @@ If you need to write a test, prefer [property-based tests] over simple unit test To find out what your tests are _not_ saying, see the [mutation testing note][mutants]; it is a report we read, not a gate that blocks anything. To find out whether they are saying what the specification asked for, see the -[specification compliance note][duvet]. +[specification compliance note][duvet]. If you write a `//=` citation, `just spec-interlock` is +what decides whether it is a claim or a comment: it mutates the code you cited and runs the test +you cited, and cites nothing on your behalf. Put the citation on the narrowest code whose +mutation would violate the requirement, not on the function whose name matches it. For inputs too large to generate directly -- a whole configuration, say -- build them from an algebra of valid operations and derive the oracles from that same algebra; see the [config algebra note][config-algebra]. If you need to handle errors, prefer `Result` types over panics in general, but see the diff --git a/development/code/mutation-testing.md b/development/code/mutation-testing.md index 3197b22a06..f2a1ccd691 100644 --- a/development/code/mutation-testing.md +++ b/development/code/mutation-testing.md @@ -98,6 +98,28 @@ Two consequences: assuming it holds only what was asked for will over-report. `scripts/spec-interlock.ts` filters the result set itself for this reason rather than trusting the flags. +## Classifying, in practice + +The release gate below asks for every mutant to be classified rather than killed. Two worked +examples exist, both found by the [citation interlock](./spec-compliance.md) and both recorded in +its `ACCEPTED` list with reasons: + +- `IpAllocator::allocate`, the exhaustion guard. Equivalent because the function it guards can only + return one error from a well-formed pool. +- `Bitmap256::allocate_port_from_bitmap`, the second-half bound. Equivalent because the caller only + enters the bitmap when the block is not full, which excludes the one value the two operators + disagree on. + +Both were settled the same way, and it is worth copying: **read the code for the invariant, then +apply the mutant by hand and run the whole crate's suite.** A passing suite is not proof of +equivalence, but a failing one is proof against it, and it costs one command. Both of these +also had a symmetric sibling that _was_ caught -- the first-half bound, the non-exhaustion arm -- +and that asymmetry is usually the clearest evidence that an invariant rather than a gap is doing +the work. + +Two of eleven survivors across that exercise were equivalent. A procedure that only knows how to +demand more tests would have produced two entrenching tests instead. + ## Cost Measured on `dataplane-net`: 2,271 mutants, about 4.5s to build and 7.6s to test each, four at a diff --git a/development/code/spec-compliance.md b/development/code/spec-compliance.md index d8b4eebc75..55811fb536 100644 --- a/development/code/spec-compliance.md +++ b/development/code/spec-compliance.md @@ -1,7 +1,7 @@ # Specification compliance with duvet Status: **three specifications tracked, the RFC corpus and its errata audited, the citation -interlock built and returning findings.** +interlock built; every cited requirement now holds.** The open-questions list below is expected to grow; it is written down so that it grows in one place rather than in four people's heads. @@ -24,11 +24,11 @@ specification, and nothing inside the suite can notice. Two specifications, roughly one afternoon. -| | outcome | -| --- | --- | +| | outcome | +| -------------------------- | ------------------------------------------------------- | | RFC 4884 length validation | **real defect**, fixed in `net/src/headers/embedded.rs` | -| RFC 5382 REQ-7, REQ-10 | already held, already tested, never named | -| RFC 5382 REQ-5, REQ-1 | conformance gaps, recorded as `todo` | +| RFC 5382 REQ-7, REQ-10 | already held, already tested, never named | +| RFC 5382 REQ-5, REQ-1 | conformance gaps, recorded as `todo` | The defect is the strongest argument for the tool: `is_full_payload()` checked the RFC 4884 length attribute in **bits** where the RFC counts 32-bit **words**, so it rejected seven of eight @@ -59,10 +59,10 @@ Latin-1 bytes. The only ones a networking project might want are RFC 1305 (NTPv3 **It is blind to lowercase normative language.** This is the important one: -| | RFC 2119 keywords | lowercase must/should | cites RFC 2119 | -| --- | --- | --- | --- | -| RFC 8200 (IPv6, STD 86) | 0 | 79 | no | -| RFC 3022 (traditional NAT) | 0 | 24 | no | +| | RFC 2119 keywords | lowercase must/should | cites RFC 2119 | +| -------------------------- | ----------------- | --------------------- | -------------- | +| RFC 8200 (IPv6, STD 86) | 0 | 79 | no | +| RFC 3022 (traditional NAT) | 0 | 24 | no | RFC 8200 says "It must obey the protocol requirements for routers when receiving (forwarding) interfaces." That is a real obligation with no uppercase token to key on. duvet is not @@ -94,18 +94,36 @@ requirement which the test that claims to check it does not notice. The unit is a (requirement, implementation, test) triple, not a file. `cargo mutants -f ` answers a weaker question -- "is this file tested" -- and buries the signal: the RFC 4884 finding -below is four mutants among the 125 that `embedded.rs` generates. +below was four mutants among the 125 that `embedded.rs` generates. -### Outcomes +### Three tiers, cheapest first + +Each catches what the one below it cannot, and running them in this order is what keeps the +expensive one affordable. + +1. **Coverage** -- `cargo llvm-cov` over the cited region, running only the cited tests. Answers + "did the test go there at all". Seconds. +2. **Mutation** -- `cargo mutants` over the same region, same tests. Answers "did it care". Minutes. +3. **Judgement** -- a person. Answers "is this the right sentence, on the right code, and is this + survivor equivalent". Not automatable, and the sections below are what it has to work with. -Four, and the last two matter as much as the first. +Coverage runs first because it can settle the question outright: a cited test that executes **no** +line of the cited region cannot be testing the requirement, and there is no reason to build a +mutant to confirm it. + +It must not become a threshold. A caught mutant was necessarily executed, so coverage adds nothing +wherever mutation already succeeds -- only zero is decisive, and only as an error. Anything above +zero is used to _explain_ a survivor rather than to judge one, which is the split below. + +### Outcomes -| | meaning | -| --- | --- | -| **held** | every mutant in the cited region was caught by the cited tests | -| **decorative** | a mutant survived; the citation does not carry the weight it claims | -| **no-mutants** | the region produced nothing testable -- usually every mutant unviable | -| **stale** | the cited test name matches no test; the citation has rotted | +| | meaning | +| -------------- | ---------------------------------------------------------------------- | +| **held** | every mutant in the cited region was caught, or accepted with a reason | +| **decorative** | a mutant survived unaccounted for | +| **uncovered** | the cited tests execute none of the cited region | +| **no-mutants** | the region produced nothing testable -- usually every mutant unviable | +| **stale** | the cited test name matches no test; the citation has rotted | `no-mutants` is not a pass. Reporting it as one would credit a citation for a check that never ran, which is the failure the tool exists to catch. `stale` exists because it is the tool's own @@ -113,33 +131,88 @@ worst failure mode: a renamed test makes the filter match nothing, nextest exits nothing, every mutant survives, and a citation that is merely out of date is reported as decorative. The test names are checked against `cargo nextest list` before any mutant runs. -### What it found, first time out +The same hazard has now appeared three times -- `stale`, `no-mutants`, and a coverage collection +that failed silently and returned an empty map, which reads as "nothing was executed" and +relabelled every survivor. **A measurement that fails silently reads as a measurement that +succeeded.** Every step here reports its own failure as a distinct outcome for that reason. + +### Why a mutant survived + +A survivor has two possible causes needing opposite fixes, and mutation alone cannot tell them +apart. Coverage of the mutated line does: -Seven requirements carry both an implementation and a test. Three hold. Six minutes. +| | meaning | the fix | +| ------------- | ----------------------------------- | ------------------------------------------------- | +| **unreached** | the cited tests never ran that line | change what the test **feeds** | +| **tolerated** | they ran it and passed anyway | change what it **asserts** -- or it is equivalent | -**RFC 4884, "at least 128 octets" -- decorative, and on the path where the original defect was.** -Four mutants survive `a_field_shorter_than_128_octets_is_refused`: +Splitting the ten survivors on RFC 4787 REQ-3 by hand took longer than the run that found them. -- The test iterates `[120, 124]`. It never tests 128, so `<` versus `<=` at the boundary is - invisible -- the same boundary class the `flow_info.rs` measurement above found. -- The citation is on **both** the ICMPv4 and the ICMPv6 branch, and there is no v6 test helper at - all. All three of the v6 branch's mutants survive. The RFC 4884 fix was applied to both - branches and tested on one. +### Accepted mutants -**RFC 4787 REQ-2, "IP address pooling: Paired" -- decorative.** `replace match guard -e.is_exhaustion() with true` survives. The guard is what separates allocator exhaustion from -every other allocator error, and the cited test never produces a non-exhaustion error. This is -the `protocol.rs` failure mode again: a test that walks a path rather than discriminating one. +Some survivors are equivalent, and the cheapest way to turn one green is to assert whatever the +code already does -- the entrenchment [mutation testing](./mutation-testing.md) warns about, +arrived at from the other direction. `scripts/spec-interlock.ts` therefore carries an `ACCEPTED` +list: a requirement, a mutant, and a reason. -**RFC 4787 REQ-3 and RFC 5382 REQ-7, "no port overloading" -- not checkable as cited.** Every -mutant of the cited region is unviable. The citation sits on `allocate_v4`, which only forwards -to `allocate_from_tables`; the code that enforces the requirement is the one it delegates to. -The citation names the wrong region, and no amount of testing would have revealed that. See -[Where a citation goes](#where-a-citation-goes-and-what-that-costs-an-abstraction) -- the wrong -region and the unviable mutants have separate causes, and only the first is a citation error. +Two rules keep it from becoming a way of not looking. An accept is **printed in full on every +run**, next to the finding it replaced. And an accept that matches no live mutant is a **failure**, +because a stale one reads as a considered judgement while silently covering whatever takes that +name next. + +Both current entries are upstream invariants rather than gaps, and both were settled the same way: +read the code, then apply the mutant by hand and run the whole crate's suite. That is the shape of +the argument to expect. + +### What it found, first time out -That last category is the one to keep in mind when extending this. A citation can be wrong about -_where_ the requirement lives, and neither duvet nor a coverage report can see it. +Seven requirements carried both an implementation and a test. duvet reported all seven green. +**Three actually held.** Closing the other four is recorded below, because what each one turned +out to be is more useful than the count. + +| finding | what it really was | +| --------------------------------- | ---------------------------------------------------- | +| RFC 4884, "at least 128 octets" | a genuine test gap, in two ways | +| RFC 4787 REQ-2, "pooling: Paired" | an equivalent mutant | +| RFC 4787 REQ-3 / RFC 5382 REQ-7 | a citation on the wrong code, then on the wrong test | + +**RFC 4884 -- a real gap, on the path where the original defect was.** The test iterated +`[120, 124]`, so `<` versus `<=` at the boundary was invisible: 128 itself was never tested. +Worse, the citation sat on **both** the ICMPv4 and ICMPv6 branches and there was no v6 fixture at +all, so three of the v6 branch's mutants had nothing to catch them. The RFC 4884 fix had been +applied to both branches and tested on one. Closed by taking the boundary from both sides in both +families. + +**RFC 4787 REQ-2 -- not a defect.** `replace match guard e.is_exhaustion() with true` survives, +and is equivalent: `reuse_allocated_ip` skips `NoFreePort` and loops, so from a well-formed pool +it can only return `NoFreeIp`, which _is_ exhaustion. Accepted with that reasoning. + +**RFC 4787 REQ-3 / RFC 5382 REQ-7 -- wrong twice over, and the most instructive.** The citation +sat on `allocate_v4`, which only forwards; every mutant of it was unviable, so the interlock could +not check the citation at all. Moving it to `Bitmap256::allocate_port_from_bitmap` -- the bit that +marks a port used, and the narrowest thing whose mutation would violate either requirement -- made +it checkable, and it immediately failed with ten survivors. + +Coverage said seven of those were **unreached**: the cited stage-level property draws a handful of +ports, so it never fills a 256-port block and never enters the second half of the bitmap. One of +the seven replaced the bit that marks a port used, which is port overloading itself. + +And the fix was **no new test**. `a_region_can_be_allocated_dry` already walks two address ranges +dry asserting no tuple is handed out twice; it simply was not cited. Adding the citation killed +nine. The tenth was an invariant and is accepted. + +### What that costs to believe + +Three things generalise, and all three are invisible to duvet: + +- **A citation can be on the wrong code.** REQ-3 pointed at a forwarding wrapper for as long as it + existed. `no-mutants` is the only signal that catches this, which is why it is not a pass. +- **A citation can be on the right code and name the wrong test.** The requirement was fully + tested, by a good test, that nobody had cited. duvet cannot see this at all: it checks that a + `type=test` citation exists, not that the test it names is the one doing the work. +- **A requirement can need more than one cited test, at different altitudes.** The stage-level + property states port overloading where it is observable -- two flows, one reply path. The + exhaustion walk reaches the code that would commit it. Neither is sufficient; both are cited. ## Where a citation goes, and what that costs an abstraction @@ -213,10 +286,10 @@ The worst failure found, because it exits 0 and reports a plausible number. 209 of 239 BCP entries in the mirror are symlinks to a single RFC and are harmless. The other 27 are concatenations, and **BCP 127 is one of them**: RFC 4787 + RFC 6888 + RFC 7857 in one file. -| | requirements | -| --- | --- | -| `bcp127.txt` | **42** | -| RFC 4787 + 6888 + 7857, extracted separately | **129** | +| | requirements | +| -------------------------------------------- | ------------ | +| `bcp127.txt` | **42** | +| RFC 4787 + 6888 + 7857, extracted separately | **129** | duvet keys requirements by section anchor, and each member document has its own `section-5`, so the last document in the concatenation wins. RFC 6888 loses all three of its sections; RFC 4787 loses @@ -272,10 +345,10 @@ for "has Verified errata"; the index is authoritative for nothing. What that says about the specifications in play: -| | errata | -| --- | --- | +| | errata | +| ------------------------------------------------ | ------------------ | | RFC 4787, RFC 5382, RFC 5508, RFC 6888, RFC 7857 | none of any status | -| RFC 4884 | one, EID 3 | +| RFC 4884 | one, EID 3 | **EID 3 does not touch us.** It corrects Section 7's description of the ICMP Extension Header checksum from "the one's complement sum of the data structure" to "...of the ICMP Extension @@ -320,21 +393,21 @@ is a `grep` to regenerate. Twenty-eight RFCs, of which three are tracked. -| RFC | crates | note | -| --- | --- | --- | -| 8200 (IPv6) | `net` | **46 mentions, the most in the tree, and duvet cannot parse it** -- 0 uppercase keywords, 79 lowercase. Synthesis or nothing. | -| 4787 (NAT/UDP) | `nat`, `dataplane` | tracked | -| 4884 (ICMP extension) | `nat`, `net` | tracked | -| 7348 (VXLAN) | `net`, `dpdk` | 15 mentions, untracked | -| 4302 (IP AH) | `net` | 11 mentions, untracked | -| 5382 (NAT/TCP) | `nat` | tracked | -| 792, 1812, 1122, 1191 | `net`, `dataplane` | foundational; expect the same lowercase problem as 8200 | -| 9293 (TCP), 2018, 7323, 3168, 6946, 3540, 2675 | `net` | TCP option and fragmentation behaviour | -| 5508 (NAT/ICMP) | `nat`, `net` | duvet-friendly, on-topic, untracked -- 92 requirements | -| 6437, 4861, 6918 | `net` | IPv6 flow label, ND | -| 1624 | `net` | checksum update | -| 7854, 8671 | `routing` | BMP | -| 3339, 9562, 7637 | `config`, `mgmt`, `id`, `dpdk` | formats, not behaviour | +| RFC | crates | note | +| ---------------------------------------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | +| 8200 (IPv6) | `net` | **46 mentions, the most in the tree, and duvet cannot parse it** -- 0 uppercase keywords, 79 lowercase. Synthesis or nothing. | +| 4787 (NAT/UDP) | `nat`, `dataplane` | tracked | +| 4884 (ICMP extension) | `nat`, `net` | tracked | +| 7348 (VXLAN) | `net`, `dpdk` | 15 mentions, untracked | +| 4302 (IP AH) | `net` | 11 mentions, untracked | +| 5382 (NAT/TCP) | `nat` | tracked | +| 792, 1812, 1122, 1191 | `net`, `dataplane` | foundational; expect the same lowercase problem as 8200 | +| 9293 (TCP), 2018, 7323, 3168, 6946, 3540, 2675 | `net` | TCP option and fragmentation behaviour | +| 5508 (NAT/ICMP) | `nat`, `net` | duvet-friendly, on-topic, untracked -- 92 requirements | +| 6437, 4861, 6918 | `net` | IPv6 flow label, ND | +| 1624 | `net` | checksum update | +| 7854, 8671 | `routing` | BMP | +| 3339, 9562, 7637 | `config`, `mgmt`, `id`, `dpdk` | formats, not behaviour | What this changes about the plan: @@ -358,9 +431,12 @@ Expected to expand. Nothing here is scheduled. the specifications that constrain us and that nothing in the tree mentions. 2. **Which of those are not RFC 2119 conforming**, and so need synthesis per the section above. RFC 8200 is the known case and the most consequential one. -3. ~~**Is a citation true?**~~ Built; see [the interlock](#is-a-citation-true-the-interlock). What - remains is deciding whether it becomes a gate. It is far too slow to run per pull request over - everything -- six minutes for seven requirements -- but `--only` on a changed citation is cheap. +3. ~~**Is a citation true?**~~ Built; see [the interlock](#is-a-citation-true-the-interlock). All + seven cited requirements hold, which means the tool currently has nothing to say and its next + real test is the next citation somebody writes. What remains is whether it becomes a gate: + seven minutes for seven requirements is too slow per pull request over everything, but + `--only` on a changed citation is cheap, and `uncovered` is now reachable without building a + single mutant. 4. **Errata.** Mostly settled; see [Errata](#errata) below. What remains is the 847 RFCs whose errata exist but are not Verified, for which the corpus holds no body. None of them is tracked today; RFC 4443 and RFC 3022 are both in that set and both plausible future targets. From fea7d1dcf8b202f410ad157c9f980378ea896ed4 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Fri, 21 Aug 2026 14:12:03 -0600 Subject: [PATCH 36/37] docs(nat): Drop the process narration from the citation comments A comment saying where a citation used to sit, or which tool moved it, is only legible to somebody who was there. What constrains the code is why the citation belongs where it is. (cherry picked from commit bcbab7a839eccfcad0ffec8a3e050c5701093de8) --- nat/src/masquerade/apalloc/mod.rs | 7 +++---- nat/src/masquerade/apalloc/pool_fuzz.rs | 10 +++------- nat/src/masquerade/apalloc/port_alloc.rs | 7 +++---- 3 files changed, 9 insertions(+), 15 deletions(-) diff --git a/nat/src/masquerade/apalloc/mod.rs b/nat/src/masquerade/apalloc/mod.rs index 61eef74b60..67cd726210 100644 --- a/nat/src/masquerade/apalloc/mod.rs +++ b/nat/src/masquerade/apalloc/mod.rs @@ -324,10 +324,9 @@ impl NatAllocator { // clearest form -- one requirement met, its neighbour missed, by the same two lines of code. // // RFC 4787 REQ-3 and RFC 5382 REQ-7 -- "MUST NOT have a Port assignment behavior of Port - // overloading" -- were cited here, and now sit on `Bitmap256::allocate_port_from_bitmap`, - // which is what would have to hand a port out twice for either to break. This function only - // forwards to `allocate_from_tables`; there is nothing here to get wrong, which is why - // `just spec-interlock` could not check the citation while it was here. + // overloading" -- are cited on `Bitmap256::allocate_port_from_bitmap`, which is what would + // have to hand a port out twice for either to break. This function only forwards to + // `allocate_from_tables` and decides nothing either requirement is about. fn allocate_v4( &self, src_vpcd: VpcDiscriminant, diff --git a/nat/src/masquerade/apalloc/pool_fuzz.rs b/nat/src/masquerade/apalloc/pool_fuzz.rs index c4992414c6..bfb962660c 100644 --- a/nat/src/masquerade/apalloc/pool_fuzz.rs +++ b/nat/src/masquerade/apalloc/pool_fuzz.rs @@ -264,13 +264,9 @@ fn re_reservation_after_a_config_change_is_honoured() { /// This carries the port-overloading citations as well as /// `distinct_flows_do_not_share_a_translation`, because the two reach different code and neither /// is sufficient alone. The stage-level property states the claim where it is observable -- two -/// flows, one reply path -- but it draws a handful of ports, so it never exhausts a 256-port -/// block and never reaches the second half of [`Bitmap256`]. Walking a region dry does, and -/// `seen` is what turns "handed out twice" into a failure. -/// -/// The interlock is what made the gap visible: with only the stage property cited, seven mutants -/// in the bitmap's second half survived, one of them replacing the bit that marks a port used -- -/// port overloading itself. See `development/code/spec-compliance.md`. +/// flows, one reply path -- but it draws a handful of ports, so it never fills a 256-port block +/// and never enters the second half of [`Bitmap256`]. Only walking a region dry reaches that +/// half, and `seen` is what turns "handed out twice" into a failure. //= https://www.rfc-editor.org/rfc/rfc5382#section-8 //= type=test //# REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port diff --git a/nat/src/masquerade/apalloc/port_alloc.rs b/nat/src/masquerade/apalloc/port_alloc.rs index 87b08a260f..0e778365c8 100644 --- a/nat/src/masquerade/apalloc/port_alloc.rs +++ b/nat/src/masquerade/apalloc/port_alloc.rs @@ -857,10 +857,9 @@ impl Bitmap256 { //# overloading". // // Port overloading is handing the same public port to two live flows, and this is the - // function that would have to do it. Every port the allocator returns is claimed here, by - // the bit set below, and nothing else marks one used -- so this is the narrowest region - // whose mutation would violate either requirement. Both were cited on `allocate_v4` until - // the interlock found nothing there it could break. + // function that would have to do it: every port the allocator returns is claimed by the bit + // set below, and nothing else marks one used. The citation belongs here rather than on a + // caller because a caller cannot violate the requirement without going through this. // // The allocator is protocol-agnostic, so the UDP and TCP requirements are one // implementation, and one test discharges both. From 762b44a116b3581bdeb93f28c4ba69ce0d582279 Mon Sep 17 00:00:00 2001 From: Daniel Noland Date: Fri, 21 Aug 2026 18:17:06 -0600 Subject: [PATCH 37/37] fix(net): Backtick a doc identifier `clippy --all-targets` fails on this branch without it, which makes it useless as a gate for everything else. Unrelated to the surrounding work. Co-Authored-By: Claude Opus 5 (1M context) (cherry picked from commit 1a797a622acd8a53b231e62b63edb8a8df5e20e8) --- net/src/headers/embedded.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/src/headers/embedded.rs b/net/src/headers/embedded.rs index 4bf5383421..95d6221f03 100644 --- a/net/src/headers/embedded.rs +++ b/net/src/headers/embedded.rs @@ -1391,7 +1391,7 @@ mod tests { /// As `v4_with_field_of`, for `ICMPv6`. /// /// `check_full_payload` implements the 128-octet minimum twice, once per address family, and - /// the two copies are not reachable by the same fixture. Without this one the ICMPv6 arm has + /// the two copies are not reachable by the same fixture. Without this one the `ICMPv6` arm has /// no test at all. fn v6_with_field_of(field_len: usize, padding_byte: u8) -> (EmbeddedHeaders, usize, Vec) { let mut buf = create_full_ipv6_tcp_packet_with_payload();