Skip to content

fix(network/dns): default-config peer-name DNS resolves (CHAOS-1478)#19

Open
chrisgeo wants to merge 5 commits into
mainfrom
feat/chaos-1478-dns-peer-name-fqdn-fix
Open

fix(network/dns): default-config peer-name DNS resolves (CHAOS-1478)#19
chrisgeo wants to merge 5 commits into
mainfrom
feat/chaos-1478-dns-peer-name-fqdn-fix

Conversation

@chrisgeo

@chrisgeo chrisgeo commented May 7, 2026

Copy link
Copy Markdown

Summary

  • nslookup probe-pg from a sibling container now resolves in the default configuration (no dns.domain set, no sudo, no container system dns create) — previously returned NXDOMAIN.
  • The fix is two cooperating layers:
    • Source fix (4 commits, 23ef3d011d7948): register peer hostnames in their bare form and strip the configured dns.domain suffix at the embedded handler so bare and FQDN queries map to the same allocator key.
    • Routing fix (50f4a43): automate the /etc/resolver/ install at .pkg postinstall time and inject the matching search directive into containers, so VM queries actually traverse the chain to 127.0.0.1:2053.
  • AttachmentAllocator gains a normalize helper that strips trailing root-label dots, making bare and FQDN-form keys equivalent at the storage boundary.
  • Utility.getAttachmentConfigurations drops the dns.domain-gated FQDN path and always registers the bare containerId; the dnsDomain parameter is removed.
  • ContainerDNSHandler receives dns.domain via init and strips both the trailing dot and the configured domain suffix from question.name before calling networkService.lookup, so both nslookup foo and nslookup foo.<dns.domain> resolve correctly.
  • DNSConfig.domain defaults to "test" (centralised in defaultDomain); container searchDomains auto-injects [domain] when empty; the .pkg postinstall writes /etc/resolver/containerization.test so the macOS resolver chain forwards *.test queries from vmnet's DNS proxy to the embedded handler.

Why

Today's behavior. Utility.getAttachmentConfigurations gates hostname registration on whether dns.domain is configured. When dns.domain is absent (the default), it stores the bare containerId (e.g., "probe-pg") in AttachmentOptions.hostname. When dns.domain is set, it stores the FQDN form ("probe-pg.containers.local"). Meanwhile, ContainerDNSHandler passes question.name — which the DNS wire format always delivers in canonical FQDN form with a trailing root-label dot ("probe-pg.") — directly to networkService.lookup, which in turn calls AttachmentAllocator.lookup as an exact-string dictionary read. The stored key "probe-pg" never matches the queried key "probe-pg.".

The user-facing failure. Running nslookup probe-pg from a sibling container on the same network returns NXDOMAIN. The container is reachable by IP; the allocation exists in the table. The failure is purely a key-form mismatch. This is the repro documented in CHAOS-1478: two containers on the default network, no dns.domain in ~/.config/container/config.json, bare peer-name lookup fails every time.

Why default-config should just work. Docker's libnetwork registers bare names by default on user-defined networks; nslookup peer resolves without any domain configuration. apple/container has all the architectural pieces — AttachmentAllocator, ContainerDNSHandler, NetworksService.lookup — to do the same. The FQDN gate made the default flow inert: it registered the right key but the handler queried the wrong one. Removing the gate and normalizing at the allocator boundary closes the gap without touching multi-alias plumbing.

Why the source fix alone wasn't enough. Live testing on macOS 26 after the source fix landed showed bare-name resolution still failed. Diagnosis: dig @127.0.0.1 -p 2053 probe-pg. succeeded (the embedded handler did register and answer correctly), but nslookup probe-pg from inside a container still returned NXDOMAIN. Containers send their queries to 192.168.x.1:53 (the vmnet bridge gateway), and vmnet's built-in DNS proxy on :53 forwards unmatched queries to upstream DNS rather than to the embedded handler — so peer names never reached 127.0.0.1:2053 to begin with.

Three architectural options for routing. All evaluated, two ruled out empirically.

Idea Verdict
A Bind embedded handler on 192.168.x.1:53 directly Infeasible. bind('192.168.x.1', 53) from uid 501 returns EACCES. Privileged port, and com.apple.security.virtualization does not grant :53 binding.
B UDP forwarder inside container-network-vmnet plugin Same problem. Plugin runs as the same uid (verified).
C Auto-install /etc/resolver/<domain> at install time Works. .pkg postinstall already runs as root; no new runtime sudo prompt; reuses the existing HostDNSResolver-style file format so system dns create / delete tooling composes cleanly.

(macOS 26 does expose vmnet_network_configuration_disable_dns_proxy() (vmnet.h L1244), which would let us silence vmnet's proxy — but that still leaves the handler unable to claim :53, so it does not unblock A or B.)

The routing chain at runtime, after this PR:

VM glibc/musl     appends ".test" via `search` directive
vmnet :53 proxy   forwards `probe-pg.test` to macOS system resolver
macOS resolver    matches /etc/resolver/containerization.test
                  forwards to 127.0.0.1 port 2053
apiserver :2053   strips ".test" (DNSRegistrationKey), looks up bare
                  "probe-pg" in AttachmentAllocator -> 192.168.x.y

Changes

Source fix (already on this branch)

Routing fix (50f4a43)

  • Sources/ContainerPersistence/ContainerSystemConfig.swiftDNSConfig.defaultDomain = "test" constant. Both the no-arg initialiser and the TOML decode path fall back to it when domain is absent. Explicit nil and explicit "" from user TOML are still preserved (opt-out path).
  • Sources/Services/ContainerAPIService/Client/Utility.swift — when constructing a container's DNSConfiguration and the user has not specified any search domains, inject [domain] so bare-name queries pick up the suffix and traverse the resolver chain. Honours user overrides; explicit empty arrays remain empty.
  • scripts/pkg-scripts/postinstall — new .pkg postinstall script. Writes /etc/resolver/containerization.test (filename / contents matching HostDNSResolver so existing system dns create | delete tooling composes cleanly) and refreshes mDNSResponder. The installer already runs as root; no new sudo prompt.
  • Makefile — pass --scripts scripts/pkg-scripts to pkgbuild so the postinstall is bundled into the installer.
  • scripts/uninstall-container.sh — best-effort removal of the resolver entry on uninstall, keeping /etc/resolver/ state symmetric with install.

Verification

Build

Build complete!

No new warnings introduced. Clean build on Apple silicon, macOS 26, Xcode 26.

Unit tests — 40 / 40 across 4 suites, all passing in 0.002s wall

  • AttachmentAllocatorTest (19 tests) — normalization symmetry across allocate / lookup / deallocate, idempotent re-allocation in canonical form, single-dot scope.
  • ContainerDNSHandlerRegistrationKeyTest (8 tests) — trailing-dot stripping, configured-domain suffix stripping, label-boundary matching, and dnsDomain == nil / empty-string passthrough.
  • UtilityRegistrationTests (5 tests) — bare-form registration for plain, trailing-dot, and dot-internal containerIds; consistent hostname across multiple network attachments.
  • ContainerSystemConfigDNSDefaultTests (8 tests, new in this commit) — DNSConfig default value, explicit overrides, the empty-string opt-out, and the TOML decode paths (missing section, empty section, explicit override).

Live repro on macOS 26 (debug build, this branch)

$ container system stop
$ sudo make all install BUILD_CONFIGURATION=debug
$ container system start --debug

$ container run --rm -d --name probe-pg -e POSTGRES_PASSWORD=test postgres:alpine
probe-pg

$ container list
ID        IMAGE                              OS     ARCH   STATE    ADDR
probe-pg  docker.io/library/postgres:alpine  linux  arm64  running  192.168.65.8/24

$ container run --rm busybox nslookup probe-pg          # the goal
Server:    192.168.65.1
Address:   192.168.65.1:53
Name:      probe-pg.test
Address:   192.168.65.8

$ container run --rm busybox nslookup probe-pg.test     # FQDN form
Name:      probe-pg.test
Address:   192.168.65.8

$ container run --rm alpine nslookup probe-pg           # musl resolver
Name:      probe-pg.test
Address:   192.168.65.8

$ dig @127.0.0.1 -p 2053 probe-pg. +short               # key-form fix intact
192.168.65.8
$ dig @127.0.0.1 -p 2053 probe-pg.test. +short          # suffix-strip works
192.168.65.8

$ dscacheutil -q host -a name probe-pg.test             # /etc/resolver host path
ip_address: 192.168.65.8

$ cat /etc/resolver/containerization.test               # installed by postinstall
domain test
search test
nameserver 127.0.0.1
port 2053

Wire compatibility

No Codable schema change to AttachmentConfiguration, AttachmentOptions, DNSConfig, or any persisted type. DNSConfig.domain remains String? and existing TOMLs decode unchanged — the change is the default fallback when the key is absent. The dropped dnsDomain parameter on Utility.getAttachmentConfigurations is internal to the API server process; it is not part of any public API, XPC message type, or on-disk format. The new APIServerCore library target is purely additive. Existing containers and networks running mid-upgrade are unaffected: AttachmentAllocator.normalize accepts both "foo" and "foo." as equivalent keys, so any entry stored in either form before the upgrade remains findable after it.

Out of scope

  • Multi-alias registration (compose-style aliases, additional names per container) is tracked separately as CHAOS-1476 and is intentionally not in this PR.
  • Upstream apple/container submission is a separate step after validating the prototype on full-chaos/container.
  • No changes to LocalhostDNSHandler — that handler serves the macOS realhost resolver and follows a different code path.
  • macOS 15 backport. The fix targets macOS 26 (container-network-vmnet's reserved-mode variant).

References

@linear

linear Bot commented May 7, 2026

Copy link
Copy Markdown

CHAOS-1478

chrisgeo added 5 commits May 24, 2026 06:18
apple/container's peer DNS returned NXDOMAIN by default because:
- Utility.getAttachmentConfigurations gated the registered hostname on
  dns.domain and stored bare containerId form when unset.
- ContainerDNSHandler passed canonical FQDN form (with trailing dot) to
  AttachmentAllocator.lookup, which did exact-string dictionary reads.
- Mismatch: stored "probe-pg" vs query "probe-pg." → NXDOMAIN.

Fix (belt-and-suspenders, no alias plumbing):
1. AttachmentAllocator normalizes trailing root-label dots in
   allocate/lookup/deallocate so registration is dot-insensitive.
2. Utility always registers the bare containerId form (drops the FQDN
   gate); the dnsDomain parameter on getAttachmentConfigurations is
   removed since registration no longer depends on it.
3. ContainerDNSHandler now receives dns.domain via init and strips
   both the trailing dot and the configured dns.domain suffix from
   question.name before lookup, so both nslookup foo and
   nslookup foo.<dns.domain> resolve to the same allocation.
4. APIServer+Start wires containerSystemConfig.dns.domain into the
   ContainerDNSHandler init.
5. NetworksService.lookup doc comment updated to reflect the new
   contract.

No public schema changes. Multi-alias support (CHAOS-1476) intentionally
out of scope.
…HAOS-1478)

Polish on top of 23ef3d0:
- Update the 'attach the first network using the fqdn' comment to
  describe the new behavior (always-bare registration, allocator-side
  normalization).
- Use registrationHostname uniformly across all network attachments;
  this makes both arms of the previous offset == 0 guard identical, so
  collapse the guard. Behavior was already equivalent thanks to allocator
  normalization, but keeping two arms produced confusing dead code.
- AttachmentAllocator: normalization symmetry across allocate/lookup/
  deallocate, single-dot scope.
- ContainerDNSHandler.registrationKey: trailing-dot stripping, dns.domain
  suffix stripping with boundary-only matching, and dns.domain=nil/empty
  passthrough.
- Utility.getAttachmentConfigurations: bare-form registration for plain,
  trailing-dot, and dot-internal containerIds; consistent hostname across
  multiple network attachments.

Adds APIServerTests target (Package.swift) and widens
ContainerDNSHandler.registrationKey to internal access for @testable
visibility.
…Server lib to APIServerCore (CHAOS-1478)

Follow-up to 19355b7. The previous test commit landed two parallel
implementations of the registrationKey logic:
  - inline 'internal func registrationKey' on ContainerDNSHandler
  - 'DNSRegistrationKey.registrationKey' in the new APIServer library
Tests covered the library copy, but production ran the inline copy —
silent-drift risk if the two ever diverged.

Changes:
- Sources/APIServer/ContainerDNSHandler.swift now imports APIServerCore
  and delegates directly to DNSRegistrationKey.registrationKey from
  answerHost / answerHost6. Inline duplicate removed.
- Library target renamed APIServer -> APIServerCore to avoid the
  module-name / type-name ambiguity with 'struct APIServer' (the
  @main executable's main type at Sources/APIServer/APIServer.swift).
- Sources/APIServerLib -> Sources/APIServerCore (path matches target).
- container-apiserver executable now depends on APIServerCore.
- Test imports updated to '@testable import APIServerCore'.

swift build clean. 32/32 tests pass across the three filtered suites.
…CHAOS-1478)

Completes the CHAOS-1478 fix: ``container run --rm busybox nslookup
<peer>`` now resolves a sibling container in the default configuration,
with no ``sudo``, no ``container system dns create``, and no manual
``dns.domain`` configuration.

The earlier source change (23ef3d0) made the embedded DNS handler
register peer hostnames in their bare form. That fix was necessary but
not sufficient: queries from inside containers still had no path to
reach ``127.0.0.1:2053`` because ``vmnet``'s built-in DNS proxy on
``192.168.x.1:53`` forwards unmatched queries upstream rather than to
the embedded handler.

Three architectural options were evaluated. (A) Bind the handler on
``192.168.x.1:53`` and (B) run a UDP forwarder there from inside the
``container-network-vmnet`` plugin both turned out to be infeasible:
``com.apple.security.virtualization`` does not grant privileged-port
binding, and every helper - apiserver and ``container-network-vmnet``
included - runs as the invoking uid (verified empirically via
``bind('192.168.x.1', 53)`` -> EACCES from uid 501). macOS 26 does
expose ``vmnet_network_configuration_disable_dns_proxy()``, but
disabling the proxy still leaves the handler unable to claim ``:53``.
We therefore fall through to (C), the documented ``/etc/resolver``
mechanism, but automate it at install time so the user never sees a
runtime sudo prompt.

The chain at runtime:

  VM glibc/musl    appends ``.test`` via ``search`` directive
  vmnet :53 proxy  forwards ``probe-pg.test`` to macOS system resolver
  macOS resolver   matches /etc/resolver/containerization.test, routes
                   to 127.0.0.1 port 2053
  apiserver :2053  strips ``.test`` (DNSRegistrationKey), looks up
                   bare ``probe-pg``, returns 192.168.x.y

Changes:

- ``ContainerPersistence/ContainerSystemConfig.swift``: ``DNSConfig``
  now defaults ``domain`` to ``\"test\"`` (centralised in
  ``defaultDomain``). Both the no-arg initialiser and the TOML decode
  path fall back to the default; an explicit nil/empty string from
  config is still honoured. The default propagates to the embedded
  handler's ``dnsDomain`` argument and to per-container DNS plumbing.

- ``ContainerAPIService/Client/Utility.swift``: when constructing a
  container's ``DNSConfiguration`` and the user has not specified any
  search domains, inject ``[domain]`` so bare-name queries actually
  pick up the suffix and traverse the resolver chain above. Honours
  user overrides; explicit empty strings remain empty.

- ``scripts/pkg-scripts/postinstall``: new ``.pkg`` postinstall script
  that writes ``/etc/resolver/containerization.test`` (filename and
  contents matching ``HostDNSResolver`` so existing add/remove tooling
  composes cleanly) and refreshes ``mDNSResponder``. The .pkg installer
  already runs as root, so this introduces no new sudo prompt.

- ``Makefile``: pass ``--scripts scripts/pkg-scripts`` to ``pkgbuild``
  so the postinstall is bundled into the installer.

- ``scripts/uninstall-container.sh``: best-effort cleanup of the
  resolver entry on uninstall, keeping the host's ``/etc/resolver``
  state symmetric with install.

- ``Tests/ContainerPersistenceTests/ContainerSystemConfigDNSDefault\
Tests.swift``: 8 new unit tests covering the default value, explicit
  overrides, the empty-string opt-out, and the TOML decode paths
  (missing section, empty section, explicit override). The 32 tests
  from the earlier CHAOS-1478 commits continue to pass (40 / 40).

Live repro on a clean container restart (debug build, macOS 26):

  $ container system stop && sudo make all install && container system start
  $ container run --rm -d --name probe-pg -e POSTGRES_PASSWORD=test postgres:alpine
  $ container run --rm busybox nslookup probe-pg
  Server:  192.168.65.1
  Address: 192.168.65.1:53
  Name:    probe-pg.test
  Address: 192.168.65.8

  $ dig @127.0.0.1 -p 2053 probe-pg. +short        # key-form fix intact
  192.168.65.8
  $ dig @127.0.0.1 -p 2053 probe-pg.test. +short   # suffix-strip works
  192.168.65.8
  $ dscacheutil -q host -a name probe-pg.test      # /etc/resolver host path
  ip_address: 192.168.65.8

Constraints honoured:

  * No new public Codable schema changes; ``DNSConfig.domain`` remains
    ``String?`` and existing TOMLs decode unchanged.
  * No runtime sudo prompts. Sudo is required only at ``.pkg`` install
    time, which the user already accepts via ``installer -pkg``.
  * The 32 pre-existing CHAOS-1478 unit tests continue to pass.
  * CHAOS-1476 (multi-alias) remains out of scope.
@chrisgeo chrisgeo force-pushed the feat/chaos-1478-dns-peer-name-fqdn-fix branch from 50f4a43 to 359d70d Compare May 24, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant