fix(ha): harden failover runtime and controller - #347
Conversation
…-seed-artifacts # Conflicts: # zig/lib/httpx/src/client/client.zig
…-failover-runtime-hardening
…time-hardening # Conflicts: # zig/pkg/antfly/src/common/http/std_http_listener.zig
|
Codex (GPT-5): This response was written and posted by Codex, not Boris. I re-evaluated the blocking review against final Antfly head The six concrete blockers are addressed:
Authentication is now one explicit fail-closed boundary: the dedicated HA bearer is required and is aligned across the outer listener and inner handler. The real handler path is exercised end to end. The Lease transport remains Zig-native. The OpenSSL C transport, Final deterministic evidence includes operator I agree with these remaining review concerns and am deferring them explicitly rather than claiming closure:
The initial final-head The PR is current with |
| restore: RestoreSource, | ||
| options: RestoreOptions, | ||
| ) !?db_mod.generation_lifecycle.StagedGeneration { | ||
| if (try restoreSnapshotAlreadyApplied(alloc, path, group_id, restore, options)) return null; |
There was a problem hiding this comment.
(not your fault but maybe you could look into) Not sure why this code is in raft/ seems a little bit of a smell? Does it get used for HA?
There was a problem hiding this comment.
nice catch this is a weird + unused merge conflict resolution artifact. i am reverting it to the state of main
| const httpx = @import("httpx"); | ||
| const common = @import("../common/http/http_common.zig"); | ||
|
|
||
| pub const ZigLeaseExecutor = struct { |
There was a problem hiding this comment.
Should this be in this package? Should it just be LeaseExecutor?
There was a problem hiding this comment.
yeah think you're right about the name - going to change it
| //! portable storage artifacts. Keep this module below both layers so decoding | ||
| //! a seed never imports the metadata control loop into storage-only binaries. | ||
|
|
||
| pub const TableRecord = struct { |
There was a problem hiding this comment.
Should this be in this package?
There was a problem hiding this comment.
codex seems to like this particular layout/separation of concerns, what do you make of this?
Codex (GPT-5): This placement is intentional. These topology wire records are consumed by both metadata/table_manager.zig and storage-only HA seed materialization. Defining them under metadata would force the storage layer to import the metadata control-plane dependency tree. Keeping the types in common provides a lower-level shared boundary, while table_manager.zig re-exports them for metadata callers. I think the current package is therefore the right dependency direction.
…time-hardening # Conflicts: # scripts/ci/zig-build-e2e-binaries.sh # zig/build.zig # zig/e2e/antfly/test_standby.py # zig/lib/httpx/src/client/client.zig # zig/pkg/antfly/src/api/http_server.zig # zig/pkg/antfly/src/api/public_table_http.zig # zig/pkg/antfly/src/api/table_writes.zig # zig/pkg/antfly/src/common/http/std_http_executor.zig # zig/pkg/antfly/src/data/runtime.zig # zig/pkg/antfly/src/metadata/table_manager.zig # zig/pkg/antfly/src/raft/storage/catalog.zig # zig/pkg/antfly/src/standalone/runtime.zig # zig/pkg/antfly/src/storage/db/db.zig # zig/pkg/antfly/src/storage/ha/http_admin.zig # zig/pkg/antfly/src/storage/ha/http_internal.zig # zig/pkg/antfly/src/storage/ha/standby.zig
3 participants
Codex (GPT-5): Delivers the Antfly runtime and operator foundation for production-ready hot-standby HA, paired with the Colony control-plane work in https://github.com/antflydb/colony/pull/346.
Current Antfly revision:
7bd93ae4b1e1d98db09731c81b19d3f2f971042c.Production-readiness scope
Swarmresources to theStandaloneruntime without changing StatefulSet, Service, PVC, selector, mount, or storage identity.Safety properties
Dependency-free runtime transport
The Lease transport remains dependency-free Zig through Antfly's existing
httpxpath. It preserves projected Kubernetes CA validation, DNS hostname verification and SNI, service-account bearer authentication, one absolute monotonic request deadline, bounded response handling, and fail-closed watchdog fencing.The checked-in Zig-native TLS compatibility fix handles an optional TLS 1.3
CertificateRequestby sending an empty client certificate while preserving transcript boundaries. The abandoned OpenSSL Lease transport, system links, package additions, and host-glibc workaround are absent.Deterministic evidence
make test.CertificateRequest.git diff --checkpassed.Review and certification boundary
backup_lsn-bound source. Safely shortening that outage requires an immutable storage-engine snapshot/checkpoint handle or equivalent two-phase boundary.zig-basegate passed. The isolatede2e-basererun reproduced only two unrelated timing failures: managed-embedding retry sequencing and automatic shard-split finalization. Neither exercises HA or the changed paths.Colony PR346 pins this exact Antfly gitlink and operator pseudo-version/checksum.