Collect health-check and UDP retry evidence#36
Closed
MaurUppi wants to merge 5 commits into
Closed
Conversation
Issue daeuniverse#1026 needs data that distinguishes exited health-check goroutines from blocked probe loops, and daeuniverse#1029 needs retry-limit warnings that carry enough context to confirm or reject selection churn. This adds atomic health-check diagnostics, Prometheus gauges, warning-level UDP retry evidence, and read-only Python collectors without changing the selected repair path for B-1/B-3. Constraint: Full eBPF/control-plane tests require a Linux runtime and currently hit generated/stub build-tag conflicts locally. Rejected: Implement self-healing or retry selection behavior in this batch | the design requires evidence collection before choosing those behavior changes. Confidence: medium Scope-risk: moderate Directive: Do not treat these metrics as a root-cause fix; use collected H1/H2/H3 evidence before changing health-check lifecycle behavior. Tested: python3 -m unittest scripts/test_issue_1026_evidence_scripts.py Tested: GOWORK=off GOCACHE=/tmp/dae-healthcheck-gocache go test control/addr.go control/udp_retry_diagnostics.go control/udp_retry_diagnostics_test.go -count=1 Tested: GOWORK=off GOOS=linux GOARCH=amd64 GOCACHE=/tmp/dae-healthcheck-gocache go test -exec /usr/bin/true ./component/outbound/dialer -count=1 Not-tested: Full ./control and ./pkg/metrics runtime tests; local environment reports bpf_stub.go and bpf_bpfel.go redeclarations plus Linux-only eBPF/runtime constraints.
DNS Benchmark Compare
Suite Status
control_dns_cacheDNS Benchmark Compare (control_dns_cache)
Failure Log (tail)component_upstream_hotpathDNS Benchmark Compare
benchstat |
added 4 commits
June 24, 2026 18:32
Audit P2 found that attempted_dialers was appended once per buffered payload, which mixed packet count with dialer-attempt count and could create false consecutive duplicates. This keeps non-consecutive repeats for churn detection while compressing only consecutive duplicates from the same selected endpoint. Constraint: Fix stays inside B-2 observability; retry selection, thresholds, and endpoint behavior remain unchanged. Rejected: Fully de-duplicate the slice | non-consecutive repeats such as dead-a, dead-b, dead-a are the B-1 signal we need to preserve. Confidence: high Scope-risk: narrow Tested: GOWORK=off GOCACHE=/tmp/dae-healthcheck-gocache go test control/addr.go control/udp_retry_diagnostics.go control/udp_retry_diagnostics_test.go -count=1 Tested: python3 -m unittest scripts/test_issue_1026_evidence_scripts.py Tested: GOWORK=off GOOS=linux GOARCH=amd64 GOCACHE=/tmp/dae-healthcheck-gocache go test -exec /usr/bin/true ./component/outbound/dialer -count=1 Not-tested: Full Linux runtime/eBPF integration; PR CI will rerun after push.
The UDP retry evidence helper should be runnable directly on a target host without first exporting a log file. The CLI now reads dae.service through journalctl by default and treats a bare timestamp argument as the journal --since value, while retaining explicit --file support for saved logs and tests. Constraint: Keep subprocess invocation shell-free and pinned to dae.service for this evidence script. Rejected: Keep positional logfile as the default | the requested operator flow is journal-first with bare time input. Confidence: high Scope-risk: narrow Tested: python3 -m unittest scripts/test_issue_1026_evidence_scripts.py Tested: python3 scripts/udp_retry_log_correlate.py --help Tested: manual --file smoke test with one synthetic retry-limit JSON line Not-tested: Live journalctl on production dae.service; user will run the script manually after deployment.
A proxy WriteTo may report the on-wire byte count including protocol encapsulation overhead (shadowsocks AES-128-GCM + IPv4 = +39 bytes), so n can legitimately exceed len(b). UDP datagrams are atomic; only n < len(b) is a genuine short write. The old n != len(b) misread every over-count as a short write, retiring healthy endpoints and exhausting the retry limit. Evidence: 10655/10655 captured drops were exactly +39 over-writes with retry exhausted and zero genuine under-writes.
healthcheck_metrics_snapshot.py: replace the wrong N×CI stall threshold with two independent detectors -- BLOCK-A (loop frozen: inflight==0 + loopAdvancedAge > CI + probe-phase ceiling) and BLOCK-B (hung probe, phase-relative, marked untested pending a real inflight>0 capture). Add raw per-dialer time-series output (--raw-output) so single-vs-monotonic is self-answerable. Replay: old N×CI flagged 3356/3356 healthy cycles; calibrated BLOCK-A flags 0/3356. udp_retry_log_correlate.py: parse enriched warning JSON, journald short-iso timestamps. Tests 8/8.
Owner
Author
|
Superseded. Root causes for both daeuniverse#1026 and daeuniverse#1029 are now identified:
This instrumentation branch ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Review notes
UnregisterAliveDialerSetwas intentionally reshaped to keep the refcount mirror symmetric without creating transient negative map entries; the observable registration lifecycle remains equivalent for valid register/unregister pairs.Validation
Not tested