-
Notifications
You must be signed in to change notification settings - Fork 84
perf: benchmark infrastructure, O(N²)→O(N) ephemeral key optimisation, and CI regression gate #3953
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: test/audit-coverage
Are you sure you want to change the base?
Changes from all commits
a418ba9
e41bb5a
80c31b8
46755d2
3cde846
1c103ac
a4075b6
62c8be5
4927cac
ee7bd17
33fdd38
f755e60
411cec8
d7a0d11
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -145,6 +145,19 @@ jobs: | |
| path: coverage/coverage.out | ||
| if-no-files-found: warn | ||
|
|
||
| - name: Check coverage gate | ||
| run: | | ||
| docker run --rm \ | ||
| -v "${{ github.workspace }}/coverage:/coverage" \ | ||
| go-build-env \ | ||
| go tool cover -func /coverage/coverage.out > /tmp/cover-func.txt | ||
| TOTAL=$(grep '^total:' /tmp/cover-func.txt | awk '{print $3}' | tr -d '%') | ||
| echo "Total coverage: ${TOTAL}%" | ||
| PASS=$(awk -v t="$TOTAL" 'BEGIN { print (t+0 >= 14) ? "yes" : "no" }') | ||
| if [ "$PASS" != "yes" ]; then | ||
| echo "::error::Coverage ${TOTAL}% is below the 14% minimum threshold" | ||
| exit 1 | ||
| fi | ||
|
|
||
| - name: Build Docker Runtime Image | ||
| if: github.event_name != 'workflow_dispatch' | ||
|
|
@@ -311,6 +324,85 @@ jobs: | |
| install-go: false | ||
| checks: "-SA1019" | ||
|
|
||
| client-bench: | ||
| needs: [client-build-test-publish] | ||
| if: github.event_name == 'push' && github.ref == 'refs/heads/main' | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| actions: read | ||
| steps: | ||
| - name: Set up Docker Buildx | ||
| uses: docker/setup-buildx-action@v3 | ||
|
|
||
| - name: Set up Go | ||
| uses: actions/setup-go@v5 | ||
| with: | ||
| go-version: "1.24" | ||
| cache: false | ||
|
|
||
| - name: Download Docker Build Image | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| name: go-build-env-image | ||
| path: /tmp | ||
|
|
||
| - name: Load Docker Build Image | ||
| run: | | ||
| docker load --input /tmp/go-build-env-image.tar | ||
|
|
||
| - name: Download previous benchmark results | ||
| id: download-prev | ||
| uses: dawidd6/action-download-artifact@v6 | ||
| continue-on-error: true | ||
| with: | ||
| name: go-bench | ||
| path: bench-prev | ||
| workflow: client.yml | ||
| branch: main | ||
| if_no_artifact_found: warn | ||
|
|
||
| - name: Run benchmarks | ||
| run: | | ||
| docker run \ | ||
| --workdir /go/src/github.com/keep-network/keep-core \ | ||
| go-build-env \ | ||
| go test -bench=. -benchmem -count=10 -run='^$' ./pkg/... \ | ||
| > bench.txt | ||
| cat bench.txt | ||
|
|
||
| - name: Install benchstat | ||
| run: go install golang.org/x/perf/cmd/benchstat@latest | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. go install golang.org/x/perf/cmd/benchstat@latest always pulls whatever is current on that repo’s default branch today. Tomorrow you might get a new release that: changes how tables are formatted (columns, spacing, wording), Pinning (@v0.0.0-… commit hash or a tagged version if they tag) means everyone gets the same benchstat binary until you bump it on purpose. |
||
|
|
||
| - name: Compare benchmarks | ||
| if: steps.download-prev.outcome == 'success' && hashFiles('bench-prev/**') != '' | ||
| run: | | ||
| benchstat bench-prev/*.txt bench.txt | tee benchstat-results.txt | ||
| python3 - <<'EOF' | ||
| import sys, re | ||
| content = open('benchstat-results.txt').read() | ||
| regressions = [] | ||
| for line in content.splitlines(): | ||
| if '~' in line or not line.strip(): | ||
| continue | ||
| m = re.search(r'\+(\d+\.\d+)%', line) | ||
| if m and float(m.group(1)) > 20: | ||
| regressions.append(line) | ||
| if regressions: | ||
| print('Performance regressions >20% detected:') | ||
| for r in regressions: | ||
| print(' ', r) | ||
| sys.exit(1) | ||
| EOF | ||
|
|
||
| - name: Upload benchmark results | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: go-bench | ||
| path: bench.txt | ||
| overwrite: true | ||
| if-no-files-found: warn | ||
|
|
||
| client-integration-test: | ||
| needs: [client-detect-changes, electrum-integration-detect-changes, client-build-test-publish] | ||
| if: | | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| # Go Profiling Runbook | ||
|
|
||
| ## Overview | ||
|
|
||
| The keep-core binary exposes Go runtime profiling endpoints via the | ||
| `clientinfo` HTTP server when `EnablePprof: true` is set in configuration. | ||
| Profiles are served at `/debug/pprof/` on the same port as metrics and | ||
| diagnostics (`ClientInfo.Port`). | ||
|
|
||
| ## Security Warning | ||
|
|
||
| The clientinfo HTTP server binds to all interfaces (`0.0.0.0`). **Never | ||
| enable pprof on a production node that is reachable from untrusted networks.** | ||
| CPU profiles, heap dumps, and goroutine traces can expose sensitive runtime | ||
| state. | ||
|
|
||
| Safe access patterns: | ||
| - Run on a private/firewalled network | ||
| - Use an SSH tunnel: `ssh -L 9601:localhost:9601 node-host` | ||
| - Restrict at the network layer (security group, firewall rule) | ||
|
|
||
| ## Enabling Profiling | ||
|
|
||
| In your config file (TOML example): | ||
|
|
||
| ```toml | ||
| [ClientInfo] | ||
| Port = 9601 | ||
| EnablePprof = true | ||
| ``` | ||
|
|
||
| Or pass via environment / flag if your deployment uses those overrides. | ||
|
|
||
| ## Standard Commands | ||
|
|
||
| Replace `9601` with your configured `ClientInfo.Port`. | ||
|
|
||
| ### CPU profile (30 seconds) | ||
|
|
||
| ```sh | ||
| go tool pprof http://localhost:9601/debug/pprof/profile?seconds=30 | ||
| ``` | ||
|
|
||
| ### Heap profile | ||
|
|
||
| ```sh | ||
| go tool pprof http://localhost:9601/debug/pprof/heap | ||
| ``` | ||
|
|
||
| ### Goroutine dump (text) | ||
|
|
||
| ```sh | ||
| curl -s http://localhost:9601/debug/pprof/goroutine?debug=2 | ||
| ``` | ||
|
|
||
| ### Trace (5 seconds) | ||
|
|
||
| ```sh | ||
| curl -o /tmp/trace.out http://localhost:9601/debug/pprof/trace?seconds=5 | ||
| go tool trace /tmp/trace.out | ||
| ``` | ||
|
|
||
| ### Mutex contention | ||
|
|
||
| ```sh | ||
| # Enable mutex profiling first (runtime call or startup flag): | ||
| # runtime.SetMutexProfileFraction(1) | ||
| go tool pprof http://localhost:9601/debug/pprof/mutex | ||
| ``` | ||
|
|
||
| ## Benchmark + Profile Workflow | ||
|
|
||
| To identify hot paths found by benchmarks: | ||
|
|
||
| ```sh | ||
| # Run benchmark and write CPU profile | ||
| go test ./pkg/tbtc/... -run=^$ -bench=BenchmarkGetRecentWindows \ | ||
| -cpuprofile=/tmp/cpu.pprof -benchtime=5s | ||
|
|
||
| # Inspect interactively | ||
| go tool pprof /tmp/cpu.pprof | ||
| (pprof) top10 | ||
| (pprof) web # requires graphviz | ||
| ``` | ||
|
|
||
| For memory allocation hot paths: | ||
|
|
||
| ```sh | ||
| go test ./pkg/bitcoin/... -run=^$ -bench=BenchmarkComputeSignatureHashes \ | ||
| -memprofile=/tmp/mem.pprof -benchtime=5s | ||
| go tool pprof /tmp/mem.pprof | ||
| (pprof) alloc_space | ||
| (pprof) top10 | ||
| ``` | ||
|
|
||
| ## Comparing Benchmarks Across Commits | ||
|
|
||
| ```sh | ||
| # Baseline (main branch) | ||
| git stash | ||
| go test ./pkg/... -run=^$ -bench=. -count=6 | tee /tmp/baseline.txt | ||
|
|
||
| # Candidate (your branch) | ||
| git stash pop | ||
| go test ./pkg/... -run=^$ -bench=. -count=6 | tee /tmp/candidate.txt | ||
|
|
||
| benchstat /tmp/baseline.txt /tmp/candidate.txt | ||
| ``` | ||
|
|
||
| Install `benchstat`: `go install golang.org/x/perf/cmd/benchstat@latest` | ||
|
|
||
| ## Available Endpoints | ||
|
|
||
| | Endpoint | Description | | ||
| |----------|-------------| | ||
| | `/debug/pprof/` | Index of available profiles | | ||
| | `/debug/pprof/cmdline` | Process command line | | ||
| | `/debug/pprof/profile` | CPU profile (30s default) | | ||
| | `/debug/pprof/symbol` | Symbol lookup | | ||
| | `/debug/pprof/trace` | Execution trace | | ||
| | `/debug/pprof/goroutine` | Goroutine stacks | | ||
| | `/debug/pprof/heap` | Heap allocations | | ||
| | `/debug/pprof/allocs` | Allocation samples | | ||
| | `/debug/pprof/block` | Goroutine blocking events | | ||
| | `/debug/pprof/mutex` | Mutex contention | | ||
|
|
||
| ## Notes | ||
|
|
||
| - CPU profiling adds ~5% overhead to the profiled binary during the sampling | ||
| window. It is safe to run against a live node for short durations. | ||
| - Heap and goroutine profiles are sampled snapshots; a single sample may | ||
| miss transient allocations. Take multiple profiles under load. | ||
| - pprof registers on `http.DefaultServeMux`. If `EnablePprof: false`, the | ||
| handlers are still compiled in but no log message is emitted and they will | ||
| not be documented in operator runbooks as intentionally exposed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
14% total is quite a loose check; lots of untested code can slip in if the average stays barely above it. Was it intentionally?