Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 92 additions & 0 deletions .github/workflows/client.yml
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,19 @@ jobs:
path: coverage/coverage.out
if-no-files-found: warn

- name: Check coverage gate
run: |
docker run --rm \
-v "${{ github.workspace }}/coverage:/coverage" \
go-build-env \
go tool cover -func /coverage/coverage.out > /tmp/cover-func.txt
TOTAL=$(grep '^total:' /tmp/cover-func.txt | awk '{print $3}' | tr -d '%')
echo "Total coverage: ${TOTAL}%"
PASS=$(awk -v t="$TOTAL" 'BEGIN { print (t+0 >= 14) ? "yes" : "no" }')
if [ "$PASS" != "yes" ]; then
echo "::error::Coverage ${TOTAL}% is below the 14% minimum threshold"
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

14% total is quite a loose check; lots of untested code can slip in if the average stays barely above it. Was it intentionally?

exit 1
fi

- name: Build Docker Runtime Image
if: github.event_name != 'workflow_dispatch'
Expand Down Expand Up @@ -311,6 +324,85 @@ jobs:
install-go: false
checks: "-SA1019"

client-bench:
needs: [client-build-test-publish]
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
permissions:
actions: read
steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: "1.24"
cache: false

- name: Download Docker Build Image
uses: actions/download-artifact@v4
with:
name: go-build-env-image
path: /tmp

- name: Load Docker Build Image
run: |
docker load --input /tmp/go-build-env-image.tar

- name: Download previous benchmark results
id: download-prev
uses: dawidd6/action-download-artifact@v6
continue-on-error: true
with:
name: go-bench
path: bench-prev
workflow: client.yml
branch: main
if_no_artifact_found: warn

- name: Run benchmarks
run: |
docker run \
--workdir /go/src/github.com/keep-network/keep-core \
go-build-env \
go test -bench=. -benchmem -count=10 -run='^$' ./pkg/... \
> bench.txt
cat bench.txt

- name: Install benchstat
run: go install golang.org/x/perf/cmd/benchstat@latest
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go install golang.org/x/perf/cmd/benchstat@latest always pulls whatever is current on that repo’s default branch today. Tomorrow you might get a new release that:

changes how tables are formatted (columns, spacing, wording),
adds/removes summary lines,
tweaks how deltas are expressed.
So CI that passed yesterday might fail tomorrow without any Go code change, or the opposite: a real regression might stop matching your script’s expectations.

Pinning (@v0.0.0-… commit hash or a tagged version if they tag) means everyone gets the same benchstat binary until you bump it on purpose.


- name: Compare benchmarks
if: steps.download-prev.outcome == 'success' && hashFiles('bench-prev/**') != ''
run: |
benchstat bench-prev/*.txt bench.txt | tee benchstat-results.txt
python3 - <<'EOF'
import sys, re
content = open('benchstat-results.txt').read()
regressions = []
for line in content.splitlines():
if '~' in line or not line.strip():
continue
m = re.search(r'\+(\d+\.\d+)%', line)
if m and float(m.group(1)) > 20:
regressions.append(line)
if regressions:
print('Performance regressions >20% detected:')
for r in regressions:
print(' ', r)
sys.exit(1)
EOF

- name: Upload benchmark results
if: always()
uses: actions/upload-artifact@v4
with:
name: go-bench
path: bench.txt
overwrite: true
if-no-files-found: warn

client-integration-test:
needs: [client-detect-changes, electrum-integration-detect-changes, client-build-test-publish]
if: |
Expand Down
5 changes: 4 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -146,4 +146,7 @@ cmd-help: build
@echo '$$ $(app_name) start --help' > docs/resources/client-start-help
./$(app_name) start --help >> docs/resources/client-start-help

.PHONY: all development sepolia download_artifacts generate gen_proto build cmd-help release build_multi
bench:
go test -bench=. -benchmem -count=10 -run='^$$' ./pkg/...

.PHONY: all development sepolia mainnet local get_artifacts generate gen_proto build cmd-help release build_multi bench
2 changes: 1 addition & 1 deletion cmd/start.go
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ func initializeClientInfo(
signing chain.Signing,
blockCounter chain.BlockCounter,
) *clientinfo.Registry {
registry, isConfigured := clientinfo.Initialize(ctx, config.ClientInfo.Port)
registry, isConfigured := clientinfo.Initialize(ctx, config.ClientInfo)
if !isConfigured {
logger.Infof("client info endpoint not configured")
return nil
Expand Down
135 changes: 135 additions & 0 deletions docs/profiling.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
# Go Profiling Runbook

## Overview

The keep-core binary exposes Go runtime profiling endpoints via the
`clientinfo` HTTP server when `EnablePprof: true` is set in configuration.
Profiles are served at `/debug/pprof/` on the same port as metrics and
diagnostics (`ClientInfo.Port`).

## Security Warning

The clientinfo HTTP server binds to all interfaces (`0.0.0.0`). **Never
enable pprof on a production node that is reachable from untrusted networks.**
CPU profiles, heap dumps, and goroutine traces can expose sensitive runtime
state.

Safe access patterns:
- Run on a private/firewalled network
- Use an SSH tunnel: `ssh -L 9601:localhost:9601 node-host`
- Restrict at the network layer (security group, firewall rule)

## Enabling Profiling

In your config file (TOML example):

```toml
[ClientInfo]
Port = 9601
EnablePprof = true
```

Or pass via environment / flag if your deployment uses those overrides.

## Standard Commands

Replace `9601` with your configured `ClientInfo.Port`.

### CPU profile (30 seconds)

```sh
go tool pprof http://localhost:9601/debug/pprof/profile?seconds=30
```

### Heap profile

```sh
go tool pprof http://localhost:9601/debug/pprof/heap
```

### Goroutine dump (text)

```sh
curl -s http://localhost:9601/debug/pprof/goroutine?debug=2
```

### Trace (5 seconds)

```sh
curl -o /tmp/trace.out http://localhost:9601/debug/pprof/trace?seconds=5
go tool trace /tmp/trace.out
```

### Mutex contention

```sh
# Enable mutex profiling first (runtime call or startup flag):
# runtime.SetMutexProfileFraction(1)
go tool pprof http://localhost:9601/debug/pprof/mutex
```

## Benchmark + Profile Workflow

To identify hot paths found by benchmarks:

```sh
# Run benchmark and write CPU profile
go test ./pkg/tbtc/... -run=^$ -bench=BenchmarkGetRecentWindows \
-cpuprofile=/tmp/cpu.pprof -benchtime=5s

# Inspect interactively
go tool pprof /tmp/cpu.pprof
(pprof) top10
(pprof) web # requires graphviz
```

For memory allocation hot paths:

```sh
go test ./pkg/bitcoin/... -run=^$ -bench=BenchmarkComputeSignatureHashes \
-memprofile=/tmp/mem.pprof -benchtime=5s
go tool pprof /tmp/mem.pprof
(pprof) alloc_space
(pprof) top10
```

## Comparing Benchmarks Across Commits

```sh
# Baseline (main branch)
git stash
go test ./pkg/... -run=^$ -bench=. -count=6 | tee /tmp/baseline.txt

# Candidate (your branch)
git stash pop
go test ./pkg/... -run=^$ -bench=. -count=6 | tee /tmp/candidate.txt

benchstat /tmp/baseline.txt /tmp/candidate.txt
```

Install `benchstat`: `go install golang.org/x/perf/cmd/benchstat@latest`

## Available Endpoints

| Endpoint | Description |
|----------|-------------|
| `/debug/pprof/` | Index of available profiles |
| `/debug/pprof/cmdline` | Process command line |
| `/debug/pprof/profile` | CPU profile (30s default) |
| `/debug/pprof/symbol` | Symbol lookup |
| `/debug/pprof/trace` | Execution trace |
| `/debug/pprof/goroutine` | Goroutine stacks |
| `/debug/pprof/heap` | Heap allocations |
| `/debug/pprof/allocs` | Allocation samples |
| `/debug/pprof/block` | Goroutine blocking events |
| `/debug/pprof/mutex` | Mutex contention |

## Notes

- CPU profiling adds ~5% overhead to the profiled binary during the sampling
window. It is safe to run against a live node for short durations.
- Heap and goroutine profiles are sampled snapshots; a single sample may
miss transient allocations. Take multiple profiles under load.
- pprof registers on `http.DefaultServeMux`. If `EnablePprof: false`, the
handlers are still compiled in but no log message is emitted and they will
not be documented in operator runbooks as intentionally exposed.
49 changes: 49 additions & 0 deletions pkg/altbn128/altbn128_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -81,3 +81,52 @@ func assertEqual(t *testing.T, n int, n2 int, msg string) {
t.Errorf("%v: [%v] != [%v]", msg, n, n2)
}
}

// --- Benchmarks ---

func BenchmarkCompressG1(b *testing.B) {
_, p, err := bn256.RandomG1(rand.Reader)
if err != nil {
b.Fatal(err)
}
b.ResetTimer()
for range b.N {
G1Point{p}.Compress()
}
}

func BenchmarkDecompressG1(b *testing.B) {
_, p, err := bn256.RandomG1(rand.Reader)
if err != nil {
b.Fatal(err)
}
buf := G1Point{p}.Compress()
b.ResetTimer()
for range b.N {
_, _ = DecompressToG1(buf)
}
}

func BenchmarkCompressDecompressRoundTripG1(b *testing.B) {
_, p, err := bn256.RandomG1(rand.Reader)
if err != nil {
b.Fatal(err)
}
b.ResetTimer()
for range b.N {
buf := G1Point{p}.Compress()
_, _ = DecompressToG1(buf)
}
}

func BenchmarkCompressDecompressRoundTripG2(b *testing.B) {
_, p, err := bn256.RandomG2(rand.Reader)
if err != nil {
b.Fatal(err)
}
b.ResetTimer()
for range b.N {
buf := G2Point{p}.Compress()
_, _ = DecompressToG2(buf)
}
}
Loading
Loading