GhostScope

⚡ Next-generation eBPF userspace runtime tracer

Printf debugging evolved — Real-time tracing without stopping your application.

中文文档

Overview

GhostScope is a source-aware userspace tracer for live Linux processes. With DWARF debug info, it lets you attach at function, source-line, or instruction granularity and print the values that matter without stopping the target.

"The most effective debugging tool is still careful thought, coupled with judiciously placed print statements." — Brian Kernighan

When To Use GhostScope

• You are diagnosing a production service that must stay online: GDB-style stop-the-world debugging would cause too much disruption, and you prefer an eBPF-based workflow with stronger safety boundaries and lower overhead than traditional kernel-module instrumentation.
• You care about source lines and real variable values, not just function entry arguments.
• You want a low-friction path from "I wish I had one more printf here" to a runnable trace script.
• You want an AI agent to turn GhostScope docs, source paths, and DWARF-backed binaries into concrete tracing commands.

When Not To Use GhostScope

• Use GDB when you want an interactive debugging experience with breakpoints, single-stepping, memory writes, or coredump debugging.
• Use perf probe when you want a quick one-off probe at a function, source line, or local variable inside the perf ecosystem.
• Use bpftrace or SystemTap when you want broad kernel + userspace event aggregation in one script.
• Do not expect source-level variable tracing to work well without DWARF debug info for the modules you care about.

GhostScope vs perf, GDB, bpftrace, and SystemTap

For the full, centrally maintained comparison, see the Tool Comparison. You can also refer to the FAQ.

AI Runtime Analysis Skill

GhostScope supports two modes: an interactive TUI mode, and a CLI mode built around --script and --script-file. The latter is the more AI-optimized workflow for automation and agent-driven runtime analysis.

Install the shared skill for Codex or Claude Code:

curl -fsSL https://raw.githubusercontent.com/swananan/ghostscope/main/scripts/skills/install_ghostscope_runtime_analysis_skill.sh | bash -s -- --copy

If you already have the repository checked out locally, ./scripts/skills/install_ghostscope_runtime_analysis_skill.sh --copy works too.

Use --codex, --claude, or --all when you want to force a target. Restart Codex or Claude Code after installation.

When the agent shares your workspace, it can often discover the source checkout path and DWARF/debug-symbol status on its own. If it cannot determine them reliably, it should ask before generating a source-backed trace. A typical session looks like this.

Context the agent can discover locally in this example:

• Source checkout: /mnt/500g/code/openresty/openresty-1.27.1.1/build/nginx-1.27.1
• Target binary: /usr/local/openresty/nginx/sbin/nginx
• Debug info: embedded DWARF is available

Prompt:

$ghostscope-runtime-analysis trace the running nginx worker and show the raw request body bytes

Generated command:

WORKER_PID=$(pgrep -n -f 'nginx: worker process')
sudo ghostscope -p "$WORKER_PID" --script-file /tmp/ghostscope-nginx-body-discard.gs --script-output plain

Generated script:

trace /mnt/500g/code/openresty/openresty-1.27.1.1/build/nginx-1.27.1/src/http/ngx_http_request_body.c:671 {
    if size > 0 {
        let req_line_len = r.request_line.len;
        let body_len = size;

        print "src=discard-preread pid={} req={:p} line={:s.req_line_len$} body_len={} body={:x.body_len$}",
            $pid, r, r.request_line.data, body_len, r.header_in.pos;
    }
}

Demo:

For best results, make sure the relevant source tree is available, the modules you care about carry DWARF debug information, and GhostScope has the privileges needed to load eBPF programs. When that information is not discoverable locally, the skill should ask for it before generating a source-backed trace. Re-run the same installer after pulling updates; the installed skill is versioned and refreshes automatically when the version changes.

The Printf That Should Have Been

GhostScope turns compiled binaries into observable systems. In the TUI, that experience unfolds progressively: first find the function or source line you care about, then inspect the variables visible at that point, enter Script Mode from that location, and finally watch the live output panel update while the target keeps running. It feels less like a generic dashboard and more like source-guided runtime printf debugging.

The demo below follows exactly that path on a DWARF-enabled nginx worker: locate the code path, drop a trace at the right line, add a small script, and immediately see conditional logic, source-oriented variable access, and live output from the running process.

_{Real-time tracing of a running nginx worker process}

How It Works

Imagine navigating a vast, uncharted forest of binary data — memory addresses, register values, stack frames — all meaningless numbers without context. DWARF debug information is our map: it tells us that stack address RSP-0x18 stores local variable count, heap address 0x5621a8c0 is a user object with string pointer user.name at offset +0x20; it tracks where each variable lives throughout program execution — parameter x is in register RDI now but will move to stack offset RSP-0x10 later.

With this map in hand, GhostScope leverages eBPF and uprobe technology to safely extract binary data from any instruction point in your running program. The combination is powerful: DWARF reveals the meaning of every byte in the process's virtual address space, while eBPF safely retrieves exactly what we need. The result? You can print variable values (local or global), function arguments, complex data structures, even stack backtraces from any point in your program — all without stopping or modifying it.

✨ Highlights

Zero Overhead One context switch + eBPF execution

Real-Time Tracing Live trace streaming

DWARF-Aware Full debug info support

Built with Rust Memory safe & blazing fast

⚠️ Experimental Tool Disclaimer

GhostScope is currently in early development and under active iteration. While we strive for data accuracy, trace information may be incorrect or incomplete in certain scenarios, primarily due to unsupported features.

Recommendation: Use GhostScope's collected data as an auxiliary reference for troubleshooting, not as the sole source of truth. Cross-validate with other debugging tools before making critical decisions.

We are continuously improving stability and accuracy, and look forward to removing this disclaimer in future versions.

See Limitations for the current list of hard and soft constraints.

📚 Documentation

🎯 Getting Started Installation Guide System requirements and setup Quick Tutorial Learn the basics in 10 minutes Tool Comparison Choose between GhostScope, perf, GDB, bpftrace, and SystemTap FAQ Common questions answered Limitations Known limitations and constraints

⚙️ Configuration Configuration Reference All configuration options TUI Reference Complete keyboard shortcuts and panel navigation Command Reference All available commands for Input Mode Script Language Write powerful trace scripts

👨‍💻 Development Architecture Overview System design and internals Development Guide Build and extend GhostScope Contributing Guide Join the community Roadmap Planned features and milestones

🤝 Contributing

We welcome contributions! Whether it's bug reports, feature requests, documentation improvements, or code contributions, we appreciate your help in making GhostScope better.

Please see our Contributing Guide for:

• Code of Conduct
• Development workflow
• Coding standards
• How to submit pull requests

📜 License

GhostScope is licensed under the GNU General Public License.

🙏 Acknowledgements

Built with amazing open source projects:

• Aya - eBPF library for Rust (using its loader functionality)
• LLVM - Compiler infrastructure
• Inkwell - Safe LLVM bindings for Rust
• Gimli - DWARF parser
• Ratatui - Terminal UI framework
• Tokio - Async runtime
• Pest - PEG parser generator

Inspired by and learned from:

• GDB - DWARF parsing optimizations
• bpftrace - eBPF tracing techniques
• cgdb - TUI design and user experience

Special thanks to these excellent resources that taught us a lot:

Blog Posts:

• The Wonderland of Dynamic Tracing
• Unwinding the Stack the Hard Way

Books:

• Crafting Interpreters