Skip to content

test: add build log memory benchmarks #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fcying merged 1 commit into from
Aug 11, 2026
Merged

test: add build log memory benchmarks #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
187 changes: 37 additions & 150 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,137 +66,39 @@ COMMANDS:
--cmd/-c value Command to be used as make executable.
```

`compiledb` provides a Go `make` wrapper which, besides executing the make
build command, updates the JSON compilation database file corresponding to that build,
resulting in a command-line interface similar to [Bear][bear].

By default, new commands update the existing compilation database. Entries are matched by
their `directory` and `file`, and a newly generated entry replaces an existing entry for
the same source file. Legacy relative directories that map to the configured build directory,
and explicit Windows drive/UNC slash and case variants, are
matched to their current representation without rewriting preserved JSON fields. Use
`--overwrite/-f` to replace the database instead:
```bash
$ compiledb --overwrite make
```
### Make Wrapper

When `--output -` is used, only commands generated by the current invocation are written
to stdout. Parser diagnostics and real Make output are written to stderr so stdout remains
a valid JSON compilation database.
Run Make and update `compile_commands.json` after a successful build:

Non-verbose diagnostics use error level and are always written to stderr. Relevant malformed compiler,
`cd`, and Make commands report the build-log line, tracked directory, and reason without printing the
full command; tokenization and backtick execution failures skip only the affected command and parsing
continues with exit status 0. Cancellation remains a nonzero error. Strict mode follows symlinks and
accepts only regular source files. Repeat `--exclude/-e value` to exclude any source path whose
beginning matches one of the supplied regular expressions.

To generate `compile_commands.json` file using compiledb's "make wrapper" script,
executing Makefile target `all`:
```bash
$ compiledb make
```

The `make` subcommand consumes `--cmd/-c command` to select the GNU Make-compatible executable,
and `--help/-h` displays its help; all other options and arguments are forwarded to Make in their
original order. The first `make --` ends wrapper option parsing and is not forwarded; every following
argument is forwarded. For example, generate `compile_commands.json` using `gmake` and
`core/main.mk`
as main makefile (`--file/-f`), starting the build from `build` directory (`--directory/-C`):
Arguments after `make` are forwarded to GNU Make. Use `make --cmd/-c` to select another compatible
executable, and `make --` when a Make argument must bypass wrapper option parsing:

```bash
$ compiledb make --cmd gmake -f core/main.mk -C build
```
An empty `--cmd` value falls back to `make`, matching the original Python CLI. The top-level
`--command-style/-c` remains separate: `compiledb -c make` enables command-style
database output, while `compiledb make -c gmake` selects the Make executable.
As in the original Click CLI, known short options are recognized inside a short-option cluster;
place `--` before attached Make options when they must bypass wrapper parsing unchanged.

To query each compiler for its predefined macros and add them to generated entries, use
`--macros/-m`. Compiler executables must be available from the command's working directory
or `PATH`. The query preserves selected safe compiler options that affect built-in macros,
such as `-std`, `-m32`, and `--target`. Macro probing is skipped for response files and
unsupported macro-affecting or compiler-forwarding options because replaying them could use
the wrong target configuration, overwrite build outputs, or load compiler plugins. It is also
skipped for multi-source commands, multi-stage languages such as CUDA/HIP, and commands whose
compiler is changed by shell, ccache, icecc, or launcher configuration. Compiler-control and
dynamic-loader injection environment variables also disable probing rather than bypassing these
checks. Compiler commands containing source inputs are recorded even when they do not use `-c`,
matching the original Python implementation. Multi-source commands generate one database entry
for each source instead of dropping all but the last one. The default matcher accepts compiler
drivers, including cross-prefixed and versioned GCC/Clang names, while excluding source-processing
tools such as `clang-format` and `clang-tidy`. Explicit `time` and `nice` launchers are supported;
arbitrary command prefixes are not scanned because output commands such as `echo gcc ...` must not
be treated as compilations. Ordinary diagnostic `-W` options are ignored by the probe:
```bash
$ compiledb --macros make
```

To add custom compiler arguments into generated entries, repeat `--add-arg/-a`. Each flag
value is appended as one argument without comma or shell-word splitting:
```bash
$ compiledb --add-arg='-DCSV=a,b' -a=-m32 make
```
The top-level `--command-style/-c` is separate from `make --cmd/-c`. Use `--no-build/-n` to skip
the real build and run discovery only:

By default, `compiledb make` generates the compilation database, runs the actual build
command requested (acting as a make wrapper), and prints wrapped `make` output using raw
stream forwarding. The discovery Make invocation starts only after the real Make process
returns successfully, so a failed build does not update the compilation database and the two
Make invocations do not run concurrently. The build step can be skipped using `--no-build/-n`.
```bash
$ compiledb -n make
```

The discovery Make invocation is forced to run serially with directory printing so compilation
commands retain a deterministic working directory, including with GNU Make 3.81. Its discovery
mode removes conflicting question, touch, print-data-base, help, version, no-keep-going, and
no-print-directory options from both command-line arguments and `MAKEFLAGS`/`GNUMAKEFLAGS`, while
preserving Make variable assignments. A Makefile supplied with `-f -`, including through those
environment variables, is read once and provided independently to the real and discovery Make
processes. Only discovery stdout is parsed as build-log input; discovery stderr is forwarded to
stderr and can never create database entries. A nonzero discovery exit leaves the compilation
database untouched, even if Make produced partial stdout before failing. Simple `mkdir -p` commands
printed by the dry run, including their implied parent directories, are tracked so a following
`cd` can use a generated build directory without depending on the real build's timing.

Build-log backtick expressions outside single quotes are executed once by an embedded POSIX shell
interpreter in the tracked command directory; their output is treated as argument data and is not
re-evaluated as shell source. Shell builtins do not require an external shell executable. Programs
explicitly invoked by the backtick body must still be available through `PATH`; a failure skips only
the affected command, emits an error diagnostic, and leaves the parser status successful.
Unsupported shell constructs and conditionals whose execution cannot be
determined safely are skipped as a whole rather than allowing commands inside groups,
functions, or control structures to escape into the database. Backslash-newline continuation
uses shell semantics without inserting extra whitespace; malformed or unterminated continuation
is skipped as a recoverable parse failure. Physical build-log lines are limited to 100 MiB.
Do not parse untrusted build logs.
With `--no-strict`, a simple inline `cd` updates the tracked directory without requiring that path
to exist on the parser host, which allows logs captured on another machine to retain their cwd.
Interrupting the CLI cancels active Make, compiler-probe, and backtick subprocesses. On Unix,
cancellation is forwarded to the subprocess group; on Windows, controlled subprocesses are placed
in a Job Object. Output is forwarded through a cancellable relay, with the inherited-pipe drain
timeout starting only after Make exits. JSON stdout writes also observe cancellation instead of
waiting indefinitely on a blocked consumer.

If your wrapped `make` output is encoded as GB18030, you can opt into decoding it while
streaming:
```bash
$ compiledb --encoding gb18030 make
```
Only successful discovery output updates the database. Make output and diagnostics go to stderr
when `--output -` is used, keeping stdout valid JSON.

To avoid repeating the option in a fixed environment, you can set an environment variable:
```bash
$ export COMPILEDB_ENCODING=gb18030
$ compiledb make
```
### Parse A Build Log

`compiledb` base command has been designed so that it can be used to parse compile commands
from arbitrary text files (or stdin), assuming it has a build log (ideally generated using
`make -Bnwk` command), and generates the corresponding JSON Compilation database.
Parse a saved build log, read stdin, or pipe Make output directly:

For example, to generate the compilation database from `build-log.txt` using `--parse/-p`:
```bash
$ compiledb --parse build-log.txt
$ compiledb < build-log.txt
$ make -Bnwk | compiledb --output -
```

The initial working directory is selected in this order:
Expand All @@ -205,56 +107,46 @@ The initial working directory is selected in this order:
2. When parsing a build-log file, its containing directory is used by default.
3. When parsing stdin, the directory where `compiledb` was started is used.

Using the build-log directory makes file parsing deterministic when the log and build tree move
together or when the command is invoked from another directory. If a log is copied separately to
another location, such as `/tmp` or an archive directory, pass the real build directory explicitly:

```bash
$ compiledb --build-dir /path/to/project/build --parse /tmp/build-log.txt
```

The selected directory is the base for relative source paths, inline `cd`, `make -C`, compiler
lookup through relative `PATH` entries, backtick expressions, strict file checks, and predefined
macro probing.
Backtick expressions are executed by an embedded POSIX interpreter. Their output is used as
argument data, but explicitly invoked programs must still be available through `PATH`. Do not parse
untrusted build logs.

or its equivalent:
```bash
$ compiledb < build-log.txt
$ compiledb --parse - < build-log.txt
```
### Output Options

By default, new entries update the existing database. Use `--overwrite/-f` to replace it:

Or even, to pipe make's output and print the compilation database to the standard output:
```bash
$ make -Bnwk | compiledb -o -
$ compiledb --overwrite make
```

By default `compiledb` generates a JSON compilation database in the "arguments" list
[format](https://clang.llvm.org/docs/JSONCompilationDatabase.html). The "command" string
format is also supported through `--command-style/-c`:
Useful output options:

```bash
$ compiledb --command-style make
$ compiledb --add-arg='-DCSV=a,b' -a=-m32 make
$ compiledb --macros make
$ PATH=/opt/buildroot/bin:$PATH compiledb --full-path make
$ compiledb --encoding gb18030 make
```

`--full-path` searches the user's $PATH for the compiler executable and in
case it's found, replaces the executable name with the full path to
executable in "arguments" section in compilation database.
This argument allows to specify the compiler path only once, when
calling compiledb, like so:
```
PATH=/opt/buildroot/bin:$PATH compiledb --full-path make
```
`--macros/-m` requires the compiler to be available from its working directory or `PATH`; unsafe or
unsupported probes are skipped. Repeat `--exclude/-e` and `--add-arg/-a` as needed.

### Windows Support
### Memory Use

Build-log processing is buffered and uses `O(build-log size + entries)` memory. Each physical line
is limited to 100 MiB. In synthetic Linux amd64 tests, direct parsing peaked at about 311 MiB/1.53
GiB RSS for 100 MiB/500 MiB logs; Make discovery peaked at about 490 MiB/1.95 GiB.

The Windows artifact supports Windows compiler paths, `arguments` output, Windows command-line
quoting for `--command-style`, and process cancellation through Job Objects. The Make wrapper still
requires a GNU Make-compatible executable; use `compiledb make --cmd gmake` or
`compiledb make --cmd mingw32-make` when it is not named `make`.
### Windows Support

Build-log parsing currently targets POSIX/MSYS shell syntax. GNU Make output produced in MSYS2 or
Git Bash is the supported Windows workflow. Native `cmd.exe` recipe grammar, including caret escapes,
`%VAR%`/`!VAR!`, `cd /d`, and `cmd /C`, is not parsed, and `nmake` is not supported. Backtick
substitution uses the embedded POSIX interpreter and does not require a shell executable.
Use a GNU Make-compatible executable and POSIX/MSYS recipe output, typically through MSYS2 or Git
Bash. Select `gmake` or `mingw32-make` with `make --cmd` when necessary. Native `cmd.exe` recipe
syntax and `nmake` are not supported.

## Testing / Contributing

Expand All @@ -269,11 +161,6 @@ could use it with some great tools, such as:
- [Neovim][neovim] + [LanguageClient-neovim][lsp] + [cquery][cquery] + [deoplete][deoplete]
- [Neovim][neovim] + [ALE][ale] + [ccls][ccls]

The release workflow supports Linux amd64/arm64, Windows amd64, and macOS arm64. Each artifact is
built on a native GitHub-hosted runner and must successfully execute `compiledb --help` before it
can be published. Linux amd64 additionally gates releases on `gofmt`, `go vet`, the full test suite,
and the race detector. Other platforms are unsupported and receive no release artifacts.

## License
GNU GPLv3

Expand Down
Loading
Loading