feat(cli): add --ignore-failure flag (#242)

[yaowubarbara]

Closes #242.

Adds --ignore-failure (short -i) to codspeed exec, inspired by hyperfine. When set, a non-zero exit status from the benchmarked command is logged as a warning instead of aborting the run.

Reworked per review: the flag now lives in the exec-harness crate (where the user command is directly executed), not in the executor layer. This correctly scopes the tolerance to the wrapped command exit code.

Changes

exec-harness (crates/exec-harness/):

• Added ignore_failure field to BenchmarkCommand (flows via stdin JSON)
• Added --ignore-failure / -i CLI arg for direct invocation
• walltime/benchmark_loop.rs: warn + continue instead of bail on non-zero exit
• analysis/mod.rs: same for perform() and perform_with_valgrind()

CLI (src/):

• Added --ignore-failure to ExecArgs only (not shared, not on codspeed run)
• Added ignore_failure: bool to BenchmarkTarget::Exec
• Threaded through multi_targets::build_exec_targets_pipe_command() into the JSON payload

Why exec only, not run?

codspeed run expects a command with built-in CodSpeed instrumentation (pytest-codspeed, cargo-codspeed, etc.). If that command fails, it's a real test failure. codspeed exec wraps arbitrary commands that may legitimately exit non-zero (interrupted servers, coverage harnesses, etc.).

[GuillaumeLagrange]

@yaowubarbara thanks for the PR

I think the idea is fine, but we should not do it at the executor level (valgrind/ walltime)

As far as I see it, the only use case of the command failing would be when running outside of a codspeed integration, i.e when going through this

That means that the allowing a non 0 exit code should be scoped to the exec-harness, so accept the cli flag only for codspeed exec and forwared it all the way to the exec harness.

Then here and here we should not bail on a failure

[yaowubarbara]

Thanks for the review @GuillaumeLagrange! That makes total sense — the executor should stay strict since it's running exec-harness (or the instrumented command), and the tolerance belongs inside exec-harness where the user's command is directly invoked.

Reworked:

• Removed ignore_failure from ExecutorConfig, OrchestratorConfig, ExecAndRunSharedArgs, and both executor impls
• Added it to BenchmarkTarget::Exec, flows through BenchmarkCommand JSON to exec-harness
• Scoped the CLI flag to codspeed exec only (via ExecArgs, not shared)
• The bail-to-warn change now lives in benchmark_loop.rs and analysis/mod.rs inside exec-harness

Force-pushed with the rework. Let me know if there's anything else to adjust!