Skip to content

feat: add harness field to cargo metadata output#17088

Open
ceronman wants to merge 1 commit into
rust-lang:masterfrom
ceronman:add-harness-to-metadata
Open

feat: add harness field to cargo metadata output#17088
ceronman wants to merge 1 commit into
rust-lang:masterfrom
ceronman:add-harness-to-metadata

Conversation

@ceronman

@ceronman ceronman commented Jun 9, 2026

Copy link
Copy Markdown

Adds the harness field to each target in cargo metadata JSON output, exposing the Cargo.toml [[test]]/[[bench]] harness setting so that external tools can tell whether a target uses the libtest harness or provides its own main().

What does this PR try to resolve?

I work on the RustRover IDE. We add small gutter buttons in the editor to run tests and benches easily. We also want to show the tests results rendered in a nice tree view. For that purpose, we send some extra arguments when running the tests (-Z unstable-options, --format=json, --show-output). However these arguments only work with the default rustc test harness (libtest). If the test does not use the default harness, but a different one, for example when using criterion and similar crates, then the custom harnesses will likely fail because they don't understand the extra arguments. We also don't want to display a tree view with the results but simply show the terminal output. There are more details about this in the RustRover tracking issue

So, for RustRover is very important to identify if a target uses or not a custom harness. For now our only workaround is to parse Cargo.toml directly, but this creates a number of problems. It would very handy to get this information from cargo metadata, which we already query.

Fixes #11846

How to test and review this PR?

Testing and reviewing should be straight forward.

@rustbot rustbot added A-cli-help Area: built-in command-line help A-documenting-cargo-itself Area: Cargo's documentation A-manifest Area: Cargo.toml issues S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 9, 2026
@rustbot

rustbot commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @epage (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

  • @rustbot author: the review is finished, PR author should check the comments and take action accordingly
  • @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue
Why was this reviewer chosen?

The reviewer was selected based on:

  • Owners of files modified in this PR: @ehuss, @epage, @weihanglo
  • @ehuss, @epage, @weihanglo expanded to ehuss, epage, weihanglo
  • Random selection from ehuss, epage, weihanglo

@ceronman ceronman force-pushed the add-harness-to-metadata branch from 90e4d19 to 9b0a85f Compare June 9, 2026 18:40
@epage epage changed the title feat: add harness field to cargo metadata output (#11846) feat: add harness field to cargo metadata output Jun 9, 2026
/// Whether tests should be run for the target (`test` field in `Cargo.toml`)
test: bool,
/// Whether the target uses the libtest harness (`harness` field in `Cargo.toml`).
harness: bool,

@epage epage Jun 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we wrote cargo metadata, we adopted a policy to omit defaults, see https://doc.crates.io/contrib/implementation/schemas.html#schema-design

However, the question becomes how do we deal with the lack of the field vs the default?

View changes since the review

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for missing that detail from the guidelines. I'll add skip_serializing_ifhere.

However, the question becomes how do we deal with the lack of the field vs the default?

I'm not sure exactly what you mean here. In RustRover we can easily assume that no value means true. Or do you mean something else?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unresolved this because of the open question.

If we always provide harness, then we have three states: true, false, and "unknown". The last one is relevant when using an old version of Cargo. If we skip on default (would it be a true? false?) then a caller can't differentiate it from "unknown".

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I understand now :). Dealing with "unknown" is unavoidable, at least for a while, because older versions of cargo won't have this field. The problem is that now there is some ambiguity: a value not present might be either true or unknown depending on the version of cargo.

For us it would be slightly better if the default was not skipped. In that way, our logic will be something like: if harness is not there, fallback to try to get it from Cargo.toml. If we skip the default value, our logic will be: if harness is not there an cargo version > x it's true otherwise fallback.

So we can manage both ways. I understand the spirit of the guidelines and the fact that extra data means extra parsing time and this can affect certain tools. But from my perspective the ambiguity introduced is worse. So I would prefer if we don't skip.

"test": true
"test": true,
/* Whether or not the target uses the libtest harness.
Corresponds to the `harness` field in `Cargo.toml`.

@epage epage Jun 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raised a higher level concern at #11846 (comment)

View changes since the review

@ceronman ceronman force-pushed the add-harness-to-metadata branch 2 times, most recently from f0b1d3e to 7dde5d8 Compare June 9, 2026 21:25
Adds the `harness` field to each target in `cargo metadata` JSON output,
exposing the Cargo.toml `[[test]]`/`[[bench]]` `harness` setting so that
external tools can tell whether a target uses the libtest harness or
provides its own `main()`.
@ceronman ceronman force-pushed the add-harness-to-metadata branch from 7dde5d8 to c9b0605 Compare June 9, 2026 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-cli-help Area: built-in command-line help A-documenting-cargo-itself Area: Cargo's documentation A-manifest Area: Cargo.toml issues S-waiting-on-review Status: Awaiting review from the assignee but also interested parties.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add harness field to cargo metadata output

3 participants