From 6223215e167f212f6855c73c4316ca40f92c4746 Mon Sep 17 00:00:00 2001 From: Mikhail Katychev Date: Sat, 11 Jul 2026 11:37:10 -0500 Subject: [PATCH 1/3] doc(BUD-2,test): `topiary test` subcommand --- buds/bud-2.md | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 buds/bud-2.md diff --git a/buds/bud-2.md b/buds/bud-2.md new file mode 100644 index 0000000..c2a0f7c --- /dev/null +++ b/buds/bud-2.md @@ -0,0 +1,64 @@ +# Proposal for `topiary test` + + +## Thoughts + +* There's value to having test cases be in one huge file for side effects. +* One should not have to recompile the `topiary` binary to test query changes for built-in languages +* external grammar repositories such as [topiary-nushell](https://github.com/blindFS/topiary-nushell) should have comparable DX to what is happening in topiary + + +## Ideas + +* initial aim to have queries organized with comment delimiters similar to [corpus tests](https://tree-sitter.github.io/tree-sitter/creating-parsers/5-writing-tests.html#writing-tests) +for [`tree-sitter test`](https://github.com/tree-sitter/tree-sitter/blob/master/crates/cli/src/test.rs) +* This doesn't establish a complete correlation whether a test query was successful but it will check to see if there was any difference in the before and after test doc for a particular query + - this is different from `topiary coverage` because it would check if the string value changed when a particular query is run v.s. if a query found matches +* to handle something akin to [topiary-nushell's pattern](https://github.com/blindFS/topiary-nushell/tree/main/test) we can do something similar to nushell's [`parse`](https://www.nushell.sh/commands/docs/parse.html) command where ranges are defined using a template string for _input_ and _expeceted_: + `topiary test --input "test/input_{case}.nu" --expected "test/expected_{case}.nu"` + +Enumerating queries and test files is a many to many complexity but could be +an acceptable complexity for now since it will improve coverage over present state. + +### Mockup + +```bash +# ./my_lang/test/test1.myl +:' + 1. use native language line comment start followed by 4 or more `=` signs to + 2. next line test name + 3. new line equivalent `=` sign comment as 1. + 4. add final line-break +' + +# ================== +# return-statement +# ================== + +ok() { + yeet 1; +} + +# ================== +# loop +# ================== + +$one by-one { + utter "the good die ${one}"; +} +``` + + +Test report could look like this in stdout: +```sh-session +$ topiary test --dir ./my_lang/test --query ./my_lang/query.scm + +FAILURE(S): +- test1.myl + * return-statement (L10-L##) + * loop (L19-L##) +``` + + +Instead of comments, one could use [`query_name!`](https://topiary.tweag.io/book/reference/capture-names/general.html#query_name) but DX may be different. + From 6167c9955c853e27dde186293623af6c8c9f3a0b Mon Sep 17 00:00:00 2001 From: Mikhail Katychev Date: Mon, 13 Jul 2026 09:30:57 -0500 Subject: [PATCH 2/3] refactor: use bud schema for topiary test --- buds/bud-2.md | 46 +++++++++++++++++++++++++++++++++++++++------- 1 file changed, 39 insertions(+), 7 deletions(-) diff --git a/buds/bud-2.md b/buds/bud-2.md index c2a0f7c..7aab6b5 100644 --- a/buds/bud-2.md +++ b/buds/bud-2.md @@ -1,25 +1,33 @@ +--- +bud: 2 (TBD) +title: "`topiary test` query tester" +author: "@mkatychev" +pr: 2 +--- + # Proposal for `topiary test` +## Summary + +Introduce a `topiary test` subcommand that lets query authors verify +formatting behaviour by running Topiary over annotated test inputs and +comparing the result against expected output. -## Thoughts +## Motivation * There's value to having test cases be in one huge file for side effects. * One should not have to recompile the `topiary` binary to test query changes for built-in languages * external grammar repositories such as [topiary-nushell](https://github.com/blindFS/topiary-nushell) should have comparable DX to what is happening in topiary - -## Ideas +## Proposed design * initial aim to have queries organized with comment delimiters similar to [corpus tests](https://tree-sitter.github.io/tree-sitter/creating-parsers/5-writing-tests.html#writing-tests) for [`tree-sitter test`](https://github.com/tree-sitter/tree-sitter/blob/master/crates/cli/src/test.rs) * This doesn't establish a complete correlation whether a test query was successful but it will check to see if there was any difference in the before and after test doc for a particular query - this is different from `topiary coverage` because it would check if the string value changed when a particular query is run v.s. if a query found matches -* to handle something akin to [topiary-nushell's pattern](https://github.com/blindFS/topiary-nushell/tree/main/test) we can do something similar to nushell's [`parse`](https://www.nushell.sh/commands/docs/parse.html) command where ranges are defined using a template string for _input_ and _expeceted_: +* to handle something akin to [topiary-nushell's pattern](https://github.com/blindFS/topiary-nushell/tree/main/test) we can do something similar to nushell's [`parse`](https://www.nushell.sh/commands/docs/parse.html) command where ranges are defined using a template string for _input_ and _expected_: `topiary test --input "test/input_{case}.nu" --expected "test/expected_{case}.nu"` -Enumerating queries and test files is a many to many complexity but could be -an acceptable complexity for now since it will improve coverage over present state. - ### Mockup ```bash @@ -59,6 +67,30 @@ FAILURE(S): * loop (L19-L##) ``` +## Alternatives considered Instead of comments, one could use [`query_name!`](https://topiary.tweag.io/book/reference/capture-names/general.html#query_name) but DX may be different. +## Drawbacks + +Enumerating queries and test files is a many to many complexity but could be +an acceptable complexity for now since it will improve coverage over present state. + +## Backwards compatibility + +`topiary test` is a new subcommand and adds no changes to existing +Topiary behaviour, so there are no backwards-compatibility concerns. + +## Testing strategy + +The subcommand will be exercised against Topiary's existing language queries +as well as [external grammar repositories](https://github.com/blindFS/topiary-nushell). + +## Documentation impact + +The Topiary Book and the generated manpages will need a new section for `topiary test`. + +## Unresolved questions + +* Do we want to apply a specific naming schema for our tests? +* How do we group multiple tests? From b7c3163a841e99f88688d46345f7782f4e48b506 Mon Sep 17 00:00:00 2001 From: Mikhail Katychev Date: Mon, 13 Jul 2026 11:04:27 -0500 Subject: [PATCH 3/3] Update buds/bud-2.md Co-authored-by: Christopher Harrison --- buds/bud-2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/buds/bud-2.md b/buds/bud-2.md index 7aab6b5..70aa0ca 100644 --- a/buds/bud-2.md +++ b/buds/bud-2.md @@ -2,7 +2,7 @@ bud: 2 (TBD) title: "`topiary test` query tester" author: "@mkatychev" -pr: 2 +pr: 10 --- # Proposal for `topiary test`