antlr-rust-gen and names for lexer, parser, start rule, and corresponding generated file names.

[kaby76]

Antlr4 and Antlr-ng follow a predictable naming scheme for source code in generated files by using the names declared in the .g4 files. It is followed consistently across all targets.

There are two exceptions to this rule. One is when there are "symbol conflicts", which were never considered systematically until a solution was added to version 4.10. Antlr4 handles this problem by looking up rule names in a table, then generating mutated names. The second is for the Go target and file names. Antlr4 generates file names in snake_case for Go.

While this solution solved the problem of having the generated code work in any target, it introduced a new problem: generating source code with an unpredictable naming scheme. The grammars-v4 repo carefully avoids "symbol conflict" issues and the unpredictable renaming solution that was chosen for version 4.10 by renaming all rule names so that it works on all targets.

For the Go target, it doesn't matter what the file name is because Go's import doesn't require a file name, just the directory where the generated files reside. Rust is different because it uses pub mod ...; syntax.

Examples of grammars and naming for Antlr4/Antlr-ng:

• For Antlr 4.13.2/Java/JSON.g4 with combined gramar name JSON and start rule json, the parser is JSONParser defined in JSONParser.java and the start rule json.
• For Antlr 4.13.2/CSharp/JSON.g4 with combined gramar name JSON and start rule json, the parser is JSONParser defined in JSONParser.cs with start rule json. The naming is Identical across the Java and CSharp targets.
• For Antlr 4.13.2/CSharp/JavaParser.g4 with parser grammar name JavaParser and start rule compilationUnit, the parser is JSONParser defined in JSONParser.cs and start rule compilationUnit.
• For Antlr 4.13.2/Go/AbbParser.g4 with parser grammar name abbParser and start rule module_, the parser is abbParser defined in abb_parser.go (renamed) and start rule module_.

In comparison, Antlr-rust-gen mutates the names of declarations and generated files to PascalCase and snake_case unconditionally.

Examples of grammars and naming for Antlr-rust-gen:

• For Antlr-rust-gen/JSON.g4 with combined gramar name JSON and start rule json, the parser is JsonParser (renamed) defined in json_parser.rs (renamed) and start rule json.
• For Antlr-rust-gen/JavaParser.g4 with parser grammar name JavaParser and start rule compilationUnit, the parser is JavaParser defined in java_parser.rs (renamed) and the start rule compilation_unit (renamed).
• For Antlr-rust-gen/AbbParser.g4 with parser grammar name abbParser and start rule module_, the parser is AbbParser (renamed) defined in abb_parser.rs (renamed) and the start rule module (renamed).

Why does this matter? We should have predictable, consistent naming for generated declarations and file names. Trash, which generates driver applications for Antlr4 grammars for any target (now including Antlr-rust-gen), requires code to implement similar renaming that Antlr-rust-gen does for Antlr-rust-runtime templates, i.e., more code bloat.

[tinovyatkin]

Thanks, Ken. We want Trash to work cleanly with antlr-rust-runtime, and I appreciate the concrete examples.

I agree that our current mapping, although deterministic, is Rust-specific and forces cross-target tooling to reproduce it. I also checked C++, Swift, and PHP: like Java and C#, they preserve recognizer filenames, type names, and parser-rule spelling except where target-specific escaping is required. So this is a real interoperability difference, rather than a parsing-correctness defect.

Our 357/0/0 runtime-suite result does not settle this question. That suite primarily tests runtime semantics and allows target-specific adapters for filenames and start-rule names; Go already has such adaptations, and our Rust harness similarly uses our normalized names. I am not aware of an upstream suite that asserts a common generated-naming contract. If one exists, please point us to it.

We do need to handle compatibility deliberately because these generated names are already public API for consumers. For example, Mehen publishes generated parser crates using module paths and calls such as kotlin_parser, KotlinParser::kotlin_file, c_sharp_parser, and compilation_unit. Changing the default naming policy in place would break source code and regeneration workflows. The same downstream concern discussed in antlr/antlr4#1070 applies here.

Could you help define the minimum contract Trash needs?

• Is it exactly the surfaces named here: generated files, recognizer types, and parser-rule methods?
• Or does Trash also depend on context, listener, visitor, and alternative-label names?
• Must grammar-declared names be callable directly, or would a generator-owned, machine-readable original-to-generated mapping eliminate the duplicated logic in Trash?
• What behavior do you expect for Rust keywords and collisions with existing recognizer methods? Raw identifiers can preserve Rust keywords, but API collisions still need deterministic escaping.

A small Trash command/corpus with the expected generated names would make an excellent acceptance test.

I do not currently see a parse-performance conflict. #247's decision tiers operate on the ATN and are independent of identifier rendering. #252 is intentionally opt-in and removes or migrates rule APIs, so its optimization manifest must describe those changes regardless of naming policy. For #125, any transformation described as tree/API-preserving would also need to preserve the selected naming contract.

Initial sizing: a grammar-spelling-preserving mode for files, recognizer types, and rule methods looks manageable because the core conversions are centralized. Making it the default while preserving existing consumers, or extending compatibility across every generated surface, is a larger versioned migration. Once the required surface is clear, we can scope the implementation and migration path precisely.

[tinovyatkin]

One clarification to my previous reply: interoperability with Trash is desirable, but it is not our primary naming objective. antlr4-rust-gen intentionally presents an idiomatic Rust API that should look and behave like handwritten Rust. That means snake-case files, modules, and methods; UpperCamelCase types with Rust acronym casing; raw identifiers for keywords; and deterministic suffixes for collisions with recognizer APIs.

Preserving .g4 spelling as the default would make Rust callers use APIs such as compilationUnit() and types such as JSONParser or abbParser. Those names are predictable across targets, but they are not idiomatic Rust. Changing the default would also break existing consumers, including Mehen. Emitting both forms as aliases would avoid the immediate breakage, but would permanently duplicate the public API, clutter rustdoc and completion, and introduce another set of collision rules. We do not want to trade Rust caller DX for a simpler implementation in external tooling.

Our preferred interoperability contract would therefore be a generator-owned, stable, machine-readable map from grammar names to generated Rust names and files. It can cover recognizers, rules, contexts, listeners, visitors, and alternative labels to whatever extent Trash needs, without requiring Trash to reproduce our conversion logic. If exact grammar-spelling compatibility has a concrete use case beyond that, we can discuss an opt-in mode, but not as the default public API.

Separately, I may be looking in the wrong place: I checked Trash main and the current head of kaby76/Trash#626. The Rust templates I found still download the antlr4rust/antlr4 fork and depend on antlr4rust = "0.5.1"; I could not find references to antlr-rust-runtime or antlr4-rust-gen. Could you point me to the exact branch or commit, plus a working command, behind “now including Antlr-rust-gen”? That would help ground the contract in the actual integration.

[kaby76]

My latest changes are always on the "dev" branch, and I currently have a PR in the works (kaby76/Trash#626), but it's going to be huge and take a while to complete. The "dev" branch is a catch-all for whatever I feel I currently need to add. The templates for using antlr-rust-runtime (gen) are there (https://github.com/kaby76/Trash/tree/432e85c23a66725954effbfae842f35f6dd6ed0f/src/trgen/templates/OphiRust), but they require a rebuilt trgen that added the "OphiRust" "target". (Note, I didn't like typing "antlr-rust-runtime" as the name of your new generator and runtime, so I just came up with "OphiRust".) I have added a -o switch to output the .errors and .tree files elsewhere so every grammar in grammars-v4 can be tested in parallel and results compared to each other,

Come up with an "interoperability contract would therefore be a generator-owned, stable, machine-readable map from grammar names to generated Rust names and files" format/file with examples, and Trash can read and use that instead.

[tinovyatkin]

Thanks, Ken. After looking at the current OphiRust integration, I think I proposed a generated names file too quickly. It would become a new OphiRust-specific public contract that we would need to version and maintain. If its only purpose is to avoid target-specific naming logic in Trash, I do not think that is the right boundary.

AWS CDK’s jsii toolchain is a useful industry precedent. Its language implementation guide explicitly says that an API authored in TypeScript should be represented “as idiomatic as possible” in each target language. jsii-pacmak converts Python methods, properties, and parameters to snake_case, while AWS CDK deliberately offers language-appropriate Java builder APIs. jsii-rosetta separately translates TypeScript documentation examples into each target language. Uniformity is provided at the semantic level, while each target retains its conventions.

I think the same boundary applies here. Trash already has an OphiRust-specific adapter for templates, dependencies, build commands, and invocation. It already maps module_ to module; applying the deterministic Rust conversion from kotlinFile to kotlin_file belongs naturally in that adapter as well.

We are happy to document the naming rules and provide fixtures so Trash does not need to guess. If there is a generated name that cannot be derived deterministically from the grammar and generator version, please show us that case and we can reconsider whether generator-produced metadata is warranted. Otherwise, I would prefer to keep the generated Rust API idiomatic and put the target-specific adaptation in the target-specific integration.

[kaby76]

Right, there are a lot of sacred cows here: every language has a preferred — or mandated — coding style for the generated code. And Antlr4 .g4 syntax is a sacred cow too, which is often not compatible with the per-language sacred cows. The best place is to just do the renaming in the Antlr4 .g4 file prior to generation using Trash. No need to document for me; I use Claude to read the code and have it devise a nice "coding style guide" for trgen or other app to help in the renaming.

[tinovyatkin]

PR #313 provides another concrete trailing-underscore case for this issue.

The pinned grammars-v4 TOML parser declares:

bool_
    : BOOLEAN
    ;

rust_function_name("bool_") currently passes through
split_identifier_words, which drops the trailing underscore and emits the
public rule method/accessor as bool, while grammar metadata correctly retains
bool_. We initially considered carrying a vendored bool_ -> boolean
rename, but that would put a Rust-target naming workaround in the grammar
instead of the generator.

This specific output compiles because bool is legal in Rust's value
namespace, so it is not a compile blocker. It does show that an intentional
grammar escape suffix is erased and that distinct source names can normalize to
the same Rust stem. The existing allocators disambiguate many resulting API
collisions, but the mapping should be covered and documented centrally rather
than handled by per-grammar edits.

A useful acceptance fixture for this issue would include bool_, module_,
type_, and pairs of rules/labels that normalize to the same Rust identifier,
then assert that:

• generated code compiles without grammar-specific renames;
• metadata preserves every original grammar name;
• every generated public surface receives a stable, unique Rust name; and
• keyword, path-keyword, predeclared primitive, and normalized-name collisions
  follow one documented policy.

[kaby76]

Hit this again with perf testing against all known generators/runtime pairs. I changed the grammar to avoid wholesale parser rule name mutation within this generator (similarly avoided for Antlr4 and Antlr-ng) by adding the rule compilation_unit : packageDeclaration? importDeclaration* typeDeclaration* EOF; to the java8 grammar, keeping both EOF-terminated start rules (compilationUnit rule), and disambiguating testing by explicitly declaring the entry point for test, and got this when compiling:

error[E0428]: the name `RULE_COMPILATION_UNIT` is defined multiple times
   --> src\gen\java8_parser.rs:160:1
    |
159 | pub const RULE_COMPILATION_UNIT: usize = 31;
    | -------------------------------------------- previous definition of the value `RULE_COMPILATION_UNIT` here
160 | pub const RULE_COMPILATION_UNIT: usize = 32;
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `RULE_COMPILATION_UNIT` redefined here
    |
    = note: `RULE_COMPILATION_UNIT` must be defined only once in the value namespace of this module

FYI: Why java/java8/? It is really an excellent test because it is (1) horribly ambiguous; (2) not only a target-agnostic grammar (definition here: https://github.com/antlr/antlr4/blob/master/doc/target-agnostic-grammars.md), not only a target-language-free grammar, but an action/predicate-free grammar ($ dotnet trash parse -t ANTLRv4 *.g4 | dotnet trash xpath ' not(//(element | lexerElement)/actionBlock)'). After working around the problem mentioned above, here are some perf data for the java/java8:exmples/*.java test suite against the different generator/runtime pairs (N=3 runs, mean ± SEM; methods and explanation of column names here: https://htmlpreview.github.io/?https://github.com/antlr/grammars-v4/blob/master/performance.html).

Grammar	Target	PT (s)	OT (s)	TT (s)	TPS	Post-warmup TPS	Post-warmup Speed Up	Actions
java8	Antlr4ng	0.6527 ± 0.004055	0.004333 ± 0.0003333	0.657 ± 0.004359	274.3 ± 1.764	219.7 ± 2.404	0.5067 ± 0.008819	0
java8	Cpp	0.3128 ± 0.004442	0.005783 ± 8.688e-05	0.3185 ± 0.004423	572.3 ± 8.11	396.3 ± 6.386	0.2367 ± 0.003333	0
java8	CSharp	0.6623 ± 0.02601	0.08155 ± 0.0004313	0.7438 ± 0.02565	270.7 ± 10.17	222 ± 11.14	0.55 ± 0.03055	0
java8	Dart	0.3127 ± 0.002186	0.006667 ± 0.0003333	0.3193 ± 0.002333	572.7 ± 3.844	422.3 ± 5.364	0.3467 ± 0.01202	0
java8	Go	0.399 ± 0.0005774	0.004 ± 0.001155	0.403 ± 0.001	449 ± 0.5774	331 ± 0.5774	0.35 ± 3.925e-17	0
java8	Java	0.3343 ± 0.003844	0.1927 ± 0.00441	0.527 ± 0.008083	535 ± 6.028	542 ± 9.165	1.033 ± 0.01453	0
java8	OphiRust	0.1267 ± 0.002186	0.004667 ± 0.0003333	0.1313 ± 0.002404	1413 ± 25.97	917.3 ± 18.22	0.13 ± 0.005774	0
java8	Python3	3.572 ± 0.01858	0.002396 ± 9.62e-05	3.574 ± 0.01862	49.67 ± 0.3333	36 ± 0	0.33 ± 0	0
java8	Rust	0.323 ± 0.002	0.005 ± 0	0.327 ± 0.002	554.7 ± 3.667	391.3 ± 2.848	0.2667 ± 0.003333	0
java8	TypeScript	0.944 ± 0.001155	0.005333 ± 0.0003333	0.9493 ± 0.001453	189.7 ± 0.3333	155.3 ± 1.856	0.5533 ± 0.02404	0

[tinovyatkin]

Thanks for the repro and the perf data @kaby76 — this is fixed in v0.32.0 (#328).

The upper-snake mangling behind generated constants is lossy (compilationUnit and compilation_unit both map to COMPILATION_UNIT), and the RULE_* constants were the one generated naming surface without collision handling. All constants in a generated module now share one collision-free namespace: the first name keeps the canonical identifier and later colliders get a deterministic _2, _3, ... suffix, matching the existing rule-method dedup. Your modified java8 grammar now generates and compiles out of the box:

pub const RULE_COMPILATION_UNIT: usize = 31;    // compilationUnit
pub const RULE_COMPILATION_UNIT_2: usize = 236; // compilation_unit

paired with compilation_unit() / compilation_unit_2() entry methods. Output is byte-identical for grammars without collisions. On the broader naming-predictability point of this issue: the mangled scheme itself (upper-snake constants, snake_case methods/files) stays — it's what makes the generated source idiomatic Rust — but the dedup is now deterministic and documented, so it can be relied on.

Two smaller notes since you're driving the perf harness:

• For your explicit-entry-point workaround: --entry-rule compilationUnit declares the entry directly (no grammar edit needed), and --prune-unreachable optionally drops what it can't reach.
• If you ever extend the dataset beyond the small examples/*.java files, we have opt-in perf flags worth knowing about. --fixed-lookahead <k> (fully behavior-preserving) compiles decisions provable within k tokens into static dispatch — on java8 the effect is small since 213 of 322 decisions are already LL(1) by default, but it shines on less ambiguous grammars (Thrift goes fully prediction-free). --optimize-precedence-ladders collapses java8's two linear expression chains (conditionalOrExpression→equalityExpression, additiveExpression→multiplicativeExpression) and measures up to 2× on large expression-heavy files in our tests — though it's recognition-preserving only (the parse tree changes shape), so it can't be used where trees are compared across targets. The numbers you published reflect our default configuration.