some "official" grammars-v4 grammars has support for unofficial rust forks

[tinovyatkin]

Analyze antlr/grammars-v4#4926 and antlr/grammars-v4#4925 note that they including Rust folder. Make our generator recognize that folder (particularly Rust/transformGrammar.py) and verify whether we can use it seamlessly or as a minimum we need to emit a warning with recommendation. #4925 even included (AI generated) Rust/c_parser_base.rs - inspect that carefully.

[tinovyatkin]

Investigated against antlr-rust-runtime 174920c8 and grammars-v4 e756f2a2, including actual generation and cargo check probes.

Conclusion

We should not automatically execute the current Rust/transformGrammar.py files. They are trgen + antlr4rust compatibility scripts, not a target-neutral protocol, and their output does not compile with this generator. We can reuse the semantic intent and much of c_parser_base.rs, but not wire either artifact in unchanged.

A no-manual-copy experience is still achievable through declarative target-support bundles and our typed semantic hooks. The generic generator must remain grammar-agnostic; C/Java behavior belongs in bundle data/support modules and parity tests.

What I verified

• C, unmodified: strict generation reports 23 unsupported coordinates: 17 predicates and 6 actions.
• C after the upstream transform: generation succeeds with --actions embedded plus both superclass acknowledgements, but the emitted crate fails with 338 Rust errors.
• Java after the upstream transform: generation succeeds, but the emitted crate fails with 61 Rust errors.
• Java, unmodified: two [[helper]] predicate mappings for IsNotIdentifierAssign and DoLastRecordComponent are enough for strict generation, and the emitted modules pass cargo check.
• C, unmodified: typed helper mappings cover all 17 predicate coordinates. The six actions currently require brittle ATN-state [[coordinate]] overrides; with those overrides the generated modules pass cargo check, and the exact upstream c_parser_base.rs also compiles unchanged as a free module. This proves its core algorithms are reusable, not that the resulting parser is semantically correct yet.

The transformed output fails for structural reasons:

• Both scripts emit the old fork's generated-parser ABI (recog.input, _localctx, JavaParser_*, get_text, get_token_type). Our generated API is intentionally different.
• C predicates contain statement sequences such as let ...; ...; the script assumes its target generator will block-wrap them, while our embedded predicate sites require an expression.
• The C script's second phase patches src/gen/cparser.rs and trgen's generated src/main.rs, adds --nopp, copies a crate-root module, and changes file loading to invoke GCC. antlr4-rust-gen emits recognizer modules and does not own an application's main.rs.
• The old-fork fixes for CParserParserContext and .into_owned() are not applicable to our output.

There is also a real runtime contract gap before C can be faithful through hooks. Named [[helper]] mappings currently work for parser predicates, but parser actions only reach a generic custom_action through coordinate overrides. More importantly, interpreted rules currently return (tree, actions) and replay those actions after the rule completes. C's EnterScope and ExitScope are both inside compoundStatement; replaying both after the body is too late for declarations/predicates inside that scope.

c_parser_base.rs needs an adapter even though it compiles: it uses thread-local global state, requires reset_symbol_table() before each translation unit, hard-codes token numbers, and puts preprocessing in a trgen driver patch. Blindly copying it would also leave the consumer to add mod c_parser_base;, which is the manual step this issue wants to eliminate.

Sources: C transform, C base, Java transform, and the two-phase trgen build template.

Integration plan

1. Detect, do not execute. When a grammar root has a sibling Rust/ directory, inventory it in semantics.json and emit one actionable warning if no supported bundle manifest was consumed. Never silently run arbitrary Python from a grammar checkout. An explicit legacy-script flag could be considered later, operating only on a staged copy.
2. Define a declarative bundle contract. Add a target manifest such as Rust/antlr-rust-runtime.toml declaring semantic-pattern files, Rust support modules, implemented grammar options, and lifecycle requirements. Auto-discovery of this data is safe; generation should stage inputs, copy declared support files, and emit one aggregator module so consumers do not manually wire extra mod declarations.
3. Close the generic parser-action gap. Extend typed helper mapping to named parser actions (EnterScope, EnterDeclaration, etc.) and execute committed actions at their grammar position before later dependent predicates, while keeping speculative prediction side-effect-free. This is generic runtime/codegen work, not C special-casing.
4. Build adapters from the original grammars. Java needs two typed predicate methods and no source rewrite. C should port the upstream symbol-table algorithms into parser-owned hook state, use generated token constants instead of numeric IDs, and reset by construction. Preprocessing should be an explicit generated parse_path/support helper returning errors; it should not silently fall back to raw input when GCC fails.
5. Prove parity before calling it seamless. Gate both bundles with --sem-unknown error --require-full-semantics, compile tests, and oracle comparisons. For C, cover repeated translation units, nested typedef shadowing, concurrent parser instances, preprocessing success/failure, examples/*.c, then the larger c-testsuite/SQLite/GCC groups that desc.xml does not currently run for Rust. For Java, cover examples/**/*.java and the JDK corpus used by the upstream PR.

Recommended split: land discovery/warning first, then the generic named-action/commit-timing support, then Java and C bundle adapters with parity proofs.

[tinovyatkin]

Follow-up after investigating RustPython and Codex's open-source sandbox implementation.

Revised conclusion

The earlier blanket recommendation to "detect, do not execute" is too conservative. We can provide zero-manual-action handling of Rust/transformGrammar.py by executing it automatically in a dedicated OS-sandboxed child over a disposable staged tree. We should still never execute it directly inside the generator process.

RustPython and WASI are not competing layers here: RustPython is the embedded interpreter; the Codex sandbox supplies the capability boundary. Native RustPython alone is not an untrusted-code sandbox (RustPython #4210).

What was verified

• Codex now has a real cross-platform codex-sandboxing Rust library, plus the codex-linux-sandbox and Windows sandbox implementations. It uses Seatbelt on macOS, bubblewrap + seccomp on Linux, and restricted-token/ACL machinery on Windows.
• A pinned Git dependency on codex-sandboxing builds successfully outside the Codex workspace after carrying Codex's root Cargo patches and exact rama-error, rama-macros, and rama-utils prerelease pins.
• A real macOS probe passed: the sandboxed child could read/write its staged directory, while reads and writes to a sibling directory were denied.
• Both this repository and the tested Codex revision use Rust 1.95, so there is no MSRV conflict.
• Making the dependencies optional under our codegen feature isolates runtime-only consumers, although the current Codex dependency graph is large.

There is one distribution constraint: optional Git dependencies do not survive cargo package for our current crates.io package. The packaging probe rewrote Codex's 0.0.0 Git dependencies to registry dependencies and failed because those versions are not published. A Git dependency is therefore fine for a prototype or Git-distributed generator, but the crates.io release needs one of: an upstream-published Codex crate, a small published wrapper/fork, or vendored/extracted sandbox modules.

Script-specific staging requirements

The two current grammars-v4 scripts use re, shutil, sys, glob, and pathlib. They rename .g4 files to .bak, so a single writable grammar file is insufficient: stage a disposable grammar directory and expose only that tree as writable.

The C script may also copy c_parser_base.rs into src. The staged tree should include an empty src/, but the transform phase should harvest only approved .g4 outputs. Support Rust modules should continue through the declarative bundle path from the earlier plan rather than being accepted as arbitrary transform output.

Integration plan

1. Add a hidden self-reexec mode to antlr4-rust-gen. The parent stages inputs and launches the same executable as the sandboxed child; the child embeds RustPython and runs the discovered script. This keeps distribution to one generator executable and requires no system Python.
2. Give the child read/write access only to the staged tree, read-only access to its executable/runtime necessities, and no network. Add a hard wall-clock timeout, bounded stdout/stderr, and OS resource limits where available.
3. Run the transform before grammar parsing, validate that all changed/created paths remain under the staging root, and copy back only the expected transformed .g4 files. Ignore .bak, src/, and other incidental outputs.
4. Fail closed with one actionable diagnostic if the sandbox cannot be established. Do not silently fall back to native unsandboxed execution.
5. Run both actual C and Java scripts under RustPython as the compatibility spike, then add hostile-script tests for outside-tree reads/writes, networking, subprocesses, infinite loops, and oversized output.
6. Preserve the generic boundary: discovery and sandbox execution are codegen infrastructure; no C/Java grammar names, rule names, or special cases belong in the generator.

Two backend details need resolution during the spike:

• Codex's macOS :minimal platform bootstrap currently grants generic /tmp writes. Strict "staged tree only" execution needs a small upstream option or a pinned patch that removes that scratch grant.
• Linux requires the helper plus bubblewrap. For zero friction we must package the helper/bubblewrap path as Codex does, rather than require users to install bwrap manually. Windows still needs an equivalent end-to-end probe.

Finally, generation-time sandboxing protects the consumer's machine from transform side effects; it does not make the resulting grammar trusted. A transform can inject embedded Rust that is later compiled, so the grammars-v4 revision remains part of the trusted input and should be pin-able/auditable.

[tinovyatkin]

Follow-up: a simpler trusted-source-first path is viable before we solve untrusted-script sandboxing. The contract should be: explicitly approve execution of the grammar's target support, then finish the entire integration in the same generator invocation with no manual transform, file copying, flags, or module wiring.

First integration path

1. Detect and identify the support bundle. When a sibling Rust/ directory exists, inventory its executable files and Rust support modules, and derive a stable source identity from the repository/revision plus content fingerprints.
2. Stop before execution and ask explicitly. An interactive invocation should offer trust-once, trust-this-exact-revision, trust-this-repository, or abort. A persisted decision must record its scope. Non-interactive generation must fail with the exact fingerprint and the explicit trust flag/config entry needed; it must never infer trust from the directory name or silently execute.
3. Continue the same invocation after approval. Stage the entire grammar directory, because the current scripts rename .g4 files and expect sibling paths. Run the script in a separate child using bundled RustPython, so no system Python or user-run preparation is required. For this first path, the grammar is trusted: staging protects the original checkout and makes outputs auditable, but is not presented as a security sandbox for the host.
4. Consume the transformed grammar automatically. Select the transformed .g4 files, enable embedded actions, derive the applicable parser/lexer superClass acknowledgements from grammar metadata and bundled support, and require fully generated parser bodies. Unknown legacy API forms must fail during generation rather than producing a broken crate later.
5. Wire shipped Rust support automatically. Copy/include applicable Rust/*.rs support modules, emit an aggregation/re-export module, and relocate integration paths such as crate::c_parser_base so the generated output resolves them without consumer mod declarations or file copies.
6. Preserve driver behavior hidden in script patches. The current C script expresses part of its contract by patching trgen's generated main.rs, not only by transforming the grammar. Since this generator emits modules, seamless support requires an equivalent generated facade or a minimal generic support convention. For C that must cover reset_symbol_table() before each translation unit, raw-input versus preprocess_input()/path parsing, and validation of the support file's hard-coded token IDs against generated token metadata. Merely compiling the parser modules would not complete the integration.
7. Gate the bundle end to end. Require strict semantics, generated-only coverage, generated-crate compilation, and Java/C behavioral parity before calling the bundle supported. Any future grammar revision that changes the fingerprint or introduces interpreted fallback must return to the trust/compatibility gate.

Relationship to #266 and #267

#267 is the critical transformed-grammar compatibility prerequisite. On the pinned grammars-v4 snapshot, its scope covers every category behind the observed 338 C and 61 Java compile errors: recog.input.la/lt, token getters and legacy token constants, _localctx/context accessors, and statement-bearing predicates. I did not find another embedded-ABI category requiring a fourth issue.

#266 is valuable but is not a blocker for this exact trusted-transform path if we enforce generated-only output. The C transform emits embedded Rust actions, and the current probe generated bodies for all 117 C parser rules, so those actions are emitted at their grammar positions. We must fail rather than silently use an interpreted body where action replay would be late. #266 remains necessary for the untransformed typed-hook path and for making interpreted fallback action timing generally correct.

Therefore #266 and #267 cover the generic codegen gaps currently exposed, but they do not by themselves provide zero-friction adoption. The discovery, trust UX, staging/execution, superclass selection, support-module wiring, generated facade, preprocessing/reset lifecycle, and end-to-end acceptance checks above remain the implementation owned by this parent issue. No additional sub-issue is indicated by the current investigation.