Initial infrastructure for types, parser, and wasm-tools component commands

[jeremyg484]

The initial scaffolding of the types, parser, and wasm-tools modules outlined in the project plan are added, and work is started on parsing of Wasm Component binaries.

To facilitate development of the parser, some initial infrastructure has been included to execute .wast tests from the Component Model spec repo (https://github.com/WebAssembly/component-model/tree/main/test). The tests from a given .wast file are executed in JUnit by running the component-model-aware wasm-tools json-from-wast command, parsing the JSON output with Jackson, and managing the output binary .wasm files via the in-memory filesystem. If a test case fails, its .wasm binary is written to the actual filesystem to facilitate analysis and debugging. This is a bit different from how .wast tests are being executed via a Java code generation approach in the main Endive project. It was chosen as a more lightweight way to facilitate incremental inclusion of the tests, and to give flexibility in how the new Component Model functionality is driven. This approach could be re-evaluated in the future to better align with the approach of Endive itself, but this felt like a quicker way to get up and running.

A shell script has been included (update-spec-tests.sh) for syncing the .wast test cases from the spec repo. Currently they are not stored in this Git repo, as their download location is included in .gitignore. This requires manually running the script once in order for the Maven build to then succeed.

Initially just a subset of the test cases from /wasm-tools/types.wast have been enabled. This has driven the initial steps in building out the ComponentParser, with a subset of the <core:type> definitions from the spec able to be parsed successfully. Parsing of valid components is tested by comparing the expected type structure for a given test with the actual types constructed by the parser, using AssertJ for convenient deep object structure comparison with nice error messages in the event of a failed comparison. The long-term goal for this project would be to eventually pass the entire suite of tests, indicating complete spec compliance.

The <core:type> section parsing was chosen as a starting point because it relies on re-used binary definitions from the core Wasm spec, thus we are able to also re-use the existing parsing infrastructure of Endive. However, most of the methods for parsing these substructures in Endive's Parser are private, so for now the needed methods have been copied to a separate CoreParser class.

Some known outstanding questions to be answered before merging this work:

1. Should we just go ahead and store a copy of the downloaded .wast tests in this repo to keep things simple? Or should we instead add a step to either CI or the Maven build itself to execute the shell script (or use some other method) to sync the files from the spec?

2. Are we content with bluntly copying the needed core parser methods from Endive, or should we consider making those methods accessible in Endive's parser?

[andreaTP]

Hey @jeremyg484 , looks really a great start thanks a lot for jumping on it!

I'm a bit swamped those days, but I can pick up the automation around testing infrastructure etc., don't worry too much about it.

For the moment, would it be possible to have a script that copy the wast files from/to a gitignored folder?
I'd try to avoid committing them when possible to keep the PR readable by humans.

[jeremyg484]

For the moment, would it be possible to have a script that copy the wast files from/to a gitignored folder?
I'd try to avoid committing them when possible to keep the PR readable by humans.

Oh, strange, I though I had already done that but I see now that the .wast files are still showing up in the PR. I must not have totally synced my branch correctly. I'll get that taken care of shortly.

[jeremyg484]

Ok @andreaTP should be cleaner now, sorry about that. I think I had mistakenly run the download script from the wrong working directory when I was testing the updated Readme instructions. To avoid that mistake, I've moved the script into the root of the repo and simplified the build instructions accordingly.

[andreaTP]

Great first step, thanks @jeremyg484 , I'll add a couple of commits to fix trivial aspects and is ready to go.

[andreaTP]

LGTM, I hope you don't mind me pushing on top of your branch, it's just faster this way.

Amazing job, good to go as a first step forward!

[jeremyg484]

LGTM, I hope you don't mind me pushing on top of your branch, it's just faster this way.

Don't mind at all!

Amazing job, good to go as a first step forward!

Awesome, thanks. More to come!