CSV import: explicit column-to-property mapping

[amirrza777]

Relates to #22, stacked on #23 (needs the CSV import contribution plugin merged first).

What this adds

An optional explicit mapping list after the CSV file path:

import CSV {
    Person from "employees.csv" [
        "fullName" = name
    ]
}

• When a mapping is given, only the listed columns are imported. This is also how a column is intentionally skipped, no separate syntax needed.
• When no mapping is given, columns are matched to properties by name, same as before.
• An unmapped CSV column or an unknown property in the mapping produces a warning rather than failing the import.

Why square brackets

The natural first attempt reused curly braces, matching import CSV { } itself:

Person from "employees.csv" {
    "fullName" -> name
}

This parsed incorrectly: the grammar's newline-aware brace handling only special-cases {/}/(/), and reusing braces here made the parser misread the mapping list's own opening brace as the start of a new class import entry. Square brackets avoid the collision entirely, and = matches the separator already used for property assignments elsewhere in this language.

Draft until #23 is merged, since this branches from it.

[amirrza777]

What's implemented:

import CSV {
Person from "employees.csv" [
"fullName" = name
]
}

• An optional mapping list after the file path, matching CSV columns to metamodel properties explicitly.

• When a mapping is given, only the listed columns are imported. That's also how a column is intentionally skipped, no separate syntax for it.

• With no mapping, columns are still matched to properties by name, same as before.

• An unmapped CSV column or an unknown property named in a mapping produces a warning instead of failing the import.

One note on the syntax: my first attempt reused curly braces to match import CSV { } itself, but that parsed incorrectly because the grammar's newline aware brace handling only special cases curly braces and parens, so reusing them made the parser misread the mapping list's own opening brace as the start of a new class import. Switched to square brackets with =, which sidesteps that and matches the separator already used for property assignments elsewhere in this language.

Verified end to end: explicit mapping applies correctly, unmapped columns are skipped, and the default name based path still works with no regression.

Stacked on #23 since it needs the CSV contribution plugin, still a draft until that one merges.

[szschaler]

No problem with the square brackets, but should this be noted as an issue with the main parsing algorithm, @nk-coding ? I must admit I don't fully understand what the issue is there, perhaps @amirrza777 you can explain in a bit more detail?

[amirrza777]

Happy to go into more detail. The grammar looks like this (csvImportRules.ts):

CsvImportContentRule: "{" many(or(add("imports", CsvClassImportRule), NEWLINE)) "}"
CsvClassImportRule: class, "from", file, optional("{", many(mapping), "}") <- original, ambiguous version

The outer import block and the per-class mapping list are nested: one entry inside a many(or(...)) loop that keeps matching either another entry or a newline until it hits the closing }, and each entry itself optionally has its own { mappings } suffix. When both levels use the same {/}, the parser can't reliably tell whether a given } closes the entry's own mapping list or the whole outer block, since chevrotain's LL(k) parser commits to an interpretation using fixed lookahead rather than backtracking. In practice this showed up as the parser splitting one entry into two: the mapping list's own { got misread as the start of a brand new CsvClassImport, and the AST had a second, bogus import whose "class" was actually one of the mapping tokens, failing to resolve as a reference.

Switching the mapping list to [ ] instead removes the ambiguity entirely, since chevrotain can now distinguish which closing bracket belongs to which nesting level by token type instead of guessing from context.

@szschaler on whether this is a main-parsing-algorithm issue: not really, in the sense that this isn't a Langium/chevrotain bug. It's a known, general limitation of LL(k) parsers: reusing the same delimiter for two nested optional/repeated structures makes the grammar locally ambiguous, and the standard fix in any LL(k)-based grammar (not specific to this codebase) is exactly what I did here, giving each nesting level its own distinct delimiter. @nk-coding might want to weigh in if there's a broader pattern here worth documenting for future grammar additions, but I don't think it needs a separate issue against the parser itself.

[nk-coding]

I really do not have a clue what is going on here
There are more then enough nested curly bracket things in the various DSLs, there should definitely be a way to do this, just have a look how it is done there
I would like to avoid the square brackets for this part of the grammar

[amirrza777]

You're right, looked at how model-transformation handles this: match { pattern } then { block } uses two adjacent brace blocks disambiguated by the keyword between them ("then"), not by using a different bracket type. Applied the same technique here: the mapping list is now introduced by a "with" keyword instead of square brackets:

Person from "file.csv" with {
"column" = property
}

Verified live: two CSV imports in the same block, the first with an explicit mapping, parse correctly now with no ambiguity errors. Pushed.

[szschaler]

Why are we writing grammars in abstract syntax notation rather than as EBNF-y text files? I find the current code utterly unreadable.

[amirrza777]

@szschaler this predates my work (it's the pattern Niklas set up for the whole codebase, e.g. the metamodel/model/model-transformation/config grammars all use it), so he can give the definitive answer, but here's my understanding of why.

A normal Langium project defines its grammar in a static .langium EBNF file, compiled once at build time. This codebase's plugins need something a .langium file can't do: contribution plugins (like this CSV one) get loaded and merged into a host language's grammar (the Model language) at runtime, and which contributions are present isn't known until then. GrammarSerializer/GrammarDeserializer (in language-common) exist specifically to take a grammar built via createRule().as(...) and turn it into a plain JSON-like structure that can be shipped over the wire from a plugin's service and merged into the host grammar's rule set at runtime, something you can't do with a .langium file since Langium compiles that ahead of time into a fixed parser.

So the abstract syntax builder isn't chosen for its own sake, it's a workaround for needing runtime-composable grammars, which is the whole point of the contribution-plugin architecture. I agree it's harder to read than an EBNF file, that trade-off is inherent to supporting plugins this way rather than something specific to the CSV grammar.

[nk-coding]

Basically that abstract syntax is an internal TS DSL for what regular Langium projecta do via its external DSL
it's not that much different, both are transpiled to the Langium AST
for the reason why I did this: exactly as described, to allow for this runtime composition
which would have been possible with a huge hack and the regular DSL
at least for me it was not that hard to read and write after an initial adjustment phase, at least compared to the actual Langium AST json structure
but if this will ever really be an issue in the future, it should be easy to add an Langium external DSL parser via its dependency too