Feature: Import large data files (CSV, DMZ) as model instances

[amirrza777]

Summary

Currently, problem instances must be manually authored as model files using the
platform's DSL. This does not scale to real-world optimisation problems, where
instances are typically provided as structured data files (CSV, DMZ, etc.).

Proposed Feature

Extend the platform with support for importing data files directly as models:

• Infer a metamodel (classes and properties) from the file structure (e.g. CSV columns)
• Populate model instances from the file content (e.g. CSV rows)
• Integrate this into the existing project/file system so the resulting files are
  immediately usable in the workbench

Initial Scope (CSV)

A first implementation targeting CSV files:

• Each column → a property (type inferred automatically: int, double, boolean, enum, string)
• Each row → a model instance
• Output: .metamodel and .model files written into the project

Related

This is the main feature for the project:

Efficiently supporting large-scale optimisation problem instances in MDEO-Cloud

[szschaler]

@amirrza777 this isn't quite right, as I have mentioned before. Metamodels must continue to be defined separately, as they are part of the general problem formulation. The CSV file only provides the data for a specific problem instance, just like a model would. So, the expected flow is:

1. User defines metamodel, constraints, objective functions, and transformations as usual.
2. User uploads a CSV file with the problem-instance data (see below)
3. User references CSV file instead of model from the config file -- everything else stays as before
4. CSV file is interpreted as an instance of the metamodel provided (or an error is thrown if this interpretation is impossible).

In particular, we do not need to infer a metamodel from the CSV. Instead, we need to map the CSV onto the metamodel or provide an explicit error if that's not possible, Note that a model may actually need to be multiple CSVs, one for each class in the metamodel, with each row representing an object.

[nk-coding]

I'm wondering if you need some metadata which defines the semantics of the CSV (basically the mapping of the CSV to the referenced metamodel parts)
this would provide some flexibility in how the CSV looks like
otherwise, the CSV would need to follow a more fixed structure
I guess both options are fine for me

Also there are two options regarding how to store these: either the CSVs are stored as is, or we have some kind of import which just converts it to a model file.

Basically I see 4options

• store CSV as is, no customizable mapping
• store CSV as is, customizable mapping "near" the CSV
  • this could be in the file metadata, which would be editable via an action dialog
  • this could be in a special col/whatever in the CSV itself
• store the CSV as is, mapping in the config file
  • would not recommend, as this would be not usable for standalone script and model transformation runs without additional work
• convert to model
  • a mapping could be simply done in the import dialog, but is optional
  • this would basically be an action in the metamodel file (import model from CSV)

[amirrza777]

Hello again Niklas @nk-coding, thank you for your prompt reply.

Going ahead with Option C (convert CSV to model on import). Before I wire it into the execution pipeline, wanted to confirm a few things:

1. Integration point: should the user reference the CSV directly in modelPath in the config file (e.g. modelPath: "data/employees.csv"), and the platform detects it's a CSV and converts it automatically at execution time? Or should there be a separate explicit import step that produces a .model file first, which the config then references as normal?

2. Multiple CSVs: for metamodels with multiple classes, the user would need one CSV per class. How should this be expressed in the config? A single modelPath can only point to one file. Should there be a modelPaths list, or a folder path, or something else?

3. Metamodel binding: when the platform converts the CSV, it needs to know which class in the metamodel each CSV maps to. Should this come from the filename (e.g. Person.csv maps to class Person), a naming convention, or something explicit in the config?

4. Error handling UX: if the CSV doesn't match the metamodel (e.g. missing a required column), should the execution fail immediately with an error, or should there be a validation step before the user can even run the optimisation?

5. Niklas's point about metadata: is there any appetite for a mapping file alongside the CSV (e.g. employees.mapping.json) that explicitly maps column names to property names? This would handle cases where the CSV column names don't exactly match the metamodel property names.

Thanks beforehand.

[nk-coding]

regarding 5. I think we need the opinion of @szschaler

regarding the rest, I think specifying the csv in the config is a bad idea, as models are also used as input for manual script and model transformation executions (and you do not want to add support for this at 3 different places)

basically I see two options

• just handle the CSV to model conversion directly during import
  • 1.: "Or should there be a separate explicit import step that produces a .model file first, which the config then references as normal"
  • 2.: the import dialog could simply take multiple files
  • 3.: either specify in the import dialog, or probably better, have the user create the model file and have the action there, because then the model file already specifies the metamodel
  • 4.: the import would simply fail, it does not reach the execution
• specify the csv file in the model file and do the import functionality via plugins (though for the approach above, it could also be implemented via plugins I guess) (see below for details)

basically, a metamodel allows already to import other metamodels (for composition)
we could introduce something similar for model files
so that basically you would have the following code:

using "./metamodel.mm"
import "partA.csv"
import "partB.csv"

this could then be done via a contribution plugin functionality: basically, you would have a separate plugin "model-csv" which

• adds support for the csv language (basically a text-only csv file)
• adds a contribution plugin to the model language so that models know that they now are able to import the csv stuff

the actual import code would still be handled by the model-csv langauge though
this is basically the composition approach also used for the config language, and would allow in the future to add support for more data sources
also a syntax like

import "partB.csv" {
  paramx = "whatever"
} 

could then be used to specify further params

the primary benefit of this approach would be that the csv file stays as it, in particular if you want to combine multiple files you could update one easily without needing to reimport everything again
but its a bit more work to implement

[amirrza777]

After reading through the codebase, here's my understanding of what Option 2 proposed by @nk-coding requires before I start building (I think it is a better approach to proceed with option 2, as looking into long-term prospects is very important)

What needs to be built:

1. Grammar extension: add import "path.csv" as ClassName syntax to .model files, alongside the existing using "metamodel.mm" statement. Each CSV import maps to one class in the metamodel, with the class name specified explicitly.

2. New language-model-csv package: a contribution plugin to the model language that registers .csv as a known file type in the workbench and provides the grammar extension above.

3. CSV-to-ModelData conversion in the model language server: extend modelDataHandler in service-model to detect CSV imports in a .model file, fetch the referenced CSV files, map their rows onto the specified metamodel class, and merge the resulting instances into the ModelData before returning it. The mapping would use column names matched to property names by convention, with an error if required properties are missing.

Open questions before I start:

1. Is import "path.csv" as ClassName the right syntax, or do you have something else in mind? Niklas mentioned a parameterised form like import "file.csv" { paramx = "whatever" } for future extensibility. Should I design for that from the start?

2. Should the CSV import be additive to existing model instances in the .model file (i.e. a .model file can have both hand-authored instances AND CSV imports), or should a CSV-importing model file contain only CSV imports?

3. For the workbench UI, should .csv files be creatable/editable in the workbench file tree, or should they only be uploadable as raw files?

4. Is there an existing contribution plugin interface for the model language (similar to ConfigContributionPlugin) that I should follow, or does this need to be designed from scratch?

Happy to start building once the approach is confirmed.

[szschaler]

Great discussion, many thanks. I agree with the suggested approach where

1. CSV files are uploaded to a project and are kept in the project as is.
2. Model files can explicitly import CSV files from the project via a specific type of syntax (though I perhaps have an alternative syntactic suggestion, see below).

This is great because it allows capturing all the metadata (column to attribute mapping etc) in place (that's @nk-coding's parametrised version). Note that one thing that needs to be managed is references between objects -- in the CSV file these need some sort of row id to identify the object to be referenced. Where references allow for multiple targets, the CSV cell would need to hold a list of such ids.

As for the model file syntax, I suggest it would not be a good idea to allow a mix of directly defined model elements and elements imported from CSV. I would, therefore, recommend a slight change of grammar, so that CSV import looks like this:

import CSV {
  class1 from "file1.csv"
  class2 from "file2.csv"
}

If the import statement is present then the grammar should forbid any other manually created objects.

[nk-coding]

@amirrza777 regarding your questions

1. The syntax @szschaler is fine, parameterization could easily be added after each of the inner liness
2. imho additive (really no reason why we should forbid manually adding another element or so)
3. The easiest way is to start with a regular language plugin for CSV (the grammar should be rather trivial I guess), this automatically brings file create support in the platform. I think upload support would be a nice feature that could be added in a future PR (either via drag and drop, and/or via an action for the model language)
4. No, you need to define a new contribution plugin for that. I would recommend using basically the same approach as for config, where the stuff inside the curly brackets (after import CSV) is basically defined by an internal language for the runtime parts. This is benefitial, as this allows to implement custom completion, validation etc. features. If you use agentic coding you can instruct the agent to basically follow how it is done for the config language, otherwise I would recommend that you have a look at this. Also as a warning, the langium linker integration might get a bit challenging, if you need support for that let me know

[amirrza777]

Hello again, would like to provide quick update on progress:

What's done:

1. .csv is now registered as a first-class file type in the workbench: "Create New CSV" appears in the file creation menu alongside Metamodel, Model, etc.
2. Backend import endpoint is working end-to-end: send a CSV file with a metamodel path and class name, and it generates a valid .m model file in the project. The resulting file opens in the workbench with both text and diagram views, no errors.
3. Edge label bug in model transformation diagrams is fixed and verified (separate PR).

What's still missing:

1. The import CSV { ClassName from "file.csv" } syntax in .model files is built but not yet wired into a proper contribution plugin. It's currently hardcoded into the model grammar, which isn't the right architecture per Niklas's feedback. This needs to be refactored into a proper contribution plugin following the config/config-mdeo pattern.
2. UI upload support: right now users trigger the import via API call. A drag-and-drop or action button in the workbench would make this usable without curl.
3. The CSV plugin needs to be manually registered with the backend after every restart. This should happen automatically on startup.

Questions (@nk-coding @szschaler):

1. For the contribution plugin refactor: modelPlugin.ts currently ignores the _contributionPlugins parameter entirely. Should I extend the model language to actually process contribution plugins, or is there a different integration point you had in mind?
2. For auto-registration: should the backend discover and register plugins automatically on startup based on a config/env variable list, or is there a different pattern you'd prefer?
3. Is UI upload support (drag and drop or action button) in scope for this PR, or should it be a follow-up?
4. For the import CSV { } syntax: should each CSV import line be able to specify a column mapping explicitly (e.g. Person from "employees.csv" { csvColumn -> modelProperty }), or is column-name matching by convention sufficient for now?

[nk-coding]

1. Yes, you need to extend the model langauge for this contribution plugin support
2. There is an env variable for default plugins, you should update that for your new service in the docker-compose / k8s files
3. imho follow-up, but I also don't mind if you do it here tbh
4. there should be a name-based matching by default, the explicit one should always be optional, it would be fine for me to only have the implicit one for this PR

[amirrza777]

Hello Dear Niklas and Steffen, here's a full update on what's been implemented so far.

What's done:

1. import CSV { ClassName from "file.csv" } syntax is now supported in .m model files, with full class reference validation against the metamodel
2. Live diagram rendering from CSV: opening a .m file with the import block renders diagram nodes directly from the CSV data, no separate import step needed
3. Node positions persist across re-renders
4. .csv is registered as a first-class file type in the workbench with "Create New CSV" in the file creation menu
5. A backend API endpoint (POST /api/projects/{id}/csv-import) that materializes a CSV import into a .m file with actual instances, as an alternative workflow
6. The model data handler (model-data) reads CSV imports correctly, so the optimizer execution pipeline works with CSV-backed models
7. Auto-registration of the CSV plugin on startup via DEFAULT_PLUGIN_URLS

What I know is still missing:

• The import CSV { } grammar is currently hardcoded into the model language rather than being a proper contribution plugin following the config/config-mdeo pattern that Niklas described. I started on this but it turned out to be architecturally complex.
• No UI for uploading CSV files yet - users currently create CSV files through the workbench text editor or via the backend endpoint. Drag-and-drop or an import action button would make this more usable.
• Column mapping parameters (Person from "employees.csv" { csvCol -> modelProp }) are not implemented. Currently only name-based matching is supported.
• Git sync / ZIP download feature documented in Issue Cyclic dependency in docker-compose build #3 but not yet built.

Questions before I continue:

1. Should I prioritise the contribution plugin refactor, or is the current approach acceptable for this PR?
2. Is UI upload support (drag-and-drop or action button for CSV files) in scope for this PR or a follow-up?
3. For the column mapping syntax, is name-based matching sufficient for now, or should I implement the explicit mapping before this is merged?
4. What's the priority for Git sync / ZIP download relative to the remaining CSV work?

[szschaler]

Great work @amirrza777! Can you point us to the PR for this?

I'm not sure what is the purpose of the API endpoint that translates .csv files into .m files? I would suggest removing it.

1. Turning this into a proper plugin would be important. We may wish to support other import formats later, so being clean from the start will be important.
2. I would move CSV upload into its own PR -- but please then open a separate issue for it
3. Explicit name mapping could be its own PR -- but please then open a separate issue for it
4. I would like to complete one piece of work before moving to the next, so let's wrap up CSV import (even in its basic form) before we go to Git sync