ATL Binding Analyzer — Evaluating LLM-Generated Model Transformation Mappings

Abstract

This tool implements a benchmark framework for evaluating the ability of Large Language Models (LLMs) to suggest structural bindings for model-to-model transformations, using manually authored ATL (ATLAS Transformation Language) rules as ground truth. Given a source and target metamodel, an LLM is asked to propose class-level and feature-level correspondences — a task that a human expert would otherwise perform by writing an ATL transformation by hand. The tool extracts the authoritative binding structure from existing ATL programs, compares it against LLM-generated suggestions, and measures precision, recall, and accuracy at both the class-mapping and structural-feature-binding levels, with explicit treatment of partial matches arising from helper-based bindings.

1. Motivation

Model-to-model (M2M) transformation is a foundational activity in Model-Driven Engineering (MDE). In the ATL language, a transformation encodes correspondences between two metamodels: each matched rule declares which source metaclass maps to which target metaclass, and each binding within a rule declares which source structural feature (or derived expression) is assigned to a target structural feature.

Authoring these correspondences requires deep knowledge of both metamodels and is a labour-intensive, error-prone task. A natural question is whether LLMs can assist engineers in proposing these correspondences automatically, given only the metamodel descriptions and a transformation name. To answer this question empirically, we need a principled evaluation methodology that: (i) defines a ground truth from existing, authoritative ATL transformations; (ii) represents LLM outputs in a comparable interchange format; and (iii) quantifies agreement between the two at multiple semantic levels.

This tool operationalises that methodology.

2. System Architecture

The framework consists of three components.

2.1 Ground Truth Extraction (`ATLBindingAnalyzer`, `MatchingExtractor`)

The ground truth is derived automatically from an existing .atl file. The ATLTransformationParser parses the ATL source into an EMF model using the Eclipse ATL Common library. The MatchingExtractor then traverses the model and produces a structured JSON document — the interchange format — that encodes:

the set of distinct class-pair matches (source metaclass → target metaclass), grouped across all rules that share the same pair;
for each class pair, the set of structural feature matches, each linking a target feature to the set of source-side paths that contribute to it.

The extraction logic resolves helper-based bindings: when a binding expression invokes a context helper (e.g., m.fullName()), the extractor traverses the helper body and collects all self.<feature> accesses, exposing the underlying source-side structural dependencies transparently. This is essential because an LLM responding to a metamodel description will reason in terms of structural features, not helper call chains.

2.2 LLM Output Format (jJodel Suggested Mappings)

LLM-generated suggestions are represented as JSON files in a suggested-mappings format produced by the jJodel model editor. Each file declares a flat array of mappings, where each entry specifies:

kind: "class" or "attribute";
source.class, source.feature: source-side endpoint;
target.class, target.feature: target-side endpoint;
confidence: "high", "medium", or "low" — an LLM self-assessed confidence label;
provider: the LLM provider, such as "GPT", "Claude", or "Mistral".

Each mapping file represents one independent run of the LLM prompt, enabling repeated-sampling analysis.

2.3 Comparator, Baseline, and Evaluator

The evaluation includes both LLM suggestions and an EMFCompare-based structural baseline:

JJodelBindingComparator aligns the ground-truth interchange document with an LLM-generated suggestion file and produces a detailed diff.
EMFCompareBaselineExtractor derives a baseline mapping from source and target Ecore metamodels using EMFCompare.
ATLBaselineComparator compares that EMFCompare baseline against the ATL ground truth.
ATLJJodelDiff exposes two-way ATL-vs-LLM comparison and three-way ATL-vs-EMFCompare-vs-LLM comparison from the command line.
ValidationPerformanceAnalyzer iterates over a validation directory tree, derives a ground truth from each .atl file found, compares it against every co-located suggestion file, and aggregates metrics per provider.

The current validation tree contains eight ATL transformations, and the RQ1 analysis combines repeated LLM runs with one deterministic EMFCompare baseline run per transformation.

3. Comparison Methodology

Comparison proceeds in two stages: class-level matching followed by feature-level matching within each matched class pair.

3.1 Class-Level Matching

A class pair (S, T) is considered a true positive (TP) at the class level if the LLM suggestion contains a class mapping from S to T and the ATL ground truth also contains a rule covering that pair. A pair present only in the ATL ground truth is a false negative (FN); a pair present only in the LLM output is a false positive (FP).

3.2 Feature-Level Matching

For each class pair (S, T) present in both the ATL ground truth and the LLM output, the comparator aligns feature bindings by target feature name. For each matched feature binding, the source-side dependency is classified into one of four categories:

Category	Condition
exact	The LLM-proposed source feature equals the unique ATL source dependency
partial	The LLM-proposed source feature matches one of several ATL source dependencies
target-only	The ATL binding has no extractable source dependency (e.g., constant or unresolvable expression)
divergent	The LLM-proposed source feature is not among the ATL source dependencies

The partial category arises when a binding is mediated by an OCL helper that depends on multiple source features (e.g., a fullName() helper that reads familyFather, name, and familyMother). In this case the LLM can, at best, predict one of the contributing features. Penalising this as a full error would be unfair; treating it as a full match would overstate correctness. The framework therefore applies a configurable partial match weight $w \in [0,1]$ (default $w = 0.5$).

4. Metric Definitions

Let $n_{\text{exact}}$, $n_{\text{partial}}$, $n_{\text{div}}$ denote the counts of exact, partial, and divergent binding matches respectively, and let $n_{\text{only\text{-}gen}}$, $n_{\text{only\text{-}atl}}$ denote bindings present only in the LLM output or only in the ATL ground truth.

4.1 Binding True Positives (with partial weighting)

$$\text{TP}_{\text{bind}} = n_{\text{exact}} + w \cdot n_{\text{partial}}$$

4.2 Binding False Positives

$$\text{FP}_{\text{bind}} = n_{\text{only\text{-}gen}} + n_{\text{div}} + (1 - w) \cdot n_{\text{partial}}$$

The partial contribution is split: the fraction $w$ is credited as correct, while the remaining $(1-w)$ fraction is penalised as a false positive and simultaneously as a false negative (see below), reflecting that the prediction is both imprecise and incomplete.

4.3 Binding False Negatives

$$\text{FN}_{\text{bind}} = n_{\text{only\text{-}atl}} + n_{\text{div}} + (1 - w) \cdot n_{\text{partial}}$$

Note that divergent bindings contribute to both FP and FN because they represent a wrong prediction (FP) for a target feature that the ATL transformation does define (FN). This double-counting is intentional: a divergent binding is simultaneously a spurious claim and a missed correct claim.

4.4 Precision, Recall, Accuracy

Standard information-retrieval metrics are applied uniformly to class and binding counts:

$$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}$$

$$\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}$$

$$\text{Accuracy} = \frac{\text{TP}}{\text{TP} + \text{FP} + \text{FN}}$$

The accuracy formula used here is the Jaccard similarity coefficient (also called Intersection over Union). It is chosen instead of the classical $(TP + TN) / N$ formula because true negatives are undefined in this setting: there is no exhaustive set of class pairs or feature bindings that are known to be absent. Any pair of metaclasses not mentioned by either system is simply outside the scope of comparison.

4.5 Overall Metrics

An overall score is computed by pooling class-level and binding-level counts:

$$\text{TP}_{\text{overall}} = \text{TP}_{\text{class}} + \text{TP}_{\text{bind}}$$

and analogously for FP and FN. This provides a single aggregate indicator of end-to-end mapping quality.

4.6 Micro vs. Macro Aggregation

For multi-transformation, multi-run comparisons the framework reports both:

Micro-aggregated metrics: TP, FP, FN are summed across all runs before computing precision/recall/accuracy. This weights each individual binding or class pair equally regardless of transformation size.
Macro-averaged metrics: precision, recall, and accuracy are computed independently for each run, then averaged. This weights each run equally and is more sensitive to performance on small transformations.

5. Experimental Dataset

The validation dataset currently comprises eight ATL transformations. Each validation subdirectory contains one ATL ground-truth transformation, the source and target Ecore metamodels, and multiple jJodel suggested-mapping JSON files.

Transformation	Source Metamodel	Target Metamodel
`book2publication`	Book	Publication
`Class2Relational`	UML Class	Relational
`Grafcet2Petrinet`	IEC 61131 Grafcet	Petri Net
`Java2Table`	Java source	Database table
`mysql2km3`	MySQL	KM3
`spreedsheet2XML`	SpreadsheetML simplified	XML
`table2spreadsheet`	Database table	SpreadsheetML simplified
`xml2ant`	XML	Apache Ant

The latest RQ1 analysis compares three LLM providers, GPT, Claude, and Mistral, against an EMFCompare structural baseline. The LLM providers are evaluated over repeated prompt runs, while EMFCompare is deterministic. The partial match weight is set to $w = 0.5$ throughout.

6. Results Summary

The table below reports the ALL rows from results/rq1_aggregate.csv, using micro-aggregated overall metrics across the eight transformations.

Provider	Precision	Recall	F1	Accuracy
Claude	0.680	0.725	0.701	0.540
GPT	0.625	0.286	0.392	0.244
Mistral	0.573	0.485	0.525	0.356
EMFCompare	0.571	0.053	0.096	0.051

Interpretation. Claude has the strongest overall balance in the current benchmark. GPT remains comparatively precise but misses many ATL-defined correspondences, while Mistral improves recall relative to GPT at the cost of lower precision. EMFCompare is included as a deterministic structural baseline; its low recall reflects that metamodel similarity alone does not recover most transformation-specific ATL bindings.

The older validation-performance.json file contains a previous four-transformation GPT/Claude-only report. Use the CSV files under results/ for the current RQ1 analysis.

7. Build and Usage

7.1 Prerequisites

Java 11 or newer.
Maven.
Eclipse ATL/EMF/EMFCompare dependencies installed in the local Eclipse p2 pool at ~/.p2/pool/plugins, matching the system-scoped JAR names declared in pom.xml.

Build the standalone JAR with:

mvn package -Dexec.mainClass=org.atlbinding.ATLBindingAnalyzer

The shaded artifact is written to:

target/atl-binding-analyzer-1.0-SNAPSHOT-standalone.jar

7.2 Extract Ground Truth from an ATL File

java -jar target/atl-binding-analyzer-1.0-SNAPSHOT-standalone.jar \
     path/to/transformation.atl [output.json]

7.3 Generate Suggested Mappings with jJodel

Suggested mappings are generated in the jJodel beta UI at https://beta.jjodel.io. The validation JSON files in this repository are exported directly from the Suggested Mappings panel.

For each transformation:

Create or open a jJodel project for the transformation, for example Grafcet2PetriNet.
Import or define both metamodels in the same megamodel. The screenshots show the source Grafcet metamodel and target PetriNet metamodel open as separate tabs.
Open a transformation tab and select the source and target metamodels from the toolbar, for example SOURCE: Grafcet and TARGET: PetriNet.
In the Suggested Mappings panel, choose the model/provider to evaluate, for example Mistral Large.
Click Analyze Metamodels. jJodel lists pending class and attribute mappings with confidence labels (high, med, low) and may mark inherited feature mappings as auto-resolved.
For inspection in the UI, select mappings to move them from pending to to insert; this draws colored links between source and target features. The repository validation files are exported suggestions, so inserting them into the transformation editor is not required for evaluation.
Click Export JSON and save the file in the corresponding validation directory, for example:

src/test/resources/validation/Grafcet2Petrinet/Grafcet_to_PetriNet_suggested_mappings (N).json

The exported JSON must keep the jJodel suggested-mappings shape:

{
  "jjodel": {
    "exportType": "suggested-mappings",
    "schemaVersion": 1
  },
  "transformation": {
    "name": "Grafcet_to_PetriNet",
    "source": "Grafcet",
    "target": "PetriNet"
  },
  "analysis": {
    "provider": "Mistral",
    "model": "mistral-large-latest",
    "modelLabel": "Mistral Large"
  },
  "mappings": []
}

Repeat the workflow for each provider/run. The analyzer groups results by analysis.provider, so keep that field consistent across exported files.

7.3.1 AI Matcher System Prompt

The jJodel AI matcher uses the following system prompt template to generate suggested mappings. The placeholders {{sourceName}}, {{sourceMetamodel}}, {{targetName}}, and {{targetMetamodel}} are filled at runtime with the selected metamodel names and their JSON representations.

You are an expert in model-driven engineering and metamodel transformations.
You generate mappings for JjTL (Jjodel Transformation Language).

Analyze these two metamodels and suggest semantic mappings between them.

## Source Metamodel: {{sourceName}}

{{sourceMetamodel}}

## Target Metamodel: {{targetName}}

{{targetMetamodel}}

## Task

Identify which elements from the Source metamodel should map to which elements in the Target metamodel.
Consider:
1. Semantic similarity (even if names are different)
2. Structural similarity
3. Type compatibility
4. Common modeling patterns

## JjTL Syntax Rules (MUST follow)

conversionHint values MUST be valid JjEL expressions following these rules:
- Equality uses '==' (NOT '===' or '=')
- Inequality uses '!='
- Conditionals: if x == 1 then "a" else "b" (NOT ternary '?:')
- Boolean operators: and, or, not, implies
- Null-safe navigation: parent?.name
- Null coalesce: parent?.name ?? "default"
- Value mappings: true=1, false=0
- Comments use '--' (NOT '#' or '//')

DO NOT use in conversionHint:
- Ternary '?:' -- use if/then/else instead
- Triple equals '===' -- use == instead
- Hash '#' or '//' for comments -- use -- instead
- JavaScript/TypeScript syntax

## Metamodel Rules

- NEVER use abstract classes as target classes -- abstract classes cannot be instantiated
- If the target metamodel has an abstract class with concrete subclasses, create SEPARATE mappings for each concrete subclass
- When mapping to different concrete subclasses, provide a "guardHint" explaining the distinguishing condition

## Response Format

Respond ONLY with a JSON array of mapping suggestions. No explanation, no markdown code blocks, just the raw JSON array:

[
    {
        "sourceClass": "ClassName",
        "sourceAttribute": null,
        "targetClass": "ConcreteClassName",
        "targetAttribute": null,
        "confidence": "high",
        "reason": "Explanation",
        "guardHint": "optional: condition for choosing this target subclass"
    },
    {
        "sourceClass": "ClassName",
        "sourceAttribute": "attrName",
        "targetClass": "ConcreteClassName",
        "targetAttribute": "attrName",
        "confidence": "medium",
        "reason": "Explanation",
        "conversionHint": "sourceAttr.toUpper()"
    }
]

Notes:
- sourceAttribute/targetAttribute should be null for class-level mappings
- confidence should be "high", "medium", or "low"
- Only suggest mappings you are confident about
- Quality over quantity
- conversionHint MUST be a valid JjEL expression (e.g., "name.toUpper()", "value.toString()", "true=1, false=0") or omitted entirely. NEVER put human-readable text or notes in conversionHint -- use the "reason" field for explanations instead. If no conversion is needed, omit conversionHint.
- guardHint is optional -- use it when the same source class maps to different target subclasses to explain the distinguishing condition

## Do Not Map Abstract Classes

Do not generate mapping entries where sourceClass is abstract. Instead, fold the attribute
bindings inherited from abstract superclasses into the mapping rules for the concrete subclasses
that inherit from them.

Example: if State and Transition both inherit `name` from abstract NamedElement, do not
generate a `NamedElement -> X` rule. Instead include `name := name` directly in the
`State -> Place` and `Transition -> Transition` rules.

## Target Attribute Must Exist

Do not generate attribute mapping entries where the target class has no matching attribute.
An entry with `targetAttribute: null` (or any name that does not exist on the target class)
is invalid -- omit the entry entirely rather than mapping to the class itself or inventing
a placeholder. This is a grammar violation of JjTL, not a style preference: emitting an
attribute mapping without a valid target attribute produces a binding with an empty left-hand
side (` := name`), which is unparseable.

Example: if State has attribute `name` and the target class Place has only `tokens`, do
NOT emit `{ sourceClass: "State", sourceAttribute: "name", targetClass: "Place",
targetAttribute: null }`. The correct action is to skip it -- Jjodie generates transformations,
it does not modify the target metamodel.

## Two-Pass Bindings for Cross-Type References

When a feature in the target metaclass is a REFERENCE (not an attribute) whose type is a metaclass
that is itself the TARGET of another mapping rule in the same transformation, that binding requires
two-pass resolution. In the JSON response, set conversionHint to the string:

    "resolve(<sourceFeature>, <TargetType>)"

where <TargetType> is the target metaclass name.

Example: if Transition.nextState has type State and State maps to Place in this transformation, set:
    conversionHint: "resolve(nextState, Place)"

## Container / Parent Bindings

When a feature in the target metaclass is a reference of type T, and the source metaclass is contained
(via a containment reference) in another metaclass that maps to T in the same transformation, generate
a SEPARATE mapping entry with:

    sourceAttribute: "parent"
    conversionHint: null
    reason: "source container maps to <T> via <SourceContainer>-><T> rule"

Example: Transition is owned by State (State.ownedTransitions is a containment), State maps to Place,
and Transition.inputPlace has type Place -> generate an entry with sourceAttribute "parent".

## Unresolvable References

If a reference in the target has no clear mapping (not direct, not two-pass, not parent), include it in
the JSON response with confidence "low" and a reason explaining what is missing.

7.4 Compare a Single LLM Suggestion File Against ATL Ground Truth

First extract the ATL ground truth to JSON, then compare it with a jJodel suggested-mappings JSON file:

java -cp target/atl-binding-analyzer-1.0-SNAPSHOT-standalone.jar \
     org.atlbinding.ATLJJodelDiff \
     --atl ground-truth.json \
     --llm suggested-mappings.json \
     --output diff.json \
     --partial-match-weight 0.5

The comparator also supports a three-way mode that compares ATL ground truth, an EMFCompare baseline, and an LLM prediction in one report:

java -cp target/atl-binding-analyzer-1.0-SNAPSHOT-standalone.jar \
     org.atlbinding.ATLJJodelDiff \
     --atl ground-truth.json \
     --baseline emfcompare-baseline.json \
     --llm suggested-mappings.json \
     --output three-way-diff.json

Programmatic usage:

String atlJson   = /* output of ATLBindingAnalyzer */;
String jjodelJson = /* content of a suggested-mappings JSON file */;
String diffJson  = new JJodelBindingComparator(0.5).compare(atlJson, jjodelJson);

7.5 Run the ValidationPerformanceAnalyzer

java -cp target/atl-binding-analyzer-1.0-SNAPSHOT-standalone.jar \
     org.atlbinding.ValidationPerformanceAnalyzer \
     src/test/resources/validation \
     validation-performance.json \
     0.5 \
     all

Arguments:

Position	Description	Default
1	Path to validation root directory	`src/test/resources/validation`
2	Output JSON file path	stdout
3	Partial match weight $w$	`0.5`
4	Confidence filter (`all`, `high`, `medium`, `low`, or comma-separated)	`all`

Each subdirectory under the validation root must contain exactly one .atl file (the ground truth) and any number of .json suggestion files (LLM outputs). The provider field inside each JSON file's analysis object determines how results are grouped.

7.6 Reproduce the RQ1 CSV Analysis

The RQ1 script generates per-run and aggregate CSV files, including the EMFCompare baseline and attribute/reference splits:

python3 results/rq1_analysis.py

Outputs:

8. Interchange Format

The ATL interchange format is specified in src/test/resources/INTERCHANGE_FORMAT.md. It is a class-first, feature-match-oriented JSON structure deliberately decoupled from ATL rule syntax, designed to be reproducible by any transformation language extractor. The semantic identity of a mapping is the tuple:

sourceClass → targetClass → targetFeature → sourcePaths

Rule names are treated as traceability metadata, not as identity keys.

9. Repository Structure

src/main/java/org/atlbinding/
  ATLBindingAnalyzer.java          — CLI: extract bindings from an ATL file
  ATLTransformationParser.java     — ATL file parser (Eclipse ATL Common)
  MatchingExtractor.java           — Semantic extraction of class and feature matches
  MatchingPrinter.java             — JSON serialisation of TransformationMatch
  JJodelBindingComparator.java     — Diff engine: ATL ground truth vs. LLM suggestion
  ATLBaselineComparator.java       — Diff engine: ATL ground truth vs. EMFCompare baseline
  ATLJJodelDiff.java               — CLI: two-way and three-way comparison reports
  EMFCompareBaselineExtractor.java — Extract baseline mappings from EMFCompare
  ValidationPerformanceAnalyzer.java — Batch evaluation across multiple transformations
  model/                           — Domain model (TransformationMatch, RuleMatch, …)

src/test/resources/validation/
  book2publication/                — ATL file, Ecore files, LLM suggestion files
  Class2Relational/                — ATL file, Ecore files, LLM suggestion files
  Grafcet2Petrinet/                — ATL file, Ecore files, LLM suggestion files
  Java2Table/                      — ATL file, Ecore files, LLM suggestion files
  mysql2km3/                       — ATL file, Ecore files, LLM suggestion files
  spreedsheet2XML/                 — ATL file, Ecore files, LLM suggestion files
  table2spreadsheet/               — ATL file, Ecore files, LLM suggestion files
  xml2ant/                         — ATL file, Ecore files, LLM suggestion files

results/
  rq1_analysis.py                  — Reproduction script for RQ1 CSV results
  rq1_per_run.csv                  — Per-run metrics
  rq1_aggregate.csv                — Aggregate metrics

validation-performance.json        — Older four-transformation GPT/Claude report
validation_table.tex               — LaTeX table of validation-set characteristics

Name		Name	Last commit message	Last commit date
Latest commit History 7 Commits
.vscode		.vscode
results		results
src		src
.gitignore		.gitignore
README.md		README.md
dependency-reduced-pom.xml		dependency-reduced-pom.xml
pom.xml		pom.xml
validation-performance.json		validation-performance.json
validation_table.tex		validation_table.tex

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

ATL Binding Analyzer — Evaluating LLM-Generated Model Transformation Mappings

Abstract

1. Motivation

2. System Architecture

2.1 Ground Truth Extraction (ATLBindingAnalyzer, MatchingExtractor)

2.2 LLM Output Format (jJodel Suggested Mappings)

2.3 Comparator, Baseline, and Evaluator

3. Comparison Methodology

3.1 Class-Level Matching

3.2 Feature-Level Matching

4. Metric Definitions

4.1 Binding True Positives (with partial weighting)

4.2 Binding False Positives

4.3 Binding False Negatives

4.4 Precision, Recall, Accuracy

4.5 Overall Metrics

4.6 Micro vs. Macro Aggregation

5. Experimental Dataset

6. Results Summary

7. Build and Usage

7.1 Prerequisites

7.2 Extract Ground Truth from an ATL File

7.3 Generate Suggested Mappings with jJodel

7.3.1 AI Matcher System Prompt

7.4 Compare a Single LLM Suggestion File Against ATL Ground Truth

7.5 Run the ValidationPerformanceAnalyzer

7.6 Reproduce the RQ1 CSV Analysis

8. Interchange Format

9. Repository Structure

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

2.1 Ground Truth Extraction (`ATLBindingAnalyzer`, `MatchingExtractor`)

Packages