GenAI-GreenML is an open-source dataset consisting of curated machine learning repositories collected for research on generative artificial intelligence and sustainable machine learning development. The dataset includes 50 small-scale (<500 MB) GitHub projects focused on tabular and natural language processing (NLP) tasks. Each repository contains both source code and datasets that enable reproducible experiments on model performance, energy consumption, and green coding practices.
This dataset was developed as part of a master’s thesis at the Norwegian University of Science and Technology (NTNU), investigating how large language models (LLMs) can be utilised in producing more energy-efficient machine learning code.
- Repository Information
- Experiment Pipeline
- Adding ML Projects
- Environment Variables Setup
- Adding LLMs
- Generate LLM Code
- Run Projects and Capture Telemetry
- Analyse Results
- Generate Code Iterations (Failed Scripts)
- Analyse Failed Code
- Analyse Errors
- Generate Code Differences
- Analyse Code Complexity (SonarQube)
- 50 open-source ML repositories (tabular and NLP tasks)
- Metadata on repository size, task type, and primary programming language
- Experiment design and evaluation scripts for code generated by LLMs
- Benchmarking tools for predictive performance, execution time, energy consumption, robustness, and cyclomatic complexity
The dataset supports reproducible research on:
- LLM-based code generation and refactoring
- Sustainable and energy-aware machine learning practices
- Comparative benchmarking of original vs. AI-generated (assisted and autonomous) implementations
If you use this dataset in academic work, please cite it as:
Bjørnevik, E. (2025). GenAI-GreenML: A Dataset for Evaluating Generative AI in Green Machine Learning Code. NTNU, Department of Computer Science.
The overall experiment pipeline is illustrated in the figure below. The following sections describe the individual scripts and stages included in the pipeline.
The experiment evaluates three different implementation conditions:
- Original: Human-written machine learning implementations collected from GitHub repositories
- Assisted: LLM-generated refactoring based on the original implementation and additional optimisation instructions.
- Autonomous: Fully LLM-generated implementations created from repository context and task descriptions without access to the original implementation
To ensure consistent and reproducible execution across all conditions, the original human-written scripts required minor standardisation changes before inclusion in the experiment pipeline. These modifications were intentionally minimal and designed to preserve the original functionality while enabling automated execution and telemetry collection. Typical changes included ensuring accuracy were explicitly printed to stdout, removing interactive inputs, removing visualisations and plotting functionality, and disabling file-writing and model persistence operations. The figure below illustrates the information provided to the LLM in each prompting condition.
To add a new ML project run the following from project root:
./scripts/import_repo.sh <repo_url>
If the repository includes multiple python files, rename the correct file to original.py
Run from repo root to create API.env:
cat <<'EOF' > API.env
# Gemini
GEMINI_API_KEY=
# OpenAI (ChatGPT/Codex)
OPENAI_API_KEY=
# Claude
export ANTHROPIC_API_KEY=
# Groq
export GROQ_API_KEY=
EOFAdd your API keys and source API.env:
source API.env
Add LLM APIs in scripts/APIs and make sure the module expose:
generate_code(mode, source_code, dataset_headers="", exampleRowDataset="", datasetPath="", projectContext="") -> str
Then in generate_llm_code.py:
-
Update DEFAULT_ALL_LLMS on line 75 with the LLM name.
-
Register the module at the bottom of the function load_llm_clients.
Add a line in the registry section starting on line 509:
registry["newllm"] = _try_load("newllm", "newllm_api")
Finally, add API key to API.env and source from project root:
source API.env
Groq is a free LLM API but is currenlty commented out because of not satisfactory performance. If wanted, uncomment where 'groq' appears in generate_llm_code.py.
To generate LLM code using the ML projects in repos, run the following from project root:
cd scripts
source venv/bin/activate
python ./generate_llm_code.py
deactivate
cd ..
CLI flags can be used together.
To choose specific LLMs to run:
python ./generate_llm_code.py --llms <llm1_name llm2_name>
To only run one generation mode, i.e., assisted or autonomous:
python ./generate_llm_code.py --modes <mode>
To overwrite existing generated code:
python ./generate_llm_code.py --force
Generate specific ML projects:
python ./generate_llm_code.py --project-regex <directory_name>
To turn off original_telemetry generation:
python ./generate_llm_code.py --no-original-telemetry
To stop after processing N projects:
python ./generate_llm_code.py --max-projects <N>
When using Gemini Free Tier, Gemini needs to pause every 10th request:
python ./generate_llm_code.py --gemini-pause
Note: there exists more flags, check the code if interested.
To delete old generated code, i.e., files with prefix GENAIGREENML, run the following from project root:
./scripts/delete_generated_ml.sh
This scripts verifies that all ML projects runs. You can skip this check for now, but it exists if there are any problems with the telemetry script below. Note: it only runs the original python code in each ML projects, i.e., it skips if there are multiple python files. That also means, any plots or visuals will be shown.
./scripts/runability_check.sh
From project root:
python scripts/run_ml_projects.py
See output in the results folder.
Flags can be used together.
Only process projects whose directory name matches this regex:
python scripts/run_ml_projects.py --project-regex <regex>
Timeout per script in seconds:
python scripts/run_ml_projects.py --timeout-s <s>
Stop after processing N projects
python scripts/run_ml_projects.py --max-projects <N>
Note: there exists more flags, check the code if interested.
The analysis compares assisted and autonomous LLM code against the original code on accuracy, execution time and energy consumption to show if they have increased, decreased or are equal. The output is saved in results/ with naming convention: <input_file>_analysis.
It also does a quantitative analysis of the data and produces box plots.
From project root:
cd scripts
source venv/bin/activate
python ./analyse_results.py
deactivate
cd ..
showfliersshow/remove outliers from visuals.showmeansshow/remove means from visuals.
Select a specific CSV file from the results folder:
python ./analyse_results.py --results-file results_20260220_135839.csv
From project root:
cd scripts
source venv/bin/activate
python ./analyse_file_sizes.py
deactivate
cd ..
The script move_failed_scripts.py moves ML projects with failed scripts, except for GENAIGREENML* files that ran correctly. The ML projects are copied to a folder failed_generated_code_iteration_N and then the failed scripts are removed from the ML projects in repos.
python ./scripts/move_failed_scripts.py
To analyse the errors occured in generated files. To analyse all iterations use:
python ./scripts/analyse_failed_scripts.py
To analyse a specific iteration:
python ./scripts/analyse_failed_scripts.py --iteration failed_generated_code_iteration_1
Analyse the error analysis from failed iterations.
cd scripts
source venv/bin/activate
python ./error_analysis.py
deactivate
cd ..
Generates code differences comparing original_telemetry with assisted and autonomous implementations by Codex and Gemini.
Output folder in results.
python scripts/code_differences.py
Add SonarQube tokens and keys to API.env, as shown in env.md.
Run the following to generate code complexity analysis:
cd scripts
source venv/bin/activate
python analyse_cyclomatic_code_complexity.py
deactivate
cd ..
To rerun the generation of boxplots without rerunning SonarQube:
cd scripts
source venv/bin/activate
python analyse_cyclomatic_code_complexity.py --reuse-cached-api
deactivate
cd ..