Skip to content

Add battery electrolyte densities benchmark - #778

Open
KMNitesh05 wants to merge 1 commit into
ddmms:mainfrom
KMNitesh05:add-battery-electrolyte-densities
Open

Add battery electrolyte densities benchmark#778
KMNitesh05 wants to merge 1 commit into
ddmms:mainfrom
KMNitesh05:add-battery-electrolyte-densities

Conversation

@KMNitesh05

Copy link
Copy Markdown

Pre-review checklist for PR author

  • I've confirmed the contribution guidelines.
  • I have reviewed and understand all AI-generated code in this PR.
  • I have added human-written tests for the new logic.
  • I have properly cited any upstream algorithms or libraries the AI utilized.
  • I have disclosed significant AI tool usage in the PR description.

Summary

NPT density benchmark for 25 battery electrolyte systems, compared against experimental densities at 298.2 K. Six neat glyme and carbonate solvents (DME, DEGDME, TEGDME, PC, DEG, DMC) plus 19 electrolytes formed from NaPF6, NaOTf, NaTFSI and KPF6 at 0.1, 0.5 and 1.0 M. Systems range from roughly 500 to 2200 atoms.

Because the same salts recur across several solvents and concentrations, the benchmark separates general liquid-density performance from the harder question of whether a potential handles concentrated ionic environments.

Structures and reference densities are published at https://github.com/KMNitesh05/sodium-ion-battery-electrolyte-dataset, and the underlying paper is arXiv:2603.20183.

Protocol: NPT with the isotropic MTK barostat at 298.2 K and 1 atm, 1 fs timestep, 150 ps per system. The first 50 ps is discarded and the remaining 100 ps averaged. Metrics are MAE, RMSE and MAPE.

Deliberate differences from liquid_densities

Four places where this diverges from the benchmark it is modelled on. All are one-line changes if you would rather I matched upstream:

  1. D3 is off (ADD_D3 = False). The reference densities come from a dispersion-inclusive OMol25-trained potential, so a runtime D3 correction on top would double-count. One consequence worth flagging: the -D3 display suffix from build_dispersion_name_map will be misleading for models not trained on dispersion. Happy to follow your preference on how that is handled.
  2. 150 ps per system rather than 1 ns, as described in Battery Electrolyte Density Benchmark (Na-ion and Zn-ion) #358. The inputs arrive already NPT-equilibrated with a potential of comparable quality, so the long re-equilibration is unnecessary.
  3. MAE and RMSE bad thresholds of 0.1 g/cm³ rather than 0.4. These densities span 0.861–1.291 g/cm³, so a 0.4 g/cm³ error is a 30–45% miss; at that scale the normalised score saturates near 1.0 for every model and stops discriminating.
  4. Densities are recomputed from the trajectory cells rather than parsed out of the text log. The log format still matches liquid_densities field-for-field, so the log-parsing route remains available if you prefer consistency across the two benchmarks.

AI tool usage

The code in this PR was drafted with AI assistance (Claude), using liquid_densities as the reference implementation, and reviewed by me before submission. The AI also generated the packaging script in the dataset repository that writes exp_density and exp_temperature into the extxyz headers.

No external algorithms were introduced. The MD driver is ASE's IsotropicMTKNPT, MAE and RMSE come from ml_peg.analysis.utils.utils, and the decorators/callbacks used are existing ML-PEG ones. mape is a five-line helper defined locally in the analysis module because analysis/utils/utils.py provides mae and rmse but not MAPE — happy to move it into the shared utils if you would prefer it there.

Linked issue

Part of #358.

This PR delivers the 25 Na-ion systems only. The 23 Zn-ion systems also scoped in #358 belong to a dataset owned by ESRA whose paper is not yet published, so I have deliberately written "Part of" rather than "Resolves" to avoid auto-closing #358 on merge. The benchmark directory is named battery_electrolyte_densities so the Zn systems can be added later without a rename. Happy to split #358 into two issues instead if that is tidier.

Note also that #358 lists the category as Physicality, whereas this code sits under molecular_dynamics alongside liquid_densities and water_density. I think molecular_dynamics is the better home for an NPT density benchmark, but say the word and I will move it.

(#773 was a duplicate issue I opened by mistake and have now closed.)

Progress

  • Calculations
  • Analysis
  • Application
  • Documentation

All four stages are implemented, but no production results exist yet — the benchmark has only been exercised with the mock model. The analysis and app stages are a first pass modelled closely on liquid_densities; @joehart2001 offered to help with those two, so please do take them over or rewrite as you see fit.

Testing

Only the mock model so far. I do not yet have GPU allocation for 25 systems × N models, so I would welcome guidance on how you would like the production runs done.

Verified end to end against stand-in configurations constructed to match the dataset's cell vectors and reference densities exactly:

  • reference densities written into the extxyz headers round-trip through ase.io.read
  • system_id 0–24 resolves to the expected systems, and the count check catches a mismatched dataset
  • velocity seeding produces a sensible temperature with zero net momentum, and is a no-op when momenta are already present
  • MAE, RMSE and MAPE all recover ~0 from trajectories centred on the reference values
  • MAPE returns exactly 10.0 for predictions uniformly 10% low, and propagates NaN rather than averaging a failed system away
  • a trajectory shorter than the discarded equilibration window returns NaN rather than a misleading number
  • the calc and analysis modules agree on the protocol arithmetic: 1500 frames written, first 500 discarded, 1000 averaged
  • the metrics emitted by the analysis table exactly match the keys in metrics.yml
  • pre-commit run --all-files is clean, including numpydoc-validation

The calc test is marked very_slow, so CI will not attempt the MD itself.

New decorators/callbacks

None. This reuses plot_parity, build_table, plot_from_table_column and struct_from_scatter as they are. The parity plot uses the existing symbol_by argument to distinguish pure solvents from the 0.1, 0.5 and 1.0 M electrolytes.

NPT density benchmark for 25 Na-ion battery electrolyte and neat solvent
systems from arXiv:2603.20183, compared against experimental densities at
298.2 K. Adds calc, analysis and app stages plus documentation.

Input structures are downloaded from
https://github.com/KMNitesh05/sodium-ion-battery-electrolyte-dataset
@ElliottKasoar ElliottKasoar added the new benchmark Proposals and suggestions for new benchmarks label Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new benchmark Proposals and suggestions for new benchmarks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants