Nuclear Gradients - #124
Conversation
|
This pull request introduces 1 alert when merging 6b92a09 into 6e7ac34 - view on LGTM.com new alerts:
|
|
This pull request introduces 1 alert when merging 7136d03 into 6e7ac34 - view on LGTM.com new alerts:
|
|
Awesome 👍 |
|
This pull request introduces 1 alert when merging 844cc1c into 6e7ac34 - view on LGTM.com new alerts:
|
|
This pull request introduces 1 alert when merging 10980bf into 6e7ac34 - view on LGTM.com new alerts:
|
|
This pull request introduces 1 alert when merging 3f82618 into 6e7ac34 - view on LGTM.com new alerts:
|
3f82618 to
b92ee1b
Compare
|
This pull request introduces 1 alert when merging b92ee1b into 041cdcb - view on LGTM.com new alerts:
|
|
This pull request introduces 1 alert when merging 1a3c802 into 041cdcb - view on LGTM.com new alerts:
|
1717c34 to
54a9181
Compare
|
This pull request introduces 1 alert when merging 54a9181 into e4ca2ed - view on LGTM.com new alerts:
|
6e67a9d to
365cd5a
Compare
|
This pull request introduces 4 alerts when merging 92e7535 into e4ca2ed - view on LGTM.com new alerts:
|
|
This pull request introduces 4 alerts when merging df675fa into e4ca2ed - view on LGTM.com new alerts:
|
f3cd5f2 to
c16ff2b
Compare
|
Of course tests need to work and all, but I wonder if a good first aim is to just get the things already implemented merged into master and release a new version? |
|
The tests are horrible to implement... ensuring highly converged eigenvectors is difficult 😞 but in principle all the implemented gradient methods are correct 👍🏻 |
|
Not sure if this is a good suggestion, but maybe we could run a couple of small molecules and save the results in a text file for pytest to compare agains (this is more or less how it's done in veloxchem). I can try out the cvs-adc gradients and find an optimal/good enough setup (I guess the question is of comparing analytical vs. numerical gradients) |
|
The generation of finite-difference reference data works fine already, for both generic and CVS-ADC 😊👍🏻 |
Can't you lower to tolerance or introduce perturbations in the molecule / orbitals to ensure they are sufficiently distinct (to avoid rotations in invariant subspaces etc.) |
|
You mean the convergence tolerance/ You mean geometry perturbations to break symmetries? Yes, I can try that for sure 👍🏻 |
Yes. It's not amazing, but it will at least catch forgotten factors of two or switched signs. |
c16ff2b to
d52ac16
Compare
|
@iubr, the CVS-ADC2-x don't seem to be correct right now, at least I get an error for H2O of ~1e-5 (compared to 5-point finite differences). Maybe I made some mistake while cleaning up the code... Which accuracy could you obtain in your tests? I'll try to run your original branch again at some point to get to the bottom of this. |
|
The ~1e-5 is similar to what I get with my code (in comparison to the 2-point finite difference numerical gradient). I checked it today with a 5-point finite difference and it is a little smaller, but still same order (1e-5). So if there is a bug, it would be in my module as well. |
I converge the SCF to 1e-12, and the ADC procedure is converged to 1e-10 in the energy. Given the fact that all other gradient methods so far can achieve at least 1e-7 accuracy, I'm convinced that one can achieve the same for CVS-ADC2-x . I'll try to add canonical ADC2-x gradients now and see how accurate one can get. |
|
Thanks a lot! |
lgtm and coveralls (hopefully) more fdiff fix
hijacked AmplitudeVector for CVS-ADC orbital response. CVS-ADC0 working! cvs-adc1 working
added cvs-adc2 2PDMs,started orbital response (need to check if the RHS is correct) cvs-adc2 gradient working. cvs-adc2x working
… vs. cvs-adc rhs. fixed 1/sqrt(2) additional comments, RHS switched to using symmetrise and antisymmetrise to construct density matrices
c1c0db5 to
65bfcc3
Compare
Performance Optimization: 2-RDM / ERI Derivative Contraction (PySCF Backend)ProblemThe current gradient implementation has two major memory bottlenecks in
Peak memory: ~5 × N⁴_AO × 8 bytes (at nao=100 → ~4 GB, at nao=150 → ~20 GB). How PySCF handles this (MP2/CCSD gradients)PySCF uses a two-part strategy that avoids full N⁴ tensors entirely: Part 1: Correlated 2-RDM → shell-batched direct contraction
Part 2: Separable 1-RDM contributions → J/K gradient machinery
Proposed Optimization StrategyNew method: Combined approach — block-wise half-transform + shell-batched ERI contraction:
Memory target: O(nocc² × nao²) + O(shell_batch × nao³) instead of O(nao⁴). Implementation Plan
Reference
|
* chore: open PR for issue 214
* implement PySCF gradient memory contraction and profiling script
* remove unused shell-batched gradient path
* extract spin sub-blocks for direct gradient AO-pair transform
Add a Tensor.export_block primitive that exports a dense rank-N
sub-block over a half-open per-axis window without materialising the
full tensor, mirroring the btod_export scatter (inverse permutation)
so it stays correct on antisymmetric same-space blocks.
Rewrite to_ao_pair_density to extract only the spin sub-block each
spin case needs via export_block and contract it with the compacted
non-zero-row coefficients, instead of densifying full zero-padded TPDM
blocks and full rank-4 AO tensors. Spin sub-blocks are pre-extracted
once and reused across pair chunks; the packed density is written once
per chunk to keep the out-of-core path I/O-efficient.
Matches the full-AO path to machine precision (RHF/UHF/CVS, cc-pVDZ/
pVTZ, memory and hdf5 storage). Peak RSS at cc-pVTZ drops ~25 percent
(in-memory) to ~37 percent (out-of-core), with the win growing with
basis size. Adds export_block and spin-subblock regression tests.
* address review findings: bound hdf5 memory, drop outcore alias, remove dead code
- hdf5 storage now picks a bounded default pair_chunk_size so the
in-memory working buffer no longer grows to the full (nao, nao, npair)
- remove the undocumented 'outcore' alias; validate storage against an
explicit {memory, hdf5} set with a matching error message
- guard the hdf5 cleanup so it cannot mask an in-flight exception
- drop dead code: numpy export_block fallback, redundant provider
ao_pair_density_from_dense wrapper, unused _empty_gradient keys,
unused cblk_size in Tensor::export_block
- fold duplicated direct/exchange spin-case loops; drop redundant copy;
hoist loop-invariant np.any(offdiag)
- fix flake8 blank-line/E402 violations
- add tests for hdf5-vs-memory equivalence, scratch cleanup, bounded
default chunk, and the new validation guards
* reframe TPDM AO-pair packing docstrings as backend-neutral
Describe the lower-triangular symmetric AO-pair packing as a generic
convention rather than a PySCF assumption; PySCF aosym="s2kl" is now
mentioned only as one concrete consumer example.
* rename gradient TEI->ERI and add nuclear gradients docs
Rename the two-electron-integral terminology to ERI throughout the
nuclear gradient API to match adcc's established convention: the
eri_contraction / eri_shell_chunk_size / eri_pair_chunk_size /
eri_pair_density_storage kwargs of nuclear_gradient, the PySCF
_contract_eri_with_packed_density method and Gradient["ERI"] key, and
the Psi4 Gradient["ERI"] key. Tests and the profiling script follow.
Add a Nuclear gradients documentation page (usage-first, followed by a
theory and algorithm section), a supported-methods table, and citations
to Rehn and Dreuw (2019) and Brumboiu et al. (2021).
* bump test data to 0.7.4 to fix download
The 0.7.2 test-data path on the upstream server is no longer available,
causing the testdata download in conftest.py to fail at pytest collection
time (and thus all CI test jobs). Pull update_testdata.sh (SOURCE bump to
0.7.4) and the regenerated SHA256SUMS from master.
* fix: add missing return in GradientComponents.total and expand gradient contraction tests
* chore: open PR for issue 219 * Merge origin/master into gradients branch Integrate the master refactor (StateView/ElectronicStates restructure, libadcc/ -> libadcc_src/ rename, pyproject.toml migration) into the gradients feature branch. Key changes for gradient code compatibility: - Add forwarding properties (ground_state, reference_state, method) on Excitation for gradient code access - Add total_energy property to ElectronicStates and StateView using the modern MethodLevel enum pattern - Use OneParticleDensity (not OneParticleOperator) for gradient density matrices — PR 200 separated these with different AO transforms (densities: C^T D C, operators: S C^T D C S) - Use OperatorSymmetry.NOSYMMETRY for CVS density matrices that require non-canonical block assignments (co/oc), with explicit transpose blocks - Update excitation_property_keys blacklist to exclude forwarding props - Modernize method.level usage to method.level.to_int() in tests - Regenerate libadcc.pyi with export_block signature Closes #219 * add backend-free total_energy and forwarding-property tests Addresses review findings 2 and 3 on PR 224: - test_total_energy (adc0/adc2) independently validates the total_energy formula across both MethodLevel branches and the Excitation/StateView delegation, instead of comparing against a self-generated snapshot. - test_excitation_forwarding_properties asserts ground_state, reference_state and method forward to the parent ElectronicStates. * fix CI: restore master test-data source and clear flake8 errors The master merge accidentally reverted the test-data sync to the old gradients-branch versions: - update_testdata.sh SOURCE reverted from 0.7.5 back to 0.7.4 - SHA256SUMS *_mpdata.hdf5 hashes reverted to 0.7.4 values The 0.7.4 server no longer hosts ch2nh2_sto3g_hfdata.hdf5, so pytest collection failed with 'missing test data file'. Restore both files to match origin/master. Also clear pre-existing flake8 errors blocking the Code style job: unused imports (unittest, itertools) and E302 in functionality_gradients_test.py, and E501/W293 in dump_fdiff_gradient.py. * fix CI test failures from master merge Three merge regressions where gradient-branch code/tests conflicted with master's state: - LazyMp_test: master's mp2_diffdm produces the canonical 'oc' block (o1o2), and the 0.7.5 reference data stores dm_o1o2 for the adcc source (dm_o2o1 for adcman). Restore master's block list and the adcman-specific label remapping (fixes KeyError 'dm_o2o1'). - workflow: revert the gradient-branch 'warn instead of fail' changes back to master's strict InputError behaviour for too-low conv_tol and too-few guesses (workflow.py and workflow_test.py now match master). - functionality_gradients_test: master's strict guess validation rejects requesting more states than available. h2o/sto3g cvs-adc0/adc1 only support 2 CVS singlets, so cap n_singlets to the number of reference states (len(energy_ref)) instead of the hard-coded 5.
* chore: open PR for issue 220 * implement restricted pair-density fast path * strengthen RHF fast-path tests from review findings 2 and 3
* chore: open PR for issue 221 * implement geometry optimization scanner core * refactor scanner API to accept PySCF SCF object * fix scanner root tracking and add MP2+ADC2 excited example Copy the PySCF Mole snapshot in the tracking descriptor so previous-step AO bases are not silently mutated by the SCF scanner; add a regression test for descriptor mol independence. Rework the excited-state example to optimise on the MP2 ground-state surface first and then run the ADC(2) excited-state optimisation from the relaxed geometry using cc-pVDZ. * switch excited example to s-trans-butadiene/6-31G* Use the s-trans-butadiene geometry optimisation showcase from Rehn & Dreuw (J. Chem. Phys. 150, 174110, 2019) with the 6-31G* basis. * add scanner unit and geomeTRIC end-to-end gradient scanner tests Extend the scanner unit tests with unconverged-SCF handling, an excited-state match against the explicit run_adc + nuclear_gradient loop, fixed-index and density-overlap root tracking, tracking-loss and ambiguity guards, and SCF guess continuity. Add functionality_geomopt_test.py with an end-to-end geomeTRIC ground- state optimisation reaching a stationary point (gradient ~ 0) and a calc_new custom-engine contract check; both skip cleanly when PySCF or geomeTRIC is unavailable. * harden scanner: reject MP3 gradients, observable root tracking, eager validation Address review findings on PR 226: - Reject ground-state MP targets with level != 2 (no MP3 gradient exists) - Make root-tracking diagnostics observable on last_tracking every call - Narrow _safe_ao_ndarray to expected unavailable-operator exceptions - Validate follow eagerly at construction - Warn on ignored run_adc kwargs for the string-MP target spelling - Add root-reorder, excited-state e2e, and validation/scoring tests * fix flake8: silence intended scanner re-export F401, wrap long example line * declare gradients __all__ instead of noqa for scanner re-exports
|
Closed in favor of #230 |
3 participants
Implemented together with @Drrehn.
Features
ToDo
master(Refactor tests #189)add equation numbers from papers (Rehn 2019, Levchenko 2005)