LESTKF RELEASE build produces different results than DEBUG

[jjokella]

Found in

• LSTDA into OMI #55

Addressed by

• Release PDAF with -fp-model=source for consistent LESTKF results TSMP2#126

Problem

When running LST-DA data assimilation experiments with the LESTKF filter, results in the RELEASE build differ from the DEBUG build. Specifically:

• LESTKF experiments using PDAF-OMI and PDAF-non-OMI produce different results in RELEASE, but are identical in DEBUG.
• EnKF experiments are identical across both build types, suggesting the issue is localized to the LESTKF localization routines.

Root Cause (Assumption)

Intel's default floating-point optimization (-O2) appears to trigger numerically inconsistent results in the LESTKF localization routines. Adding -fp-model=precise to the Intel Fortran compiler flags for RELEASE builds restores identical results.

Reference: https://www.intel.com/content/www/us/en/docs/fortran-compiler/developer-guide-reference/2023-0/fp-model-fp.html

[jjokella]

I could see precision problems in the simple testcase clm5_pdaf_10x10 as well. The differences start with the PDAF-update after 5 days and they are removed by the PR

• Release PDAF with -fp-model=source for consistent LESTKF results TSMP2#126

Figure

[kvrigor]

just FYI, a related issue in eCLM: Numerical issues caused by unsafe IntelLLVM flags