Bug: Model crashes when skipping Balance Check

[s-poll]

Location:
src/clm5/main/clm_driver.F90

---

Description

The model crashes when skipping the balance check routine BalanceCheckMod.F90. The crash occurs while checking the internal coupler variable Flrl_rofgwl. Testcase is EUR-12.

---

Observed Behavior

The model runs normally until reaching the balance check. The model crashes in the cpl7 check, because of the variable Flrl_rofgwl. Even though 'Rnet' seems to be calculated at the energy balance check (should be moved), I could not find modifications in the water balance.

---

Expected Behavior

A balance check should be non-intrusive and must not modify or corrupt model state, as its role is supposed to be diagnostic.

[kvrigor]

@s-poll do you have a backtrace? You may need to recompile eCLM with -DCMAKE_BUILD_TYPE="Debug"

[s-poll]

Sure, but as it is stopping during the coupler check, it might not be meaningful.

backtrace:

 ERROR:
 lnd_export                       ERROR: One or more of the output from CLM to t
 he coupler are NaN
Image              PC                Routine            Line        Source
eclm               0000000000DED3E9  shr_abort_abort           114  shr_abort_mod.F90
eclm               000000000050A8A3  endrun_vanilla             50  abortutils.F90
eclm               00000000005092E9  lnd_export                424  lnd_import_export.F90
eclm               0000000000504E8B  lnd_run_mct               500  lnd_comp_mct.F90
eclm               0000000000439F42  component_run             728  component_mod.F90
eclm               000000000042B83A  cime_run                 2800  cime_comp_mod.F90
eclm               0000000000436485  cime_driver               125  cime_driver.F90

here the message from the log:

 # of NaNs =            1
 Which are NaNs =  F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F
 F F F F F F F F F F F F F F T F F F F F F F F F F F F F F F F F F F F F F F
 Flrl_rofgwl
 gridcell index =        xxx
 ENDRUN:
 ERROR:
 lnd_export                       ERROR: One or more of the output from CLM to t
 he coupler are NaN

, wereby the gridcell index

xxx= 42252 99360 45469 102176 45066 57525 102578 48687 83673 41862 80052 48284 51089 43460 80454 9657 60743 50686 57927 46675 84879 27367 94533 44250 77236 39434 97759 49088 67581 46275 76431 49489 18506 90108 47468 59134 45868 78041 63961 15289 67179 2415 92925 44665 83270 38630 79649 26160 69995 29378 89304 35816 54307 404 79247 47884 66375 37021 97350 22535 99762 40247 58732 6840 61950 30183 80856 12875 77639 28172 52296 13276 56720 32193 88901 16494 90913 43057 84074 35010 84476 30584 74421 33003 70800 33802 28573 10058 54709 36216 87695 31389 95338 35412 74822 26964 55916 17708 20931 87293 25758 75225 28976 96544 49882 93729 32597 95741 24149 61548 22941 81661 7253 91315 19712 69593 39034 98958 26563 86086 34607 68788 94131 38228 86488 6444 5634 81258 19317 98557 806 101776 10460 86893 57123 36619 55514 13680 71604 20127 53502 12069 39838 3221 66777 29780 3622 76842 14483 52700 23345 100567 16093 92122 41446 65570 21737 70397 41043 51491 24954 51893 11265 73615 4829 58329 6036 93327 31792 55111 15696 96947 62352 1611 74018 61145 33401 60341 10863 64363 22140 82465 14083 73213 92522 4426 71202 9254 64766 20517 88097 17299 100974 23746 63157 18915 7644 65168 12472 59938 16908 47077 1208 68386 43858 72811 8856 98158 37825 82867 90510 24550 40642 27769 85683 75627 5231 56318 53100 8059 53905 100166 82063 72006 25355 85281 18106 11667 96147 21323 63559 2821 67984 78444 4024 89706 59536 91719 37427 76029 72409 50284 94936 14886 65972 101376 2013 62754 42660 78845 34206 30988 8461 88499 69191

[kvrigor]

Thanks Stefan. From the logs I see that Flrl_rofgwl = NaN could likely cause the crash, but I fail to see how this crash is connected to a "skipped balance check routine". The backtrace does not show BalanceCheckMod.F90. Can you provide more details on how this crash is linked to balance check (mis)behavior? You might have more relevant log files to share.

[s-poll]

I think I need to explain myself more clearly. As mentioned in the description above, the model crashes whilst checking the internal coupler variable Flrl_rofgwl. Thus, I unfortunately do not have more relevant log files to share. I found the connection to the BalanceCheck while debugging the error. This commit seem to introduce the error, and the model is working when revoking it. This skipping of the BalanceCheck in eCLM-sta leads to the same error (not shown). One possible solution for this issue might be to suppress all log-writes in the balance check, however, this might be not the smartest solution and do not tackle the origin of the error.

[kvrigor]

Initial investigation reveals that an unitialized variable, i.e. qflx_liq_dynbal_grc, causes Flrl_rofgwl to be NaN.

1. The only place where Flrl_rofgwl (liquid runoff from glaciers and wetlands) gets set is at:

eCLM/src/clm5/cpl/lnd_import_export.F90

Lines 394 to 396 in 98054f5

	! qgwl sent individually to coupler
	l2x(index_l2x_Flrl_rofgwl,i) = lnd2atm_inst%qflx_rofliq_qgwl_grc(g)

2. Next we trace how qflx_rofliq_qgwl_grc is set. Further grep-ping reveals the line below:

eCLM/src/clm5/main/lnd2atmMod.F90

Line 424 in 98054f5

lnd2atm_inst%qflx_rofliq_qgwl_grc(g) = lnd2atm_inst%qflx_rofliq_qgwl_grc(g) - waterflux_inst%qflx_liq_dynbal_grc(g)

3. Now waterflux_inst%qflx_liq_dynbal_grc looks something that might be used in a water balance check. Indeed it is:

eCLM/src/clm5/biogeophys/BalanceCheckMod.F90

Line 259 in 98054f5

qflx_liq_dynbal => waterflux_inst%qflx_liq_dynbal_grc , & ! Input: [real(r8) (:) ] liq runoff due to dynamic land cover change (mm H2O /s)

But as you've noted @s-poll, it appears that BeginWaterBalance does not mutate any global state. Actually, qflx_liq_dynbal_grc is neither read nor written in BalanceCheckMod.F90; all the time it kept its NaN initial value, which eventually propagates to Flrl_rofgwl by how it's calculated (see Step 2).

4. There's this odd remark too:

eCLM/src/clm5/main/clm_driver.F90

Lines 329 to 338 in 98054f5

	! ============================================================================
	! Initialize the column-level mass balance checks for water, carbon & nitrogen.
	!
	! For water: Currently, I believe this needs to be done after weights are updated for
	! prescribed transient patches or CNDV, because column-level water is not generally
	! conserved when weights change (instead the difference is put in the grid cell-level
	! terms, qflx_liq_dynbal, etc.). In the future, we may want to change the balance
	! checks to ensure that the grid cell-level water is conserved, considering
	! qflx_liq_dynbal; in this case, the call to BeginWaterBalance should be moved to
	! before the weight updates.

I don't fully understand that remark; however it seems like there's a distinction between column- and gridcell-level balance checks. Anyway I need to further investigate how qflx_liq_dynbal_grc gets set so that it isn't NaN—and therefore would not cause the Flrl_rofgwl = NaN crash.

@s-poll as a workaround, just revert e0a6a97. Even without that commit balance check errors will still be ignored in BalanceCheckMod.F90

[kvrigor]

Side remark: CLM5's transient PFTs and "dynamic land units" feature might be relevant to our general issues with coupled water/energy balance issues. From the CLM5 tech note:

Water and Energy Conservation

When subgrid areas change, the water and energy states remain unchanged on a per-area basis. This can lead to changes in the total gridcell water and energy content.

For example, consider a gridcell with two columns: column 1 has a water mass of 1 kg m^2 and column 2 has a water mass of 2 kg m^2 for a given water state variable, where these are expressed per unit column area. If column 1 increases in area at the expense of column 2, then column 1 will still have a water mass of 1 kg m^2, but now expressed over the new column area. This results in a decrease in the total gridcell water content.

Water and energy are conserved by summing up the total water and energy content of each gridcell before and after a change in area. Differences in liquid and ice water content are balanced by liquid and ice runoff terms, which can be either positive or negative. (Negative runoff is effectively a withdrawal of water from the ocean.) Differences in energy content are balanced by a sensible heat flux term, which again can be either positive or negative. These balancing fluxes are spread evenly throughout the following year.

There is a special case when a given crop column type newly comes into existence - for example, when temperate corn first comes into existence in a gridcell. In this case, the column’s below-ground temperature and water states are copied from the natural vegetated column in its gridcell, so that these state variables begin in a close-to-spun-up state. Other state variables (most of which spin up relatively quickly) begin at their cold start initialization values. This initialization is not necessary for the two other land unit types that currently can grow - natural vegetation and glacier: Those land unit types are always active, even when they have zero area on the gridcell, so their state variables will be spun up immediately when they come into existence. After this initialization, the conservation code described above takes effect.