Fix 32-bit integer overflow in AllOutData byte calculation

[andrew-platt]

Ready to merge pending review

Feature or improvement description

Fix 32-bit integer overflow in the AllOutData binary output buffer byte-count calculation, and add a pre-allocation safeguard with an actionable error message.

When (NumOuts-1) * NOutSteps * BYTES_IN_RxKi exceeds ~2.1 GB, the 32-bit integer arithmetic in AllocAry error messages overflowed, producing confusing negative byte counts (e.g., -1501695564 bytes). This commonly occurs with small DT_Out values (or DT_Out defaulting to DT) combined with long simulation times (e.g., DT=0.0002, TMax=6400s → 32 million output steps).

This PR:

1. Casts all dimension arguments to B8Ki (64-bit integer) before multiplication in the NWTC library's AllocAry error messages (all 2D–5D variants).
2. Adds a pre-allocation check in FAST_InitOutput that computes the required AllOutData size using 64-bit arithmetic and fails early with a clear message including the required memory, array dimensions, and user-actionable suggestions.

Related issue, if one exists

#3371

Impacted areas of the software

• modules/nwtc-library/src/NWTC_IO.f90 — All multi-dimensional AllocAry routines (error message byte-count calculations)
• modules/openfast-library/src/FAST_Subs.f90 — FAST_InitOutput subroutine (binary output buffer allocation)

Additional supporting information

Example error before this fix:

FAST_InitializeAll:Error allocating -1501695564 bytes of memory for the AllOutData array.

Example error after this fix:

The AllOutData array would require 2560000800 bytes of memory (2441 MB). This exceeds the 2 GB limit for binary output stored in memory. Reduce TMax, increase DT_Out, reduce output channels, or switch to text output (OutFileFmt=1). (NumOuts=20, NOutSteps=32000001)

The checkpoint save/restore code (RegPackAlloc/RegUnpackAlloc) stores array bounds as individual B4Ki values per dimension and uses Fortran native I/O for data, so it is unaffected by this change and correctly handles large arrays on 64-bit systems.

Generative AI usage

This PR was substantially generated with AI assistance for analysis, diagnosis, and implementation:

Co-authored-by: Anthropic Claude claude@anthropic.com
Assisted by: GitHub Copilot support@github.com

Estimated resource usage for this session:

Resource	Estimate
Total tokens	~200,000
API cost (open market)	~$4.50
Electricity	~1 kWh
Water (data center cooling)	~2 liters
Engineer iv in the loop	1.5 hours

Test results, if applicable

• Compilation verified (clean build of nwtclibs and openfastlib with gfortran on aarch64 Linux).
• No regression test changes expected — this only affects error reporting and adds an early-exit guard for oversized allocations.
• r-test branch merging required

[Copilot]

Pull request overview

Fixes misleading negative byte-count values caused by 32-bit integer overflow in allocation error messages, and adds an early guard in FAST_InitOutput to fail fast with a clear, actionable message when the in-memory binary output buffer would exceed ~2 GB.

Changes:

• Updates NWTC AllocAry-style allocation error messages (2D–5D variants shown) to compute byte counts using 64-bit integer arithmetic.
• Adds a pre-allocation size check for AllOutData in FAST_InitOutput to prevent oversized in-memory binary output allocations and provide user guidance.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
modules/openfast-library/src/FAST_Subs.f90	Adds a 64-bit pre-check for AllOutData buffer size and aborts early with an actionable error message when exceeding the in-memory binary limit.
modules/nwtc-library/src/NWTC_IO.f90	Prevents 32-bit overflow in allocation error-message byte-count calculations by casting dimensions to 64-bit integers before multiplication.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.