[Compatibility] Avoid stacking WarmCache with FasterFileCopy prefault on 32 GB systems

[nwn900]

Summary

This is not a claim that NextGen Disk Cache has an unbounded private-memory leak or that it is the sole cause of my Skyrim freezes.

On a 32 GB, very large Skyrim SE load order, NextGen Disk Cache 1.1.0 cache warming ran alongside Faster File Copy 1.7.0's decompression-cache prefaulting. The two optimizations appear to have stacked several gigabytes of intentional page-cache/working-set pressure, leaving almost no physical-memory headroom and producing prolonged paging/stutter.

Disabling NextGen Disk Cache, deleting the generated stale cache directories, and putting Faster File Copy into mmap-only mode made the apparent RAM-growth problem go away. Objective freeze snapshots improved from roughly 0.4–0.7 GiB available RAM to 6.6–8.9 GiB available RAM. Some freezes still occur afterward, so this interaction was a major memory-pressure amplifier, not the sole underlying freeze cause.

Environment

• Skyrim SE 1.5.97
• Windows 10 build 19044
• Ryzen 7 5700X
• 32 GB DDR4-3200
• NVMe storage
• GTX 1080 Ti
• NextGen Disk Cache 1.1.0 at the time of the incident
• Faster File Copy 1.7.0, with decompression cache and prefault enabled
• Very large MO2/Nolvus-based load order

I see that the current repository is already at 1.3.0 and that releases after 1.1.0 added important safeguards such as the atomic global warm budget, an automatic memory reserve, stopping on memory pressure, and disabling working-set expansion by default. This report is therefore mainly about the remaining cross-plugin compatibility gap in current code.

Evidence from the affected session

NextGen Disk Cache logged:

• Working-set range: min 128 MB, max 16,346 MB
• Warm plan: 256 files, 3,252 MB budget, 64 MB/file, 4 threads
• Result: 3,444 MB warmed across 54 files in 46 ms

The 192 MB budget overrun is consistent with the pre-1.2 global-budget race that appears to have since been fixed.

Faster File Copy logged:

• Automatic cache ceiling: 8,173 MiB (25% of 32 GB)
• Decompression cache, prefault, cache serving, and save/load acceleration enabled
• Initial full prefault: about 3.98 GB
• Cache later grew to about 4.78 GB
• Individual loading bursts included approximately 0.8–2.1 GB

At the worst pre-mitigation freezes:

• Available physical RAM: about 0.40–0.66 GiB
• Skyrim working set: about 23.8–24.2 GiB
• Skyrim private bytes: about 26.0–27.9 GiB

The associated crash snapshot showed:

• Physical RAM: 27.79 / 31.93 GB
• Working set: 20.24 GB
• Private bytes: 29.93 GB
• Page faults: 44.6 million

After the mitigation described above, later freeze snapshots showed:

• Available physical RAM: 6.59–8.85 GiB
• Skyrim working set: 15.49–18.56 GiB
• Skyrim private bytes: 12.90–20.40 GiB

That is a large, repeatable improvement in memory headroom. Because I changed the interacting cache settings together and cleared their generated caches, this is not a clean isolated A/B proof against NextGen alone.

Remaining current-code gap

Current DetectSiblingPlugins() checks several related SKSE plugins but does not appear to check for FasterFileCopy.dll. Detection already happens before the warm thread is started, so this looks like a natural place to prevent duplicate aggressive warming.

Faster File Copy's shipped behavior can reserve/prefault up to 25% of total RAM when its decompression cache and prefault options are enabled. NextGen's independent warm cache cannot currently account for that peer-owned memory commitment.

The current reserve logic is much safer than 1.1.0, but there are still edge cases:

• The worker checks memory headroom before each file, not during each 32 MB mapped/read chunk.
• Multiple workers may already have sizable in-flight reservations while available RAM is falling.
• An explicitly configured warm budget does not appear to receive the same plan-time available - reserve cap as the automatic budget.
• If GlobalMemoryStatusEx fails, the headroom check currently fails open.

Proposed fix

1. Detect FasterFileCopy.dll.
2. Locate/read the effective FasterFileCopy.ini, where practical.
3. If both bEnableDecompCache=1 and bPrefaultDecompCache=1, suppress NextGen's WarmCache by default and emit a clear compatibility message.
4. Add an explicit override such as bAllowWarmCacheWithPeer=1 for users who deliberately want both.
5. Keep NextGen's core CreateFile flag hooks active; only suppress the overlapping warm-cache pass.
6. Cap both automatic and manually configured warm budgets to the current available physical - reserve allowance.
7. Include aggregate in-flight worker reservations in the live allowance.
8. Recheck memory pressure within the 32 MB mapped/read loop so a large file can stop promptly.
9. When memory-pressure protection is enabled, fail closed if the memory-status query fails.

Suggested regression cases:

• 32 GB system, 5 GB available, 4 GB reserve, 8 workers: no more than 1 GB total scheduled/in flight.
• Memory-status query failure: no warming when pressure protection is enabled.
• Faster File Copy present with decompression prefault active: NextGen warm thread does not start.
• Faster File Copy present in mmap-only mode: NextGen behavior remains configurable/normal.

Reproduction caveat

The mitigation confirms that the combined cache configuration caused severe memory pressure, but it does not establish NextGen as the sole source of every freeze. Freezes continued after memory headroom was restored, and older freezes predate NextGen's installation. I can provide the relevant sanitized NextGen, Faster File Copy, crash, and freeze-logger excerpts if useful.