Fix jvmkill racing with HeapDumpOnOutOfMemoryError leading to truncated heap dumps

[grzegorz-wal]

Overview

jvmkill calls kill(getpid(), SIGKILL) from every JVMTI ResourceExhausted event. On JDK 22+, this races with the JDK's own -XX:+HeapDumpOnOutOfMemoryError handler and truncates the resulting .hprof.

On JDK ≤ 21, the entire heap dump ran inside a single VM_HeapDumper VM operation at safepoint. While that safepoint was held, all Java threads - including threads that lost the report_java_out_of_memory CAS latch and were carrying the JVMTI callback were frozen. The agent's SIGKILL could only fire after the dump finished and the safepoint was released, so the .hprof was always complete on disk.
JDK-8306441 (JDK 22) split the dump so that the segment-merge phase runs outside the safepoint. Loser-of-CAS hreads now unfreeze while the merge is still writing to disk, reach the JVMTI callback, and jvmkill kills the process mid-write truncating the .hprof.

Changes

Add a VMInit callback that probes HotSpotDiagnosticMXBean at startup to detect whether ExitOnOutOfMemoryError or CrashOnOutOfMemoryError is set. When neither ExitOnOutOfMemoryError nor CrashOnOutOfMemoryError is set, the agent continues to SIGKILL for backward compatibility. Route ResourceExhausted events based on the flags argument :

OOM source	Flags	Old behavior	New behavior
Java heap (memAllocator.cpp)	OOM_ERROR|JAVA_HEAP (0x3)	SIGKILL	log only — JDK handles shutdown
Metaspace(metaspace.cpp)	OOM_ERROR (0x1)	SIGKILL	log only — JDK handles shutdown
Native thread create (jvm.cpp)	OOM_ERROR|THREADS (0x5)	SIGKILL	SIGKILL

For Java heap and metaspace OOM, report_java_out_of_memory runs upstream of the JVMTI callback - the JDK has already started its dump and will terminate the process. Killing unconditionally races with the dump write (heapDumper.cpp) and truncates the .hprof.

Native thread OOM has no JDK-side handler; SIGKILL there is unchanged.

Fixes trinodb/trino#29301

[electrum]

This is far too complicated. We can simply change it to only kill if the VM is out of threads.

We don't need a wall of comments. This looks like poorly generated code from AI when you don't give it any guidance. You can always ask the AI to explain the code in the future or do the analysis again.

[grzegorz-wal]

The if statements are there to show the intent of only killing the JVM when resource exhaustion comes from thread allocation. The rest of the added code is for backward compatibility. I went through the airlift org docs and couldn't find any explicit advice to set -XX:+ExitOnOutOfmemoryError or -XX:+CrashOnOutOfMemoryError alongside jvmkill (correct me if I'm wrong). So there may be users who rely solely on this agent to terminate the JVM on OOM. Simply skipping the kill for non-thread OOM without checking those flags first would silently break them, leaving the JVM in an ambiguous state. The flag check is here to provide information in the logs and leave the behavior unchanged for the users without those flag set. On the implementation side the jvmti doesn't give direct access to JVM startup flags, so I'm using HotSpotDiagnosticMXBean to read them. It's the least hacky approach I found, but I'm happy to adjust or completely change it if you know of a better way or think this doesn't make sense.

[findepi]

We can simply change it to only kill if the VM is out of threads.

Although https://trino.io/docs/current/installation/deployment.html#jvm-config includes -XX:+ExitOnOutOfMemoryError i think it's reasonable to worry about users who run Trino without that flag, leveraging the fact that jvmkill does the job equally well.

This user empathy led to the code being complicated.
In Java, this is as simple as

HotSpotDiagnosticMXBean bean = ManagementFactory.getPlatformMXBean(HotSpotDiagnosticMXBean.class);
String value = bean.getVMOption("ExitOnOutOfMemoryError").getValue();            
boolean isEnabled = Boolean.parseBoolean(value);

In C, using JMVTI, it's obviously not as concise, but that's all the code does.

@electrum do I read correctly that you believe we shouldn't be worried about someone running without -XX:+ExitOnOutOfMemoryError set?

[VictorAtPL]

@electrum second PR with simplified logic: #10 ; pls review