[Profiler] Raise the crashing-app flag even when crashtracking is disabled

[dkattan]

Summary of changes

Raise the mmap-shared "application is crashing" flag (introduced in #7657) on the execve interception path even when the call is not redirected to the Datadog crash handler — i.e. when DD_CRASHTRACKING_ENABLED=false, or when dd-dotnet is not found next to the wrapper. The original createdump invocation is then passed through unchanged.

Reason for change

#7657 exists because the profiler's signal-based stack collector can interrupt a process that is in the middle of its crash handling ("crash on a crash"). But the flag is only ever set inside the ShouldCallCustomCreatedump != 0 branch of the execve wrapper, and that branch is unreachable when crashtracking is disabled:

• With DD_CRASHTRACKING_ENABLED=false, initLibrary() takes the early return, so crashHandler stays NULL and ShouldCallCustomCreatedump returns 0 unconditionally.
• Independently, the --name datadog_crashtracking marker is only planted into DOTNET_DbgMiniDumpName when crashtracking is enabled, so the marker check can never match either.

Result: disabling crashtracking (a legitimate configuration for users who want the stock runtime createdump minidumps) silently re-exposes the exact pre-#7657 behavior — the profiler keeps sending sampling signals throughout the entire crash sequence.

Production evidence (dd-trace-dotnet 3.51.1, .NET 10.0.11, Ubuntu 24.04 AKS via k8s single-step injection, all profiler samplers enabled, DD_CRASHTRACKING_ENABLED=false, DOTNET_DbgEnableMiniDump=1): a process with a 12–17 GB heap crashing on its GC hard limit terminated with exit code 139 (second SIGSEGV during crash handling) and no dump produced, on 4 consecutive crashes. The final log line each time was the runtime's

Problem reading from createdump child_read_pipe: Success (0)

which is the fork-handshake in PROCCreateCrashDump: read() returning 0 with errno 0 means the crashing parent died between fork() and writing the go-byte, so createdump was never even exec'd. Disabling the profiler makes dumps reliably appear again.

Implementation details

profiler/src/ProfilerEngine/Datadog.Linux.ApiWrapper/functions_to_wrap.c:

• initLibrary() now captures the user-configured DOTNET_DbgMiniDumpName/COMPlus_DbgMiniDumpName into userMiniDumpName before the crashtracking-disabled early return (the real_getenv/real_setenv lookup moved above the check to make that possible; the check itself is unchanged).
• New helper IsCreatedump() extracts the existing /createdump path-suffix test (now shared with ShouldCallCustomCreatedump, which is otherwise unchanged).
• New helper IsCrashTriggeredCreatedump() distinguishes a crash-triggered createdump from an on-demand dump request when the datadog_crashtracking marker mechanism is not in place: following a crash the runtime forwards the raw value of DbgMiniDumpName through --name (and passes no --name at all when the variable is not set), whereas dump-generation requests (ex: dotnet-dump collect) carry a client-chosen file name. This mirrors what the marker check does for the crashtracking-enabled path, and avoids permanently muting the profiler after a live dotnet-dump collect (the flag is sticky).
• In execve(): when ShouldCallCustomCreatedump says no, but IsCrashTriggeredCreatedump says yes, set *is_app_crashing = 1 and fall through to the original execve(pathname, argv, envp) unchanged — no dd-dotnet substitution, no argv/envp rewriting, so user-visible createdump behavior in this configuration is exactly as before, minus the sampling during the crash window.

No reader-side changes were needed: the shared-memory flag is already mmap'd unconditionally in init(), and the profiler consumes it through the weak dd_inside_wrapped_functions() symbol (LinuxStackFramesCollector, TimerCreateCpuProfiler, SystemCallsShield) with no crashtracking gating.

Behavior with crashtracking enabled and functioning is unchanged: the crash path still goes through the marker match and the custom-createdump branch. The one intentional extension is the "crashtracking enabled but dd-dotnet binary not found" case, where the flag is now also raised (the guard was equally dead there).

Test coverage

• New profiler/test/Datadog.Linux.ApiWrapper.Tests/CrashFlag.cpp (runs under the existing RunNativeWrapperNativeTests target with the wrapper preloaded): forks, execve's a (nonexistent) .../createdump the way the runtime's crash path does, and asserts dd_inside_wrapped_functions() becomes non-zero in the parent. This fails against master and passes with this change. Note the flag is sticky for the remainder of the test process (documented in the test).
• The existing CreatedumpTests.DoNothingIfNotEnabled(enableCrashDumps: true) integration test covers this exact configuration and asserts stock createdump still runs — the pass-through here keeps it intact.
• Manual verification in the gleocadie/centos7-clang16 CI image (wrapper built and LD_PRELOADed into a fork/execve harness mimicking PROCCreateCrashDump):
  • crashtracking disabled + --name matching DOTNET_DbgMiniDumpName → flag set (was: not set on master);
  • crashtracking disabled + dotnet-dump-style foreign --name → flag not set;
  • DbgMiniDumpName unset + no --name → flag set;
  • non-createdump exec → flag not set;
  • COMPlus_ variants → flag set.
• Build verification: functions_to_wrap.c compiles clean (clang 16, -std=c11 -Wall, glibc and -DDD_ALPINE) in the CI image and the full .so links; nm -D confirms no new exported symbols (the wrapper-symbols snapshot test should be unaffected). I could not run the full Nuke/CMake pipeline locally (macOS host), so relying on CI for the complete build and test matrix.

Other details

• Residual gap this does not fix: the flag is raised at execve() interception time, but the vulnerable window opens at fork() — the PROCCreateCrashDump pipe handshake happens before exec, and in our production crashes the parent died inside that pre-exec window. So even with crashtracking enabled, the current mechanism cannot protect the fork→exec gap; closing it (e.g. raising the flag from the runtime's SIGSEGV handler side, or intercepting the crash-path fork) is follow-up material.
• Related known limitation (pre-existing, shared with the marker mechanism): a direct diagnostics-IPC GenerateCoreDump request whose target file name happens to equal DbgMiniDumpName would be treated as a crash.
• We can readily test candidate builds against the reproducing production workload.

Refs #7657

🤖 Generated with Claude Code