Core/Batch: hardcoded 8192-byte msg_receive() buffer causes a CPU-spin poison message when kernel.msgmax is raised above the default

[ivanrey]

Environment details

• OS: Linux (Debian/Ubuntu), any distro where kernel.msgmax is tunable via sysctl
• PHP version: 8.x (not version-specific — this is a kernel/IPC-level issue)
• Package name and version: google/cloud-core v1.72.3 (Core/src/Batch), used indirectly via google/cloud-logging's LoggingClient::psrBatchLogger() / BatchDaemon

Summary

Core\Batch\BatchJob::run() reads from the SysV message queue with a hardcoded 8192-byte buffer and without MSG_NOERROR:

google-cloud-php/Core/src/Batch/BatchJob.php

Lines 117 to 126 in 720551e

	if (msg_receive(
	$q,
	0,
	$type,
	8192,
	$message,
	true,
	0, // blocking mode
	$errorcode
	)) {

if (msg_receive(
    $q,
    0,
    $type,
    8192,
    $message,
    true,
    0, // blocking mode
    $errorcode
)) {

Meanwhile SysvProcessor::submit() never checks the message size before calling msg_send() — it relies entirely on the kernel to reject oversized messages and fall back to the temp-file path:

google-cloud-php/Core/src/Batch/SysvProcessor.php

Lines 47 to 81 in 720551e

	public function submit($item, $idNum)
	{
	if (!array_key_exists($idNum, $this->sysvQs)) {
	$this->sysvQs[$idNum] =
	msg_get_queue($this->getSysvKey($idNum));
	}
	$result = @msg_send(
	$this->sysvQs[$idNum],
	self::$typeDirect,
	$item,
	true,
	false
	);
	if ($result === false) {
	// Try to put the content in a temp file and send the filename.
	$tempFile = tempnam(sys_get_temp_dir(), 'Item');
	$result = file_put_contents($tempFile, serialize($item));
	if ($result === false) {
	throw new \RuntimeException(
	"Failed to write to $tempFile while submiting the item"
	);
	}
	$result = @msg_send(
	$this->sysvQs[$idNum],
	self::$typeFile,
	$tempFile,
	true,
	false
	);
	if ($result === false) {
	@unlink($tempFile);
	throw new QueueOverflowException();
	}
	}
	}

This makes the library's correctness implicitly dependent on kernel.msgmax being ≤ 8192 (the Linux default), a constraint that is undocumented and unenforced anywhere in the code.

Steps to reproduce

1. Raise kernel.msgmax above 8192 on the host, e.g. sysctl -w kernel.msgmax=10240 (a plausible, unrelated tuning change — e.g. to accommodate other software's IPC needs on the same box).
2. Run the batch daemon (vendor/bin/google-cloud-batch daemon) with a job using SysvProcessor (e.g. via LoggingClient::psrBatchLogger()).
3. Submit an item whose serialized size is between 8193 bytes and the new msgmax (e.g. a moderately large log entry/context payload). msg_send() now succeeds as a direct message, where before the tuning change it would have failed and gone through the temp-file fallback.
4. When the worker's msg_receive() reaches that message, it fails with E2BIG (message larger than the requested 8192-byte buffer) and, critically, the message is not removed from the queue (that's the POSIX/Linux semantics of msgrcv without MSG_NOERROR).
5. BatchJob::run()'s while (true) loop immediately retries the same blocking msg_receive() call, which fails again with E2BIG instantly (the syscall fails validation before it would need to block) → tight loop, ~100% CPU in the worker process.
6. The oversized message is now a permanent "poison pill": nothing behind it in the queue can be consumed (FIFO head-of-line blocking), so the queue fills up until producers start hitting QueueOverflowException.

Impact

• Silent, sustained CPU exhaustion in a background worker process, with no error surfaced anywhere in the calling application — from the app's point of view msg_send() succeeded, so logging/batch calls appear to work fine. The only visible symptom is anomalous CPU usage on the host, which is hard to correlate back to this library.
• Any operator who raises kernel.msgmax above the Linux default of 8192 — for any reason, not necessarily related to this library — can silently trigger this the next time a slightly-larger-than-usual item is submitted. Since kernel.msgmax is a machine-wide setting, this can be caused by completely unrelated software sharing the host.
• We hit this in production: a StackdriverLogger-driven psrBatchLogger() call encoded to somewhere between 8KB and 10KB (an httpRequest context payload) became the poison message as soon as we raised kernel.msgmax to 10240 for unrelated reasons. ipcrm-ing the affected queues was the only way to recover, and the daemon needed a restart since the running workers held stale queue handles.

Suggested fixes

Any of these would resolve it; happy to hear which direction you'd prefer:

1. Pass MSG_NOERROR to msg_receive() so an oversized message gets truncated and dequeued instead of stalling the loop — ideally combined with logging a warning so the drop is at least visible.
2. Don't hardcode 8192; read the buffer size from msg_stat_queue() (msg_qbytes), or make it configurable similar to the existing GOOGLE_CLOUD_SYSV_ID env var, so msg_receive()'s buffer always matches the queue's actual configured capacity instead of assuming the kernel default.
3. At minimum, treat a false return from msg_receive() with $errorcode === MSG_E2BIG as fatal for that specific message — actively drain-and-discard it (e.g. retry with MSG_NOERROR) rather than retrying the identical call forever.
4. Document the kernel.msgmax ≤ 8192 requirement explicitly (e.g. in Core/src/Batch's docblocks and any Batch daemon setup guide), and/or have BatchDaemon/SysvConfigStorage validate it at startup and fail fast with a clear message instead of degrading into an unexplained CPU spin under production load.

Related but distinct from #1336 / PR #3176, which addressed the producer side stalling on submit() when the queue is full. This report is about the consumer side (msg_receive()) silently spinning on an oversized message — a different failure mode with a different fix.