escaping C++ exception / unclosable close race

[r2evans]

When I use websocket multiple times back to back, sometimes simultaneously, R crashes. My basic use is websocket-1.4.4 on R-4.5.2, confirmed on both macos and ubuntu-24.04. The server-end is fine, other websocket clients work just fine.

I did some stress-testing (including fable-5) and found some specific examples causing crash and/or mis-referencing.

One takeaway (claude's words):

ClientImpl::close() uses websocketpp's throwing overload while
the R-side state cache races the network thread; invoke_function_callback()
has no exception barrier, so the throw crosses a C boundary (UB — on current
cpp11-based later it surfaces as an async R error injected into whatever is
running the loop, plus a leaked heap closure per escape; on a toolchain
without a catch frame there it terminates the process). Fix: the
non-throwing error_code close overload (no-op on closed, browser
semantics) + a try/catch barrier in invoke_function_callback.

Reprex 1:

### R -q -f 01-async-error-kills-computation.R
install.packages("websocket")
# Installing package into ‘/Users/r2/Library/R/4.5’
# (as ‘lib’ is unspecified)
# trying URL 'https://packagemanager.posit.co/cran/latest/bin/macosx/big-sur-arm64/contrib/4.5/websocket_1.4.4.tgz'
# Content type 'binary/octet-stream' length 3130974 bytes (3.0 MB)
# ==================================================
# downloaded 3.0 MB
# The downloaded binary packages are in
#         /var/folders/3f/0hlmq7rx34ddjqtlc1lqbxl80000gn/T//Rtmp9RiYTU/downloaded_packages
packageVersion("websocket")
# [1] ‘1.4.4’
library(websocket)
cat("websocket", as.character(packageVersion("websocket")),
    "from", find.package("websocket"), "\n")
# websocket 1.4.4 from /Users/r2/Library/R/4.5/websocket
s <- httpuv::startServer("127.0.0.1", httpuv::randomPort(), list(
  onWSOpen = function(ws) ws$close()   # server closes as soon as it opens
))
url <- paste0("ws://127.0.0.1:", s$getPort(), "/")
# Open some websockets and close them immediately; every call here succeeds.
for (i in 1:20) {
  ws <- WebSocket$new(url)
  ws$close()
  Sys.sleep(0.01)   # bg thread finishes handshake + processes server close
}
rm(ws)

# Completely unrelated work that services the event loop, as Shiny, promises,
# or any later-based code does. The websocket events queued above fire here
# and their escaping exception kills this loop.
result <- 0
for (i in 1:100) {
  result <- result + i
  later::run_now(0.01)
}
# Error: invalid state
# Execution halted

### echo $? -> 1

When run interactively (not headless/backend) the R process does not exit though the "invalid state" still occurs:

Error in eval(ei, envir) (from 01-async-error-kills-computation.R#67) : invalid state
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.
Error: invalid state
later: exception occurred while executing callback.

---

Reprex 2, slightly different:

### R -q -f 02-close-after-server-close-error.R

library(websocket)
cat("websocket", as.character(packageVersion("websocket")),
    "from", find.package("websocket"), "\n")
# websocket 1.4.4 from /Users/r2/Library/R/4.5/websocket
# >
s <- httpuv::startServer("127.0.0.1", httpuv::randomPort(), list(
  onWSOpen = function(ws) {
    ws$onMessage(function(binary, msg) ws$close())  # server closes on first message
  }
))
url <- paste0("ws://127.0.0.1:", s$getPort(), "/")
# >
ws <- WebSocket$new(url)
while (ws$readyState() < 1L) later::run_now(0.05)   # wait until OPEN
# >
ws$send("bye")
later::run_now(0.2)   # the (in-process) server handles "bye" and closes
Sys.sleep(0.5)        # client bg thread processes the close frame; the close
                      # event is queued for R but not yet processed
cat("readyState:", ws$readyState(), "\n")   # still 1 (OPEN)
# readyState: 1
ws$close()                                  # Error: invalid state
# Error: invalid state
# Execution halted

### echo $? -> 1