Introduction

A lightweight, experimental NGINX module that forwards HTTP messages to a sidecar process using System V shared memory. Built as a playground to explore IPC and NGINX internals. It’s functional and well-structured but intentionally minimal and not suited for production use cases.

The Protocol

Overview

The system links an NGINX server with a sidecar process via System V shared memory and event-driven signaling to delegate HTTP request handling to the sidecar. Transaction timeouts and other related settings are configurable in the NGINX config per server block (See the example config).

At runtime the sidecar first performs a handshake with NGINX and receives an eventfd over a UNIX domain socket. After a successful handshake, NGINX accepts client requests at configured endpoints and writes data of requests to those endpoints into shared memory. The sidecar reads the request, writes its response into shared memory, and signals NGINX by writing the eventfd. NGINX then constructs the HTTP response and returns it to the client. Transactions may time out and be cancelled; any signals or responses received after a timeout are ignored for that transaction.

Handshakes

• A process (sidecar) that wishes to communicate via shared memory initiates a handshake by sending an HTTP request to an NGINX location configured with the NGINX directive shm_handshake on.
• The NGINX internal handler first ensures no handshake is already established or in progress; if one is, it responds immediately with 409 Conflict and stops further processing of the request.
• NGINX sets up an eventfd and attempts to share its file descriptor with the sidecar over a UNIX domain socket.
• When the sidecar successfully receives the eventfd, it attaches the shared memory region and polls a status field in an allocator structure kept at the first location in shared memory for ready requests written by NGINX.
• After successfully sharing the eventfd, NGINX registers the eventfd with its epoll event loop and continues to monitor it so the sidecar can signal NGINX when it writes responses for future shared-memory request transactions.
• NGINX updates the internal status on success; if the handshake times out or is aborted, NGINX reverts the internal status to allow future handshake attempts.

Forwarding

• After a successful handshake, NGINX and the sidecar are ready to exchange messages via the shared memory region.
• Requests sent to endpoints configured for shared-memory forwarding are accepted by the NGINX internal handler only if no other message transaction is in progress; otherwise NGINX responds with 409 Conflict.
• For an accepted request NGINX responds to the client with 102 Processing and writes the request into shared memory.
• The sidecar detects the arrived request, reads it from shared memory, writes back its response, and signals NGINX by writing to the eventfd.
• Transactions may time out and be cancelled after a set period; any signals or responses received afterward are ignored for that transaction.
• NGINX reads the response and sends it to the client and updates internal state so the next transaction can proceed.

Limitations

• Reduced transfer rate: Transactions are written serially to shared memory in a hierarchical structure; throughput could be improved by introducing parallelism (for example, multiple streams or packetized transfers).
• Single worker: Because NGINX workers are separate processes, an eventfd notification can wake a worker that has not attached the shared-memory region or lacks access to the associated NGINX request structures. Introducing a shared memory zone between all workers could help resolve the issue.
• Signaling the sidecar: The sidecar currently polls a status field; using signals or a kernel-driven notification mechanism would be more efficient and reduce CPU usage.
• Prone to dead locks: When the sidecar process stops after having completed the handshake, no further handshakes will be allowed requiring NGINX to restart as well.
• Restricted to a single sidecar per server: The reason makes up a subset of the above. Because request notification uses an indiscriminate signaling mechanism, allowing multiple sidecars would create races to serve the same request and require synchronization before either can claim the shared memory, since transactions are expected to be handled serially.

Build Environment

The included Containerfile builds a container that contains the NGINX source and the required build tools.

• The Container filesystem can be mounted to view the code with an editor installed on the host.
• Lsp support is possible by leveraging cmake compile_commands.json, however the lsp server may not resolve the includes in the code correctly because of the difference in path prefix between host and container.
• The setup script ./devcontainer/setup.sh resolves this issue in total by auto mounting the container filesystem and writing the mount path back to the container as an environment variable used as cmake install prefix. This ensures CMake records host-visible paths (including the mount prefix) in its configuration so LSP servers on the host can resolve includes correctly.

I used this setup with my personal Neovim configuration.

To build the project inside the container:

/project/devcontainer/ngx_build.sh
/project/devcontainer/ngx_mkdirs.sh
(cd /project/build && make)
(cd /usr/local/src/nginx && make)

You can then run:

rm -f /tmp/shmod/handshakes; /usr/local/src/nginx/objs/nginx -c /project/ngx_http_shm_module/deploy/nginx.conf
/project/build/shm_sidecar/src/sidecar >> log &

And should then be able to receive forwarded responses from the sidecar:

root@bf149d33bf6c:/project# curl -d "hello" 0.0.0.0/shm; echo ''
hello