Google Cloud Load Balancer: Web Bot Authentication using Service Extensions

An enterprise-ready Google Cloud Service Extensions callout service implementing RFC 9421 HTTP Message Signature for cryptographically verifying Web Bots and AI Crawlers at the Google Cloud Application Load Balancer.

---

📖 Table of Contents

• Overview & Architecture
• Repository Structure & Dependencies
• Prerequisites
• Quick Start
  • 1. Environment Setup
  • 2. Generate & Install Protobuf Stubs
  • 3. Run the Callout Server
• Running Tests & Utilities
• Interactive Demo UI
• Docker & Cloud Run Deployment
• Terraform Infrastructure Deployment
• Configuration Options

---

🏛️ Overview & Architecture

Modern AI crawlers and search indexers sign their HTTP requests per the Web Bot Auth IETF internet draft using asymmetric cryptography.

This repository provides a Service Extensions callout server that:

1. Intercepts incoming HTTP request headers via Service Extensions running on Google Cloud Load Balancer.
2. Validates HTTP Message Signatures (Signature-Agent, Signature-Input, Signature) against public keys fetched dynamically from the bot's JWKS directory.
3. Caches JWKS Public Keys in a thread-safe LRU cache with configurable TTL to minimize latency overhead.
4. Mutates Request Headers:
   • x-verified-bot: true / false
   • x-bot-agent: The authenticated bot agent URI.
   • x-bot-key-id: The key identifier used to sign the request.
   • x-bot-algorithm: The cryptographic algorithm used (e.g., ed25519, rsa-pss-sha512).
5. Enforcement Modes:
   • Observability / Tagging Mode: Annotates headers for downstream routing.
   • Enforce Mode: Returns an immediate bot-friendly HTTP 403 Forbidden response with JSON error details and IETF documentation link.

---

📦 Repository Structure & Dependencies

This repository packages the core web_bot_auth module alongside all necessary Service Extensions frameworks, protobuf definitions, and Terraform:

WebBotAuth-on-google-cloud-load-balancer/
├── terraform/                    # Complete Terraform IAC for Cloud Run, Service Extensions, and Load Balancer
│   ├── provider.tf               # GCP provider configurations
│   ├── variables.tf              # Configurable variables
│   ├── cloud_run.tf              # Cloud Run v2 Callout Service with h2c gRPC support
│   ├── callout_service.tf        # Serverless NEG, Callout Backend Service & LB Traffic Extension
│   ├── sample_lb.tf              # Sample External Application Load Balancer & sample upstream origin
│   ├── outputs.tf                # Load Balancer IP, test curl commands
│   ├── terraform.tfvars.example  # Example variable values
│   └── README.md                 # Terraform deployment and testing guide
├── extproc/
│   ├── web_bot_auth/             # Core Web Bot Auth Implementation
│   │   ├── server.py             # WebBotAuthCalloutServer (gRPC Service)
│   │   ├── verifier.py           # RFC 9421 Verification Engine & KeyCache
│   │   ├── README.md             # Module architecture documentation
│   │   ├── TESTING.md            # Detailed testing guide
│   │   ├── test/                 # Test suite & verification utilities
│   │   │   ├── test_web_bot_auth.py    # Unit tests
│   │   │   ├── generate_jwks.py        # Keypair & JWKS generator
│   │   │   └── send_signed_request.py  # RFC 9421 client request signer
│   │   └── ui/                   # Interactive Web UI Playground
│   │       ├── index.html, styles.css, favicon.ico
│   │       ├── js/               # Modular frontend scripts
│   │       ├── Dockerfile        # Nginx Cloud Run container
│   │       └── README.md         # UI documentation
│   ├── service/                  # Service Extensions Callout SDK Framework
│   │   ├── callout_server.py     # Base CalloutServer class & health checks
│   │   └── callout_tools.py      # Envoy header mutation and response helpers
│   └── ssl_creds/                # Default SSL certificates for secure gRPC
├── protodef/                     # Generated Envoy ext_proc Protobuf & gRPC stubs
│   ├── setup.py                  # Local package installer for proto definitions
│   ├── envoy/                    # Envoy external processor v3 definitions
│   ├── udpa/                     # Universal Data Plane API annotations
│   ├── validate/                 # Protobuf validation stubs
│   └── xds/                      # xDS protocol annotations
├── buf.gen.yaml                  # Buf configuration for regenerating proto stubs
├── Dockerfile                    # Root multi-stage Dockerfile
├── requirements.txt              # Core Python dependencies
└── requirements-test.txt         # Test dependencies (pytest)

---

⚙️ Prerequisites

• Python: 3.11 or later
• Buf CLI (Optional, only needed if regenerating protobuf stubs): Install Buf
• Docker (Optional, for containerized execution)

---

🚀 Quick Start

1. Environment Setup

Create and activate a Python virtual environment:

python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt -r requirements-test.txt

2. Generate & Install Protobuf Stubs

Install the bundled Envoy protobuf stubs as a local Python package:

pip install ./protodef

(Optional) To regenerate the protobuf stubs from the official Envoy API:

buf -v generate https://github.com/envoyproxy/envoy.git#subdir=api \
  --path envoy/service/ext_proc/v3/external_processor.proto \
  --include-imports
pip install ./protodef

3. Run the Callout Server

Start the Web Bot Auth callout server on plaintext port 8080 with combined health checks:

python -m extproc.web_bot_auth.server --plaintext_port=8080 --combined_health_check

To enable Enforce Mode (fail-closed HTTP 403 on invalid signatures):

python -m extproc.web_bot_auth.server --plaintext_port=8080 --combined_health_check --enforce

---

🧪 Running Tests & Utilities

Run Unit Tests

Execute the test suite using Python unittest or pytest:

python -m unittest discover -s extproc/web_bot_auth/test -p "test_*.py"

Or:

pytest extproc/web_bot_auth/test/test_web_bot_auth.py

Utility 1: Generate JWKS and Keys

Generate an Ed25519 or RSA key pair and matching jwks.json:

python -m extproc.web_bot_auth.test.generate_jwks \
  --alg ed25519 \
  --key_id my-crawler-key-1 \
  --out_key private_key.pem \
  --out_jwks jwks.json

Utility 2: Send Signed HTTP Requests

Test your live deployment by sending an RFC 9421 signed request:

python -m extproc.web_bot_auth.test.send_signed_request \
  --url https://your-domain.com/data \
  --key private_key.pem \
  --key_id my-crawler-key-1 \
  --agent_uri https://my-crawler.com/.well-known/jwks.json \
  --alg ed25519

---

🎨 Interactive Demo UI

An interactive web playground is included in extproc/web_bot_auth/ui/ to test signature generation, header inspection, and verification workflows visually in your browser.

To open the UI:

# Open directly in your browser
google-chrome extproc/web_bot_auth/ui/index.html
# Or serve with Python
python -m http.server 8000 --directory extproc/web_bot_auth/ui

---

🐳 Docker & Cloud Run Deployment

Build the container image using the multi-stage Dockerfile:

docker build -t google-cloud-web-bot-auth:latest .

Run locally:

docker run -p 8080:8080 google-cloud-web-bot-auth:latest

Deploy to Google Cloud Run:

gcloud run deploy web-bot-auth-callout \
  --image gcr.io/PROJECT_ID/google-cloud-web-bot-auth:latest \
  --platform managed \
  --region us-central1 \
  --use-http2 \
  --port 8080 \
  --no-allow-unauthenticated

---

🏗️ Terraform Infrastructure Deployment

A complete Terraform module is provided in terraform/ to automatically provision:

1. Cloud Run Callout Service: HTTP/2 gRPC ext_proc server with dedicated Service Account and auto-scaling.
2. Service Extensions Infrastructure: Serverless NEG, Global Backend Service (EXTERNAL_MANAGED), and google_network_services_lb_traffic_extension.
3. Sample External Application Load Balancer: Global IPv4 address, upstream origin Cloud Run app, URL map, and Forwarding Rule.

Deploy with Terraform:

cd terraform
cp terraform.tfvars.example terraform.tfvars
# Edit project_id and region in terraform.tfvars
terraform init
terraform plan
terraform apply

See terraform/README.md for full configuration details and testing instructions.

---

⚙️ Configuration Options

The callout server accepts the following command-line flags:

Flag	Default	Description
--plaintext_port	8080	Port for plaintext gRPC / HTTP2 traffic.
--combined_health_check	False	Serves health check endpoints on the gRPC port.
--enforce	False	If set, rejects unauthenticated/invalid bot requests with HTTP 403.
--allow_http	False	Allows http:// URLs for Signature-Agent (testing only).
--key_ttl	3600	JWKS public key cache TTL in seconds.
--doc_url	RFC 9421	URL included in HTTP 403 responses and Link headers.

---

📄 License

Licensed under the MIT License. See LICENSE for details.