s3-vectorbucket-api

FastAPI HTTP gateway for Amazon S3 Vectors — unified insert, query, and delete operations over multiple vector indexes.

Overview

s3-vectorbucket-api is a lightweight Python microservice that wraps the AWS S3 Vectors service behind a simple, authenticated REST API. It belongs to the shared infrastructure pillar of the WavebreakMedia ecosystem, acting as the central write and read interface for vector embeddings produced by the various vectorization services (clip-vectorization-api, convnext-vectorization-api, openai-vectorization-api, etc.) and consumed by similarity-search services (qdrant-vector-search-api, category-vector-search, waveflow-vector-search). See the workspace map at ../WBM_ECOSYSTEM_ARCHITECTURE.md for the full picture.

What It Does

• Exposes a single unified POST /api/vectors endpoint that routes to S3 Vectors PutVectors, QueryVectors, or DeleteVectors based on the operation field.
• Converts all input vectors to float32 before calling the AWS SDK, and validates dimension if configured.
• Supports WBM-specific metadata helper filters (wbmType, orientation, istransparent, liveyear) that are automatically translated into S3 Vectors filter expressions and AND-ed together.
• Accepts raw S3 Vectors filter syntax alongside the helper fields.
• Provides admin-only endpoints (POST /api/admin/vector-bucket, POST /api/admin/vector-index) for provisioning vector infrastructure, protected by an optional X-Admin-Token header.
• Supports multiple indexes per bucket; bucket and index can be specified per request or defaulted from environment variables.
• Returns structured JSON responses for all operations, including matched vectors with distances and metadata on query.

How It Works

Request Lifecycle

HTTP client
  → POST /api/vectors  (VectorOperationRequest)
  → src/api.py: vectors_router()
      validates Pydantic schema (VectorOperationRequest)
  → src/service.py: handle_operation()
      resolves bucket/index (request body → env defaults)
      dispatches to handle_insert / handle_query / handle_delete
  → src/s3vectors_client.py: get_s3vectors_client()
      lazy-initialised boto3 "s3vectors" singleton
  → AWS S3 Vectors API
      put_vectors / query_vectors / delete_vectors
  → VectorOperationResponse (JSON)

Insert Flow (operation=insert)

handle_insert in src/service.py:

1. Converts the input vector list to a NumPy float32 array and back to a Python list.
2. Optionally validates dimension against INDEX_DIMENSION if ENFORCE_DIMENSION=true.
3. Calls client.put_vectors() with key=req.id, data.float32=vector, and optional metadata dict.
4. Returns {operation, success, message}.

Query Flow (operation=query)

handle_query in src/service.py:

1. Converts query vector to float32.
2. Builds a combined metadata filter via _build_query_filter():
   • Any raw filter dict (S3 Vectors syntax) is included as-is.
   • Each set helper field (wbmType, orientation, istransparent, liveyear) becomes an {"field": {"$eq": value}} condition.
   • Multiple conditions are wrapped in {"$and": [...]}.
3. Calls client.query_vectors() with topK, queryVector, returnMetadata, returnDistance, and the combined filter.
4. Returns up to top_k matches as QueryMatch objects (key, distance, metadata). AWS S3 Vectors caps top_k at 30.

Delete Flow (operation=delete)

handle_delete in src/service.py accepts a single id or a delete_ids list and calls client.delete_vectors() with the resolved keys array.

Admin Endpoints

Both admin endpoints in src/api.py check the X-Admin-Token request header against the ADMIN_TOKEN environment variable (when set) before delegating to service.create_vector_bucket() or service.create_vector_index(). Both are idempotent — a ConflictException from AWS is treated as success.

• POST /api/admin/vector-bucket: calls client.create_vector_bucket(), optionally with KMS encryption.
• POST /api/admin/vector-index: calls client.create_index() with dataType=float32, configurable dimension, distanceMetric (cosine or euclidean), and an optional list of non-filterable metadata keys.

Tech Stack

Component	Technology
Language	Python 3.12
Web framework	FastAPI
ASGI server	Uvicorn
AWS SDK	boto3 >= 1.40.68 (requires s3vectors client support)
Data validation	Pydantic v2 (pydantic-settings for config)
Numeric processing	NumPy (float32 conversion)
Testing	pytest, pytest-cov, pytest-mock, httpx

Project Layout

s3-vectorbucket-api/
├── src/
│   ├── main.py              # FastAPI app factory; registers /api router
│   ├── api.py               # Route handlers: /api/vectors, /api/admin/*
│   ├── service.py           # Business logic: handle_operation, handle_insert/query/delete, create_vector_bucket/index
│   ├── schemas.py           # Pydantic models: VectorOperationRequest/Response, CreateBucketRequest, CreateIndexRequest
│   ├── config.py            # Pydantic BaseSettings: loads env vars / .env file
│   └── s3vectors_client.py  # Lazy singleton boto3 s3vectors client
├── tests/
│   ├── conftest.py          # Pytest fixtures (mock S3 client, test environments)
│   ├── test_service.py      # Unit tests for service layer
│   ├── test_schemas.py      # Unit tests for Pydantic models and validators
│   ├── test_config.py       # Unit tests for config loading
│   ├── test_api.py          # Integration tests for HTTP endpoints
│   ├── test_metadata.py     # Tests specific to metadata filter building
│   ├── benchmark_queries.py # Manual benchmark script (80 queries against 50k vectors)
│   └── manual_test_metadata.py # Manual metadata filter tests
├── request.py               # Interactive manual test script (all endpoints)
├── requests.http            # VS Code REST Client file for manual testing
├── requirements.txt         # Python dependencies
├── run_tests.sh             # Test runner with coverage reporting
├── pytest.ini               # Pytest configuration
└── TODO.md                  # Pending tasks: Docker, ECS, Route53, CI/CD

Configuration

Environment variables are loaded from .env (via pydantic-settings) or the shell environment. AWS credentials are resolved through the standard boto3 credential chain.

Variable	Required	Default	Notes
AWS_REGION	Yes	—	AWS region for the S3 Vectors service. In development, eu-west-1 is used.
VECTOR_BUCKET_NAME	No	None	Default vector bucket name. Can be overridden per request via vector_bucket_name body field.
VECTOR_INDEX_NAME	No	None	Default vector index name. Can be overridden per request via index_name body field.
ENFORCE_DIMENSION	No	true	When true, rejects vectors whose length != INDEX_DIMENSION (if INDEX_DIMENSION > 0).
INDEX_DIMENSION	No	0 (disabled)	Expected vector dimension. 0 disables the dimension check even when ENFORCE_DIMENSION=true.
ADMIN_TOKEN	No	None	If set, admin endpoints require X-Admin-Token: <value> header. Omit to leave admin endpoints open.

AWS credentials must be supplied via one of:

• Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, optionally AWS_SESSION_TOKEN)
• IAM role attached to the EC2 instance / ECS task / Lambda function
• AWS CLI config (~/.aws/credentials)

Example .env for local development:

AWS_REGION=eu-west-1
VECTOR_BUCKET_NAME=wbm-prod-vectors
VECTOR_INDEX_NAME=faces_512_cosine
ENFORCE_DIMENSION=true
INDEX_DIMENSION=512
ADMIN_TOKEN=your-secret-token

Data Flow and Relationships

This service is a leaf node in the WBM workspace — no sibling repository is detected as a direct upstream or downstream via SQS queues or shared S3 buckets. It is called over HTTP by other services.

Inputs / Upstream

• Vectorization APIs (callers inferred from profile): clip-vectorization-api, convnext-vectorization-api, openai-vectorization-api, videomae-vectorization-api — these services produce embedding vectors and are expected to call POST /api/vectors with operation=insert to store them.
• HTTP callers — any service that needs to perform a KNN search calls POST /api/vectors with operation=query.

Outputs / Downstream

• AWS S3 Vectors service — the sole storage backend. All vectors, metadata, bucket structures, and indexes are persisted there.
• Search APIs (callers inferred from profile): qdrant-vector-search-api, category-vector-search, waveflow-vector-search — these services are expected to query this API for nearest-neighbour results.

No SQS Usage

This service does not consume or produce any SQS queues. All interaction is synchronous HTTP.

API Reference

POST /api/vectors

Unified vector operations endpoint.

Insert:

{
  "operation": "insert",
  "id": "asset-12345",
  "vector": [0.1, 0.2, ...],
  "metadata": {"wbmType": "image", "liveyear": 2024, "istransparent": false, "orientation": "landscape"},
  "vector_bucket_name": "wbm-prod-vectors",
  "index_name": "clip_1024_cosine"
}

Query:

{
  "operation": "query",
  "vector": [0.1, 0.2, ...],
  "top_k": 10,
  "wbmType": "image",
  "orientation": "landscape",
  "istransparent": false,
  "liveyear": 2024,
  "return_metadata": true,
  "return_distance": true
}

Helper fields (wbmType, orientation, istransparent, liveyear) and the optional raw filter dict are automatically AND-ed into a single filter expression. A raw filter alone (S3 Vectors syntax) is also accepted.

Delete:

{
  "operation": "delete",
  "id": "asset-12345"
}

or batch:

{
  "operation": "delete",
  "delete_ids": ["asset-1", "asset-2", "asset-3"]
}

POST /api/admin/vector-bucket

Requires X-Admin-Token header if ADMIN_TOKEN is set.

{
  "bucket_name": "wbm-prod-vectors",
  "kms_key_arn": "arn:aws:kms:..."
}

kms_key_arn is optional; omit for SSE-S3 encryption. Idempotent.

POST /api/admin/vector-index

Requires X-Admin-Token header if ADMIN_TOKEN is set.

{
  "bucket_name": "wbm-prod-vectors",
  "index_name": "clip_1024_cosine",
  "dimension": 1024,
  "distance_metric": "cosine",
  "non_filterable_metadata_keys": ["raw_caption"]
}

distance_metric must be "cosine" or "euclidean". Idempotent.

Local Development

Prerequisites

• Python 3.12
• AWS credentials configured
• boto3 >= 1.40.68 (S3 Vectors support was added in this version)

Running

# Install dependencies
pip install -r requirements.txt

# Copy and edit env
cp .env.example .env  # or create manually from the Configuration section above

# Run development server (hot reload)
uvicorn src.main:app --reload

# Run production server
uvicorn src.main:app --host 0.0.0.0 --port 8000

The service starts on port 8000. OpenAPI docs are available at http://localhost:8000/docs.

Testing

# Run all tests with coverage (outputs htmlcov/index.html)
./run_tests.sh

# Unit tests only (no AWS calls)
./run_tests.sh unit
# or
pytest tests/test_service.py tests/test_schemas.py tests/test_config.py -v

# Integration tests (mocked AWS client)
./run_tests.sh integration
# or
pytest tests/test_api.py -v

# Quick run without coverage
./run_tests.sh quick

108 unit tests and 23 metadata-specific tests pass against mocked S3 Vectors clients — no live AWS connection required for the test suite.

Manual Testing

Start the server, then use either:

python request.py

or open requests.http in VS Code with the REST Client extension.

Caveats and Notes

• No Dockerfile yet: containerization is pending (see TODO.md). The service must currently be run directly with uvicorn. Docker + ECS/Fargate deployment is planned.
• AWS S3 Vectors top_k cap: the AWS service enforces a maximum top_k of 30. Requests with higher values will fail at the AWS layer, not at this service.
• Sequential insertion throughput: benchmarked at ~7 vectors/second via sequential HTTP requests. Batch insert via a single put_vectors call with multiple vectors is not yet exposed through this API (each request inserts one vector). For bulk loading, the API must be called in parallel.
• boto3 version requirement: boto3 >= 1.40.68 is required for the s3vectors client to exist. Earlier versions will raise Unknown service: 's3vectors' at startup.
• Pydantic v2: BaseSettings is imported from pydantic-settings, not from pydantic directly. Both packages must be installed.
• VECTOR_INDEX_NAME vs INDEX_NAME: src/config.py stores the setting as vector_index_name and exposes it via an index_name property alias. The env var to set is VECTOR_INDEX_NAME.
• Admin endpoint security: if ADMIN_TOKEN is not set, admin endpoints are open to any caller. Set it in production.
• Idempotent admin operations: creating a bucket or index that already exists returns success=true rather than an error, making provisioning scripts safe to re-run.