Public Data Access BB

The Public Data Access Building Block (BB) is a unified gateway service that provides standardized access to multiple European public data APIs through a single, consistent interface. By default, it connects to EUROSTAT (European Statistical Office) APIs, with the capability to integrate additional public data sources by simply editing the endpoints.json configuration file.

Design Document

See the design document here.

Requirements

Software Requirements

Python 3.11+ (tested with Python 3.13)
pip (Python package installer)
Docker (optional, for containerized deployment)
Docker Compose (optional, for orchestrated deployment)

Python Dependencies

All Python dependencies are listed in requirements.txt:

FastAPI 0.104.1
Uvicorn 0.24.0
httpx 0.25.2
python-multipart 0.0.6
requests 2.31.0

Building instructions

The Public Data Access BB can be built and deployed using multiple approaches. Choose the method that best fits your infrastructure requirements.

Docker (Recommended for Production)

docker build -t public-data-access:latest .

Local Development Environment

python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt

Adding New Public Data APIs

To integrate additional public data sources, edit the config/endpoints.json file:

{
  "ESTAT": { 
    "code": "ESTAT",
    "url": "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data",
    "description": "EUROSTAT Statistical Data API",
    "requires_auth": false,
    "auth_token": null
  },
  "NEW_API": { 
    "code": "NEW_API",
    "url": "https://new-public-api.gov/api",
    "description": "New Public Data API",
    "requires_auth": true,
    "auth_token": "your-api-token-here"
  }
}

Configuration Fields:

code: Unique identifier for the endpoint
url: Base URL of the external API
description: Human-readable description
requires_auth: Set to true if the external API requires authentication
auth_token: Bearer token to authenticate with the external API (used only if requires_auth is true)
example_usage: Example request path (optional)

Running instructions

The BB supports multiple deployment scenarios from local development to production containerized environments.

Docker Compose (Recommended for Production)

docker compose up

Docker Container

docker run -p 8000:8000 \
  -v $(pwd)/config:/app/config \
  -v $(pwd)/logs:/app/logs \
  public-data-access:latest

Local Development & Testing

./scripts/start.sh

or manually:

python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt
python3 src/main.py

The BB will start on http://localhost:8000 with interactive API documentation available at http://localhost:8000/docs

Example usage

The Public Data Access BB provides a unified interface to access multiple European public data APIs. Below are examples for the default integrated API (EUROSTAT):

Endpoint	endpoint_code	data_object	Expected Output
GET /	-	-	200, BB service information and available endpoints
GET /endpoints	-	-	200, List of configured public data APIs (ESTAT, custom)
GET /usage-stats	-	-	200, Usage analytics and access statistics
GET /{endpoint_code}/{data_object}	ESTAT	NAMA_10_GDP/A.CP_MEUR.B1GQ.LU?startPeriod=2020&endPeriod=2024&format=json	200, JSON with EUROSTAT statistical data
GET /{endpoint_code}/{data_object}	CUSTOM_CODE	Varies by integrated API	200, Response from custom public data API

BB Authentication Methods

The BB supports multiple authentication approaches for flexibility across different integration scenarios:

# Method 1: X-API-Key header (recommended for applications)
curl -H "X-API-Key: dev-api-key-1234567890abcdef" \
  "http://localhost:8000/ESTAT/NAMA_10_GDP/A.CP_MEUR.B1GQ.LU?startPeriod=2020&endPeriod=2024&format=json"


# Method 2: Bearer token (OAuth 2.0 compatible)
curl -H "Authorization: Bearer dev-api-key-1234567890abcdef" \
  "http://localhost:8000/ESTAT/NAMA_10_GDP/A.CP_MEUR.B1GQ.LU?startPeriod=2020&endPeriod=2024&format=json"


# Method 3: Query parameter (simple integration)
curl "http://localhost:8000/ESTAT/NAMA_10_GDP/A.CP_MEUR.B1GQ.LU?startPeriod=2020&endPeriod=2024&format=json&api_key=dev-api-key-1234567890abcdef"


# Open access (if enabled for specific endpoints)
curl "http://localhost:8000/ESTAT/NAMA_10_GDP/A.CP_MEUR.B1GQ.LU?startPeriod=2020&endPeriod=2024&format=json"

Request Timeout Configuration

The BB implements configurable request timeouts to balance performance and reliability:

Open Access: 30 seconds (default timeout for requests without API key)
API Key Access: Configurable per API key in config/api_keys.json

Example API Key Configuration:

{
  "dev-api-key-1234567890abcdef": {
    "institution": "rejustify",
    "contact_name": "rejustify",
    "contact_email": "info@rejustify.com",
    "department": "Data analysis team",
    "created_date": "2024-11-01",
    "rate_limit": 10000,
    "allowed_endpoints": ["ESTAT"],
    "timeout": 60
  }
}

Users with valid API keys can have extended timeout limits (e.g., 60 seconds) for long-running data queries, while open access maintains a 30-second limit to prevent resource exhaustion.

Unit testing

The Public Data Access BB includes comprehensive testing to ensure reliability and proper functionality of all BB components.

Setup test environment

Refer to the Local Development Environment section above for full details. The essential steps are:

python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt

Important: Before running the tests, make sure the Public Data Access BB server is running (e.g. setup the BB in the background or in a separate window).

Run tests

python tests/test_api.py

Expected results

FastAPI Pass-through API Test Suite
==================================
🧪 Testing FastAPI Pass-through API at http://localhost:8000
============================================================

📋 Testing: Root Endpoint
✅ Root endpoint working

📋 Testing: Endpoints List
✅ Endpoints list working (1 endpoints)

📋 Testing: Usage Stats
✅ Usage stats endpoint working

📋 Testing: Invalid Endpoint
✅ Invalid endpoint correctly returns 404

📋 Testing: API Key Authentication
✅ Valid API key authentication working
✅ Invalid API key correctly returns 401

📋 Testing: Different Auth Methods
✅ X-API-Key header authentication working
✅ Authorization Bearer header authentication working
✅ Query parameter authentication working

📋 Testing: Open Access
✅ Open access working

📋 Testing: Usage Stats Update
✅ Usage stats updating correctly

============================================================
📊 Test Results: 8/8 tests passed
🎉 All tests passed! API is working correctly.

Component-level testing

Test the Public Data Access BB as an integrated component, validating its ability to provide unified access to multiple public data APIs with proper BB functionality.

Setup test environment

Refer to the Local Development Environment section above for full details. The essential steps are:

python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt

Important: Before running the component-level tests, make sure the Public Data Access BB server is running in a separate terminal or background process:

./scripts/start.sh

Then verify external public data APIs (EUROSTAT) are accessible and ensure BB configuration files are properly loaded.

Run tests

# Test BB integration with EUROSTAT public API  
curl "http://localhost:8000/ESTAT/NAMA_10_GDP/A.CP_MEUR.B1GQ.LU?startPeriod=2023&endPeriod=2024&format=json"


# Test BB authentication and authorization
curl -H "X-API-Key: dev-api-key-1234567890abcdef" \
  "http://localhost:8000/ESTAT/NAMA_10_GDP/A.CP_MEUR.B1GQ.LU?startPeriod=2023&endPeriod=2024&format=json"

# Test BB endpoint discovery
curl "http://localhost:8000/endpoints"

# Test BB usage analytics
curl "http://localhost:8000/usage-stats"

# Test BB error handling for non-existent endpoints
curl "http://localhost:8000/NONEXISTENT?param=value"

Expected results

The following are example outputs based on the current configuration and usage statistics. Actual results may vary depending on your deployment parameters, API key usage, and data updates.

Below are the actual outputs you should expect from each run command above (output truncated for brevity):

EUROSTAT Integration

curl "http://localhost:8000/ESTAT/NAMA_10_GDP/A.CP_MEUR.B1GQ.LU?startPeriod=2023&endPeriod=2024&format=json"

Output:

{"version":"2.0","class":"dataset","label":"Gross domestic product (GDP) and main components (output, expenditure and income)", ... "value":{"0":82115.5,"1":86180.3}, ...}

(Returns JSON with statistical data in SDMX format for the requested table and period.)

BB Authentication

curl -H "X-API-Key: dev-api-key-1234567890abcdef" "http://localhost:8000/ESTAT/NAMA_10_GDP/A.CP_MEUR.B1GQ.LU?startPeriod=2023&endPeriod=2024&format=json"

Output:

{"version":"2.0","class":"dataset","label":"Gross domestic product (GDP) and main components (output, expenditure and income)", ... "value":{"0":82115.5,"1":86180.3}, ...}

(Returns the same data as above if the API key is valid. If the key is invalid, you will get a 401 error.)

BB Endpoint Discovery

curl "http://localhost:8000/endpoints"

Output:

{"endpoints":{"ESTAT":{"code":"ESTAT","url":"https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data","description":"EUROSTAT Statistical Data API", ...}},"total_count":1}

(Lists all configured endpoints.)

BB Usage Analytics

curl "http://localhost:8000/usage-stats"

Output:

{"open_access":{"count":1,"last_accessed":"2026-01-27T18:49:35.356764"},"api_key_usage":{"dev-api-key-1234567890abcdef":{"count":1,"last_accessed":"2026-01-27T18:49:42.367582"}}}

(Shows usage statistics for open access and API key usage.)

BB Error Handling

curl "http://localhost:8000/NONEXISTENT?param=value"

Output:

{"detail":"Endpoint code 'NONEXISTENT' not found. Available endpoints: ESTAT"}

(Returns a standardized 404 error response in JSON format.)

BB Logging All requests above are also logged to logs/api_requests.log with request details.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Public Data Access BB

Design Document

Requirements

Software Requirements

Python Dependencies

Building instructions

Docker (Recommended for Production)

Local Development Environment

Adding New Public Data APIs

Running instructions

Docker Compose (Recommended for Production)

Docker Container

Local Development & Testing

Example usage

BB Authentication Methods

Request Timeout Configuration

Unit testing

Setup test environment

Run tests

Expected results

Component-level testing

Setup test environment

Run tests

Expected results

About

Uh oh!

Releases

Packages

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 32 Commits
config		config
docs		docs
logs		logs
scripts		scripts
src		src
tests		tests
.gitignore		.gitignore
Dockerfile		Dockerfile
README.md		README.md
docker-compose.yml		docker-compose.yml
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation

Public Data Access BB

Design Document

Requirements

Software Requirements

Python Dependencies

Building instructions

Docker (Recommended for Production)

Local Development Environment

Adding New Public Data APIs

Running instructions

Docker Compose (Recommended for Production)

Docker Container

Local Development & Testing

Example usage

BB Authentication Methods

Request Timeout Configuration

Unit testing

Setup test environment

Run tests

Expected results

Component-level testing

Setup test environment

Run tests

Expected results

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Packages