Apache Druid Docker Compose Setup

This Docker Compose configuration sets up a complete Apache Druid cluster with all necessary components for real-time analytics and data ingestion.

Architecture Overview

This setup includes:

• Apache Druid 27.0.0 - Complete cluster with all services
• PostgreSQL - Metadata storage
• Apache Zookeeper - Coordination service
• Apache Kafka - Streaming data platform

Services

Core Infrastructure

• PostgreSQL (druid_postgres) - Stores Druid metadata on port 5432
• Zookeeper (druid_zookeeper) - Manages cluster coordination on port 2181
• Kafka (cp-kafka) - Message streaming platform on ports 29092-29093

Druid Services

• Coordinator (druid_coordinator) - Manages data availability and replication (port 8081)
• Broker (druid_broker) - Handles queries from external clients (port 8082)
• Historical (druid_historical) - Serves historical data segments (port 8083)
• MiddleManager (druid_middlemanager) - Manages data ingestion tasks (port 8091, 8100-8105)
• Router (druid_router) - Routes requests to appropriate services (port 8888)

Prerequisites

1. Docker and Docker Compose installed
2. Create a .env-druid file with Druid configuration (see Configuration section)
3. Ensure ports 2181, 5432, 8081-8083, 8091, 8100-8105, 8888, 29092-29093 are available

Quick Start

1. Clone this repository and navigate to the directory
2. Create the required environment file:

   touch .env-druid

3. Start the cluster:

   docker-compose up -d

4. Wait for all services to be healthy (this may take a few minutes)
5. Access the Druid console at: http://localhost:8888

Configuration

Create a .env-druid file with your Druid configuration. Example:

# Database
druid_metadata_storage_type=postgresql
druid_metadata_storage_connector_connectURI=jdbc:postgresql://postgres:5432/druid
druid_metadata_storage_connector_user=druid
druid_metadata_storage_connector_password=FoolishPassword

# Zookeeper
druid_zk_service_host=zookeeper:2181

# Extensions
druid_extensions_loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-kafka-indexing-service"]

# Logging
druid_log_level=INFO

# Java heap settings (adjust based on your system)
DRUID_XMX=1g
DRUID_XMS=1g

Port Reference

Service	Port	Description
PostgreSQL	5432	Database access
Zookeeper	2181	Coordination service
Coordinator	8081	Cluster management UI
Broker	8082	Query endpoint
Historical	8083	Historical data service
MiddleManager	8091	Task management
MiddleManager Tasks	8100-8105	Individual task ports
Router	8888	Main Druid Console
Kafka	29092	Internal broker access
Kafka External	29093	External broker access

Data Persistence

The following Docker volumes ensure data persistence:

• metadata_data - PostgreSQL database files
• druid_shared - Shared Druid segments and logs
• coordinator_var, broker_var, historical_var, middle_var, router_var - Service-specific data

Usage

Accessing the Web Console

• Primary Interface: http://localhost:8888 (Router - recommended)
• Coordinator UI: http://localhost:8081
• Broker Queries: http://localhost:8082

Kafka Integration

• Internal: Connect to kafka:9092 from within Docker network
• External: Connect to localhost:29092 from host machine
• External with custom IP: Set EXT_IP environment variable

Sample Query

curl -X POST 'http://localhost:8082/druid/v2/sql' \
-H 'Content-Type: application/json' \
-d '{"query":"SELECT * FROM INFORMATION_SCHEMA.TABLES"}'

Scaling and Production Notes

This configuration is designed for development and testing. For production:

1. Security: Change default passwords and add authentication
2. Resources: Adjust heap sizes in .env-druid
3. Persistence: Use external databases for metadata storage
4. Monitoring: Add monitoring and alerting
5. Load Balancing: Consider multiple instances of broker/historical nodes

Troubleshooting

Common Issues

• Services not starting: Check if ports are already in use
• Out of memory errors: Increase Docker memory allocation or reduce heap sizes
• Connection refused: Wait for all dependencies to be fully ready

Useful Commands

# Check service logs
docker-compose logs -f [service_name]

# Restart a specific service
docker-compose restart [service_name]

# Check service health
docker-compose ps

Data Ingestion

Once running, you can ingest data via:

1. Kafka: Stream data to Kafka topics and create Kafka ingestion specs
2. Batch: Upload files through the web console
3. API: Use the ingestion API endpoints

Refer to the Apache Druid documentation for detailed ingestion guides.

Shutdown

# Stop all services
docker-compose down

# Stop and remove volumes (WARNING: This deletes all data)
docker-compose down -v