🚀 Messenger Clone API

A high-performance, horizontally scalable REST API built with Go and Fiber v3, featuring database sharding, connection pooling, and streaming replication.

📊 Performance Highlights

Tested on MacBook Air M2 with 10,000 concurrent users:

Metric	Value
Success Rate	100%
Throughput	~2,100 req/sec
P50 Latency	1ms
P95 Latency	3ms
P99 Latency	11ms
Max Response	105ms

🏗️ Architecture

┌────────────────────────────────────────────────────────────────────────────────┐
│                       API Server (Fiber v3 with Prefork)                       │
│                   7 Worker Processes • Sonic JSON • :8080                      │
└────────────────────────────────────────────────────────────────────────────────┘
         │              │              │              │              │
         ▼              ▼              ▼              ▼              ▼
   ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐
   │PgBouncer │   │PgBouncer │   │PgBouncer │   │PgBouncer │   │PgBouncer │
   │ Pool 0   │   │ Pool 1   │   │ Pool 2   │   │ Pool 3   │   │ Pool 4   │
   │  :6430   │   │  :6431   │   │  :6432   │   │  :6433   │   │  :6434   │
   └────┬─────┘   └────┬─────┘   └────┬─────┘   └────┬─────┘   └────┬─────┘
        │              │              │              │              │
        ▼              ▼              ▼              ▼              ▼
   ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐
   │ Shard 0  │   │ Shard 1  │   │ Shard 2  │   │ Shard 3  │   │ Shard 4  │
   │ Primary  │   │ Primary  │   │ Primary  │   │ Primary  │   │ Primary  │
   │  :5440   │   │  :5441   │   │  :5442   │   │  :5443   │   │  :5444   │
   └────┬─────┘   └────┬─────┘   └────┬─────┘   └────┬─────┘   └────┬─────┘
        │              │              │              │              │
        ▼              ▼              ▼              ▼              ▼
   ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐
   │ Shard 0  │   │ Shard 1  │   │ Shard 2  │   │ Shard 3  │   │ Shard 4  │
   │ Replica  │   │ Replica  │   │ Replica  │   │ Replica  │   │ Replica  │
   │  :5450   │   │  :5451   │   │  :5452   │   │  :5453   │   │  :5454   │
   └──────────┘   └──────────┘   └──────────┘   └──────────┘   └──────────┘

Key Components

• Fiber v3: High-performance HTTP framework with prefork mode for multi-core utilization
• Sonic JSON: 2-5x faster JSON encoding/decoding than standard library
• PgBouncer: Connection pooling in transaction mode (10K connections → 300 DB connections)
• PostgreSQL 16: Sharded across 5 instances with streaming replication
• Consistent Hashing: MD5-based shard routing for even data distribution
• GORM: ORM with optimized settings (skip default transaction, prepared statements disabled)

🚀 Quick Start

Prerequisites

• Docker & Docker Compose
• Make
• Go 1.25+ (for local development)

Start the Application

# Start everything (infrastructure + API server with prefork)
make dev

The API will be available at http://localhost:8080

Verify Everything is Running

# Check health
make health

# View shard statistics
make shards

# Show running containers
make ps

📋 Available Commands

Run make help to see all available commands:

Infrastructure Commands

Command	Description
make up	Start infrastructure (5 shards + replicas + pgbouncers)
make down	Stop all services
make down-clean	Stop all and remove volumes (clean slate)
make ps	Show running containers
make stats	Show resource usage (CPU, memory, network)
make logs	View API server logs
make logs-all	View all service logs

API Server Commands

Command	Description
make dev	Start everything (infra + API with prefork)
make restart	Restart API server
make rebuild	Rebuild and restart API server

Database Commands

Command	Description
make shards	Show shard statistics (user counts per shard)
make health	Check API health endpoint
make db-shell SHARD=0	Connect to PostgreSQL shell on specific shard

Migration Commands

Command	Description
make migrate-make NAME=<name>	Create a new migration file
make migrate-up	Run all pending migrations
make migrate-down	Rollback the last migration
make migrate-latest	Run migrations to latest version

Load Testing Commands

Run make loadtest-help to see all load testing commands:

Command	Description
make loadtest-smoke	Quick validation (10 users, 30s)
make loadtest	Standard load test (100 users, 60s)
make loadtest-custom USERS=N	Custom user count load test
make loadtest-10k	10,000 concurrent users test
make loadtest-crud	Full CRUD operations test
make loadtest-stress	High load test (500 users, 120s)
make loadtest-spike	Sudden traffic burst test
make loadtest-soak	Extended duration test (5 min)
make loadtest-pagination	Pagination endpoint test
make loadtest-read	Read-heavy workload (80% reads)
make loadtest-write	Write-heavy workload
make loadtest-report	Open latest HTML report
make loadtest-clean	Remove test results

🔌 API Endpoints

Users

Method	Endpoint	Description
POST	/api/v1/users/	Create a new user
GET	/api/v1/users/	List users (paginated)
GET	/api/v1/users/:id	Get user by ID
PUT	/api/v1/users/:id	Update user
DELETE	/api/v1/users/:id	Delete user

System

Method	Endpoint	Description
GET	/health	Health check
GET	/api/v1/shards/stats	Shard statistics

Example Requests

# Create a user
curl -X POST http://localhost:8080/api/v1/users/ \
  -H "Content-Type: application/json" \
  -d '{"name": "John Doe", "email": "john@example.com"}'

# List users
curl http://localhost:8080/api/v1/users/?limit=20&offset=0

# Get user by ID
curl http://localhost:8080/api/v1/users/123e4567-e89b-12d3-a456-426614174000

# Update user
curl -X PUT http://localhost:8080/api/v1/users/123e4567-e89b-12d3-a456-426614174000 \
  -H "Content-Type: application/json" \
  -d '{"name": "Jane Doe"}'

# Delete user
curl -X DELETE http://localhost:8080/api/v1/users/123e4567-e89b-12d3-a456-426614174000

# Check shard statistics
curl http://localhost:8080/api/v1/shards/stats | jq

📁 Project Structure

messenger-clone/
├── Makefile                    # Main build and run commands
├── docker-compose.yml          # Docker orchestration (5 shards + replicas)
├── README.md                   # This file
│
├── server/                     # Go API Server
│   ├── Dockerfile              # Multi-stage build (dev + production)
│   ├── Makefile                # Server-specific commands
│   ├── go.mod                  # Go modules
│   ├── migration.yml           # Migration configuration
│   │
│   ├── cmd/
│   │   └── api/
│   │       └── main.go         # Application entry point
│   │
│   ├── config/
│   │   └── config.go           # Configuration management
│   │
│   ├── internal/
│   │   ├── application/        # Business logic layer
│   │   │   └── services/       # Application services
│   │   │
│   │   ├── bootstrap/          # Service initialization
│   │   │   ├── application_service.go
│   │   │   └── shard_service.go
│   │   │
│   │   ├── domain/             # Domain models
│   │   │   ├── entities/       # Entity definitions
│   │   │   └── repositories/   # Repository interfaces
│   │   │
│   │   ├── infra/              # Infrastructure layer
│   │   │   └── database/
│   │   │       └── shard/      # Sharding implementation
│   │   │
│   │   ├── persistence/        # Data access layer
│   │   │   └── repositories/   # Repository implementations
│   │   │
│   │   └── presentation/       # HTTP layer
│   │       ├── handlers/       # Request handlers
│   │       └── routes/         # Route definitions
│   │
│   ├── migrations/             # Database migrations
│   │
│   └── tools/
│       └── migration/          # Migration tool
│
├── database/                   # Database configuration
│   ├── pgbouncer/              # PgBouncer config
│   └── postgres/
│       ├── init/               # Initialization scripts
│       ├── replica/            # Replica setup scripts
│       └── pg_hba.conf         # Authentication config
│
├── loadtest/                   # Load testing
│   ├── Makefile                # Load test commands
│   ├── README.md               # Load test documentation
│   └── gatling/
│       ├── Dockerfile          # Gatling container
│       ├── entrypoint.sh       # Test runner
│       ├── conf/               # Gatling configuration
│       ├── results/            # Test results (HTML reports)
│       └── simulations/        # Test scenarios
│           ├── CreateUserSimulation.scala
│           └── UserSimulation.scala
│
├── docs/                       # 📖 Documentation
│   ├── INDEX.md                # Documentation index
│   ├── ARCHITECTURE.md         # Complete system overview
│   ├── MIGRATIONS.md           # Database migrations guide
│   ├── FOREIGN_KEYS.md         # Relations in sharded DBs
│   ├── DATABASE_SCALING.md     # Database scaling guide
│   ├── PARTITIONING_GUIDE.md   # Table partitioning guide
│   ├── PERFORMANCE_TUNING.md   # Performance optimization
│   └── SHARDING_GUIDE.md       # Sharding implementation
│
└── scripts/                    # Utility scripts
    ├── loadtest-create-user.sh
    ├── loadtest-env.sh
    ├── loadtest-sharded.sh
    ├── migrate-shards.sh
    ├── partition-shards.sh
    └── tune-postgres.sh

⚙️ Configuration

Environment Variables

Variable	Default	Description
PORT	8080	API server port
SHARDING_ENABLED	true	Enable database sharding
SHARDING_COUNT	5	Number of database shards
GOMAXPROCS	7	Number of prefork worker processes
GOGC	50	Garbage collection target percentage
GOMEMLIMIT	4GiB	Go memory limit
DB_WRITE_MAX_OPEN_CONNS	500	Max open DB connections per shard
DB_WRITE_MAX_IDLE_CONNS	250	Max idle DB connections per shard

PgBouncer Configuration

Setting	Value	Description
pool_mode	transaction	Connection pooling mode
max_client_conn	10000	Max client connections
default_pool_size	200	Default pool size per database
max_db_connections	300	Max database connections
reserve_pool_size	50	Reserve pool for burst traffic

🧪 Running Load Tests

Quick Smoke Test

make loadtest-smoke

Standard Load Test

make loadtest

10K Concurrent Users

make loadtest-10k

View Results

# Open HTML report in browser
make loadtest-report

Custom Load Test

# Run with custom user count and duration
make loadtest-custom USERS=5000 DURATION=120

🔧 Development

Hot Reload

The development mode includes hot reload via Air:

make dev

Edit any Go file in server/ and the server will automatically rebuild and restart.

Adding New Migrations

# Create a new migration
make migrate-make NAME=add_messages_table

# Run migrations
make migrate-up

Running Tests

cd server
make test

# With coverage
make test-coverage

📈 Monitoring

Resource Usage

make stats

Shard Distribution

make shards

Example output:

{
  "total_users": 9398603,
  "shard_count": 5,
  "shards": [
    { "shard_name": "shard-0", "user_count": 2132135 },
    { "shard_name": "shard-1", "user_count": 2368920 },
    { "shard_name": "shard-2", "user_count": 2424517 },
    { "shard_name": "shard-3", "user_count": 2382946 },
    { "shard_name": "shard-4", "user_count": 90085 }
  ]
}

Container Logs

# API server logs
make logs

# All service logs
make logs-all

🐳 Docker Services

Service	Port	Description
messenger_server	8080	API Server
pgbouncer-0 to pgbouncer-4	6430-6434	Connection poolers
shard-0 to shard-4	5440-5444	Primary databases
shard-0-replica to shard-4-replica	5450-5454	Read replicas
redis	6379	Cache (optional)

📚 Documentation

Start Here: docs/INDEX.md - Complete documentation index

Topic	Guide	Description
🏗️ Architecture	ARCHITECTURE.md	Complete system overview, use cases, who uses this
🔀 Sharding	SHARDING_GUIDE.md	How we distribute data across databases
🔄 Migrations	MIGRATIONS.md	Database schema versioning explained
🔗 Foreign Keys	FOREIGN_KEYS.md	Relationships in distributed databases
⚡ Performance	PERFORMANCE_TUNING.md	How we achieved 100% success rate
📊 Scaling	DATABASE_SCALING.md	Comprehensive scaling strategies
📦 Partitioning	PARTITIONING_GUIDE.md	PostgreSQL table partitioning
🧪 Load Testing	LOADTEST_INSTRUCTIONS.md	Running performance tests

Learning Paths

Beginner: ARCHITECTURE → MIGRATIONS → FOREIGN_KEYS

Scaling: DATABASE_SCALING → SHARDING_GUIDE → PARTITIONING_GUIDE

Performance: PERFORMANCE_TUNING → DATABASE_SCALING → SHARDING_GUIDE

🤝 Contributing

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Built with ❤️ using Go, Fiber, PostgreSQL, and Docker