Open telemetry

DOCKER_SERVICES.md

@@ -0,0 +1,147 @@
+# Docker Compose Services for Arroyopy
+
+This docker-compose setup provides a complete observability stack for development.
+
+## Services
+
+### 1. Kvrocks (Redis-compatible storage)
+- **Port**: 6379
+- **Purpose**: Redis-compatible key-value store for testing
+- **Access**: `redis-cli -h localhost -p 6379`
+
+### 2. Jaeger (Distributed Tracing)
+- **UI Port**: 16686
+- **OTLP gRPC Port**: 4317
+- **OTLP HTTP Port**: 4318
+- **Purpose**: Collect and visualize distributed traces
+- **Access**: http://localhost:16686
+
+### 3. Prometheus (Metrics Collection)
+- **Port**: 9090
+- **Purpose**: Scrape and store metrics from Arroyopy applications
+- **Access**: http://localhost:9090
+- **Configuration**: `prometheus.yml`
+
+### 4. Grafana (Metrics Visualization)
+- **Port**: 3000
+- **Purpose**: Create dashboards to visualize Prometheus metrics
+- **Access**: http://localhost:3000
+- **Default Credentials**: admin/admin
+
+## Quick Start
+
+### Start all services:
+```bash
+docker-compose up -d
+```
+
+### Start specific services:
+```bash
+# Just Jaeger for tracing
+docker-compose up -d jaeger
+
+# Full observability stack
+docker-compose up -d jaeger prometheus grafana
+
+# Just Redis
+docker-compose up -d kvrocks
+```
+
+### View logs:
+```bash
+docker-compose logs -f [service_name]
+```
+
+### Stop services:
+```bash
+docker-compose down
+```
+
+### Stop and remove volumes:
+```bash
+docker-compose down -v
+```
+
+## Configuration
+
+### Prometheus
+Edit `prometheus.yml` to configure scrape targets. By default, it scrapes:
+- Arroyopy app on `host.docker.internal:8000`
+- Prometheus itself on `localhost:9090`
+
+**Linux Note**: Replace `host.docker.internal` with your Docker bridge IP (usually `172.17.0.1`). Find it with:
+```bash
+ip addr show docker0 | grep inet
+```
+
+### Grafana
+Datasources are automatically configured via `grafana-datasources.yml`.
+
+To create dashboards:
+1. Go to http://localhost:3000
+2. Login (admin/admin)
+3. Create → Dashboard
+4. Add panels with PromQL queries:
+   - `arroyopy_messages_per_second`
+   - `arroyopy_avg_processing_seconds`
+   - `rate(arroyopy_messages_received_total[5m])`
+
+**Quick Start Dashboard:**
+Import the pre-configured dashboard:
+1. Go to Dashboards → Import
+2. Upload `grafana-dashboard.json`
+3. Select the Prometheus datasource
+4. Click Import
+
+This dashboard includes:
+- Messages per second by listener type
+- Average processing time by operator type
+- Total messages received
+- Processing time percentiles (p50, p95, p99)
+
+## Using with Arroyopy
+
+### 1. Start your Arroyopy application with telemetry:
+```python
+from arroyopy import init_telemetry
+from prometheus_client import start_http_server
+
+# Initialize OpenTelemetry
+init_telemetry(service_name="my-app")
+
+# Start metrics server
+start_http_server(8000)
+```
+
+### 2. View traces in Jaeger:
+- Open http://localhost:16686
+- Select service "my-app"
+- Click "Find Traces"
+
+### 3. View metrics in Prometheus:
+- Open http://localhost:9090
+- Go to Graph
+- Enter query: `arroyopy_avg_processing_seconds`
+
+### 4. Create dashboards in Grafana:
+- Open http://localhost:3000
+- Create new dashboard
+- Add Prometheus queries for visualization
+
+## Data Persistence
+
+Metrics and dashboard data are persisted in Docker volumes:
+- `prometheus-data`: Prometheus time-series database
+- `grafana-data`: Grafana dashboards and settings
+
+## Ports Summary
+
+| Service    | Port  | Purpose           |
+|------------|-------|-------------------|
+| Kvrocks    | 6379  | Redis API         |
+| Jaeger UI  | 16686 | Web Interface     |
+| Jaeger OTLP| 4317  | Trace Collection  |
+| Prometheus | 9090  | Metrics & UI      |
+| Grafana    | 3000  | Dashboards & UI   |
+
+All ports are bound to `127.0.0.1` for security on development machines.

README.md

@@ -249,6 +249,35 @@ pixi run -e dev pre-commit-install
 pixi run -e dev clean
 ```
 
+### Docker Services for Development
+
+The repository includes a docker-compose setup with services for testing and observability:
+
+```bash
+# Start all services (Redis, Jaeger, Prometheus, Grafana)
+docker-compose up -d
+
+# Start specific services
+docker-compose up -d kvrocks     # Redis-compatible storage
+docker-compose up -d jaeger      # Distributed tracing
+docker-compose up -d prometheus  # Metrics collection
+docker-compose up -d grafana     # Metrics visualization
+
+# View logs
+docker-compose logs -f
+
+# Stop services
+docker-compose down
+```
+
+**Service Access:**
+- Jaeger UI: http://localhost:16686
+- Prometheus: http://localhost:9090
+- Grafana: http://localhost:3000 (admin/admin)
+- Redis: localhost:6379
+
+See [DOCKER_SERVICES.md](DOCKER_SERVICES.md) for detailed documentation.
+
 ### Using Different Environments
 
 ```bash

TELEMETRY_QUICKSTART.md

@@ -0,0 +1,153 @@
+# OpenTelemetry Integration - Quick Reference
+
+## What was added:
+
+### 1. New Module: `telemetry.py`
+- **`@traced` decorator**: Creates Jaeger spans for functions
+- **`init_telemetry()`**: Initialize OpenTelemetry with Jaeger
+- **`get_metrics_tracker()`**: Access the global metrics tracker
+- **Automatic Prometheus metrics** for listeners and operators
+
+### 2. Dependencies Added (pyproject.toml)
+- `opentelemetry-api`
+- `opentelemetry-sdk`
+- `opentelemetry-exporter-jaeger`
+- `prometheus-client`
+
+### 3. Automatic Metrics Collection
+
+#### Listeners (RedisListener, ZMQListener)
+- **Messages per second** - Real-time message rate
+- **Total messages received** - Cumulative counter
+- Labeled by listener type (class name)
+
+#### Operators
+- **Average processing time** - Running average
+- **Processing time histogram** - Full distribution
+- Labeled by operator type (class name)
+
+## Quick Start
+
+### Use the @traced decorator:
+
+```python
+from arroyopy import traced, Operator
+
+class MyOperator(Operator):
+    @traced()  # Simple usage
+    async def process(self, message):
+        return message
+
+    @traced(span_name="validate", attributes={"version": "1.0"})
+    async def validate(self, data):
+        return data
+```
+
+### Initialize tracing (optional):
+
+```python
+from arroyopy import init_telemetry
+
+init_telemetry(
+    service_name="my-app",
+    otlp_endpoint="http://localhost:4317"
+)
+```
+
+### Expose Prometheus metrics:
+
+```python
+from prometheus_client import start_http_server
+
+# Start metrics server
+start_http_server(8000)
+# Metrics at http://localhost:8000/metrics
+```
+
+## Prometheus Metrics Available
+
+| Metric | Type | Description | Labels |
+|--------|------|-------------|--------|
+| `arroyopy_messages_received_total` | Counter | Total messages received | listener_type |
+| `arroyopy_messages_per_second` | Gauge | Current message rate | listener_type |
+| `arroyopy_processing_seconds` | Histogram | Processing time distribution | operator_type |
+| `arroyopy_avg_processing_seconds` | Gauge | Average processing time | operator_type |
+
+## Files Created
+
+1. **`src/arroyopy/telemetry.py`** - Main telemetry module
+2. **`src/_test/test_telemetry.py`** - Unit tests
+3. **`examples/telemetry_example.py`** - Usage examples
+4. **`examples/prometheus_example.py`** - Metrics server example
+5. **`docs/telemetry.md`** - Complete documentation
+
+## Files Modified
+
+1. **`pyproject.toml`** - Added dependencies
+2. **`src/arroyopy/__init__.py`** - Exported telemetry functions
+3. **`src/arroyopy/operator.py`** - Added processing time tracking
+4. **`src/arroyopy/redis.py`** - Added message rate tracking
+5. **`src/arroyopy/zmq.py`** - Added message rate tracking
+
+## Next Steps
+
+1. **Install new dependencies:**
+   ```bash
+   pixi install  # or pip install -e .
+   ```
+
+2. **Start observability stack:**
+   ```bash
+   # Option 1: Full stack (recommended) - Jaeger + Prometheus + Grafana
+   docker-compose up -d
+
+   # Option 2: Just Jaeger for tracing
+   docker-compose up -d jaeger
+
+   # Option 3: Using docker directly
+   docker run -d -p 4317:4317 -p 16686:16686 jaegertracing/all-in-one
+   ```
+
+   **Access points:**
+   - Jaeger UI: http://localhost:16686
+   - Prometheus: http://localhost:9090
+   - Grafana: http://localhost:3000 (admin/admin)
+
+3. **Add metrics endpoint to your app:**
+   ```python
+   from prometheus_client import start_http_server
+   start_http_server(8000)
+   ```
+
+4. **Use @traced decorator in your operators:**
+   ```python
+   @traced(span_name="my_process")
+   async def process(self, message):
+       # your code
+   ```
+
+See [DOCKER_SERVICES.md](DOCKER_SERVICES.md) for detailed docker-compose documentation.
+
+## How It Works
+
+### Automatic Collection
+- **No code changes needed** for basic metrics
+- Listeners automatically track message rates when messages are received
+- Operators automatically track processing times when messages are processed
+- All metrics exported to Prometheus automatically
+
+### Manual Tracing
+- Use `@traced` decorator for custom spans
+- Supports nested spans
+- Works with sync and async functions
+- Automatically captures errors and exceptions
+
+## Documentation
+
+See [docs/telemetry.md](../docs/telemetry.md) for complete documentation including:
+- Detailed usage examples
+- Configuration options
+- Prometheus query examples
+- Grafana dashboard setup
+- Troubleshooting guide
+- Best practices