Skip to content

Vrishali34/logsight

BranchesTags

Repository files navigation

LogSight

I built LogSight to understand how log monitoring tools work, how you ingest thousands of logs per minute, query them efficiently, and surface meaningful patterns to users.

Send logs via HTTP API → See error trends on a dashboard → Click "Analyze" to get AI explanations.

Live: https://logsight.onrender.com

Why I Built This

I was curious about how companies handle log monitoring at scale:

  • How do you detect errors from thousands of services simultaneously?
  • How do you pinpoint the exact place where something broke?
  • How do you query billions of logs efficiently?
  • How do monitoring tools like Datadog and New Relic actually work under the hood?

This project taught me:

  • How to handle real-time data ingestion
  • Why database queries are the bottleneck (not the API)
  • What JWT auth needs that API keys don't (and vice versa)
  • How to write SQL that doesn't destroy your database under load
  • Why testing actually matters (caught me breaking auth 3 times)

Quick Start

Prerequisites

  • Node.js v24+
  • PostgreSQL 18 (local) or Supabase (cloud)
  • Groq API key (free)

Setup

git clone https://github.com/Vrishali34/logsight.git
cd logsight

# Install dependencies
npm install
cd client && npm install && cd ..

# Configure environment
cp .env.example .env
# Edit: DATABASE_URL, JWT_SECRET, GROQ_API_KEY

# Start development
npm run dev          # Backend @ localhost:3000
npm run client       # Frontend @ localhost:5173

# Run tests
npm test             # 26 tests passing

How It Works

Send a Log

Your app makes an HTTP request:

curl -X POST https://logsight.onrender.com/api/logs \
  -H "x-api-key: ls_2xf1d8f8d372d08a381fa1b42b69a8256c5fcf87a" \
  -H "Content-Type: application/json" \
  -d '{
    "level": "error",
    "message": "Payment gateway timeout after 30s",
    "service": "checkout",
    "metadata": {"user_id": 123, "amount": 99.99, "gateway": "stripe"}
  }'

See Trends

Dashboard queries aggregated data:

SELECT 
  COUNT(*) as total_logs,
  COUNT(*) FILTER (WHERE level = 'error') as error_count,
  ROUND(100.0 * COUNT(*) FILTER (WHERE level = 'error') / COUNT(*), 2) as error_rate
FROM logs
WHERE app_id = 1 AND created_at > NOW() - INTERVAL '24 hours';

Result: "50% error rate. 7 errors in the last 24 hours."

Get AI Explanation

Click "Analyze":

curl https://logsight.onrender.com/api/ai/insights?app_id=1&hours=24 \
  -H "Authorization: Bearer your-jwt-token"

Groq analyzes the logs and responds:

"Your payment service is timing out waiting for Stripe. The timeouts spike at 3 AM UTC — likely a batch job running. You're not retrying failed payments. Recommend: (1) increase timeout to 60s, (2) add exponential backoff, (3) queue failed payments for async retry."

System Design

This diagram shows how the application layers communicate:

graph TB
    subgraph Browser["Client Layer"]
        UI["React + Vite<br/>Dashboard, Auth UI"]
    end

    subgraph API["Express API Layer"]
        Auth["Auth Service<br/>JWT, bcrypt"]
        Logs["Log API<br/>Ingestion"]
        Analysis["Analytics Service<br/>Trends, Summary"]
        Alerts["Alert Service<br/>Threshold Check"]
        AI["AI Service<br/>Groq Integration"]
    end

    subgraph DB["Data Layer"]
        Users[("users")]
        Apps[("apps")]
        LogsDB[("logs")]
        AlertsDB[("alert_rules")]
    end

    subgraph External["External Services"]
        Groq["Groq API<br/>Llama 3.3 70B"]
    end

    UI -->|POST /auth| Auth
    UI -->|GET /analysis| Analysis
    UI -->|POST /alerts| Alerts

    Auth -->|Read/Write| Users
    Auth -->|Read/Write| Apps

    Logs -->|INSERT| LogsDB
    Logs -->|Verify| Apps

    Analysis -->|SELECT| LogsDB
    Analysis -->|SELECT| Apps

    Alerts -->|SELECT| LogsDB
    Alerts -->|Read/Write| AlertsDB

    AI -->|SELECT| LogsDB
    AI -->|Request| Groq

    Browser -->|API Key| Logs

    style Browser fill:#6366f1,color:#fff,stroke:#4f46e5
    style API fill:#10b981,color:#fff,stroke:#059669
    style DB fill:#f59e0b,color:#fff,stroke:#d97706
    style External fill:#ef4444,color:#fff,stroke:#dc2626
Loading

Why These Choices

Raw SQL instead of ORM: I wanted to practice writing complex queries myself. An ORM would've been faster, but I needed to understand DATE_TRUNC for hourly bucketing and FILTER for conditional counts — things most ORMs hide. Raw SQL makes it obvious when a query is slow.

JWT + API keys (two auth systems): User login (JWT) and log ingestion (API keys) have different needs. JWT is short-lived, revocable, session-specific. API keys are long-lived, server-to-server, bulk operations. Splitting them makes security clearer.

PostgreSQL for time-series logs: A real system would use a time-series database (ClickHouse, TimescaleDB). PostgreSQL works fine for demos but starts struggling above ~1k logs/minute. This tradeoff helped me understand why specialized databases exist.

Groq for AI: It's cheap ($0.10 per 1M tokens) and fast (1-3 seconds). Easy to swap to OpenAI or Anthropic later.

The Stack

Component Technology Why
Backend Node.js + Express 5 Lightweight, good async error handling
Frontend React 18 + Vite Fast, familiar, works with dashboards
Database PostgreSQL 18 Mature, good for time-series queries
Auth JWT + bcrypt Stateless, no session storage needed
Validation Zod Type-safe input validation
Charts Recharts Simple line/bar charts for trends
AI Groq SDK + Llama 3.3 70B Cheap and surprisingly good at log analysis
Testing Jest + Supertest 26 tests covering all critical paths
Deployment Render + Supabase Both have free tiers, work together well

API Reference

Authentication

# Register
curl -X POST https://logsight.onrender.com/api/auth/register \
  -H "Content-Type: application/json" \
  -d '{"email":"user@example.com","password":"Secure@123"}'

# Login (returns JWT)
curl -X POST https://logsight.onrender.com/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"user@example.com","password":"Secure@123"}'

Apps & API Keys

TOKEN="your-jwt-token"

# Create app (auto-generates API key)
curl -X POST https://logsight.onrender.com/api/apps \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"My Service"}'

Log Ingestion

API_KEY="your-api-key"

# Send a log
curl -X POST https://logsight.onrender.com/api/logs \
  -H "x-api-key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "level": "error",
    "message": "Database connection failed",
    "service": "auth-service",
    "metadata": {"user_id": 123, "retry_count": 3}
  }'

Analytics

TOKEN="your-jwt-token"

# Summary metrics (24 hours)
curl "https://logsight.onrender.com/api/analysis/summary?app_id=1&hours=24" \
  -H "Authorization: Bearer $TOKEN"

# Hourly trends
curl "https://logsight.onrender.com/api/analysis/trends?app_id=1&hours=24" \
  -H "Authorization: Bearer $TOKEN"

# Per-service breakdown
curl "https://logsight.onrender.com/api/analysis/services?app_id=1&hours=24" \
  -H "Authorization: Bearer $TOKEN"

Alerts

# Create alert (trigger when error_count > 10)
curl -X POST https://logsight.onrender.com/api/alerts \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "app_id": 1,
    "metric": "error_count",
    "threshold": 10,
    "cooldown_minutes": 15
  }'

# Get alerts
curl "https://logsight.onrender.com/api/alerts?app_id=1" \
  -H "Authorization: Bearer $TOKEN"

# Delete alert
curl -X DELETE "https://logsight.onrender.com/api/alerts/1?app_id=1" \
  -H "Authorization: Bearer $TOKEN"

AI Insights

curl "https://logsight.onrender.com/api/ai/insights?app_id=1&hours=24" \
  -H "Authorization: Bearer $TOKEN"

Endpoints

Method Endpoint Auth Purpose
POST /api/auth/register None Create user
POST /api/auth/login None Get JWT token
GET /api/auth/verify JWT Verify token
POST /api/apps JWT Create monitored app
GET /api/apps JWT List apps
POST /api/logs API Key Ingest log
GET /api/logs JWT Query logs
GET /api/analysis/summary JWT Error rate, counts
GET /api/analysis/trends JWT Hourly distribution
GET /api/analysis/services JWT Per-service breakdown
POST /api/alerts JWT Create alert rule
GET /api/alerts JWT List alert rules
DELETE /api/alerts/:id JWT Delete alert rule
GET /api/ai/insights JWT AI analysis

Database Schema

Core Tables

-- Users (authentication)
CREATE TABLE users (
  id SERIAL PRIMARY KEY,
  email VARCHAR UNIQUE NOT NULL,
  password_hash VARCHAR NOT NULL,
  created_at TIMESTAMP DEFAULT NOW()
);

-- Apps (monitored applications)
CREATE TABLE apps (
  id SERIAL PRIMARY KEY,
  user_id INT REFERENCES users(id),
  name VARCHAR NOT NULL,
  api_key VARCHAR UNIQUE NOT NULL,
  created_at TIMESTAMP DEFAULT NOW()
);

-- Logs (immutable event stream)
CREATE TABLE logs (
  id SERIAL PRIMARY KEY,
  app_id INT REFERENCES apps(id),
  level VARCHAR NOT NULL,
  message TEXT NOT NULL,
  service VARCHAR,
  metadata JSONB,
  created_at TIMESTAMP DEFAULT NOW()
);
-- Key index: (app_id, created_at) for time-series queries

-- Alert Rules (thresholds)
CREATE TABLE alert_rules (
  id SERIAL PRIMARY KEY,
  app_id INT REFERENCES apps(id),
  metric VARCHAR NOT NULL,
  threshold INT NOT NULL,
  cooldown_minutes INT,
  last_triggered_at TIMESTAMP,
  created_at TIMESTAMP DEFAULT NOW()
);

Query Strategy

All analytics computed in single round-trip:

SELECT 
  COUNT(*) as total_logs,
  COUNT(*) FILTER (WHERE level = 'error') as error_count,
  COUNT(*) FILTER (WHERE level = 'warning') as warning_count,
  ROUND(
    100.0 * COUNT(*) FILTER (WHERE level = 'error') / COUNT(*),
    2
  ) as error_rate
FROM logs
WHERE app_id = $1
  AND created_at > NOW() - INTERVAL '1 hour' * $2;

Why single query?

  • Minimize database round-trips
  • Atomic snapshot of metrics
  • Lower latency for dashboard

Performance

Operation Latency Notes
API request <100ms After warm-up (excludes AI)
Summary query 5–20ms Indexed on (app_id, created_at)
Authorization check 2–5ms Single index lookup
Trends query 10–30ms GROUP BY with window functions
AI analysis 1–3s Groq network latency

Optimization Details

  • Indexes: (app_id, created_at) on logs table (time-series pattern)
  • Strategy: GROUP BY with FILTER aggregates (no temporary tables)
  • Window functions: For percentiles without joining back
  • Partitioning: Ready for future scaling (by month)

Security

Every endpoint that returns user data checks: "Does this user own this app?"

const ownershipCheck = await pool.query(
  'SELECT id FROM apps WHERE id = $1 AND user_id = $2',
  [appId, req.user.userId]
);

if (ownershipCheck.rows.length === 0) {
  return res.status(403).json({ error: 'Access denied' });
}

Tests verify:

  • User A cannot see User B's logs ✅
  • Brute force attacks are rate-limited (10 attempts per 15 minutes) ✅
  • Passwords are bcrypt-hashed, never logged ✅
  • Invalid input is rejected (Zod validation) ✅
  • API keys can be revoked per app ✅

Screenshots

Sign Up & Authentication

Sign Up

Dashboard Overview

Dashboard Overview

Real-time Logs

Recent Logs

Alert Management

Alert Rule Fired

AI-Powered Insights

AI Insights

Testing

npm test

Results:

  • ✅ Auth (register, login, JWT verification)
  • ✅ Apps (create, list apps, API key generation)
  • ✅ Logs (ingest, query, filter by level)
  • ✅ Analysis (summary, trends, service breakdown)
  • ✅ Alerts (create, trigger on threshold, delete)
  • ✅ Security (ownership checks, rate limiting)

26 tests, 0 failures.

Project Structure

logsight/
├── client/                          # React + Vite frontend
│   └── src/
│       ├── Dashboard.jsx            # Main layout
│       ├── Summary.jsx              # Error rate metrics
│       ├── TrendsChart.jsx          # Hourly line chart
│       ├── ServicesTable.jsx        # Per-service breakdown
│       ├── LogViewer.jsx            # Raw logs view
│       ├── AlertsPanel.jsx          # Alert rule management
│       └── AIInsights.jsx           # AI analysis display
│
├── src/                             # Node.js + Express backend
│   ├── features/
│   │   ├── auth/                    # Login, JWT, password hashing
│   │   ├── apps/                    # App CRUD, API key generation
│   │   ├── logs/                    # Log ingestion, query, filter
│   │   ├── analysis/                # Summary, trends, services queries
│   │   ├── alerts/                  # Alert rule creation & checking
│   │   └── ai/                      # Groq integration
│   ├── config/
│   │   └── db.js                    # PostgreSQL connection pool
│   ├── middleware/
│   │   └── errorHandler.js          # Global error handling
│   └── utils/
│       └── logger.js                # Structured logging
│
├── tests/                           # Jest test suite (26 tests)
│   ├── auth.test.js
│   ├── apps.test.js
│   ├── logs.test.js
│   ├── analysis.test.js
│   └── alerts.test.js
│
├── migrations/                      # Database schema
├── app.js                           # Express app setup
├── server.js                        # Entry point
├── Dockerfile                       # Multi-stage Docker build
├── docker-compose.yml               # Local development
├── .env.example                     # Environment template
├── package.json                     # Dependencies
├── LICENSE                          # MIT
├── docs/
│   └── screenshots/                 # Project screenshots
└── README.md                        # This file

Environment Variables

Create .env file:

# Server
NODE_ENV=development
PORT=3000

# Database (get from Supabase)
DATABASE_URL=postgresql://postgres:password@aws-0-region.pooler.supabase.com:6543/postgres

# JWT (generate: node -e "console.log(require('crypto').randomBytes(32).toString('hex'))")
JWT_SECRET=your-32-character-hex-secret-key-here
JWT_EXPIRES_IN=7d

# AI (get from console.groq.com)
GROQ_API_KEY=gsk_your_key_here

# Frontend
FRONTEND_URL=http://localhost:5173

Testing

npm test

What I Learned

Database queries are the bottleneck. I spent 3 weeks optimizing queries before even touching Express. The difference between a 500ms query and a 20ms query is bigger than anything else.

Raw SQL taught me things. I learned FILTER, DATE_TRUNC, window functions, and index strategy by necessity. An ORM would've hidden that. Now I write faster queries.

JWT vs API keys is a useful split. I initially tried to use JWT for everything. Turned out log ingestion needs different properties (fire-and-forget, no session state, revocable per app).

Testing is real. The 26 tests caught me breaking authorization checks 3 times. After that, I won't ship without tests.

Groq is actually good. I expected the AI analysis to be generic. It's not — it catches patterns I'd miss manually (like "this service has 0 logs but high error rate" = something is silently failing).

License

MIT

About

Logsight is a real time log monitoring platform with Ai powered analysis.

logsight.onrender.com

Topics

react docker express dashboard alerting full-stack restapi jwt-authentication devops-tools observability grok log-monitoring postrgresql llm ai-insights realtimeanalysis

Resources

Readme

License

MIT license

Security policy

Security policy
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages