We collect your personal information including name, email, and browsing data. We share this data with third-party advertisers. You have no right to delete your data. We retain data indefinitely. We use industry-standard security.
" - }' -``` - ---- - -## Testing - -### Unit Tests - -Run all unit tests (no external services needed): - -```bash -go test ./backend/... -``` - -With coverage: - -```bash -go test ./backend/... -cover -``` - -### Integration Tests - -Integration tests exercise the full pipeline against real services. They require: -- Running Qdrant instance (or in-memory store depending on test) -- Valid `ANTHROPIC_API_KEY` and `OPENAI_API_KEY` in your environment - -```bash -# Start Qdrant if needed (via Compose or standalone container) - -# Run integration tests with the build tag: -go test -tags=integration ./backend/internal/integration/... -v -timeout 120s -``` - -Integration tests are gated behind the `//go:build integration` build tag, so they won't run during normal `go test ./backend/...`. - ---- - -## Deploying to a Server - -### Build the Docker Image - -```bash -docker build -t smolterms-backend ./backend -``` - -Or with Podman: - -```bash -podman build -t smolterms-backend ./backend -``` - -The Dockerfile uses a multi-stage build: -1. **Build stage:** Compiles a static Go binary with `CGO_ENABLED=0` -2. **Runtime stage:** Uses `gcr.io/distroless/static-debian12` (minimal, no shell, ~2MB base) - -The resulting image is small and contains only the compiled binary. - -### Deploying with Docker Compose on a Server - -**1. Copy project files to your server:** - -```bash -scp -r docker-compose.yml backend/Dockerfile .env.example user@server:/opt/smolterms/ -``` - -**2. On the server, create your `.env` file:** - -```bash -cd /opt/smolterms -cp .env.example .env -# Edit .env with production API keys and settings -``` - -**3. Start the services:** - -```bash -# Backend only (in-memory vector store): -docker compose up --build -d - -# With Qdrant (set VECTOR_STORE=qdrant and QDRANT_URL=qdrant:6334 in .env): -docker compose --profile qdrant up --build -d -``` - -**4. (Optional) Set up a reverse proxy:** - -Put Nginx, Caddy, or Traefik in front of port 8080 to handle TLS and domain routing. Example Nginx config: - -```nginx -server { - listen 443 ssl; - server_name smolterms.example.com; - - ssl_certificate /path/to/cert.pem; - ssl_certificate_key /path/to/key.pem; - - location / { - proxy_pass http://127.0.0.1:8080; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_read_timeout 120s; # analysis can take 10-30s - } -} -``` - -### Deploying the Standalone Binary - -If you prefer not to use containers on the server: - -**1. Build the binary locally (cross-compile for Linux):** - -```bash -CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o smolterms-server ./backend/cmd/server/main.go -``` - -For ARM servers (e.g., AWS Graviton): - -```bash -CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o smolterms-server ./backend/cmd/server/main.go -``` - -**2. Copy to server and run:** - -```bash -scp smolterms-server user@server:/opt/smolterms/ -ssh user@server - -# Set environment variables -export ANTHROPIC_API_KEY="sk-ant-..." -export OPENAI_API_KEY="sk-..." -# VECTOR_STORE defaults to "memory" — no Qdrant needed - -# Run -/opt/smolterms/smolterms-server -``` - -**3. (Optional) Run Qdrant for persistent vector storage:** - -Set `VECTOR_STORE=qdrant` and `QDRANT_URL=localhost:6334`, then: - -```bash -docker run -d --name qdrant \ - -p 6333:6333 -p 6334:6334 \ - -v /opt/smolterms/qdrant_data:/qdrant/storage \ - --restart unless-stopped \ - qdrant/qdrant -``` - -**4. (Optional) Create a systemd service:** - -```ini -# /etc/systemd/system/smolterms.service -[Unit] -Description=SmolTerms Backend -After=network.target - -[Service] -Type=simple -User=smolterms -WorkingDirectory=/opt/smolterms -ExecStart=/opt/smolterms/smolterms-server -EnvironmentFile=/opt/smolterms/.env -Restart=on-failure -RestartSec=5 - -[Install] -WantedBy=multi-user.target -``` - -```bash -sudo systemctl daemon-reload -sudo systemctl enable --now smolterms -``` - -### Production Checklist - -- [ ] Set `LOG_LEVEL=warn` or `LOG_LEVEL=error` for production -- [ ] Set spending limits on both Anthropic and OpenAI dashboards -- [ ] Put a reverse proxy (Nginx/Caddy) in front for TLS termination -- [ ] If using Qdrant (`VECTOR_STORE=qdrant`): secure it -- by default it has no authentication; consider binding to localhost only or adding an API key via [Qdrant security config](https://qdrant.tech/documentation/guides/security/) -- [ ] Set up monitoring for the `/api/v1/health` endpoint -- [ ] Configure firewall rules -- only expose ports 80/443 publicly, keep 8080/6333/6334 internal -- [ ] If using Qdrant: back up the Qdrant volume periodically (or use [Qdrant snapshots](https://qdrant.tech/documentation/concepts/snapshots/)) -- [ ] Consider rate limiting at the reverse proxy level to prevent abuse - ---- - ## Project Structure ``` diff --git a/backend/README.md b/backend/README.md index 7af4f65..9eb1311 100644 --- a/backend/README.md +++ b/backend/README.md @@ -1,7 +1,427 @@ +## Getting API Keys + +### Anthropic API Key (for Claude LLM) + +SmolTerms uses Claude Sonnet 4.5 for analyzing privacy policies and generating structured scores. + +**1. Create an account:** +- Go to [console.anthropic.com](https://console.anthropic.com) and sign up +- Verify your email address + +**2. Add credits:** +- Navigate to **Settings > Billing** in the Console +- Add a payment method and purchase credits +- A minimum $5 purchase gets you to **Tier 1** (50 RPM, 30K input tokens/min) +- $40 cumulative gets you to **Tier 2** (1,000 RPM, 450K input tokens/min), which is more than enough for development + +**3. Generate an API key:** +- Go to [console.anthropic.com/settings/keys](https://console.anthropic.com/settings/keys) +- Click **Create Key**, give it a name (e.g., "smolterms-dev") +- Copy the key immediately -- it starts with `sk-ant-` and won't be shown again + +**4. Set spending limits (recommended):** +- Go to **Settings > Limits** in the Console +- Set a monthly spend limit (e.g., $10-20 for development) +- You can also configure per-workspace limits for tighter control + +**Pricing for SmolTerms usage (Claude Sonnet 4.5):** +- Input: $3 / million tokens +- Output: $15 / million tokens +- A typical privacy policy analysis uses ~5K-15K input tokens and ~1K-2K output tokens +- Estimated cost per analysis: ~$0.02-$0.06 + +**Rate limits by tier:** + +| Tier | Credit Purchase | RPM | Input Tokens/min | +|---|---|---|---| +| Tier 1 | $5 | 50 | 30,000 | +| Tier 2 | $40 | 1,000 | 450,000 | +| Tier 3 | $200 | 2,000 | 800,000 | + +For development and testing, Tier 1 is sufficient. You'll automatically advance tiers as your cumulative credit purchases increase. + +### OpenAI API Key (for Embeddings) + +SmolTerms uses OpenAI's `text-embedding-3-small` model (1536 dimensions) to generate vector embeddings for the RAG pipeline. + +**1. Create an account:** +- Go to [platform.openai.com](https://platform.openai.com) and sign up +- Verify your email and phone number + +**2. Add credits:** +- Navigate to **Settings > Billing** in the dashboard +- Add a payment method and purchase credits +- $5 gets you to **Tier 1** (which is sufficient for development) + +**3. Generate an API key:** +- Go to [platform.openai.com/api-keys](https://platform.openai.com/api-keys) +- Click **Create new secret key**, give it a name (e.g., "smolterms-dev") +- Copy the key immediately -- it starts with `sk-` and won't be shown again + +**4. Set usage limits (recommended):** +- Go to **Settings > Limits** in the dashboard +- Set a monthly budget limit (e.g., $5-10 for development) +- You can set both a soft limit (email alert) and a hard limit (requests blocked) + +**Pricing for SmolTerms usage (text-embedding-3-small):** +- $0.02 / million tokens +- A typical privacy policy (~10K tokens across all chunks) costs ~$0.0002 to embed +- Embeddings are extremely cheap -- the OpenAI costs for SmolTerms are negligible + +**Rate limits by tier:** + +| Tier | Qualification | Usage Limit/month | +|---|---|---| +| Free | Allowed geography | $100/month | +| Tier 1 | $5 paid | $100/month | +| Tier 2 | $50 paid + 7 days | $500/month | + +For development, the free tier or Tier 1 is more than adequate. + +--- + +## Environment Setup + +Copy the example environment file and fill in your API keys + +## Local Development + +### Option A: Docker Compose + +Requires [Docker Engine](https://docs.docker.com/engine/install/) and [Docker Compose](https://docs.docker.com/compose/install/) (v2+). + +**Start the backend (uses in-memory vector store by default):** + +```bash +docker compose up --build +``` + +This will: +1. Build the Go backend from `backend/Dockerfile` (multi-stage, distroless image) +2. Expose the backend on `http://localhost:8080` +3. Use the in-memory vector store (no external dependencies needed) + +**Start with Qdrant (for persistent vector storage):** + +```bash +docker compose --profile qdrant up --build +``` + +This additionally starts the Qdrant container. You must also set `VECTOR_STORE=qdrant` and `QDRANT_URL=qdrant:6334` in your `.env` file. + +**Run in the background:** + +```bash +docker compose up --build -d +# Or with Qdrant: +docker compose --profile qdrant up --build -d +``` + +**View logs:** + +```bash +docker compose logs -f # all services +docker compose logs -f backend # backend only +docker compose logs -f qdrant # qdrant only (requires --profile qdrant) +``` + +**Stop everything:** + +```bash +docker compose down # stop containers +docker compose down -v # stop containers AND remove Qdrant data volume +``` + +### Option B: Podman Compose + +Requires [Podman](https://podman.io/docs/installation) and [Podman Compose](https://github.com/containers/podman-compose). + +**Start the backend (uses in-memory vector store by default):** + +```bash +podman compose up --build +``` + +**Start with Qdrant:** + +```bash +podman compose --profile qdrant up --build +``` + +Set `VECTOR_STORE=qdrant` and `QDRANT_URL=qdrant:6334` in your `.env` file when using Qdrant. + +**Run in the background:** + +```bash +podman compose up --build -d +# Or with Qdrant: +podman compose --profile qdrant up --build -d +``` + +**View logs:** + +```bash +podman compose logs -f +podman compose logs -f backend +podman compose logs -f qdrant # requires --profile qdrant +``` + +**Stop everything:** + +```bash +podman compose down +podman compose down -v # also remove Qdrant data volume +``` + +> **Note:** Podman Compose reads the same `docker-compose.yml` file. If you encounter networking issues with Podman, you may need to ensure the `podman` socket is running (`systemctl --user start podman.socket`) or use `podman-compose` (the Python-based variant) instead of the Go-based `podman compose`. + +### Option C: Run Backend Directly (without containers) + +Run the Go backend directly. By default it uses the in-memory vector store, so no external services are needed. + +**Run the backend:** + +```bash +go run ./backend/cmd/server/main.go +``` + +**(Optional) Use Qdrant for persistent vector storage:** + +Set `VECTOR_STORE=qdrant` and `QDRANT_URL=localhost:6334` in your `.env`, then start Qdrant: + +```bash +# Docker: +docker run -d --name qdrant -p 6333:6333 -p 6334:6334 \ + -v qdrant_data:/qdrant/storage qdrant/qdrant + +# Podman: +podman run -d --name qdrant -p 6333:6333 -p 6334:6334 \ + -v qdrant_data:/qdrant/storage qdrant/qdrant +``` + +### Verifying the Setup + +Once everything is running, check the health endpoint: + +```bash +curl http://localhost:8080/api/v1/health +``` + +Expected response (with in-memory vector store): + +```json +{ + "status": "healthy", + "services": { + "vectorstore": { "status": "healthy" }, + "anthropic": { "status": "configured" }, + "openai": { "status": "configured" } + } +} +``` + +Try an analysis (requires valid API keys): + +```bash +curl -X POST http://localhost:8080/api/v1/analyze \ + -H "Content-Type: application/json" \ + -d '{ + "url": "https://example.com/privacy", + "html": "We collect your personal information including name, email, and browsing data. We share this data with third-party advertisers. You have no right to delete your data. We retain data indefinitely. We use industry-standard security.
" + }' +``` + +--- + +## Testing + +### Unit Tests + +Run all unit tests (no external services needed): + +```bash +go test ./backend/... +``` + +With coverage: + +```bash +go test ./backend/... -cover +``` + +### Integration Tests + +Integration tests exercise the full pipeline against real services. They require: +- Running Qdrant instance (or in-memory store depending on test) +- Valid `ANTHROPIC_API_KEY` and `OPENAI_API_KEY` in your environment + +```bash +# Start Qdrant if needed (via Compose or standalone container) + +# Run integration tests with the build tag: +go test -tags=integration ./backend/internal/integration/... -v -timeout 120s +``` + +Integration tests are gated behind the `//go:build integration` build tag, so they won't run during normal `go test ./backend/...`. + +--- + +## Deploying to a Server + +### Build the Docker Image + +```bash +docker build -t smolterms-backend ./backend +``` + +Or with Podman: + +```bash +podman build -t smolterms-backend ./backend +``` + +The Dockerfile uses a multi-stage build: +1. **Build stage:** Compiles a static Go binary with `CGO_ENABLED=0` +2. **Runtime stage:** Uses `gcr.io/distroless/static-debian12` (minimal, no shell, ~2MB base) + +The resulting image is small and contains only the compiled binary. + +### Deploying with Docker Compose on a Server + +**1. Copy project files to your server:** + +```bash +scp -r docker-compose.yml backend/Dockerfile .env.example user@server:/opt/smolterms/ +``` + +**2. On the server, create your `.env` file:** + +```bash +cd /opt/smolterms +cp .env.example .env +# Edit .env with production API keys and settings +``` + +**3. Start the services:** + +```bash +# Backend only (in-memory vector store): +docker compose up --build -d + +# With Qdrant (set VECTOR_STORE=qdrant and QDRANT_URL=qdrant:6334 in .env): +docker compose --profile qdrant up --build -d +``` + +**4. (Optional) Set up a reverse proxy:** + +Put Nginx, Caddy, or Traefik in front of port 8080 to handle TLS and domain routing. Example Nginx config: + +```nginx +server { + listen 443 ssl; + server_name smolterms.example.com; + + ssl_certificate /path/to/cert.pem; + ssl_certificate_key /path/to/key.pem; + + location / { + proxy_pass http://127.0.0.1:8080; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_read_timeout 120s; # analysis can take 10-30s + } +} +``` + +### Deploying the Standalone Binary + +If you prefer not to use containers on the server: + +**1. Build the binary locally (cross-compile for Linux):** + +```bash +CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o smolterms-server ./backend/cmd/server/main.go +``` + +For ARM servers (e.g., AWS Graviton): + +```bash +CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o smolterms-server ./backend/cmd/server/main.go +``` + +**2. Copy to server and run:** + +```bash +scp smolterms-server user@server:/opt/smolterms/ +ssh user@server + +# Set environment variables +export ANTHROPIC_API_KEY="sk-ant-..." +export OPENAI_API_KEY="sk-..." +# VECTOR_STORE defaults to "memory" — no Qdrant needed + +# Run +/opt/smolterms/smolterms-server +``` + +**3. (Optional) Run Qdrant for persistent vector storage:** + +Set `VECTOR_STORE=qdrant` and `QDRANT_URL=localhost:6334`, then: + +```bash +docker run -d --name qdrant \ + -p 6333:6333 -p 6334:6334 \ + -v /opt/smolterms/qdrant_data:/qdrant/storage \ + --restart unless-stopped \ + qdrant/qdrant +``` + +**4. (Optional) Create a systemd service:** + +```ini +# /etc/systemd/system/smolterms.service +[Unit] +Description=SmolTerms Backend +After=network.target + +[Service] +Type=simple +User=smolterms +WorkingDirectory=/opt/smolterms +ExecStart=/opt/smolterms/smolterms-server +EnvironmentFile=/opt/smolterms/.env +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=multi-user.target +``` + +```bash +sudo systemctl daemon-reload +sudo systemctl enable --now smolterms +``` + +### Production Checklist + +- [ ] Set `LOG_LEVEL=warn` or `LOG_LEVEL=error` for production +- [ ] Set spending limits on both Anthropic and OpenAI dashboards +- [ ] Put a reverse proxy (Nginx/Caddy) in front for TLS termination +- [ ] If using Qdrant (`VECTOR_STORE=qdrant`): secure it -- by default it has no authentication; consider binding to localhost only or adding an API key via [Qdrant security config](https://qdrant.tech/documentation/guides/security/) +- [ ] Set up monitoring for the `/api/v1/health` endpoint +- [ ] Configure firewall rules -- only expose ports 80/443 publicly, keep 8080/6333/6334 internal +- [ ] If using Qdrant: back up the Qdrant volume periodically (or use [Qdrant snapshots](https://qdrant.tech/documentation/concepts/snapshots/)) +- [ ] Consider rate limiting at the reverse proxy level to prevent abuse + +--- + + ## Todo - [ ] Make the request timeout an env config option - [ ] Decide how to prevent multiple requests by same user or different users - how to set up limits and prevent DOS - [ ] Testing metrics with some privacy policy datasets to evaluate model and chunking/retrieval strategies - [ ] For website, have a slider section, right side t&c left side scoring result, to demonstrate the results -- [ ] Add application-side retry logic with backoff for Qdrant connection on startup \ No newline at end of file +- [ ] Add application-side retry logic with backoff for Qdrant connection on startup diff --git a/backend/internal/extractor/detector.go b/backend/internal/extractor/detector.go index 7c14bcc..fe88258 100644 --- a/backend/internal/extractor/detector.go +++ b/backend/internal/extractor/detector.go @@ -28,6 +28,8 @@ var privacyKeywords = []keywordEntry{ {phrase: "data protection", weight: 2}, {phrase: "terms of service", weight: 2}, {phrase: "terms and conditions", weight: 2}, + {phrase: "conditions of use", weight: 2}, + {phrase: "privacy notice", weight: 2}, {phrase: "opt-out", weight: 2}, {phrase: "third party", weight: 2}, // Single-word terms (weight 1) @@ -44,9 +46,11 @@ var privacyKeywords = []keywordEntry{ // when found in a section heading. var headingIndicators = []string{ "privacy policy", + "privacy notice", "terms of service", "terms and conditions", "terms of use", + "conditions of use", "cookie policy", "data protection", } diff --git a/backend/internal/extractor/detector_test.go b/backend/internal/extractor/detector_test.go index aceda21..e39650b 100644 --- a/backend/internal/extractor/detector_test.go +++ b/backend/internal/extractor/detector_test.go @@ -211,6 +211,40 @@ func TestIsPrivacyPolicy_TermsAndConditions(t *testing.T) { } } +func TestIsPrivacyPolicy_PrivacyNotice(t *testing.T) { + padding := strings.Repeat("word ", 40) + text := padding + "This privacy notice describes how we collect and use your personal information." + + content := &ParsedContent{ + Text: text, + Sections: []Section{{Title: "Amazon.com Privacy Notice", Content: text}}, + WordCount: countWords(text), + } + + isPolicy, confidence := IsPrivacyPolicy(content) + + if !isPolicy { + t.Errorf("IsPrivacyPolicy(privacy notice) = false, want true (confidence=%.2f)", confidence) + } +} + +func TestIsPrivacyPolicy_ConditionsOfUse(t *testing.T) { + padding := strings.Repeat("word ", 40) + text := padding + "Please read these conditions of use carefully before using our services." + + content := &ParsedContent{ + Text: text, + Sections: []Section{{Title: "Conditions of Use", Content: text}}, + WordCount: countWords(text), + } + + isPolicy, confidence := IsPrivacyPolicy(content) + + if !isPolicy { + t.Errorf("IsPrivacyPolicy(conditions of use) = false, want true (confidence=%.2f)", confidence) + } +} + func TestIsPrivacyPolicy_ConfidenceInRange(t *testing.T) { padding := strings.Repeat("word ", 60)