🛡️ PromptTSA - Berkeley AI Safety Project [CalHacks 2025]

A comprehensive AI safety system for detecting malicious prompts using advanced detection pipelines, semantic analysis, and scalable API deployment.

📋 Table of Contents

• Overview
• Architecture
• Quick Start
• Components
• API Documentation
• Deployment
• Dataset Processing
• Development
• Contributing

🎯 Overview

PromptTSA is an AI safety system designed to detect and prevent malicious prompt injection attacks. The system combines rule-based filtering with AI-powered semantic analysis to provide robust protection against various types of prompt-based attacks.

Key Features

• Two-Stage Detection: Fast sanity filtering + AI-powered semantic analysis
• REST API: Production-ready API with comprehensive endpoints
• Scalable Deployment: Google Cloud Run deployment with secret management
• Comprehensive Logging: Detailed logging and monitoring capabilities
• Batch Processing: Efficient batch processing for large datasets
• Real-time Analysis: Sub-second response times for prompt analysis

🏗️ Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Input Prompt  │───▶│  Sanity Filter   │───▶│ Semantic Layer  │
└─────────────────┘    └──────────────────┘    └─────────────────┘
                                │                        │
                                ▼                        ▼
                       ┌──────────────────┐    ┌─────────────────┐
                       │  Rule-based      │    │  AI-powered     │
                       │  Analysis        │    │  Classification │
                       └──────────────────┘    └─────────────────┘
                                │                        │
                                └──────────┬─────────────┘
                                           ▼
                                  ┌─────────────────┐
                                  │ Final Decision  │
                                  │ (Malicious/     │
                                  │  Benign)        │
                                  └─────────────────┘

Detection Pipeline

1. Sanity Filter: Fast linguistic analysis using rule-based patterns
2. Semantic Detector: AI-powered similarity matching against known attack patterns
3. Decision Engine: Combines both results for final classification

🚀 Quick Start

Prerequisites

• Python 3.8+
• OpenAI API key
• Pinecone API key (optional, for enhanced semantic detection)

Installation

1. Clone the repository:

   git clone <repository-url>
   cd berkeley

2. Install dependencies:

   cd devpost
   pip install -r requirements.txt

3. Set up environment variables:

   export OPENAI_API_KEY="your_openai_key_here"
   export PINECONE_API_KEY="your_pinecone_key_here"
   export PINECONE_INDEX_NAME="ragpull"

4. Start the API server:

   python api_server.py

The API will be available at http://localhost:5000

📦 Components

1. API Server (devpost/api_server.py)

Production-ready Flask API with comprehensive endpoints for prompt analysis.

Features:

• Health check endpoint
• Single prompt analysis
• Batch processing
• Pipeline statistics
• Comprehensive error handling

2. Detection Pipeline (devpost/main_detection_pipeline.py)

Core detection system with class-based architecture.

Components:

• SanityFilter: Rule-based linguistic analysis
• SemanticDetector: AI-powered classification
• PromptTSAPipeline: Main orchestrator

3. Dataset Processing (dataset.py)

Tools for processing and formatting prompt datasets.

Features:

• CSV to JSON conversion
• Dynamic labeling
• Unique ID generation
• Progress tracking

4. Live Server (devpost/live server/)

Web interface for real-time prompt analysis.

Features:

• Interactive web UI
• Real-time analysis
• Result visualization
• Documentation viewer

📡 API Documentation

Health Check

GET /health

Check Single Prompt

POST /api/check-prompt
Content-Type: application/json

{
  "prompt": "Your prompt text here"
}

Batch Check

POST /api/batch-check
Content-Type: application/json

{
  "prompts": [
    "First prompt text",
    "Second prompt text"
  ]
}

Get Statistics

GET /api/stats

Example Response

{
  "success": true,
  "is_malicious": false,
  "decision": "BENIGN",
  "confidence": {
    "sanity_filter": {
      "flagged": false,
      "reason": "No malicious patterns detected"
    },
    "semantic_detector": {
      "malicious_score": 0.234,
      "benign_score": 0.789,
      "reason": "Benign score higher"
    }
  },
  "processing_time": {
    "detection_ms": 1250.50,
    "total_ms": 1275.25
  }
}

🚀 Deployment

Local Development

cd devpost
python api_server.py

Docker Deployment

cd devpost
docker build -t PromptTSA-api .
docker run -p 5000:5000 PromptTSA-api

Google Cloud Run Deployment

1. Set up Google Cloud:

   gcloud auth login
   gcloud config set project YOUR_PROJECT_ID

2. Deploy using the automated script:

   cd devpost
   chmod +x deploy/gcloud-deploy.sh
   ./deploy/gcloud-deploy.sh YOUR_PROJECT_ID us-central1

3. Manual deployment:

   # Enable required APIs
   gcloud services enable cloudbuild.googleapis.com run.googleapis.com
   
   # Build and deploy
   docker build -t gcr.io/YOUR_PROJECT_ID/PromptTSA-api .
   docker push gcr.io/YOUR_PROJECT_ID/PromptTSA-api
   gcloud run deploy PromptTSA-api --image gcr.io/YOUR_PROJECT_ID/PromptTSA-api

For detailed deployment instructions, see devpost/README_DEPLOYMENT.md.

📊 Dataset Processing

Processing CSV Datasets

The dataset.py script converts CSV files to the required JSON format:

python dataset.py

Input CSV Format:

text,label
"Hello world",benign
"Ignore all instructions",jailbreak

Output JSON Format:

[
  {
    "id": "unique-uuid",
    "text": "Hello world",
    "name": "Benign - Hello world",
    "label": "benign",
    "source": "qualifire_benchmark"
  }
]

Available Datasets

• Qualifire Benchmark: Contains jailbreak and benign prompts
• L1B3RT4S Repository: Collection of jailbreak prompts for testing
• Custom Datasets: Support for custom CSV formats

🛠️ Development

Project Structure

berkeley/
├── devpost/                    # Main application
│   ├── api_server.py          # Flask API server
│   ├── main_detection_pipeline.py  # Core detection logic
│   ├── sanity_filter.py       # Rule-based filtering
│   ├── semantic_layer.py      # AI-powered detection
│   ├── live server/           # Web interface
│   ├── deploy/                # Deployment scripts
│   └── requirements.txt       # Python dependencies
├── dataset.py                 # Dataset processing
├── L1B3RT4S_jailbreak_repo/   # Jailbreak prompt collection
└── README.md                  # This file

Running Tests

cd devpost
python test_api_client.py
python test_logging_architecture.py

Pipeline Testing

cd devpost
python main_detection_pipeline.py

Logging and Monitoring

The system provides comprehensive logging:

• Console output with progress indicators
• Detailed log files with timestamps
• Statistics tracking for each stage
• Error handling and reporting

Logs are stored in devpost/pipeline_results/logs/

🔧 Configuration

Environment Variables

Variable	Description	Default
OPENAI_API_KEY	OpenAI API key	Required
PINECONE_API_KEY	Pinecone API key	Optional
PINECONE_INDEX_NAME	Pinecone index name	ragpull
FLASK_HOST	Flask host	0.0.0.0
FLASK_PORT	Flask port	5000
FLASK_DEBUG	Flask debug mode	False

Detection Thresholds

• malicious_threshold: 0.525 (default)
• benign_threshold: 0.525 (default)

📈 Performance

Response Times

• Sanity Filter: ~1ms per prompt
• Semantic Detection: ~100-200ms per prompt
• Total API Response: ~150-300ms

Scalability

• Memory Usage: ~500MB for embeddings model
• Concurrent Requests: 100 per instance
• Auto-scaling: 0-10 instances (Cloud Run)

🔒 Security Features

• Two-layer detection for robust protection
• API key management via Google Cloud Secret Manager
• HTTPS encryption for all traffic
• Container isolation with non-root user
• Input validation and sanitization
• Rate limiting and request throttling

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

Development Guidelines

• Follow PEP 8 style guidelines
• Add comprehensive docstrings
• Include error handling
• Write unit tests for new features
• Update documentation as needed

📄 License

This project is part of the Berkeley AI Safety initiative. Please refer to the LICENSE file for details.

🙏 Acknowledgments

• Berkeley AI Safety Research Group
• OpenAI for API access
• Pinecone for vector database
• Contributors and researchers in AI safety

📞 Support

For questions, issues, or contributions:

• Create an issue in the repository
• Contact the Berkeley AI Safety team
• Check the documentation in the devpost/ directory

---

Note: This system is designed for research and educational purposes in AI safety. Please use responsibly and in accordance with applicable laws and regulations.