Direct Prompt Customization

[Entropy-rgb]

Problem Statement

Glossary and catalog exist, but no way to directly customize the system prompt for domain-specific tuning. Users cannot add custom instructions, domain-specific rules, or prompt templates that modify the AI's behavior.

Proposed Solution

Allow users to add custom instructions, domain-specific rules, and prompt templates that modify the AI's behavior.

Acceptance Criteria

• "Custom Instructions" field in Settings
• Prompt template preview
• Domain-specific presets (e.g., "E-commerce", "Finance", "Healthcare")
• Variable interpolation (schema, glossary, examples)
• Version history for custom prompts
• Share custom prompts with team
• A/B testing for prompt variants
• Prompt injection protection

Technical Approach

Backend Changes

1. Custom prompt storage (backend/app/config.py):

# Add to config
CUSTOM_PROMPT_CONFIG = {
    "custom_instructions": "",  # User's custom instructions
    "prompt_preset": None,  # Selected domain preset
    "prompt_version": 1,
    "prompt_history": []  # Previous versions
}

2. Prompt builder update (backend/app/llm.py):

def build_sql_system_prompt(schema_text, dialect, config=None):
    base_prompt = """You are an expert {dialect} analyst. Convert the user's question into exactly one read-only SELECT query.
    
    Rules:
    1. SELECT (or WITH ... SELECT) only — never modify data.
    2. Use ONLY the tables and columns listed in the schema below. Never invent names.
    3. Join tables via the foreign keys shown as col->table.column.
    4. Add LIMIT 100 unless the question asks for a single total/count or the dialect uses TOP.
    5. When filtering on a column whose example values are shown in [brackets], match those values exactly.
    6. Qualify column names with table aliases whenever more than one table is involved.
    7. {dialect_hint}
    """
    
    # Add custom instructions if provided
    if config and config.get("custom_instructions"):
        base_prompt += f"\n\nCustom Instructions:\n{config['custom_instructions']}"
    
    # Add domain preset if selected
    if config and config.get("prompt_preset"):
        preset = get_preset(config["prompt_preset"])
        base_prompt += f"\n\nDomain-Specific Rules:\n{preset}"
    
    return base_prompt

3. Domain presets (backend/app/services/presets.py):

DOMAIN_PRESETS = {
    "ecommerce": """
    - When referring to "revenue", use SUM(total_amount) or SUM(price * quantity)
    - "Orders" typically means the orders table
    - "Customers" typically means the customers table
    - Date ranges are often fiscal years (April-March)
    - Currency amounts should be formatted with 2 decimal places
    """,
    "finance": """
    - Use DECIMAL(19,4) for financial calculations
    - "Revenue" is typically net_revenue after returns
    - "Profit" is revenue minus costs
    - Fiscal year typically runs January-December
    - Always include currency code in output
    """,
    "healthcare": """
    - Use HIPAA-compliant queries (no PHI in output)
    - Patient IDs are sensitive - mask in output
    - "Visits" typically means patient encounters
    - Date ranges are often calendar years
    - Include appropriate aggregations for privacy
    """,
    "saas": """
    - "MRR" = Monthly Recurring Revenue = SUM(monthly_amount)
    - "ARR" = Annual Recurring Revenue = MRR * 12
    - "Churn" = cancelled / total subscriptions
    - "Active users" = last_login >= CURRENT_DATE - INTERVAL '30 days'
    - Use cohort analysis for retention queries
    """
}

4. API routes (backend/app/routes/prompts.py):

• GET /api/prompts/custom - Get custom instructions
• PATCH /api/prompts/custom - Update custom instructions
• GET /api/prompts/presets - List available presets
• POST /api/prompts/preview - Preview prompt with customizations
• GET /api/prompts/history - Prompt version history

Frontend Changes

1. Custom instructions editor (frontend/src/lib/components/PromptEditor.svelte):

<script>
    export let config;
    
    let customInstructions = config.custom_instructions || '';
    let selectedPreset = config.prompt_preset || '';
    let previewPrompt = '';
    
    async function updatePreview() {
        const response = await apiCall('/api/prompts/preview', {
            method: 'POST',
            body: JSON.stringify({
                custom_instructions: customInstructions,
                prompt_preset: selectedPreset
            })
        });
        previewPrompt = response.prompt;
    }
    
    async function saveInstructions() {
        await apiCall('/api/prompts/custom', {
            method: 'PATCH',
            body: JSON.stringify({
                custom_instructions: customInstructions,
                prompt_preset: selectedPreset
            })
        });
    }
</script>

<div class="prompt-editor">
    <h1>Custom Instructions</h1>
    
    <section class="instructions">
        <h2>Custom Instructions</h2>
        <p>Add instructions that will be included in every AI prompt.</p>
        <textarea 
            bind:value={customInstructions}
            on:change={updatePreview}
            placeholder="e.g., Always use JOIN instead of subqueries. Prefer LEFT JOIN when unsure."
            rows="6"
        ></textarea>
    </section>
    
    <section class="presets">
        <h2>Domain Presets</h2>
        <p>Select a preset for domain-specific instructions.</p>
        <select bind:value={selectedPreset} on:change={updatePreview}>
            <option value="">None</option>
            <option value="ecommerce">E-commerce</option>
            <option value="finance">Finance</option>
            <option value="healthcare">Healthcare</option>
            <option value="saas">SaaS</option>
        </select>
    </section>
    
    <section class="preview">
        <h2>Prompt Preview</h2>
        <pre>{previewPrompt}</pre>
    </section>
    
    <button on:click={saveInstructions}>Save Instructions</button>
</div>

2. Version history - Show previous prompt versions with diff

3. A/B testing - Allow testing different prompt variants

Key Files

• backend/app/config.py - Prompt config
• backend/app/llm.py - Prompt builder
• backend/app/services/presets.py - Domain presets (new)
• backend/app/routes/prompts.py - API routes (new)
• frontend/src/lib/components/PromptEditor.svelte - Editor UI

Related Issues

• Part of Feature Roadmap (Feature Roadmap: Next Iteration of BoloDB #71)
• Related to Custom prompt tuning for domain-specific customization
• Related to Workspace-scoped Memory (Anomaly Alerts #231) for shared prompts