PHI-Weaver

🧬 AI-Powered Pathogen-Host Interaction Curation System

A comprehensive framework for automating PHI-base and PHI-Canto curation workflows using Large Language Models and intelligent automation tools.

📂 New colleague? Start with docs/STORAGE-CONFIGURATION.md — it explains where the pipeline reads/writes literature files and how to point it at your own location (via the PHI_LITERATURE_ROOT environment variable).

🚀 Want to try it with zero local setup? Follow docs/DEMO-CODESPACES.md to curate an open-access article entirely in a GitHub Codespace.

🎯 Overview

PHI-Weaver streamlines the complex process of curating pathogen-host interaction data from scientific literature. It integrates Claude Code, automated PDF processing, database tracking, and quality assurance tools to accelerate high-quality curation for the PHI-base database.

Key Features

✅ Complete Automation Pipeline - End-to-end workflow from PDF intake to curation completion
✅ LLM-Powered Analysis - Claude integration for intelligent entity extraction and annotation
✅ Professional PDF Processing - Academic formatting with figure/table caption extraction
✅ Database Integration - SQLite tracking for progress analytics and session management
✅ Timeline System - Development tracking with incremental updates
✅ External Storage Architecture - Scalable content management with performance optimization
✅ Quality Assurance Tools - Validation workflows and error detection
✅ Training Materials - Comprehensive onboarding and quick reference guides

📁 Repository Structure

phi-weaver/
├── 00-Inbox/                   # Incoming tasks and project coordination
├── 02-Projects/                # Project-specific curation work
├── 04-Literature/              # Literature stubs (content lives in external storage)
├── 05-Protocols/               # Standard operating procedures
├── 06-Training/                # Curator onboarding and quick references
├── 07-Standards/               # Genetic nomenclature and ontology guides
├── 08-Wiki/                    # Curation protocols, registries, templates
├── 11-CLAUDE-AI/               # Automation tools and session management
│   ├── curation_pipeline.py    # Master automation script
│   ├── pdf-convert-skill/      # PDF processing system
│   ├── db/                     # SQLite tracking database + helper scripts
│   ├── SESSION-LOGS/           # Development history and context
│   └── *.py                    # Timeline tracking and automation tools
├── content-links/              # References to external literature storage
├── archive/                    # Historical / superseded documents
├── docs/                       # Storage config, Codespaces demo, PDF-converter usage
├── CLAUDE.md                   # System configuration and guidelines
└── README.md                   # This file

🚀 Quick Start

Prerequisites

• Python 3.8+ with pip
• Claude Code CLI (installation guide)
• Git for version control
• WSL/Linux environment recommended for full functionality

Installation

1. Clone the repository:

git clone https://github.com/PHI-base/phi-weaver.git
cd phi-weaver

2. Install Python dependencies:

pip install -r requirements.txt   # PyMuPDF for PDF conversion; sqlite3 is built into Python

3. Set up Claude Code:

# WSL users may need permission bypass
claude --dangerously-skip-permissions

4. Initialize the tracking database (optional):

# Creates 11-CLAUDE-AI/db/phi_canto_tracking.db with demo data
python3 11-CLAUDE-AI/db/phi_canto_sqlite.py

First Curation Session

# Process a new PDF through the complete pipeline
python3 11-CLAUDE-AI/curation_pipeline.py auto-process ~/Downloads/paper.pdf

# Start an interactive curation session
python3 11-CLAUDE-AI/db/session_logger.py quick "Project Name" "Summary"

# Generate development timeline
python3 11-CLAUDE-AI/vault-ops/update_timeline_incremental.py

📚 Core Workflows

1. PDF Processing

# Convert PDF with academic formatting
python3 11-CLAUDE-AI/pdf-convert-skill/pdf-convert.py paper.pdf

# Process for curation with file organization
python3 11-CLAUDE-AI/convert-for-curation.py paper.pdf

2. Session Management

# Quick session logging
python3 11-CLAUDE-AI/db/session_logger.py quick 'Fusarium effectors' 'Added FgTPP1 analysis' 3 5 2.0

# View recent progress
python3 11-CLAUDE-AI/db/daily_curation.py progress

# Check productivity analytics
python3 11-CLAUDE-AI/db/daily_curation.py gaps

3. Timeline Tracking

# Update development timeline (preserves manual edits)
python3 11-CLAUDE-AI/vault-ops/update_timeline_incremental.py

# Full regeneration when needed
python3 11-CLAUDE-AI/vault-ops/generate_dev_timeline.py

🏗️ System Architecture

Modular Framework Design

Literature → Document Processing → Entity Recognition → Ontology Mapping → 
Relationship Analysis → Validation & Learning → Database Output → PHI-base

Module 1: Document Processing

• PDF to markdown conversion with figure extraction
• Caption and table extraction for academic papers
• Quality validation and formatting

Module 2: Entity Recognition

• LLM-powered gene/protein identification
• Organism and strain classification
• Experimental method detection

Module 3: Ontology Mapping

• UniProtKB accession lookup and validation
• PHIPO/GO term suggestion and mapping
• Quality assurance with confidence scoring

Module 4: Relationship Analysis

• Protein interaction detection
• Genotype-phenotype association mapping
• Cross-pathway analysis capabilities

Module 5: Validation & Learning

• Memory system for curator feedback
• Progress tracking and analytics
• Quality metrics and improvement suggestions

Module 6: Database Output

• PHI-Canto ready annotation records
• Structured data export capabilities
• Integration with external storage systems

External Storage Integration

The framework uses external storage for scalable content management:

• Development Vault (this repository): Tools, documentation, automation
• Literature Storage (external): Active work, completed curations, media files
• Content Links: References connecting development tools to literature content

🛠️ Configuration

Claude Code Setup

Edit CLAUDE.md for system-specific configuration:

• Session startup protocol: Automatic context loading
• Database integration settings: SQLite configuration
• Permission and capabilities: Research access authorization
• External storage paths: Literature storage configuration

Environment Variables

export CURATION_ROOT="/path/to/curation/workspace"
export LITERATURE_STORAGE="/path/to/external/literature"
export PHI_CANTO_DB="/path/to/phi_canto_tracking.db"

Database Configuration

The system uses SQLite for tracking:

• Articles: Literature pipeline status (queued → curated → published)
• Proteins: Gene IDs, functions, UniProtKB links, species relationships
• Sessions: Daily curation work with metrics and timestamps
• Progress: Analytics on productivity and data completeness

📖 Documentation

User Guides

• Complete System Guide - Comprehensive framework documentation
• Automation Guide - Workflow automation usage
• Timeline System Guide - Development tracking

Training Materials

• PHI-Canto Curation Protocol
• Curator Onboarding
• Quick References: Genotype creation, phenotype annotation, UniProtKB lookup, ontology terms

Technical Documentation

• PDF Conversion System
• Tracking Database (SQLite) - schema, scripts, and example queries
• Session Logs - Development history and context

🤝 Contributing

We welcome contributions to improve PHI-Weaver!

Getting Started

1. Read the session logs in 11-CLAUDE-AI/SESSION-LOGS/INDEX.md for context
2. Check the development timeline for current priorities
3. Review the architecture documentation in CLAUDE.md

Development Workflow

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-improvement)
3. Use the session logging system to track your work
4. Update the timeline with development progress
5. Commit your changes (git commit -m 'Add amazing improvement')
6. Push to your branch (git push origin feature/amazing-improvement)
7. Open a Pull Request

Areas for Contribution

• 🔬 Module Enhancement: Improve entity recognition, ontology mapping
• 🤖 LLM Integration: Enhanced agent workflows, memory systems
• 📊 Analytics: Advanced progress tracking, quality metrics
• 🔗 API Integration: Direct PHI-Canto submission, external database connections
• 📝 Documentation: User guides, video tutorials, training materials
• 🧪 Testing: Automated testing frameworks, validation systems

📞 Support

Getting Help

• Issues: Report bugs or request features via GitHub Issues
• Discussions: Join community discussions for usage questions
• Documentation: Check CLAUDE.md and guides in 11-CLAUDE-AI/

Contact

• Maintainer: martin2urban
• Research Domain: Pathogen-Host Interactions, PHI-base, PHI-Canto
• LLM Integration: Claude Code, Agent Networks, Automation Workflows

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🏆 Acknowledgments

• PHI-base Team - Database infrastructure and curation standards
• PHI-Canto Platform - Community curation system
• Claude AI/Anthropic - LLM integration and automation capabilities
• Research Community - Domain expertise and collaborative curation efforts

📈 Project Status

Current Version: Development Framework
Development Timeline: 26 days (Apr 11 - May 7, 2026)
Major Milestones: 5 infrastructure additions
Session Logs: 12 development sessions tracked
Automation Level: Complete pipeline with external storage integration

---

Built for accelerating scientific curation through intelligent automation 🚀