Feature/deeper underutilization metrics

.gitignore

@@ -0,0 +1,4 @@
+data/raw/slurm_data.db
+data/raw/slurm_data_small.db
+data/processed/csv_output/slurm_data.csv
+data/processed/csv_output/jobs.csv

DOCUMENTATION_MIGRATION.md

@@ -0,0 +1,181 @@
+# Documentation Migration: pdoc3 → MkDocs
+
+This document outlines the migration from pdoc3 to MkDocs for the SLURM Job Analytics project documentation system.
+
+## Migration Summary
+
+**Before**: Documentation was generated using pdoc3, which created basic API documentation from Python docstrings.
+
+**After**: Documentation now uses MkDocs with the Material theme, providing:
+- ✅ Modern, responsive web interface
+- ✅ Comprehensive navigation structure
+- ✅ Auto-generated API documentation (via mkdocstrings)
+- ✅ Tutorial and user guides
+- ✅ Beginner-friendly documentation guide
+- ✅ Jupyter notebook integration
+- ✅ Search functionality
+- ✅ GitHub Pages deployment support
+
+## New Documentation Structure
+
+```
+docs_new/
+├── index.md                          # Project homepage
+├── getting-started/
+│   ├── installation.md               # Installation guide
+│   ├── quick-start.md                # Quick start tutorial
+│   └── documentation-guide.md        # 📖 Beginner's guide for docs
+├── user-guide/
+│   ├── dashboard.md                   # Web dashboard usage
+│   ├── command-line-tools.md          # CLI tools reference
+│   ├── jupyter-notebooks.md           # Jupyter integration
+│   └── email-outreach.md              # Email outreach system
+├── api-reference/
+│   ├── analytics.md                   # Analytics module API
+│   ├── dashboard.md                   # Dashboard module API
+│   └── outreach.md                    # Outreach module API
+├── technical-notes/
+│   ├── date-filtering-fixes.md        # Technical implementation details
+│   └── database-schema.md             # Database schema reference
+└── development/
+    ├── contributing.md                # Contributing guidelines
+    └── testing.md                     # Testing framework guide
+```
+
+## Key Features
+
+### 1. Auto-Generated API Documentation
+- Uses `mkdocstrings[python]` to automatically generate API docs from Python docstrings
+- Maintains all existing docstring content
+- Provides better formatting and cross-references
+
+### 2. Beginner's Documentation Guide
+- **New**: Comprehensive guide for newcomers to the documentation system
+- Located at `docs_new/getting-started/documentation-guide.md`
+- Covers:
+  - How to read and navigate the documentation
+  - Editing and contributing to docs
+  - Local development setup
+  - Best practices for documentation
+
+### 3. Modern Web Interface
+- Material Design theme with responsive layout
+- Advanced search functionality
+- Tabbed content and admonitions
+- Code syntax highlighting
+- Mobile-friendly design
+
+### 4. Enhanced Content Organization
+- Clear separation between user guides and API reference
+- Progressive disclosure from basic to advanced topics
+- Cross-references between related sections
+
+## Migration Changes
+
+### Files Added
+- `mkdocs.yml` - Main configuration file
+- `docs_new/` - Complete new documentation structure
+- `scripts/generate_docs.py` - New documentation build script
+
+### Files Deprecated
+- `scripts/generate_all_docs.py` - Old pdoc3 script (kept for reference)
+- `docs/auto_docs/` - Old pdoc3 output (can be removed after verification)
+
+### Dependencies Added
+```bash
+pip install mkdocs mkdocs-material mkdocstrings[python] mkdocs-jupyter
+```
+
+## Usage
+
+### Building Documentation
+```bash
+# Using the new script
+python scripts/generate_docs.py build
+
+# Or directly with MkDocs
+mkdocs build
+```
+
+### Serving Locally
+```bash
+# Using the new script
+python scripts/generate_docs.py serve
+
+# Or directly with MkDocs
+mkdocs serve
+```
+
+### Validation
+```bash
+# Validate setup and test build
+python scripts/generate_docs.py validate
+```
+
+### Deployment
+```bash
+# Deploy to GitHub Pages
+python scripts/generate_docs.py deploy
+```
+
+## Development Workflow
+
+1. **Edit Documentation**: Modify files in `docs_new/`
+2. **Preview Changes**: Run `mkdocs serve` to see live preview
+3. **Build for Production**: Run `mkdocs build` to generate static site
+4. **Deploy**: Use `mkdocs gh-deploy` for GitHub Pages
+
+## Benefits of Migration
+
+### For Users
+- **Better Navigation**: Clear structure and search functionality
+- **Comprehensive Guides**: Step-by-step tutorials and examples
+- **Responsive Design**: Works well on desktop and mobile
+- **Faster Loading**: Optimized static site generation
+
+### For Developers
+- **Easier Maintenance**: Markdown-based content is easier to edit
+- **Better Integration**: Supports Jupyter notebooks and complex layouts
+- **Extensible**: Plugin ecosystem for additional features
+- **Version Control Friendly**: Plain text files work well with Git
+
+### For Contributors
+- **Lower Barrier**: Markdown is more accessible than reStructuredText
+- **Live Preview**: Immediate feedback during editing
+- **Template System**: Consistent formatting across all pages
+- **Documentation Guidelines**: Clear contribution process
+
+## Configuration
+
+The MkDocs configuration (`mkdocs.yml`) includes:
+
+- **Site Information**: Title, description, repository links
+- **Theme**: Material theme with custom colors and features
+- **Navigation**: Structured menu with logical grouping
+- **Plugins**: API documentation, Jupyter notebooks, search
+- **Markdown Extensions**: Code blocks, admonitions, tables, etc.
+
+## Next Steps
+
+1. **Verify Migration**: Review all documentation sections
+2. **Update References**: Update any external links to old docs
+3. **Train Team**: Share the documentation guide with contributors
+4. **Set Up CI/CD**: Automate documentation deployment
+5. **Gather Feedback**: Collect user feedback on the new system
+
+## Rollback Plan
+
+If needed, the old pdoc3 system can be restored:
+
+1. Use `scripts/generate_all_docs.py` to regenerate pdoc3 docs
+2. Serve from `docs/auto_docs/` directory
+3. Update any deployment configurations
+
+However, the new MkDocs system provides significant advantages and should be the preferred solution going forward.
+
+---
+
+**Migration Date**: January 2025  
+**Migrated By**: AI Assistant  
+**Status**: ✅ Complete  
+**Next Review**: After 30 days of usage