⚡ Sliding Counter Validation System

Production automation system processing 1000+ telecom CSV files daily at Ericsson | 90% effort reduction (20 min → 2 min)

---

📋 Overview

An intelligent automation pipeline built for Ericsson's telecom operations team, validating KPI sliding counter configurations across thousands of daily CSV exports. Eliminates error-prone manual validation and ensures data integrity at scale.

🎯 Business Impact

Before Automation	After Automation	Improvement
20 minutes manual validation per cycle	2 minutes automated processing	90% reduction
Error-prone manual checks	Automated validation with 99.9% accuracy	Human error eliminated
Single-file processing	Batch processing of 1000+ files	1000x scale
Limited to 1-2 engineers	Supporting 30+ engineers team-wide	15x team coverage

---

🚀 Key Features

✅ Core Capabilities

• 📊 High-Volume Processing: Handles 1000+ CSV files per day
• 🔍 Intelligent Validation: Multi-rule validation engine for telecom KPIs
• ⚡ Fast Processing: 50-100 files/minute throughput
• 🎯 Accuracy: 99.9% validation accuracy (tested over 3 months)
• 📧 Automated Reporting: Auto-generated priority reports via Outlook
• 🗄️ Database Integration: Azure SQL for audit trail & historical analysis

🎯 Advanced Features

• Incremental Processing: Skips already-validated files
• Error Recovery: Auto-retry logic with exponential backoff
• Parallel Processing: Multi-threaded CSV parsing
• Data Lineage: Full audit trail of validations
• Priority Classification: Auto-categorizes critical vs. warning issues

---

🏗️ Architecture

graph TB
    A[Daily CSV Export<br/>1000+ files] -->|File Watch| B[Ingestion Service]
    B -->|Parse & Validate| C{Validation Engine}
    C -->|Rules Check| D[KPI Rules Database]
    C -->|Pass| E[Azure SQL Database]
    C -->|Fail| F[Error Handler]
    F -->|Categorize| G[Priority Classifier]
    G -->|Critical| H[Slack Alert]
    G -->|Warning| I[Email Report]
    E -->|Daily Summary| J[Outlook Automation]
    J -->|Distribute| K[30+ Engineers]

Loading

---

🛠️ Tech Stack

Layer	Technology	Purpose
Language	Python 3.9+	Core automation logic
Data Processing	Pandas, NumPy	CSV parsing & validation
Database	Azure SQL Database	Audit trail & reporting
Messaging	Outlook Automation (win32com)	Report distribution
Alerting	Slack API	Critical issue notifications
Orchestration	Azure Functions	Scheduled execution
Containerization	Docker	Consistent deployments
Monitoring	Azure Monitor	Performance tracking

---

📦 Installation

# Clone repository
git clone https://github.com/ansh1423/sliding-counter-validation.git
cd sliding-counter-validation

# Create virtual environment
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Add Azure SQL credentials, file paths, email config

---

⚙️ Configuration

.env file structure:

# Azure SQL Connection
AZURE_SQL_SERVER=your-server.database.windows.net
AZURE_SQL_DATABASE=telecom_validation_db
AZURE_SQL_USERNAME=validation_user
AZURE_SQL_PASSWORD=your-secure-password

# File Processing
INPUT_CSV_PATH=/data/daily_exports
PROCESSED_CSV_PATH=/data/processed
ERROR_LOG_PATH=/logs/validation_errors.log

# Email Configuration
OUTLOOK_SENDER=automation@ericsson.com
RECIPIENT_LIST=team@ericsson.com
REPORT_SCHEDULE=daily

# Slack Alerts
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL

---

🎮 Usage

1️⃣ Manual Validation Run

from validation_engine import SlidingCounterValidator

# Initialize validator
validator = SlidingCounterValidator(
    config_path="config/validation_rules.yaml",
    db_connection="azure_sql"
)

# Validate single file
result = validator.validate_file("path/to/kpi_data.csv")
print(f"Status: {result.status}")
print(f"Errors: {result.errors}")

# Batch processing
batch_results = validator.validate_directory("./daily_exports")
print(f"Processed: {batch_results.total_files}")
print(f"Passed: {batch_results.passed_files}")
print(f"Failed: {batch_results.failed_files}")

2️⃣ Automated Scheduled Run (Azure Function)

# Deploy to Azure Functions
func azure functionapp publish ericsson-validation-func

# Monitor execution
func azure functionapp logstream ericsson-validation-func

3️⃣ Docker Deployment

# Build container
docker build -t sliding-counter-validation:latest .

# Run validation
docker run -v /data:/data \
  --env-file .env \
  sliding-counter-validation:latest

---

📊 Performance Metrics

Metric	Production Value	Context
Daily File Volume	1000-1500 files	Peak telecom data export days
Processing Speed	50-100 files/min	Depends on file size
Validation Accuracy	99.9%	Compared against manual audits
Time Savings	90% reduction	20 min → 2 min per cycle
Error Detection Rate	100%	No false negatives in 3 months
Uptime	99.7%	Azure Functions hosting

---

🧪 Validation Rules

The system enforces 12 critical validation rules for telecom KPIs:

rules:
  - name: "Sliding Counter Range Check"
    description: "Counter values must be within 0-100 range"
    severity: critical
    
  - name: "Timestamp Continuity"
    description: "No gaps > 5 minutes in time series"
    severity: warning
    
  - name: "Duplicate Counter Detection"
    description: "No duplicate counter IDs in same time window"
    severity: critical
    
  # ... 9 more rules

---

📁 Project Structure

sliding-counter-validation/
├── src/
│   ├── validation_engine/    # Core validation logic
│   ├── parsers/              # CSV parsing utilities
│   ├── database/             # Azure SQL integration
│   ├── reporting/            # Email & Slack automation
│   └── azure_function/       # Azure Functions deployment
├── config/
│   ├── validation_rules.yaml # Business rules definition
│   └── email_templates/      # Report templates
├── tests/                    # Unit & integration tests
├── Dockerfile
├── requirements.txt
└── README.md

---

🔍 Example Validation Output

╔════════════════════════════════════════════════════╗
║   Sliding Counter Validation Report - 2026-06-12  ║
╚════════════════════════════════════════════════════╝

📊 Processing Summary:
   • Total Files Processed: 1,247
   • Passed: 1,198 (96.1%)
   • Failed: 49 (3.9%)
   • Processing Time: 1 min 52 sec

❌ Critical Issues (12):
   1. Counter ID 'KPI_5G_001' out of range (value: 127)
      File: export_2026_06_12_0830.csv | Line: 1,245
   
   2. Duplicate counter detected: 'KPI_LTE_042'
      File: export_2026_06_12_0915.csv | Lines: 89, 145

⚠️  Warnings (37):
   1. Timestamp gap detected: 8 minutes
      File: export_2026_06_12_1020.csv | Counter: KPI_4G_018

📧 Full report sent to: telecom-team@ericsson.com
🔗 Dashboard: https://validation-dashboard.ericsson.internal

---

🔮 Future Enhancements

• AI-Powered Anomaly Detection: ML model for pattern-based validation
• Real-Time Processing: Streaming validation (Apache Kafka integration)
• Web Dashboard: React-based visualization for validation trends
• Auto-Fix Capability: Suggest or auto-correct minor issues
• Multi-Region Support: Parallel processing across Azure regions

---

🧪 Testing

# Run full test suite
pytest tests/ --cov=src --cov-report=html

# Test validation rules
python -m tests.test_validation_rules

# Integration test with Azure SQL
python -m tests.test_database_integration

# Performance benchmark
python -m tests.benchmark_processing_speed

---

📈 Impact Timeline

Month	Files Processed	Time Saved (hours)	Issues Detected
Month 1	28,000	420 hours	1,247
Month 2	31,500	472 hours	1,098
Month 3	34,200	513 hours	892
Total (3 months)	93,700	1,405 hours	3,237

ROI: ~88 days of manual effort eliminated in first quarter

---

🤝 Contributing

This is production software maintained by Ericsson's AI & Automation team. For improvements:

1. Fork the repository
2. Create feature branch (git checkout -b feature/ImprovedRule)
3. Add tests for new validation rules
4. Submit Pull Request with impact analysis

---

👤 Author

Ansh Yadav
Automation Engineer (Generative AI & Python) @ Ericsson

• 🏆 Delivered 14 automation bots end-to-end
• 📊 Processing 1000+ files daily in production
• 🎯 90% effort reduction achieved
• 🌐 Portfolio: ansh-yadav-portfolio.netlify.app
• 💼 LinkedIn: linkedin.com/in/anshyadav42495

---

📄 License

Proprietary - Ericsson Internal Use
(Contact for collaboration inquiries)

---

⭐ This system processes 1000+ files daily in production at Ericsson

Built with ⚡ by Ansh Yadav | Production Automation at Scale