Integrating Machine Learning with Live Network Telemetry
Developed as a core technical project to demonstrate competency in Big Data processing and Real-Time Systems. This project serves as a foundation for my transition into graduate-level research in Computer Vision and Robotics, specifically focusing on securing machine-to-machine communication.
This project is contained within a single comprehensive Jupyter Notebook, covering:
- Data Engineering: Cleaning ~700,000 flows from the CICIDS2017 dataset, handling high-cardinality features, and resolving infinity/NaN values.
- Machine Learning: Training a Random Forest Classifier with an overall accuracy of 99.94%.
- Interpretability: Utilizing Feature Importance to identify specific behavioral indicators for DDoS and DoS attacks.
- Live Deployment: Using the Scapy library to sniff live traffic and apply the pre-trained model for real-time anomaly detection.
| Attack Category | Precision | Recall | F1-Score |
|---|---|---|---|
| Benign | 1.00 | 1.00 | 1.00 |
| DoS/DDoS | 1.00 | 1.00 | 1.00 |
| Heartbleed | 1.00 | 1.00 | 1.00 |
A key part of this project was identifying why the model flags certain traffic. As shown in the graph below, Max Packet Length and Avg Bwd Segment Size were the most critical features for distinguishing malicious traffic from benign user activity.
- Modeling: Scikit-learn (Random Forest), Pandas, NumPy
- Networking: Scapy (Packet Sniffing & Injection)
- Visualization: Matplotlib, Seaborn
- Deployment: Joblib (Model Serialization)
- Data: Due to GitHub's size limits, the raw 1GB CSV is excluded. Use the included
.pklfiles for instant testing. - Environment: Ensure
scapyandjoblibare installed. - Live Monitoring: Run the final cells in the notebook to start the real-time detection engine.
- Data: Due to GitHub's size limits, the raw 1GB CSV is excluded. Use the included
.pklfiles for instant testing. - Environment: Ensure
scapyandjoblibare installed. - Live Monitoring: Run the final cells in the notebook to start the real-time detection engine.