Datacenter Resource Efficiency & Carbon Footprint Analysis

This repository contains the code and data analysis for an empirical study on resource over-provisioning and its environmental impact in production datacenters. The study utilizes the Alibaba Open Cluster Trace v2018.

Project Overview

Modern datacenters often employ static resource allocation to prevent application crashes during traffic spikes. While this guarantees Quality of Service (QoS), it leads to severe hardware over-provisioning. This project aims to:

1. Identify Bottlenecks: Analyze 247 million utilization records across 4,000 servers to identify structural bottlenecks (e.g., memory vs. CPU).
2. Temporal Analysis: Uncover diurnal patterns in resource usage.
3. Carbon Footprint Quantification: Map empirical utilization data to linear power models to estimate wasted static power and project the corresponding $CO_2$ emissions under various grid intensities.

Key Findings

• Memory Bottleneck: The cluster exhibits a severe memory bottleneck (averaging 88.21% utilization), which artificially throttles CPU utilization to just 38.17%.
• Stranded Capacity: On average, 61.83% of the cluster's computational capacity remains idle, drawing static power without performing useful work.
• Environmental Impact: The persistent static memory allocations prevent physical servers from entering deep sleep states during off-peak hours, resulting in significant carbon emissions that could be mitigated via carbon-aware scheduling.

Repository Structure

• tahap2_alibaba_analysis.ipynb: Jupyter Notebook detailing the chunk-based processing and temporal analysis of the 9GB trace dataset.
• tahap3_energy_carbon.ipynb: Jupyter Notebook containing the linear power modeling and carbon footprint projections.
• Paper_Workspace/: Contains the generated figures and the LaTeX (main.tex) source code for the associated research paper draft.

Methodology

To process the 9GB raw telemetry data without encountering Out-Of-Memory (OOM) errors, we implemented a memory-safe chunking pipeline in Pandas. A representative sample (machine_usage_sample_clean.csv) was generated to perform rapid statistical modeling and visualization.

Technologies Used

• Python 3
• Pandas & NumPy (Data processing and aggregation)
• Matplotlib & Seaborn (Data visualization)

---

Note: This is an independent research project conducted as preparation for academic research in Sustainable Computing.