Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 98 additions & 42 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,55 +1,111 @@
# video-processing-distributed
# 🎥 Video Processing Distributed

## Master Controller
## 🧠 Master Controller

This project provides a **distributed video processing system** using:
- **Streamlit** for the UI
- **Flask** for the worker-facing API
- **FFmpeg** for merging
- **MP4Box** for video splitting
- **Multiple worker nodes** that pick up chunks, encode them, and return processed video parts.
This project implements a **distributed video processing system** using a master–worker model.
You control everything from a simple UI, while worker nodes handle video chunks in parallel.

The master automatically:
1. Accepts a large input MP4
2. Splits it into chunks
3. Assigns each chunk to workers
4. Waits for all jobs to complete
5. Merges encoded chunks into a final MP4
6. Deletes all temporary chunk folders (chunks + results)
### 🔧 Tech Stack

* 🖥️ Streamlit for the master UI
* 🌐 Flask for worker communication
* 🎞️ FFmpeg for encoding and merging
* ✂️ MP4Box for video splitting
* 🧩 Multiple worker nodes for parallel processing

---

## ⚙️ How the System Works

The master node performs the following steps automatically:

1. 📥 Accepts a large input MP4 file
2. ✂️ Splits the video into time-based chunks
3. 📤 Assigns chunks to available worker nodes
4. ⏳ Waits for all workers to finish processing
5. 🔗 Merges processed chunks into a final MP4
6. 🧹 Cleans up all temporary chunk and output folders

---

## 📦 Requirements

### 🐍 Python Libraries (Master Node)

Install using pip:

* flask
* werkzeug
* streamlit
* streamlit-autorefresh
* pandas
* requests

### 🧑‍💻 Worker Node Requirements

Only the following are required:

* requests
* ffmpeg

---

## 🛠️ System Dependencies

### ✂️ MP4Box (Master Node Only)

Used for splitting videos into chunks.

* Command used:
MP4Box -split `<seconds>`
* Download via official GPAC site or YouTube guides
* Ensure MP4Box is added to system PATH

### 🎞️ FFmpeg (Master + Worker Nodes)

Used for encoding and merging video chunks.

Install on Windows:

```
winget install ffmpeg
```

Verify both `ffmpeg` and `MP4Box` are accessible from the terminal.

---

## Requirements

### Python libraries (install with pip)
- flask
- werkzeug
- streamlit
- streamlit-autorefresh
- pandas
- requests

For worker node need to install only `requests` and `ffmpeg`

### System Dependencies (must be installed manually)
- **MP4Box (GPAC)-(Only needed in master node)**
Required for splitting video into chunks: MP4Box -split <seconds>
Refer google or youtube for downloading it
- **FFmpeg(Needed both in master and worker node)**
Required for merging processed chunks.
Install in windows using
```
winget ffmpeg
```
Ensure both `ffmpeg` and `MP4Box` are available in your system PATH.

Run the master node using streamlit
## ▶️ Running the Master Node

Start the master controller UI:

```
streamlit run master.py
```
Ensure that the system is configured for getting inbound through the specified port. Used here is **5000**.
You can set up in windows by adding a new inbound tcp for port 5000 rule in firewall.

---

## 🔐 Network Configuration

* Master node listens on port **5000**
* Ensure inbound TCP traffic is allowed on this port

### Windows Firewall Setup

* Add a new inbound rule
* Protocol: TCP
* Port: 5000
* Action: Allow

---

## 🚀 Use Case

This project demonstrates:

* Distributed computing fundamentals
* Parallel video encoding
* Master–worker architecture
* Practical use of FFmpeg in Python systems

Ideal for academic experiments, demos, and learning distributed systems through multimedia processing.