DPI Engine - Deep Packet Inspection System

This document explains everything about this project, from basic networking concepts to the complete Python code architecture. After reading this, you should understand exactly how packets flow through the system without needing to read the code.

---

Table of Contents

1. What is DPI?
2. Networking Background
3. Project Overview
4. File Structure
5. The Journey of a Packet
6. Deep Dive: Each Component
7. How TLS SNI and HTTP Host Extraction Works
8. How Blocking Works
9. Running
10. Understanding the Output
11. Extending the Project

---

1. What is DPI?

Deep Packet Inspection (DPI) is a technology used to examine the contents of network packets as they pass through a checkpoint. Unlike simple firewalls that only look at packet headers such as source and destination IP, DPI looks inside the packet payload.

Real-World Uses

• ISPs: Throttle or block certain applications such as BitTorrent
• Enterprises: Block social media on office networks
• Parental controls: Block inappropriate websites
• Security: Detect malware or intrusion attempts

What This DPI Engine Does

User Traffic (PCAP) -> [Python DPI Engine] -> Filtered Traffic (PCAP)
                               |
                               +-- Identifies apps (YouTube, Facebook, etc.)
                               +-- Blocks based on rules
                               +-- Generates reports

---

2. Networking Background

The Network Stack (Layers)

When you visit a website, data travels through multiple layers:

┌─────────────────────────────────────────────────────────┐
│ Layer 7: Application    │ HTTP, TLS, DNS               │
├─────────────────────────────────────────────────────────┤
│ Layer 4: Transport      │ TCP (reliable), UDP (fast)   │
├─────────────────────────────────────────────────────────┤
│ Layer 3: Network        │ IP addresses (routing)       │
├─────────────────────────────────────────────────────────┤
│ Layer 2: Data Link      │ MAC addresses (local network)│
└─────────────────────────────────────────────────────────┘

A Packet's Structure

Every network packet is like a Russian nesting doll, with headers wrapped inside headers:

┌──────────────────────────────────────────────────────────────────┐
│ Ethernet Header (14 bytes)                                       │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ IPv4 Header (20 bytes)                                       │ │
│ │ ┌──────────────────────────────────────────────────────────┐ │ │
│ │ │ TCP Header (20 bytes)                                    │ │ │
│ │ │ ┌──────────────────────────────────────────────────────┐ │ │ │
│ │ │ │ Payload (Application Data)                           │ │ │ │
│ │ │ │ e.g., TLS Client Hello with SNI                      │ │ │ │
│ │ │ └──────────────────────────────────────────────────────┘ │ │ │
│ │ └──────────────────────────────────────────────────────────┘ │ │
│ └──────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────┘

The Five-Tuple

A connection, or flow, is uniquely identified by 5 values:

Field	Example	Purpose
Source IP	192.168.1.100	Who is sending
Destination IP	172.217.14.206	Where it's going
Source Port	54321	Sender's application identifier
Destination Port	443	Service being accessed (443 = HTTPS)
Protocol	TCP (6)	TCP or UDP

Why this matters:

• All packets with the same 5-tuple belong to the same connection
• If we block one packet of a connection, we should block all of them
• This is how we track conversations between computers

What is SNI?

Server Name Indication (SNI) is part of the TLS/HTTPS handshake. When you visit https://www.youtube.com:

1. Your browser sends a Client Hello message
2. This message includes the domain name in plaintext before encryption starts
3. The server uses this to know which certificate to send

TLS Client Hello:
├── Version: TLS 1.2
├── Random: [32 bytes]
├── Cipher Suites: [list]
└── Extensions:
    └── SNI Extension:
        └── Server Name: "www.youtube.com"  ← We extract this

This is the key to DPI: even though HTTPS is encrypted, the domain name is visible in the first packet.

---

3. Project Overview

What This Project Does

┌─────────────┐     ┌─────────────────┐     ┌─────────────┐
│ Wireshark   │     │ Python DPI      │     │ Output      │
│ Capture     │ ──► │ Engine          │ ──► │ PCAP        │
│ (input.pcap)│     │ - Parse         │     │ (filtered)  │
└─────────────┘     │ - Classify      │     └─────────────┘
                    │ - Block         │
                    │ - Report        │
                    └─────────────────┘

Two Entry Points

Version	File	Use Case
Packet viewer	analyze_pcap.py	Learning, debugging, inspecting packets
DPI filter	dpi_engine.py	Blocking and writing filtered PCAPs

---

4. File Structure

packet_analyzer/
├── pcap.py             # PCAP reader and writer
├── parser.py           # Ethernet / IPv4 / TCP / UDP parsing
├── extractors.py       # TLS SNI, HTTP Host, DNS name extraction
├── rules.py            # Blocking rules and rule file loading
├── types.py            # AppType, FiveTuple, Connection, stats objects
└── dpi_engine.py       # Main DPI engine logic

analyze_pcap.py         # Packet-by-packet inspection CLI
dpi_engine.py           # DPI filtering CLI entry point
generate_test_pcap.py   # Creates test data
test_dpi.pcap           # Sample capture with various traffic
filtered.pcap           # Example output capture
README.md               # This file!

---

5. The Journey of a Packet

Let's trace a single packet through the Python engine.

Step 1: Read PCAP File

from packet_analyzer.pcap import PcapReader

with PcapReader("capture.pcap") as reader:
    for raw in reader.packets():
        ...

What happens:

1. Open the file in binary mode
2. Read the 24-byte global header (magic number, version, etc.)
3. Verify it's a valid PCAP file

Step 2: Read Each Packet

for raw in reader.packets():
    # raw.data contains the packet bytes
    # raw.header contains timestamp and length
    ...

What happens:

1. Read a 16-byte packet header
2. Read N bytes of packet data (N = header.incl_len)
3. Stop when no more packets remain

Step 3: Parse Protocol Headers

from packet_analyzer.parser import PacketParser

parsed = PacketParser.parse(raw)

What happens in packet_analyzer/parser.py:

raw.data bytes:
[0-13]   Ethernet Header
[14-33]  IPv4 Header
[34-53]  TCP Header
[54+]    Payload

After parsing:
parsed.src_mac   = "00:11:22:33:44:55"
parsed.dest_mac  = "aa:bb:cc:dd:ee:ff"
parsed.src_ip    = "192.168.1.100"
parsed.dest_ip   = "172.217.14.206"
parsed.src_port  = 54321
parsed.dest_port = 443
parsed.protocol  = 6 (TCP)
parsed.has_tcp   = True

Parsing details:

• Ethernet header: destination MAC, source MAC, EtherType
• IPv4 header: version, header length, TTL, protocol, source IP, destination IP
• TCP header: source port, destination port, sequence number, acknowledgment number, flags
• UDP header: source port and destination port

Step 4: Build the Five-Tuple and Look Up the Connection

from packet_analyzer.types import FiveTuple

tuple_ = FiveTuple(
    parsed.src_ip,
    parsed.dest_ip,
    parsed.src_port,
    parsed.dest_port,
    parsed.protocol,
)

What happens:

• The connection table is a dictionary: FiveTuple -> Connection
• If the tuple already exists, the existing connection is reused
• If not, a new connection is created
• All packets with the same 5-tuple share the same connection state

Step 5: Extract SNI and Host Information

from packet_analyzer.extractors import extract_sni, extract_http_host, extract_dns_query

sni = extract_sni(parsed.payload)
host = extract_http_host(parsed.payload)
domain = extract_dns_query(parsed.payload)

What happens:

1. Check if the payload is a TLS Client Hello
2. Skip through the handshake fields
3. Find the SNI extension and extract the hostname
4. For HTTP, extract the Host header
5. For DNS, extract the queried domain name

Step 6: Check Blocking Rules

reason = rules.should_block(src_ip, dst_port, app_type, domain)
if reason:
    state = "BLOCKED"

What happens in packet_analyzer/rules.py:

if src_ip in blocked_ips:
    return BlockReason("IP", src_ip)

if dst_port in blocked_ports:
    return BlockReason("PORT", str(dst_port))

if app in blocked_apps:
    return BlockReason("APP", app.value)

if domain and is_domain_blocked(domain):
    return BlockReason("DOMAIN", domain)

return None

Step 7: Forward or Drop

if reason:
    dropped_packets += 1
    # Don't write to output
else:
    forwarded_packets += 1
    # Write packet to output file
    writer.write_packet(job)

Step 8: Generate Report

After processing all packets:

for app, count in app_counts.items():
    print(f"{app}: {count}")

---

6. Deep Dive: Each Component

pcap.py

Purpose: read and write network captures saved by Wireshark.

Key structures:

@dataclass
class PcapGlobalHeader:
    magic_number: int
    version_major: int
    version_minor: int
    thiszone: int
    sigfigs: int
    snaplen: int
    network: int
    raw: bytes
    endian: str


@dataclass
class PcapPacketHeader:
    ts_sec: int
    ts_usec: int
    incl_len: int
    orig_len: int

Key classes:

• PcapReader: Opens the capture and yields RawPacket objects
• PcapWriter: Writes packets back to a PCAP file

parser.py

Purpose: extract protocol fields from raw bytes.

Key function:

parsed = PacketParser.parse(raw)

Important concepts:

Network byte order is handled with struct.unpack("!...") and ipaddress to decode values safely.

extractors.py

Purpose: extract domain names from TLS, HTTP, and DNS payloads.

For TLS (HTTPS):

sni = extract_sni(payload)

For HTTP:

host = extract_http_host(payload)

For DNS:

domain = extract_dns_query(payload)

types.py

Purpose: define data structures used throughout the engine.

FiveTuple:

@dataclass(frozen=True)
class FiveTuple:
    src_ip: str
    dst_ip: str
    src_port: int
    dst_port: int
    protocol: int

AppType:

class AppType(Enum):
    UNKNOWN = "Unknown"
    HTTP = "HTTP"
    HTTPS = "HTTPS"
    DNS = "DNS"
    GOOGLE = "Google"
    FACEBOOK = "Facebook"
    YOUTUBE = "YouTube"
    # ... more apps

sni_to_app_type function:

def sni_to_app_type(sni: str) -> AppType:
    if "youtube" in sni.lower():
        return AppType.YOUTUBE
    if "facebook" in sni.lower():
        return AppType.FACEBOOK
    # ... more patterns

rules.py

Purpose: store and evaluate blocking rules.

Key class:

rules = RuleManager()
rules.block_ip("192.168.1.50")
rules.block_app("YouTube")
rules.block_domain("*.example.com")
rules.block_port(443)

---

7. How TLS SNI and HTTP Host Extraction Works

The TLS Handshake

When you visit https://www.youtube.com:

┌──────────┐                              ┌──────────┐
│ Browser  │                              │ Server   │
└────┬─────┘                              └────┬─────┘
     │                                         │
     │ ──── Client Hello ─────────────────────►│
     │      (includes SNI: www.youtube.com)    │
     │                                         │
     │ ◄─── Server Hello ───────────────────── │
     │      (includes certificate)             │
     │                                         │
     │ ──── Key Exchange ─────────────────────►│
     │                                         │
     │ ◄═══ Encrypted Data ══════════════════► │
     │      (from here on, everything is       │
     │       encrypted - we can't see it)      │

We can only extract SNI from the Client Hello.

TLS Client Hello Structure

Byte 0:     Content Type = 0x16 (Handshake)
Bytes 1-2:  Version = 0x0301 (TLS 1.0)
Bytes 3-4:  Record Length

-- Handshake Layer --
Byte 5:     Handshake Type = 0x01 (Client Hello)
Bytes 6-8:  Handshake Length

-- Client Hello Body --
Bytes 9-10:  Client Version
Bytes 11-42: Random (32 bytes)
Byte 43:     Session ID Length (N)
Bytes 44 to 44+N: Session ID
... Cipher Suites ...
... Compression Methods ...

-- Extensions --
Bytes X-X+1: Extensions Length
For each extension:
    Bytes: Extension Type (2)
    Bytes: Extension Length (2)
    Bytes: Extension Data

-- SNI Extension (Type 0x0000) --
Extension Type: 0x0000
Extension Length: L
  SNI List Length: M
  SNI Type: 0x00 (hostname)
  SNI Length: K
  SNI Value: "www.youtube.com" ← THE GOAL!

Our Extraction Code (Simplified)

from packet_analyzer.extractors import extract_sni

sni = extract_sni(payload)
if sni:
    print(sni)

HTTP Host Extraction

HTTP requests usually include a Host header:

GET /index.html HTTP/1.1
Host: www.example.com
User-Agent: ...

The Python extractor looks for that header and returns the host name.

---

8. How Blocking Works

Rule Types

Rule Type	Example	What it Blocks
IP	192.168.1.50	All traffic from this source
App	YouTube	All YouTube connections
Domain	tiktok	Any SNI containing "tiktok"
Port	443	Any connection to that destination port

The Blocking Flow

Packet arrives
      |
      v
┌─────────────────────────────────┐
│ Is source IP in blocked list?  │──Yes──► DROP
└───────────────┬─────────────────┘
                │No
                v
┌─────────────────────────────────┐
│ Is destination port blocked?   │──Yes──► DROP
└───────────────┬─────────────────┘
                │No
                v
┌─────────────────────────────────┐
│ Is app type in blocked list?   │──Yes──► DROP
└───────────────┬─────────────────┘
                │No
                v
┌─────────────────────────────────┐
│ Does domain match a rule?      │──Yes──► DROP
└───────────────┬─────────────────┘
                │No
                v
            FORWARD

Flow-Based Blocking

Important: we block at the connection level, not packet level.

Connection to YouTube:
  Packet 1 (SYN)           -> No SNI yet, FORWARD
  Packet 2 (SYN-ACK)       -> No SNI yet, FORWARD
  Packet 3 (ACK)           -> No SNI yet, FORWARD
  Packet 4 (Client Hello)  -> SNI: www.youtube.com
                            -> App: YOUTUBE (blocked!)
                            -> Mark connection as BLOCKED
                            -> DROP this packet
  Packet 5 (Data)          -> Connection is BLOCKED -> DROP
  Packet 6 (Data)          -> Connection is BLOCKED -> DROP
  ...all subsequent packets -> DROP

Why this approach:

• We cannot identify the app until we see the Client Hello or Host header
• Once identified, we block all future packets of that connection
• The connection will fail or time out on the client side

---

9. Running

Prerequisites

• Python 3.10 or newer
• No external libraries are required for the core parser and DPI engine

Basic Usage

python dpi_engine.py test_dpi.pcap filtered.pcap

With Blocking Rules

python dpi_engine.py test_dpi.pcap filtered.pcap \
    --block-app YouTube \
    --block-app TikTok \
    --block-ip 192.168.1.50 \
    --block-domain facebook \
    --block-port 443

Load Rules from a File

python dpi_engine.py test_dpi.pcap filtered.pcap --rules rules.txt

Packet Viewer

python analyze_pcap.py test_dpi.pcap

Create Test Data

python generate_test_pcap.py

Notes on --lbs and --fps

The Python CLI accepts --lbs and --fps for compatibility with the original project layout, but the port runs as a single-process pipeline.

---

10. Understanding the Output

Sample Output

====================================
DPI ENGINE v1.0
Configuration: Load Balancers=2, FPs per LB=2 (Python port runs single-process)

[RuleManager] Blocked app: YouTube
[RuleManager] Blocked IP: 192.168.1.50

[DPIEngine] Processing: test_dpi.pcap
[DPIEngine] Output to:  filtered.pcap

Opened PCAP file: test_dpi.pcap
  Version: 2.4
  Snaplen: 65535 bytes
  Link type: 1 (Ethernet)

====================================
DPI ENGINE STATISTICS
====================================
Total Packets:   77
Total Bytes:     5738
TCP Packets:     73
UDP Packets:     4
Forwarded:       69
Dropped/Blocked: 8
Drop Rate:       10.39%
Connections:     21

Application Distribution:
  HTTPS           16
  Unknown         4
  YouTube         2
  DNS             1
  Facebook        1

Top Domains:
  www.youtube.com                          2
  www.facebook.com                         1
  www.google.com                           1

What Each Section Means

Section	Meaning
Configuration	CLI options used for the run
Rules	Which blocking rules are active
Total Packets	Packets read from input file
Forwarded	Packets written to output file
Dropped/Blocked	Packets blocked and not written
Connections	Unique five-tuples tracked
Application Distribution	Traffic classification results
Top Domains	Actual domains found in payloads

---

11. Extending the Project

Ideas for Improvement

1. Add more app signatures in packet_analyzer/types.py

2. Add bandwidth throttling instead of immediate dropping in packet_analyzer/dpi_engine.py

3. Add live statistics updates in a background thread if you want to evolve the Python port into a streaming engine

4. Add QUIC/HTTP3 support for UDP-based HTTPS traffic

5. Add persistent rules loading and saving

---

Summary

This Python DPI engine demonstrates:

1. Network protocol parsing - understanding packet structure
2. Deep packet inspection - looking inside TLS, HTTP, and DNS payloads
3. Flow tracking - managing stateful connections
4. Rule-based filtering - blocking traffic by IP, app, domain, or port
5. Python packet processing - reading, classifying, and writing PCAP files

The key insight is that even HTTPS traffic can leak useful metadata in the TLS handshake, allowing network operators to identify and control application usage.

---

Questions?

If you have questions about any part of this project, the code is organized to follow the same flow described in this document. Start with analyze_pcap.py to understand packet parsing, then move to packet_analyzer/dpi_engine.py to see how filtering and reporting work.

Happy learning!