FastDL Processor

A high-performance C++ application for Source Engine game servers that monitors server directories and automatically creates compressed BZIP2 copies of game files for FastDL web hosting.

Overview

FastDL Processor is designed to be significantly faster than Python implementations, utilizing multi-threading and efficient C++ I/O operations. It preserves original files while creating compressed copies in a separate destination directory.

Features

• Multi-threaded processing - Auto-detects CPU cores for optimal performance
• Multiple server monitoring - Process several game servers simultaneously
• Directory structure preservation - Maintains relative paths from source to destination
• Incremental processing - Only processes new or changed files
• BZIP2 compression - Configurable compression level for optimal speed/size balance
• SFTP upload - Sends compressed files straight to your hosting server (IP, user, password, remote path)
• Zero manual installs - The SFTP library (libssh2) is fetched and built automatically by CMake
• File stability checking - Ensures files aren't being written during compression
• Cross-platform support - Native support for Linux and Windows
• JSON configuration - Flexible configuration via config.json
• Configurable extensions - Customize which file types to process
• Persistent database - Tracks processed files to avoid redundant work

Supported Games

• Counter-Strike: Source (CS:S)
• Counter-Strike: Global Offensive (CS:GO)
• Team Fortress 2 (TF2)
• Left 4 Dead 2 (L4D2)
• Garry's Mod (GMod)
• Any Source Engine based game

Requirements

Requirement	Version
CMake	3.16+
C++ Compiler	C++17 compatible (GCC 8+, Clang 7+, MSVC 2019+)
Git	Any recent version (used once to fetch libssh2 on first build)
libbz2	Optional (a bundled copy is used when not found)
nlohmann-json	Optional (a bundled copy is used when not found)
libssh2	Not required, fetched and built automatically
mbedTLS	Not required, fetched automatically if the system package is missing

Building

Linux (Ubuntu/Debian)

sudo apt update
sudo apt install -y cmake g++ build-essential libbz2-dev nlohmann-json3-dev

mkdir build && cd build
cmake ..
make -j$(nproc)

The first cmake .. run downloads and builds libssh2 (and mbedTLS if your system lacks it), so it needs an internet connection and git once. Everything else is automatic; no libraries are installed globally.

Windows (Visual Studio 2022 or MinGW)

# Build with CMake (vcpkg is optional; bundled copies are used when deps are missing)
mkdir build
cd build
cmake ..
cmake --build . --config Release

Or with MinGW on the command line:

mkdir build && cd build
cmake .. -G "MinGW Makefiles"
mingw32-make -j

The first configure downloads and builds libssh2 plus mbedTLS automatically (git and internet needed once). Nothing is installed on your system.

Configuration

Create config.json in the executable directory:

{
  "check_interval": 300,
  "chunk_size_mb": 8,
  "max_workers": 0,
  "stability_checks": 1,
  "stability_interval_ms": 100,
  "processed_files_db": "processed_files.txt",
  "extensions": [".nav", ".bsp", ".mdl", ".phy", ".vvd", ".vtf", ".vmt", ".wav", ".mp3", ".vtx", ".ani", ".dds", ".pcf", ".svg"],
  "multi_extensions": [".dx80.vtx", ".dx90.vtx", ".sw.vtx"],
  "servers": [
    {
      "name": "MyCSServer",
      "source": "/path/to/csserver/cstrike",
      "destination": "/var/www/fastdl/cstrike",
      "sftp": {
        "enabled": true,
        "host": "203.0.113.10",
        "port": 22,
        "username": "root",
        "password": "your-password-here",
        "remote_path": "/var/www/fastdl/cstrike"
      }
    }
  ]
}

SFTP Upload

Each server can optionally push its compressed files to a remote hosting server over SFTP. Add an sftp block inside any server entry:

Field	What it does
enabled	Set to true to turn on uploading for this server
host	IP address or hostname of your hosting server
port	SSH port, normally 22
username	The user that logs in, often root
password	The password for that user
remote_path	Folder on the remote server where files go, created automatically

Files keep their relative structure on the remote side. A file at maps/de_dust2.bsp becomes remote_path/maps/de_dust2.bsp.bz2. Compressed copies are still written to the local destination folder as a cache, then uploaded. If an upload fails, the local copy is removed so the next cycle tries again.

When enabled is false or the sftp block is omitted, the program behaves exactly as before (local destination only).

Usage

# Run with default config.json
./FastDLProcessor

# Custom config file
./FastDLProcessor --config /path/to/config.json

# Quiet mode (errors only)
./FastDLProcessor --quiet

# Debug mode
./FastDLProcessor --debug

# Single pass (compress and upload once, then exit, handy for cron)
./FastDLProcessor --once

If config.json is missing, the program creates a default one for you, tells you to fill in your server details, and exits. Edit the file and run it again.

Performance

Metric	C++ vs Python
File scanning	10-50x faster
Compression	5-10x faster
Memory usage	Lower
CPU utilization	Better

Author

SyntX

License

MIT License - See LICENSE for details.