🌐 DaNetworkCode Streamer Scraper

A high-performance, multi-platform livestream monitoring service.

Actively tracks live status, viewer counts, metadata, and VODs across Kick, Twitch, YouTube, Vaughn, Parti, and Pump.fun.

---

✨ Key Features

• Multi-Platform Integration: Connects to Kick, Twitch, YouTube, Vaughn, Parti, and Pump.fun via API and web scraping to build a unified streamer database.
• Pump.fun Crypto Streaming: Tracks streams tied to Solana mint addresses, complete with intelligent WebSocket/React rendering bypass using Puppeteer to scrape live viewer counts.
• Advanced YouTube Quota Management: Features a built-in admin dashboard (/youtube-dashboard) to track daily/monthly usage, archive history, and prevent API exhaustion. Operates on a strict budget target (default 85%).
• Kick OAuth2 & PKCE: Fully automated Kick token generation, client credentials usage, and refreshing for reliable data extraction.
• Intelligent Puppeteer Fallback: Uses headless browser scraping for platforms that require JS rendering, specifically tuned for low-core VPS environments (auto-restarts to prevent memory leaks).
• Crash-Proof Data: Utilizes better-sqlite3 in WAL mode for persistent, lightning-fast reads and writes.
• Security Hardened: Routes are protected using helmet and express-rate-limit.
• Interactive Frontend: Serves a fully responsive UI displaying real-time live status updates, avatars, premiere badges, and viewer counts.

🚀 Installation & Setup

Ensure you have Node.js version 18 or higher installed on your system, or use Docker for a containerized deployment.

Option 1: Docker Compose (Recommended)

# 1. Clone and navigate to the directory
git clone https://github.com/Riotcoke123/danetworkcode.git
cd danetworkcode
2. Configure environment variables (see below)

cp .env.example .env
3. Build and run the container

docker-compose up -d --build

Option 2: Standard Node.js / PM2

# 1. Clone the repository
git clone https://github.com/Riotcoke123/danetworkcode.git
cd danetworkcode
2. Install dependencies

npm install
3. Configure environment variables

cp .env.example .env
4. Start the server

npm start
OR start with PM2 for production deployments

npm run pm2-start

⚙️ Environment Variables (.env)

You will need to create a .env file in the root directory. Below are the required configurations:

# Server Settings
PORT=3000
CHECK_INTERVAL_SECONDS=300
ADMIN_TOKEN=your_secure_random_string_here
TRUST_PROXY_HOPS=1
USER_AGENT=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36

# Concurrency Limits (Tune based on your VPS cores/RAM)
CONCURRENT_LIMIT=4
PUPPETEER_CONCURRENT_LIMIT=2
MAX_SCRAPES_BEFORE_RESTART=20
SCRAPE_TIMEOUT_MS=240000

# Streamer Lists (Comma separated)
KICK_USERNAMES=streamer1,streamer2
TWITCH_USERNAMES=streamer1,streamer2
YOUTUBE_USERNAMES=streamer1,streamer2
VAUGHN_USERNAMES=streamer1,streamer2
PARTI_USER_IDS=12345,67890
PUMPFUN_MINTS=21rKrtBzibPAZHAHQRzGiGDSh7XimCKB2a8VgsjZpump,AnotherMint

# API Keys & Auth Setup
YOUTUBE_API_KEY=your_google_api_key
YOUTUBE_QUOTA_DAILY_LIMIT=1010000
YOUTUBE_MONTHLY_BUDGET_PERCENT=0.85

TWITCH_CLIENT_ID=your_twitch_client_id
TWITCH_CLIENT_SECRET=your_twitch_client_secret

KICK_CLIENT_ID=your_kick_client_id
KICK_CLIENT_SECRET=your_kick_client_secret
KICK_REDIRECT_URI=http://localhost:3000/auth/kick/callback

# Optional Parti API Token
PARTI_AUTH_TOKEN=your_parti_token

# Optional Pump.fun API Key (if required by frontend API)
PUMPFUN_API_KEY=your_pumpfun_api_key

🛡️ Admin Dashboard & Endpoints

This project exposes several endpoints. Admin routes require your ADMIN_TOKEN passed as a Bearer token or as a query parameter (?token=YOUR_TOKEN).

Endpoint	Description	Access
/api/streamers	Returns JSON list of all tracked streamers & their current live status.	Public
/api/stats	Returns aggregated viewer counts and total active streams.	Public
/login/kick	OAuth entry point to authorize the Kick API integration.	Public (Rate Limited)
/youtube-dashboard	Visual HTML dashboard showing YouTube API quota consumption.	Admin Only
/api/youtube/quota	Returns current YouTube quota usage and budget status data.	Admin Only
/api/youtube/audit	Generates JSON audit history for API compliance checks.	Admin Only
/healthz	System health, active scrapers, quota snapshots, and error logs.	Admin Only

---

Developed & Maintained by Riotcoke123. Released under the GNU General Public License v3.0 (GPL-3.0).