Skip to content
View osamaaltaf-pk's full-sized avatar
🎯
Focusing
🎯
Focusing

Block or report osamaaltaf-pk

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
osamaaltaf-pk/README.md

LinkedIn Email HuggingFace WhatsApp


🚀 Professional Profile

I am a specialized AI Systems Engineer focused on building offline-first AI media production tools, self-hosted LLM infrastructure, and multi-agent automation pipelines. My recent work spans the full spectrum — from ONNX model export pipelines and local inference routing to multi-lane video editors, 2D lip sync studios, and LangGraph-orchestrated content automation — all running entirely on local hardware.

  • 40+ Active Codebases spanning offline video production, local AI runtimes, voice synthesis, LLM fine-tuning, and enterprise automation.
  • Offline-First Philosophy: Every tool I build runs without cloud dependencies — TTS, SFX, video stitching, lip sync, and sketch animation all on-device.
  • Production AI Deployments: Veteran of enterprise SaaS backends, browser automation agents, local AI OS builds, and ONNX inference pipelines.

🎯 Target Role Alignment & Capabilities

Target Pillar Core Alignment & Competencies Key Evidence (In my Repos)
Offline AI Media Engineer Multi-lane video stitching, local TTS/SFX generation, FFmpeg pipeline engineering, 2D lip sync, whiteboard sketch animation. Auto_StitchOmni-Lip-SyncWisko
LLM Infrastructure Engineer Self-hosted AI runtime OS, multi-backend inference routing (Ollama/vLLM/SGLang), ONNX export pipelines, hybrid vector + graph memory. NeuralFolkonnx-sfx-music-stable-audio
AI Automation Engineer LangGraph multi-agent DAG pipelines, YouTube content automation, APScheduler orchestration, Supabase state + GCS media management. CMSOmni_Automator
Voice AI Engineer Real-time TTS streaming (~200ms), voice cloning, WebRTC audio, multi-engine offline voice pipelines. Pocket_TTSAura-TTSOrpheusAssistant
LLM Research / Fine-tuning LoRA/QLoRA fine-tuning, GRPO reasoning training without critic models, MoE kernels, model alignment with Unsloth. LLMs-Unslothsmol-course

🛠️ Technical Ecosystem

AI & LLM Systems

PyTorch HuggingFace ONNX FastAPI Transformers Gemini API Anthropic SDK

Offline Media & Production Pipeline

FFmpeg OpenCV Whisper LangGraph

Real-Time & Backend Infrastructure

Node.js NestJS Apache Kafka Redis Docker PostgreSQL Supabase

Frontend Development

React 19 TypeScript Vite Tailwind CSS Framer Motion


📂 Featured Deep-Dives

Here is a curated overview of my primary repositories representing deep engineering focus and production capabilities:

A locally-run, offline-first Windows desktop application that automates voiceover generation, sound effect creation, and multi-lane video stitching — entirely without any cloud dependency.

  • Engineering Highlights:
    • Multi-Lane Timeline Editor: Drag, reorder, split, merge, and edit clips across 3 independent lanes with a browser-based UI.
    • Local TTS Engine: Generates realistic voiceovers from text scripts entirely on-device using PocketTTS.
    • Local SFX Engine: Generates sound effects from text prompts using ONNX-exported Stable Audio 3 Small, running fully on CPU.
    • Auto-Captions: Word-by-word caption overlays burned into final video via FFmpeg drawtext filters.
    • Background Music Mixer: Mixes ambient music tracks with adjustable volume into the final rendered output.
  • Key Stack: Python, FFmpeg, ONNX Runtime, PocketTTS, FastAPI, WebSockets.

A self-hosted, open-source, air-gapped AI runtime Operating System — enabling companies, developers, and researchers to orchestrate state-of-the-art AI agents and multi-model workflows completely locally, from an RTX 2070 laptop to enterprise clusters.

  • Engineering Highlights:
    • Universal Inference Layer: Abstracts backend engines (Ollama, vLLM, SGLang, llama.cpp, TensorRT) under a single dynamic routing API.
    • Hybrid Memory System: Merges semantic vector embeddings (Qdrant) and entity-relationship knowledge graphs (FalkorDB) for human-like agent context recall.
    • Distributed Task Orchestration: Celery + Valkey Streams drive parallelized multi-agent loops and complex DAG workflows.
    • Zero API Keys, 100% Privacy: No external phone-homes, no vendor lock-in, fully air-gapped deployment.
  • Key Stack: Python, FastAPI, Celery, Valkey Streams, Qdrant, FalkorDB, Ollama, vLLM, Docker.

A locally-run, offline-first Windows desktop application for 2D animators — automatically generating frame-accurate lip-sync video from character sprites and a dialogue script.

  • Engineering Highlights:
    • Phoneme Detection: Rhubarb Lip Sync extracts mouth shape timings (phonemes A–X) at frame-level accuracy from generated or provided WAV audio.
    • Character Studio: Upload full-body character images, define face regions, and map custom mouth sprites to each phoneme.
    • Script-Driven Voice: [Character Name] tagged dialogue scripts converted to audio locally via PocketTTS.
    • ONNX Face Tracking: Optional ONNX-based face/gaze tracking for characters with animated heads.
    • Video Compositor: Composites frames + mouth sprites + audio into final MP4s via FFmpeg — no external rendering software needed.
  • Key Stack: Python, Rhubarb Lip Sync, ONNX Runtime, FFmpeg, PocketTTS, FastAPI.

A fully automated, sequential content pipeline that discovers viral YouTube videos, writes original scripts, generates TTS audio + AI images + whiteboard sketch animations, and uploads to YouTube — orchestrated by LangGraph multi-agents.

  • Engineering Highlights:
    • LangGraph DAG Agents: Full viral_finder → await_approval → data_collection → script_generation → media_generation → upload pipeline with crash-safe resumption via Supabase pipeline_stage state.
    • Thin-State Architecture: LangGraph state holds only IDs — every agent reads inputs fresh from Supabase/GCS and writes outputs back, eliminating context bloat.
    • Human-in-the-Loop Gate: await_approval polls yt_results.approved — a human reviews generated scripts before media production begins.
    • Integrated Sketch Animation: Converts AI-generated images into whiteboard sketch animations using Omni Sketches on GCP VM.
  • Key Stack: Python, LangGraph, FastAPI, APScheduler, Supabase, Google Cloud Storage, FFmpeg, Gemini.

An open-source ONNX conversion pipeline for Stability AI's Stable Audio 3 Small — enabling CPU/GPU audio generation inference without a PyTorch runtime dependency.

  • Engineering Highlights:
    • Exports the full diffusion pipeline into 4 standalone ONNX modules: Text Encoder → Conditioner → DiT (Diffusion Transformer) → Decoder (Oobleck).
    • Supports both stable-audio-3-small-music and stable-audio-3-small-sfx model variants.
    • Runtime-agnostic deployment on CUDA, DirectML, and CPU via ONNX Runtime — no PyTorch installation required at inference time.
  • Key Stack: Python, ONNX Runtime, HuggingFace Diffusers, Stability AI, PyTorch.

A tightly integrated suite of offline voice engineering tools covering real-time TTS streaming, multi-engine workstations, and full voice-driven AI assistants.

  • Engineering Highlights:
    • Pocket_TTS: Low-latency real-time TTS streaming via WebSockets (~200ms to first chunk), voice cloning support, CPU-only, glassmorphism web UI.
    • Aura-TTS: Unified offline speech workstation managing engine lifecycles (Kokoro-TTS, Pocket-TTS, Supertonic) via custom mutual exclusion locking under RAM/VRAM constraints.
    • OrpheusAssistant: Offline voice agent with WebRTC bi-directional streams (aiortc), Whisper Large STT, and LLaMA 3.2 3B — integrated with n8n for workflow tool execution.
  • Key Stack: Python, FastAPI, WebRTC, WebSockets, PipeCat, PyTorch, n8n, Docker.

A specialized research-to-production pipeline repository detailing high-throughput fine-tuning, reasoning models, and edge optimizations.

  • Engineering Highlights:
    • Fine-tuning pipelines utilizing Unsloth kernels for parameter-efficient optimizations (LoRA / QLoRA).
    • RLHF training using Group Relative Policy Optimization (GRPO) without a separate Critic model (pioneered by DeepSeek-R1), targeting Qwen3 8B.
    • Horizontal engineering recipes for Mixture-of-Experts (MoE) kernels and ModernBERT dense classification systems.
  • Key Stack: Unsloth, PyTorch, LoRA, QLoRA, HuggingFace Transformers, TRL, JAX.

📊 Developer Metrics & Impact

  • 🎬 Offline-First Media Architect: Building complete AI video production stacks that run 100% on local hardware — ONNX inference, FFmpeg rendering, TTS streaming, and lip sync compositing.
  • 🧠 AI Pipeline Engineer: Expert in LangGraph DAG orchestration, Kafka event streaming, Celery distributed workers, and multi-agent system design.
  • 📦 Docker-first Deployer: Containerizing complex AI stacks with multi-stage builds and clean Docker Compose networking configs.

💼 Open to Remote Opportunities

I am actively exploring Remote AI Engineer, LLM Infrastructure Engineer, AI Media Engineer, and AI Automation opportunities globally. Let's build the future of offline, production-grade AI systems.

📧 Get In Touch💬 Chat on WhatsApp

Pinned Loading

  1. LLMs-Unsloth LLMs-Unsloth Public

    LLM fine-tuning pipeline using Unsloth, LoRA and QLoRA for efficient domain-specific training

    Jupyter Notebook

  2. OrpheusAssistant OrpheusAssistant Public

    AI assistant built with LLM orchestration, tool use, and conversational memory

    Python 1

  3. Pocket_TTS Pocket_TTS Public

    Lightweight Python TTS pipeline with multi-engine voice support and real-time audio processing

    Python

  4. davidbrowne17/csm-streaming davidbrowne17/csm-streaming Public

    Forked from SesameAILabs/csm

    Realtime demo, Streaming and Finetuning code for CSM

    Python 454 73