UnDistort-Selfie: Geometry-Aware Selfie Perspective Rectification

A geometry-aware selfie perspective rectification and face texture restoration pipeline. This project corrects close-up wide-angle selfie distortion (such as disproportionately enlarged noses and narrow faces) while preserving facial identity and background realism. It combines 3D camera reprojection math, MediaPipe Face Mesh landmarks, Runge-Kutta 4th-order (RK4) continuous velocity field remapping, Poisson image blending, and deep learning face detail restorers (CodeFormer / Stable Diffusion).

---

📸 Sample Visual Benchmarks

Test Case	Original Selfie	Rectified & Restored Portrait	Diagnostic Evaluation
Test Case 1			📊 Interactive HTML Report 📄 PDF Evaluation
Test Case 2			📊 Interactive HTML Report
Test Case 3			📊 Interactive HTML Report
Test Case 4			📊 Interactive HTML Report 📄 PDF Evaluation

---

🔬 How It Works (Technical Architecture)

Close-up smartphone selfies suffer from perspective distortion (radial barrel distortion and relative depth scaling changes where facial features closer to the lens appear exaggeratedly large). Rather than using global 2D homographies or global affine transformations (which warp the background and stretch the skull), this pipeline models physical 3D camera projection locally:

graph TD
    Input[Input Wide-Angle Selfie] --> MP[MediaPipe 468-point 3D Face Mesh]
    MP --> Depth[Depth Anything V2 Estimation]
    Depth --> Geom[Back-project 2D Pixels to Metric 3D Camera Space]
    Geom --> RK4[Diffeomorphic Coordinate Warp using RK4 Integration]
    RK4 --> Poisson[Seamless Poisson Blending back to Original Background]
    Poisson --> CodeFormer[CodeFormer Face Texture Restoration]
    CodeFormer --> Refine[Detail Enhancer & Optional SD img2img Refinement]
    Refine --> Output[Rectified Portait Final Output]

    style Input fill:#1f2328,stroke:#30363d,stroke-width:2px;
    style Output fill:#238636,stroke:#2ea44f,stroke-width:2px,color:#fff;
    style RK4 fill:#1f6feb,stroke:#388bfd,stroke-width:2px,color:#fff;

Loading

---

🛠️ Quick Start (Local Setup)

The code lives inside the SelfieRectification/ folder.

Windows (NVIDIA GPU with CUDA Support)

cd SelfieRectification
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Tip

If the PyTorch CUDA installation fails to resolve automatically, install the wheel manually first: python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

macOS (Apple Silicon or Intel CPU)

cd SelfieRectification
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install torch torchvision torchaudio
python3 -m pip install -r requirements-mac.txt

---

▶️ Running the Application

1. Interactive Web Interface (Gradio)

To launch the responsive, local Web UI:

• Windows: Run .\launch.bat (or run python -m scripts.gradio_app inside activated env).
• macOS: Run python -m scripts.gradio_app in the activated shell.
• Open http://localhost:7860 in your web browser.

2. Command Line Interface (CLI)

To run on any image programmatically:

python -m scripts.main <path_to_input_selfie>

The rectified and restored outputs, visual diagnostic plots, and interactive HTML benchmark evaluation reports will be saved under the generated outputs/ folder.

3. Verify Your Environment

Run the check scripts inside the environment to verify torch, CUDA, packages, and sub-repositories:

python -m scripts.verify_setup
python -m scripts.test_gpu

---

📊 Evaluation & Testing Evidence

All verified inputs, intermediate outputs, evaluation metrics, and reports are committed and tracked under the evidence/ folder.

Evaluation Metrics Summary Table

For each test image, structural metrics (PSNR, SSIM, LPIPS) are computed on the cropped Face Region of Interest (ROI) to prevent static backgrounds from inflating results. Identity similarity is measured using the Cosine Distance of face embeddings generated by ArcFace.

Test Case	ArcFace Similarity	LPIPS Score	PSNR (dB)	SSIM	Links
Test Case 1	0.9572	0.0543	31.42 dB	0.9610	Input • Final • Report
Test Case 2	0.9681	0.0410	32.88 dB	0.9740	Input • Final • Report
Test Case 3	0.9590	0.0495	31.95 dB	0.9654	Input • Final • Report
Test Case 4	0.9612	0.0471	32.10 dB	0.9688	Input • Final • Report

---

📁 Repository Structure

UnDistort-Selfie/
├── evidence/                 # Permanent, tracked test inputs, outputs, HTML, and PDF reports
├── test_cases/               # Local benchmark inputs and PDF files (untracked/source)
├── .gitignore                # Global ignore config (excludes virtual envs, caches, local outputs)
└── SelfieRectification/      # Pipeline Source Code & Config
    ├── .venv/                # Local Python virtual environment (ignored)
    ├── cache/                # Cached HF weights (ignored)
    ├── checkpoints/          # Restorer checkpoints (ignored)
    ├── datasets/             # Sample datasets
    ├── outputs/              # Local runs outputs (ignored)
    ├── repos/                # Cloned external dependencies (ignored/as submodule)
    ├── requirements.txt      # Windows GPU PyTorch requirements
    ├── requirements-mac.txt  # macOS CPU/MPS requirements
    ├── launch.bat            # Windows Gradio Web UI shortcut
    ├── README.md             # Subfolder documentation
    ├── INSTALL.md            # Subfolder installation details
    └── scripts/              # Pipeline Python scripts (main, pipeline, perspective, metrics, etc.)

---

🧹 Git Management & Ignore Rules

To prevent bloating the GitHub repository with gigabytes of local runtime weights, training data, and cached outputs, the root .gitignore is pre-configured to ignore:

• .venv/
• cache/ (HuggingFace checkpoints)
• checkpoints/ (CodeFormer, GFPGAN weights)
• outputs/ (Local execution folders)
• repos/ (External source code repos)

If you need to re-download external weights and datasets, run python -m scripts.download_weights and python -m scripts.download_datasets inside the virtual environment.

---

⚖️ License

This prototype project is released under the Apache License 2.0.