diff --git a/README.en.md b/README.en.md new file mode 100644 index 0000000..4a4f934 --- /dev/null +++ b/README.en.md @@ -0,0 +1,507 @@ +

VisionSR

+ +

+ 한국어 · + English · + 中文 · + 日本語 +

+ +

+ Enterprise AI super resolution · upscaling, restoration, face recovery and detail reconstruction
+ Automatic model routing · 9 models · tiled inference from 4 GB VRAM · fully local
+ CLI · HTTP API · Web UI · Electron desktop app +

+ +

+ CI + Security + Python 3.11+ + Ruff + mypy + License: Apache-2.0 +

+ +Give it an image; it works out what the image *is* (photograph, portrait, line art, +scan), how badly it is damaged (noise, blur, JPEG), picks the model that suits, sizes +its tiles to the VRAM it actually has, and runs. No dials to turn unless you want to. + +```bash +visionsr enhance photo.jpg --scale 4 # that is the whole command +``` + +--- + +## Background + +Every upscaler faces the same fork: a photograph, a scanned document and a cel-shaded +drawing each need a *different* model, and running the wrong one is not a small error. +A photo model over line art stipples texture into flat regions; an anime model over a +portrait turns skin to plastic. Most tools push that decision onto the user as a +dropdown, which only works if the user already knows the answer. + +VisionSR makes the decision itself. It measures the image — colour statistics, edge +structure, blind noise/blur/JPEG estimates, face detection — and routes to the model +those measurements imply, then sizes its tiles to the VRAM actually present. The +dropdown still exists, but it is an override rather than a prerequisite. + +Three constraints shaped the rest: + +- **It runs where the images are.** No upload, no per-image cost, no queue. A 4 GB + laptop GPU is the design target, which is why tiling and VRAM estimation are core + rather than optional. +- **The claims are measured.** `scripts/verify_quality.py` degrades a known original, + reconstructs it, and scores against the truth with LPIPS. The figures in + [Status](#status) come from that script and you can re-run it. +- **The output is honest.** A correctly-sized image is not evidence of anything, so + the tests assert things about pixels and metadata rather than about shapes. + +## Quick start + +```bash +python -m venv .venv && . .venv/Scripts/activate # Windows; use bin/activate on Unix +pip install -e ".[api,onnx,metrics,dev]" + +# PyTorch, matched to your driver — check `nvidia-smi` first (>=525 -> cu126, 452-525 -> cu118) +pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126 + +python scripts/download_weights.py --all # ~485 MB of checkpoints +visionsr doctor # confirm it found your GPU +``` + +Then pick a surface: + +```bash +visionsr enhance photo.jpg --scale 4 # CLI +uvicorn backend.app.main:app --port 8000 # HTTP API, docs at /docs +cd frontend && npm install && npm run dev # web UI on :3000 +cd desktop && npm install && npm start # desktop app (starts its own engine) +``` + +Full detail in [Install](#install) and [Use](#use). For the test and end-to-end +suites, see [Tests](#tests). + +## Directory + +``` +ai/ The engine. Knows nothing about HTTP or the UI. + visionsr/ + analysis/ What is this image, and how damaged? Classifier, blind + quality estimators, face detection, model selection. + backends/ One interface, many runtimes: torch, ONNX, OpenVINO. + core/ Types, config, errors, and the model/architecture registries. + inference/ The engine loop and the tiler (overlap, feathering, VRAM). + models/ Network definitions — RRDBNet, SRVGG, GFPGAN, StyleGAN2. + pipelines/ Multi-model flows: face restoration, alpha matting. + preprocessing/ Image decode/encode. The only place BGR/RGB and EXIF are handled. + postprocessing/ Sharpening and other output filters. + exporters/ Torch -> ONNX, so the packaged app can ship without PyTorch. + configs/models.yaml The model registry. Single source of truth; no model is + named in code. + checkpoints/ Downloaded weights (gitignored). + +backend/app/ FastAPI service: routes, job queue, SSE progress, schemas. +frontend/ Next.js workspace (static export) + the e2e scripts. +desktop/ Electron shell: spawns the engine as a sidecar, hot folder, + native open/save. +packaging/ PyInstaller spec for the no-PyTorch server binary. +scripts/ Weights, assets, ONNX export, quality gate, fixtures, installer. +tests/ Two tiers: unmarked (no weights, no GPU) and @pytest.mark.weights. +.github/workflows/ CI (lint, types, tests, builds) and Security (CVEs, CodeQL, + secrets, filesystem). +``` + +## Models + +The registry is [`ai/configs/models.yaml`](ai/configs/models.yaml). Nothing in the +engine names a model; it asks for "something that does X for content type Y", so +adding a checkpoint of a known architecture takes no code. + +### Deep learning + +| Model | Task | Architecture | Scale | Notes | License | +|---|---|---|---|---|---| +| `realesrgan-x4plus` | Super resolution | RRDBNet (16.7M) | 4x | Strongest detail, slowest. The default on a capable GPU. | BSD-3-Clause | +| `realesrgan-x2plus` | Super resolution | RRDBNet | 2x | Native 2x; sharper than downscaling a 4x result. | BSD-3-Clause | +| `realesr-general-x4v3` | Super resolution | SRVGG (1.2M) | 4x | ~8x faster, far lighter on VRAM. Right default on a 4 GB card. | BSD-3-Clause | +| `realesrgan-x4plus-anime` | Anime / illustration | RRDBNet 6B | 4x | Preserves flat cel shading instead of stippling it. | BSD-3-Clause | +| `realesr-animevideov3` | Anime / illustration | SRVGG | 4x | Lightweight anime path. | BSD-3-Clause | +| `gfpgan-v1.4` | Face restoration | GFPGAN + StyleGAN2 | 1x | Runs on aligned 512px crops and blends back. Recovers hair, lashes, iris. | Apache-2.0 | +| `isnet-general` | Background removal | IS-Net (ONNX) | 1x | Alpha matting; holds thin structure like wheel spokes. | Apache-2.0 | +| `u2netp` | Background removal | U²-Net lite (ONNX) | 1x | 4 MB fallback. | Apache-2.0 | +| YuNet | Face detection | OpenCV DNN (ONNX) | — | 340 KB. Gates face restoration and portrait classification. | Apache-2.0 | + +Super resolution is GAN-based, so it is expected to *lose* PSNR while being +perceptually much closer — see the LPIPS note in [Status](#status). + +### Classical ML and signal processing + +Not everything here is a network, and the parts that are not are deliberate: + +| Component | Method | Why not a network | +|---|---|---| +| Content classification | Heuristic ensemble over colour-count, saturation, edge and frequency statistics | Inspectable — `ImageAnalysis.scores` names the signal that decided. No weights, no GPU, ~10 ms. | +| Noise estimation | Immerkær Laplacian-of-Laplacian kernel | Blind, closed-form, no training data needed. | +| Blur estimation | Variance of Laplacian + gradient statistics | Same. | +| JPEG artefact estimation | Block-boundary discontinuity at the 8x8 grid | Directly measures the artefact rather than inferring it. | +| Model selection | Threshold rules over the above, priority-ordered | A wrong route is the most visible failure mode; it needs to be explainable. | + +A learned classifier slots in behind the same `classify()` signature — the scores +above are what it would be trained to reproduce. + +--- + +## Status + +The engine is real and verified. On a laptop RTX 3050 Ti (4 GB), against Kodak +reference images degraded with blur + noise + JPEG-50 and upscaled 4x: + +| | LPIPS ↓ (SR) | LPIPS ↓ (bicubic) | improvement | +|---|---|---|---| +| kodim04 | **0.391** | 0.685 | **43 %** | +| kodim05 | **0.359** | 0.724 | **50 %** | + +LPIPS is the gate, not PSNR. Real-ESRGAN is a *perceptual* model: it synthesises +texture that is right to a human eye and wrong to a pixel metric, so it loses ~1 dB of +PSNR while being dramatically closer perceptually. Details in +[`scripts/verify_quality.py`](scripts/verify_quality.py) — run it yourself. + +Face restoration works: from a 128×192 JPEG-50 crop, GFPGAN recovers individual hair +strands, eyelashes and iris detail that super resolution alone smooths away. Background +removal works: IS-Net keeps the spokes of a bicycle wheel. + +**Built and verified end to end:** the AI engine, the pluggable model/backend registries, +streaming tiled inference, analysis and model selection, the face and matting pipelines, +the CLI, the HTTP API, the Next.js workspace, and an Electron desktop app that ships with +no PyTorch in it. + +**Not built:** the training pipeline and the cloud/multi-tenant layer. See +[Roadmap](#roadmap). + +--- + +## Install + +Requires Python 3.11+ and, for GPU, an NVIDIA card. + +```bash +python -m venv .venv && . .venv/Scripts/activate # Windows +pip install -e ".[api,onnx,metrics,dev]" + +# PyTorch, matched to your driver — check `nvidia-smi` first: +# driver >= 525 -> CUDA 12.x +pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126 +# driver 452-525 -> CUDA 11.8 (older laptops; CUDA 11 minor-version compatibility) +pip install torch==2.7.1+cu118 torchvision==0.22.1+cu118 --index-url https://download.pytorch.org/whl/cu118 + +python scripts/download_weights.py --all # ~485 MB +visionsr doctor # confirm it found your GPU +``` + +`visionsr doctor` is the first thing to run when anything looks wrong. It reports every +backend it can see, the runtimes that are installed, and whether CUDA actually +initialised — which is not the same as whether you have an NVIDIA card. + +--- + +## Use + +### CLI + +```bash +visionsr enhance photo.jpg # 4x, auto everything +visionsr enhance photo.jpg -s 2 -o out.png # 2x, explicit output +visionsr enhance ./album -r -o ./upscaled # batch, recursive +visionsr enhance scan.png --model realesr-general-x4v3 # pin a model +visionsr enhance photo.jpg --remove-bg # upscale, restore, and cut the subject out + +visionsr analyze photo.jpg # what it sees and which model it would choose +visionsr models # what is registered, and what can actually run here +visionsr doctor # what hardware it can use +``` + +### Python + +```python +from visionsr import enhance_file, EnhanceOptions + +result = enhance_file("photo.jpg", EnhanceOptions(scale=4)) + +result.image # HWC uint8 RGB numpy array +result.analysis # what the classifier and quality estimators found +result.runs # which models ran, on what backend, for how long +``` + +### HTTP API + +```bash +uvicorn backend.app.main:app --port 8000 # docs at /docs +``` + +| | | +|---|---| +| `GET /api/v1/health` | backends, device, resident models | +| `GET /api/v1/models` | the registry, with install status | +| `POST /api/v1/analyze` | profile an image; no enhancement | +| `POST /api/v1/jobs` | queue an enhancement → `202 {job_id}` | +| `GET /api/v1/jobs/{id}/events` | live progress over SSE | +| `GET /api/v1/jobs/{id}/result` | the enhanced image | + +`POST /api/v1/jobs` takes `remove_background`, and refuses `output_format=jpeg` with it — +JPEG has no alpha channel, so the cutout would be silently flattened onto white. A request +that succeeds while throwing away what it was asked for is the worst possible outcome. + +### Desktop + +```bash +cd desktop && npm install && npm start +``` + +Electron's job here is narrow, and stays that way: it starts the Python engine, serves +the same frontend the browser build uses, and adds the things a web page cannot do. +There is no second implementation of anything — the desktop and web apps run identical +UI code against an identical HTTP API. The API just happens to be a child process. + +- **Hot folder.** Point it at a directory; anything dropped in comes back enhanced + beside the original. This is what makes a desktop build worth having: a background + service with a window attached, rather than a page. It runs in the main process, so a + crashed renderer does not stop it, and it shares the backend's single-worker queue, so + it cannot starve an interactive job. +- **Native open and save.** The browser can only drop a file into Downloads. "Save this + 192 MP PNG next to the original" is the actual workflow. +- **The engine dies with the app.** An orphaned `uvicorn` holding a GPU is invisible, and + it makes the *next* launch fail for reasons that look nothing like the cause. It is + killed through the process tree, and there is a test that asserts the port is dead + after the window closes. + +The sidecar takes an ephemeral port (hard-coding 8000 means the app cannot start +alongside anything else, including a second copy of itself), which is why the frontend +resolves its API base URL at runtime rather than at build time. + +### Shipping: one .exe, and no PyTorch in it + +```bash +bash scripts/build_installer.sh # -> desktop/release/VisionSR-0.1.0-x64.exe +``` + +The shipped app does not contain PyTorch, and that is the central packaging decision. + +| | size | runs on | +|---|---|---| +| torch + CUDA | ~2.5 GB | NVIDIA only, and only with a recent driver | +| **ONNX Runtime + DirectML** | **~250 MB** | **any DX12 GPU — NVIDIA, AMD, Intel — no CUDA at all** | + +Nobody ships a desktop image tool by freezing a 2.5 GB CUDA stack into it. So torch is a +*development* dependency — training, research, and exporting the graphs — and the +shipped runtime is ONNX. Verified on the development machine, whose NVIDIA driver is too +old for CUDA 12: the full chain (Real-ESRGAN → GFPGAN → IS-Net) runs on DirectML with no +torch present at all, and a packaged-app test asserts exactly that (`cuda` and `cpu` are +torch-only backends, so their *absence* from `/health` is the proof). + +What it costs is a build gate. An ONNX export can be silently wrong — the graph builds, +the session loads, the output is the right size and subtly degraded. So +`scripts/export_onnx.py` checks every graph against the torch model it came from, on real +input, **on every execution provider the app might use**, and fails the build if any +deviates by more than 2/255 (below the quantisation step of the 8-bit image either would +be written to, so anything under it is provably invisible). Measured: 0.001/255 for the SR +models, 0.005/255 for GFPGAN. + +"On every provider" is not belt-and-braces. Verifying on CPU alone nearly shipped a +catastrophically broken GFPGAN — see the bug list below. + +The installer bundles the graphs (652 MB) rather than the `.pth` checkpoints, which the +shipped runtime cannot use. Building it needs ~4 GB of free disk. + +--- + +## How it decides + +``` +upload → analyse → classify → select model → tiled inference + → face restore → post-process → remove background → export +``` + +**Analysis** measures what is actually wrong with the image, not what it looks like: +noise via Immerkær's estimator, JPEG blocking by comparing gradients across the 8×8 +grid against gradients inside it, blur as Laplacian variance normalised by the image's +own contrast (so a low-contrast but sharp photo is not misread as blurred). + +**Classification** is a documented heuristic ensemble — flatness, palette +concentration, axis-alignment of gradients, colourfulness — not a learned model. That +is a deliberate trade: it is inspectable (`analysis.scores` says exactly which signal +drove the decision), it needs no weights, and the boundary that matters most +(illustration vs photograph) is genuinely separable by these statistics. Getting it +wrong is the most *visible* failure this system can have — a photo model stipples fake +texture into flat cel-shaded regions — so the fallbacks are conservative and +`--model` always wins. + +**Selection** asks the registry for "a model that does X", never for a model by name. +It matches model *capacity* to how much repair the image needs: a pristine 24 MP photo +gets the 1.2 M-parameter model and a result in seconds; a ruined 0.2 MP JPEG earns the +16.7 M-parameter one. + +**Background removal** is last, and that ordering is not incidental. The segmenter +downsamples whatever it is handed to a fixed 1024 px square, so a 512 px upscaled canvas +resized to 1024 is a far better-defined input than a 128 px original resized to 1024 — +and the edge quality of a cutout is entirely a question of detail. Cutting first would +throw away the detail super resolution had just reconstructed. The alpha stays *soft*: +hair and fur are genuinely semi-transparent, and a hard threshold turns them into a +jagged silhouette. + +**Tiling** is what makes a 4 GB card work at all. Tiles overlap and are blended with a +raised-cosine feather (not cropped), so there is no seam; tile size is derived from +*live* free VRAM; on OOM the runner halves the tile and retries rather than failing. +Verified numerically: tiled output matches untiled output to within one 8-bit level. + +**Assembly streams.** The obvious way to blend overlapping tiles is a float32 +accumulator the size of the output plus a weight map — 16 bytes per output pixel. At +4x on a 12 MP photo that is a 2.9 GB allocation, and when it fails there is no +exception to catch: the process is killed. But tiles arrive in row-major order, so +once the next tile row starts, every row above it is final. Only the rows in play need +to exist in float. Memory drops from O(output pixels) to O(tile height × width), the +canvas stays `uint8`, and each tile is converted to float individually on its way into +the network — which is the only place float was ever needed. + +The result: a 12 MP photo upscales to **192 MP (16000 × 12000) in 36 s on a 4 GB laptop +GPU**. The old assembler could not have allocated the buffer at all. + +--- + +## Architecture + +``` +ai/visionsr/ + core/ types, errors, config, device probing, registries + backends/ torch (CPU/CUDA/TensorRT), ONNX Runtime, DirectML, OpenVINO + models/ RRDBNet, SRVGGNetCompact, GFPGAN — architectures only + inference/ tiled runner, engine + analysis/ quality estimators, classifier, face detection, model selector + pipelines/ face restoration + pre/postprocessing/ + exporters/ torch -> ONNX, on demand +ai/configs/models.yaml the model registry — the only place a model is named +backend/app/ FastAPI: jobs, SSE progress, models, analyze +frontend/ Next.js workspace +``` + +Two rules hold the whole thing together. + +**No model is named in engine code.** Models live in `ai/configs/models.yaml`; adding a +checkpoint of a known architecture is a YAML entry and no code at all. Adding a new +*architecture* is one file and one `@register_architecture` decorator. This is enforced +by tests that build a registry of entirely fictional models and assert the engine still +routes correctly. + +**Backends expose one contract: NHWC float32 in [0,1], in and out.** Networks disagree +about their value range — SR models want [0,1], anything with a StyleGAN decoder wants +[-1,1] — so each `ModelSpec` declares its `value_range` and the backend converts at the +boundary. Nothing above that layer knows or cares. + +--- + +## Things that were wrong, and are documented so they stay fixed + +Every one of these produced a correctly-sized image or a green test run. None raised. +They were found by putting the output on screen and by driving the real app in a real +browser. Each now has a regression test that asserts something about the *pixels* or the +*behaviour*, not the plumbing. + +- **GFPGAN's `[-1,1]` output was clamped to `[0,1]`** by the backend, deleting every + shadow and returning a flat pink smear where the face was. Fixed by making value range + a declared property of the model, not an assumption of the backend. +- **GFPGAN collapses in fp16.** Not a crash — the StyleGAN2 decoder's demodulation plus + a per-layer √2 gain exceeds half precision's dynamic range, and the output silently + degenerates to a near-constant dark field (measured: fp32 std 0.343 → fp16 std 0.123). + It is pinned to fp32 in the registry; the SR models keep fp16. +- **PyTorch's caching allocator hid free VRAM.** `mem_get_info` reports what the *driver* + has left, but torch does not return freed blocks to the driver — so after one inference + the tiler concluded it was out of memory and dropped to 64 px tiles for the rest of the + process. Correct output, several times slower. +- **The model selector could not do what its own comment claimed.** "Prefer the cheap + model on clean input" was unreachable: `priority` contributed up to 100 points and the + quality term at most 20. Rewritten around cost-in-proportion-to-work, and the policy + reversed once measured: a *small* image gets the best model, because there the expensive + one costs a second; a large one gets the fast model, because there it costs minutes. +- **`asyncio.create_task` without a strong reference.** The event loop holds only a weak + one, so a fire-and-forget job could be garbage-collected mid-flight — leaving the + client's progress bar frozen forever, and its abandoned coroutine still holding GPU + tensors. That leak starved VRAM until the tiler fell to 64 px tiles and a routine job + took **ten minutes**. Found by running the actual UI, not the tests. +- **`POST /analyze` blocked the event loop.** It is `async def`, but it decoded the image + and ran face detection inline: 100–500 ms during which every other request — including + the health check the frontend uses to decide the backend is alive — stalled behind it. + `async def` is a promise not to block; that function could not keep it. +- **Progress meant two different things.** The engine emitted a fraction *within the + current stage*, so a progress bar hit 100% four times. It is now a single monotonic + 0→100% across the whole job, with each stage weighted by how long it actually takes. +- **The ONNX export gate was measuring the wrong machine.** The shipped runtime is ONNX + Runtime on DirectML, and the export was verified on ONNX Runtime's *CPU* provider. Those + are not the same computer. In fp16, ONNX Runtime's CPU kernels quietly compute much of a + graph in fp32 — so the fp16 GFPGAN measured **1.5/255** from torch and passed. Run the + identical graph on DirectML and its StyleGAN2 decoder collapses exactly as torch's fp16 + did: output std 0.343 → 0.123, **179/255** from the truth, no error and no NaN — a dark + smear where the face was. The same CPU-only gate also hid that *every* fp16 SR graph + fails outright on DirectML, on its final `Conv`, with an HRESULT the Python binding + cannot even decode. The graphs were fine. The gate was wrong. It now verifies on every + provider that ships, and demotes a model to fp32 the moment fp16 misbehaves anywhere it + will run. + +- **A big scale killed the process.** The tiler assembled into a full-size float32 + accumulator plus weight map — 16 bytes per output pixel — so a 12 MP photo at 4x wanted + 2.9 GB of host RAM just to stitch. When that allocation fails there is nothing to catch: + the worker dies and takes every queued job with it. Fixed properly rather than papered + over: assembly now streams row-bands into a `uint8` canvas, so the float buffer is + bounded by tile height and does not grow with the image. Same output, to the byte — + the LPIPS figures above are unchanged after the refactor. + +--- + +## Tests + +```bash +pytest # 91 tests, 2 skipped (DirectML/OpenVINO not installed here) +pytest -m "not weights" # the subset needing no checkpoints and no GPU +python scripts/verify_quality.py # quality gate against ground truth + +python scripts/download_assets.py # ground-truth photos; without them ~19 tests skip +python scripts/make_e2e_fixtures.py # build the generated inputs the e2e scripts upload + +cd frontend && node e2e-smoke.mjs # drives the real UI in Chromium against the real backend +cd frontend && node e2e-orientation.mjs # a phone photo (EXIF-rotated) survives the round trip +cd desktop && npm run e2e # launches the real Electron app, runs a real job +cd desktop && npm run e2e:hotfolder # drops a file in a watched folder, waits for the result +``` + +The end-to-end tests are not decoration. Five of the bugs above were invisible to the +Python suite and only appeared when something actually drove the app: three when a +browser ran a job through a long-running server, one when a file was dropped into a +watched folder a moment too early, and one that only a browser could see at all — the +result was rotated by its own EXIF on the way to the screen, which nothing that +inspects the pixel array in memory can detect. + +The browser scripts prefer Playwright's pinned Chromium and fall back to an installed +Chrome or Edge if it will not start; `PW_CHANNEL=chrome|msedge|chromium` pins one. + +--- + +## Roadmap + +Built and verified end to end: the engine, the model and backend registries, streaming +tiled inference, analysis and selection, the face pipeline, the CLI, the HTTP API, the +Next.js workspace, and the Electron desktop shell with its hot folder. + +Next, in the order they unlock the most: + +1. **More architectures** — SwinIR, HAT, DAT, CodeFormer. Each is one file plus a YAML + entry; the engine does not change. +3. **Training pipeline** — losses, metrics, AMP, DDP, EMA, resume. +4. **Cloud** — Postgres, Redis, RabbitMQ workers, S3, multi-tenancy, billing. The job API + is already the shape a queue-backed deployment would expose, so the frontend does not + change. + +--- + +## Licence + +Apache-2.0. Model weights carry their own licences (Real-ESRGAN: BSD-3-Clause; +GFPGAN: Apache-2.0) and are downloaded, not redistributed. diff --git a/README.ja.md b/README.ja.md index b5b5e1a..8eaa4dc 100644 --- a/README.ja.md +++ b/README.ja.md @@ -1,8 +1,8 @@

VisionSR

- English · - 한국어 · + 한국어 · + English · 中文 · 日本語

@@ -31,7 +31,7 @@ visionsr enhance photo.jpg --scale 4 # コマンドはこれだけ ``` > 以下はプロジェクトの概要です。ベンチマーク、設計上の判断、API リファレンス、 -> パッケージングを含む全文は [英語版 README](README.md) を参照してください。 +> パッケージングを含む全文は [英語版 README](README.en.md) を参照してください。 --- @@ -92,7 +92,7 @@ visionsr models # 登録済みモデルと実行可否 visionsr doctor # 利用可能なハードウェア ``` -テストと E2E の実行方法は英語版 README の [Tests](README.md#tests) を参照してください。 +テストと E2E の実行方法は英語版 README の [Tests](README.en.md#tests) を参照してください。 ## ディレクトリ diff --git a/README.ko.md b/README.ko.md deleted file mode 100644 index 6e2b2a1..0000000 --- a/README.ko.md +++ /dev/null @@ -1,160 +0,0 @@ -

VisionSR

- -

- English · - 한국어 · - 中文 · - 日本語 -

- -

- AI 초해상도 엔진 · 이미지 업스케일링, 복원, 얼굴 복구, 디테일 재구성
- 모델 자동 라우팅 · 모델 9종 · 4GB VRAM부터 타일 추론 · 완전 로컬 실행
- CLI · HTTP API · 웹 UI · Electron 데스크톱 앱 -

- -

- CI - Security - Python 3.11+ - Ruff - mypy - License: Apache-2.0 -

- -이미지를 넣으면 그것이 무엇인지(사진, 인물, 선화, 스캔), 얼마나 손상됐는지(노이즈, -블러, JPEG)를 스스로 판단해 알맞은 모델을 고르고, 실제 가용 VRAM에 맞춰 타일 크기를 -정한 뒤 실행합니다. 원한다면 직접 지정할 수도 있지만, 기본값만으로 동작합니다. - -```bash -visionsr enhance photo.jpg --scale 4 # 이게 전부입니다 -``` - -> 아래는 프로젝트 개요입니다. 벤치마크, 아키텍처 설계 근거, API 레퍼런스, 패키징 등 -> 전체 문서는 [영문 README](README.md)를 참고하세요. - ---- - -## 프로젝트 배경 - -업스케일러는 모두 같은 갈림길을 만납니다. 사진과 스캔 문서와 셀 셰이딩 그림은 각각 -**다른** 모델이 필요하고, 잘못 고르면 결과가 조금 나빠지는 정도로 끝나지 않습니다. -선화에 사진용 모델을 돌리면 평평해야 할 면에 잡텍스처가 끼고, 인물 사진에 애니용 -모델을 돌리면 피부가 플라스틱처럼 변합니다. 대부분의 도구는 이 선택을 드롭다운으로 -사용자에게 넘기지만, 그건 사용자가 이미 정답을 알고 있을 때만 작동하는 방식입니다. - -VisionSR은 그 판단을 직접 합니다. 색상 통계, 엣지 구조, 무참조 노이즈·블러·JPEG -추정, 얼굴 검출로 이미지를 측정하고, 그 측정값이 가리키는 모델로 라우팅한 뒤 실제로 -존재하는 VRAM에 맞춰 타일을 자릅니다. 드롭다운은 그대로 있지만 전제 조건이 아니라 -재정의 수단입니다. - -나머지 설계를 결정한 세 가지 제약: - -- **이미지가 있는 곳에서 실행됩니다.** 업로드도, 장당 과금도, 대기열도 없습니다. - 설계 기준은 4GB 노트북 GPU이며, 그래서 타일링과 VRAM 추정이 선택 기능이 아니라 - 핵심입니다. -- **주장은 측정합니다.** `scripts/verify_quality.py`가 원본을 의도적으로 열화시키고 - 복원한 뒤 LPIPS로 채점합니다. README의 수치는 그 스크립트 출력이고 직접 재현할 수 - 있습니다. -- **결과는 정직해야 합니다.** 크기가 맞는 이미지는 아무것도 증명하지 못하므로, - 테스트는 형태가 아니라 픽셀과 메타데이터를 검증합니다. - -## 실행 방법 - -```bash -python -m venv .venv && . .venv/Scripts/activate # Windows (Unix는 bin/activate) -pip install -e ".[api,onnx,metrics,dev]" - -# 드라이버에 맞춘 PyTorch — 먼저 `nvidia-smi` 확인 (>=525 -> cu126, 452~525 -> cu118) -pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126 - -python scripts/download_weights.py --all # 체크포인트 약 485MB -visionsr doctor # GPU 인식 확인 -``` - -이후 원하는 인터페이스를 고르면 됩니다: - -```bash -visionsr enhance photo.jpg --scale 4 # CLI -uvicorn backend.app.main:app --port 8000 # HTTP API (문서: /docs) -cd frontend && npm install && npm run dev # 웹 UI (:3000) -cd desktop && npm install && npm start # 데스크톱 앱 (엔진을 직접 기동) -``` - -주요 CLI 명령: - -```bash -visionsr enhance photo.jpg --remove-bg # 업스케일 + 얼굴 복원 + 배경 제거 -visionsr enhance ./album -r -o ./out # 폴더 일괄 처리 -visionsr analyze photo.jpg # 무엇으로 인식했고 어떤 모델을 고를지 -visionsr models # 등록된 모델과 실행 가능 여부 -visionsr doctor # 사용 가능한 하드웨어 -``` - -테스트와 E2E 실행은 영문 README의 [Tests](README.md#tests) 절을 참고하세요. - -## 디렉토리 - -``` -ai/ 엔진. HTTP나 UI를 전혀 알지 못합니다. - visionsr/ - analysis/ 이 이미지는 무엇이고 얼마나 손상됐는가 — 분류기, 무참조 - 품질 추정, 얼굴 검출, 모델 선택. - backends/ 하나의 인터페이스, 여러 런타임: torch, ONNX, OpenVINO. - core/ 타입, 설정, 예외, 모델/아키텍처 레지스트리. - inference/ 엔진 루프와 타일러(오버랩, 페더링, VRAM 계산). - models/ 신경망 정의 — RRDBNet, SRVGG, GFPGAN, StyleGAN2. - pipelines/ 다중 모델 흐름: 얼굴 복원, 알파 매팅. - preprocessing/ 이미지 디코드/인코드. BGR/RGB와 EXIF를 다루는 유일한 곳. - postprocessing/ 샤프닝 등 출력 필터. - exporters/ Torch -> ONNX. 배포판이 PyTorch 없이 동작하기 위한 경로. - configs/models.yaml 모델 레지스트리. 유일한 진실 공급원이며, 코드에는 모델 - 이름이 등장하지 않습니다. - checkpoints/ 내려받은 가중치 (gitignore 대상). - -backend/app/ FastAPI 서비스: 라우트, 작업 큐, SSE 진행률, 스키마. -frontend/ Next.js 워크스페이스(정적 export)와 E2E 스크립트. -desktop/ Electron 셸: 엔진을 사이드카로 기동, 핫 폴더, 네이티브 열기/저장. -packaging/ PyTorch 없는 서버 바이너리를 만드는 PyInstaller 스펙. -scripts/ 가중치·에셋 다운로드, ONNX 익스포트, 품질 게이트, 픽스처, 인스톨러. -tests/ 두 계층: 무표식(가중치·GPU 불필요)과 @pytest.mark.weights. -.github/workflows/ CI(린트·타입·테스트·빌드)와 Security(CVE, CodeQL, 시크릿, 파일시스템). -``` - -## 사용 모델 - -레지스트리는 [`ai/configs/models.yaml`](ai/configs/models.yaml)입니다. 엔진은 모델을 -이름으로 부르지 않고 "콘텐츠 유형 Y에 대해 X를 하는 무언가"를 요청하므로, 이미 아는 -아키텍처의 체크포인트를 추가하는 데는 코드가 필요 없습니다. - -### 딥러닝 - -| 모델 | 작업 | 아키텍처 | 배율 | 특징 | 라이선스 | -|---|---|---|---|---|---| -| `realesrgan-x4plus` | 초해상도 | RRDBNet (16.7M) | 4x | 디테일 복원 최강, 가장 느림. 성능이 되는 GPU의 기본값. | BSD-3-Clause | -| `realesrgan-x2plus` | 초해상도 | RRDBNet | 2x | 네이티브 2x. 4x 결과를 축소하는 것보다 선명. | BSD-3-Clause | -| `realesr-general-x4v3` | 초해상도 | SRVGG (1.2M) | 4x | 약 8배 빠르고 VRAM 부담이 훨씬 적음. 4GB 카드의 적정 기본값. | BSD-3-Clause | -| `realesrgan-x4plus-anime` | 애니 / 일러스트 | RRDBNet 6B | 4x | 평평한 셀 셰이딩을 잡텍스처로 만들지 않고 보존. | BSD-3-Clause | -| `realesr-animevideov3` | 애니 / 일러스트 | SRVGG | 4x | 경량 애니 경로. | BSD-3-Clause | -| `gfpgan-v1.4` | 얼굴 복원 | GFPGAN + StyleGAN2 | 1x | 정렬된 512px 크롭에서 동작 후 원본에 블렌딩. 머리카락·속눈썹·홍채 복원. | Apache-2.0 | -| `isnet-general` | 배경 제거 | IS-Net (ONNX) | 1x | 알파 매팅. 자전거 바퀴살 같은 얇은 구조를 유지. | Apache-2.0 | -| `u2netp` | 배경 제거 | U²-Net lite (ONNX) | 1x | 4MB 폴백. | Apache-2.0 | -| YuNet | 얼굴 검출 | OpenCV DNN (ONNX) | — | 340KB. 얼굴 복원과 인물 분류의 관문. | Apache-2.0 | - -초해상도 모델은 GAN 기반이라 PSNR은 **떨어지면서** 지각적으로는 훨씬 가까워지는 것이 -정상입니다. 자세한 내용은 영문 README의 LPIPS 설명을 참고하세요. - -### 고전 머신러닝 / 신호처리 - -전부가 신경망은 아니며, 신경망이 아닌 부분은 의도적인 선택입니다: - -| 구성 요소 | 방법 | 왜 신경망이 아닌가 | -|---|---|---| -| 콘텐츠 분류 | 색상 수·채도·엣지·주파수 통계에 대한 휴리스틱 앙상블 | 설명 가능 — `ImageAnalysis.scores`가 어떤 신호가 결정했는지 알려줌. 가중치도 GPU도 불필요하고 약 10ms. | -| 노이즈 추정 | Immerkær의 Laplacian-of-Laplacian 커널 | 무참조이고 해석적으로 닫힌 형태라 학습 데이터가 필요 없음. | -| 블러 추정 | Laplacian 분산 + 그래디언트 통계 | 위와 동일. | -| JPEG 아티팩트 추정 | 8x8 격자 경계의 불연속 측정 | 추론이 아니라 아티팩트를 직접 측정. | -| 모델 선택 | 위 지표에 대한 임계값 규칙 + 우선순위 | 잘못된 라우팅이 가장 눈에 띄는 실패이므로 설명 가능해야 함. | - -학습 기반 분류기는 동일한 `classify()` 시그니처 뒤로 그대로 대체할 수 있으며, 위 -점수들이 곧 그 모델이 재현하도록 학습될 목표입니다. diff --git a/README.md b/README.md index 0974d1f..cd7dfd2 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,16 @@

VisionSR

- English · - 한국어 · + 한국어 · + English · 中文 · 日本語

- Enterprise AI super resolution · upscaling, restoration, face recovery and detail reconstruction
- Automatic model routing · 9 models · tiled inference from 4 GB VRAM · fully local
- CLI · HTTP API · Web UI · Electron desktop app + AI 초해상도 엔진 · 이미지 업스케일링, 복원, 얼굴 복구, 디테일 재구성
+ 모델 자동 라우팅 · 모델 9종 · 4GB VRAM부터 타일 추론 · 완전 로컬 실행
+ CLI · HTTP API · 웹 UI · Electron 데스크톱 앱

@@ -22,486 +22,139 @@ License: Apache-2.0

-Give it an image; it works out what the image *is* (photograph, portrait, line art, -scan), how badly it is damaged (noise, blur, JPEG), picks the model that suits, sizes -its tiles to the VRAM it actually has, and runs. No dials to turn unless you want to. +이미지를 넣으면 그것이 무엇인지(사진, 인물, 선화, 스캔), 얼마나 손상됐는지(노이즈, +블러, JPEG)를 스스로 판단해 알맞은 모델을 고르고, 실제 가용 VRAM에 맞춰 타일 크기를 +정한 뒤 실행합니다. 원한다면 직접 지정할 수도 있지만, 기본값만으로 동작합니다. ```bash -visionsr enhance photo.jpg --scale 4 # that is the whole command +visionsr enhance photo.jpg --scale 4 # 이게 전부입니다 ``` ---- - -## Background - -Every upscaler faces the same fork: a photograph, a scanned document and a cel-shaded -drawing each need a *different* model, and running the wrong one is not a small error. -A photo model over line art stipples texture into flat regions; an anime model over a -portrait turns skin to plastic. Most tools push that decision onto the user as a -dropdown, which only works if the user already knows the answer. - -VisionSR makes the decision itself. It measures the image — colour statistics, edge -structure, blind noise/blur/JPEG estimates, face detection — and routes to the model -those measurements imply, then sizes its tiles to the VRAM actually present. The -dropdown still exists, but it is an override rather than a prerequisite. - -Three constraints shaped the rest: - -- **It runs where the images are.** No upload, no per-image cost, no queue. A 4 GB - laptop GPU is the design target, which is why tiling and VRAM estimation are core - rather than optional. -- **The claims are measured.** `scripts/verify_quality.py` degrades a known original, - reconstructs it, and scores against the truth with LPIPS. The figures in - [Status](#status) come from that script and you can re-run it. -- **The output is honest.** A correctly-sized image is not evidence of anything, so - the tests assert things about pixels and metadata rather than about shapes. - -## Quick start - -```bash -python -m venv .venv && . .venv/Scripts/activate # Windows; use bin/activate on Unix -pip install -e ".[api,onnx,metrics,dev]" - -# PyTorch, matched to your driver — check `nvidia-smi` first (>=525 -> cu126, 452-525 -> cu118) -pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126 - -python scripts/download_weights.py --all # ~485 MB of checkpoints -visionsr doctor # confirm it found your GPU -``` - -Then pick a surface: - -```bash -visionsr enhance photo.jpg --scale 4 # CLI -uvicorn backend.app.main:app --port 8000 # HTTP API, docs at /docs -cd frontend && npm install && npm run dev # web UI on :3000 -cd desktop && npm install && npm start # desktop app (starts its own engine) -``` - -Full detail in [Install](#install) and [Use](#use). For the test and end-to-end -suites, see [Tests](#tests). - -## Directory - -``` -ai/ The engine. Knows nothing about HTTP or the UI. - visionsr/ - analysis/ What is this image, and how damaged? Classifier, blind - quality estimators, face detection, model selection. - backends/ One interface, many runtimes: torch, ONNX, OpenVINO. - core/ Types, config, errors, and the model/architecture registries. - inference/ The engine loop and the tiler (overlap, feathering, VRAM). - models/ Network definitions — RRDBNet, SRVGG, GFPGAN, StyleGAN2. - pipelines/ Multi-model flows: face restoration, alpha matting. - preprocessing/ Image decode/encode. The only place BGR/RGB and EXIF are handled. - postprocessing/ Sharpening and other output filters. - exporters/ Torch -> ONNX, so the packaged app can ship without PyTorch. - configs/models.yaml The model registry. Single source of truth; no model is - named in code. - checkpoints/ Downloaded weights (gitignored). - -backend/app/ FastAPI service: routes, job queue, SSE progress, schemas. -frontend/ Next.js workspace (static export) + the e2e scripts. -desktop/ Electron shell: spawns the engine as a sidecar, hot folder, - native open/save. -packaging/ PyInstaller spec for the no-PyTorch server binary. -scripts/ Weights, assets, ONNX export, quality gate, fixtures, installer. -tests/ Two tiers: unmarked (no weights, no GPU) and @pytest.mark.weights. -.github/workflows/ CI (lint, types, tests, builds) and Security (CVEs, CodeQL, - secrets, filesystem). -``` - -## Models - -The registry is [`ai/configs/models.yaml`](ai/configs/models.yaml). Nothing in the -engine names a model; it asks for "something that does X for content type Y", so -adding a checkpoint of a known architecture takes no code. - -### Deep learning - -| Model | Task | Architecture | Scale | Notes | License | -|---|---|---|---|---|---| -| `realesrgan-x4plus` | Super resolution | RRDBNet (16.7M) | 4x | Strongest detail, slowest. The default on a capable GPU. | BSD-3-Clause | -| `realesrgan-x2plus` | Super resolution | RRDBNet | 2x | Native 2x; sharper than downscaling a 4x result. | BSD-3-Clause | -| `realesr-general-x4v3` | Super resolution | SRVGG (1.2M) | 4x | ~8x faster, far lighter on VRAM. Right default on a 4 GB card. | BSD-3-Clause | -| `realesrgan-x4plus-anime` | Anime / illustration | RRDBNet 6B | 4x | Preserves flat cel shading instead of stippling it. | BSD-3-Clause | -| `realesr-animevideov3` | Anime / illustration | SRVGG | 4x | Lightweight anime path. | BSD-3-Clause | -| `gfpgan-v1.4` | Face restoration | GFPGAN + StyleGAN2 | 1x | Runs on aligned 512px crops and blends back. Recovers hair, lashes, iris. | Apache-2.0 | -| `isnet-general` | Background removal | IS-Net (ONNX) | 1x | Alpha matting; holds thin structure like wheel spokes. | Apache-2.0 | -| `u2netp` | Background removal | U²-Net lite (ONNX) | 1x | 4 MB fallback. | Apache-2.0 | -| YuNet | Face detection | OpenCV DNN (ONNX) | — | 340 KB. Gates face restoration and portrait classification. | Apache-2.0 | - -Super resolution is GAN-based, so it is expected to *lose* PSNR while being -perceptually much closer — see the LPIPS note in [Status](#status). - -### Classical ML and signal processing - -Not everything here is a network, and the parts that are not are deliberate: - -| Component | Method | Why not a network | -|---|---|---| -| Content classification | Heuristic ensemble over colour-count, saturation, edge and frequency statistics | Inspectable — `ImageAnalysis.scores` names the signal that decided. No weights, no GPU, ~10 ms. | -| Noise estimation | Immerkær Laplacian-of-Laplacian kernel | Blind, closed-form, no training data needed. | -| Blur estimation | Variance of Laplacian + gradient statistics | Same. | -| JPEG artefact estimation | Block-boundary discontinuity at the 8x8 grid | Directly measures the artefact rather than inferring it. | -| Model selection | Threshold rules over the above, priority-ordered | A wrong route is the most visible failure mode; it needs to be explainable. | - -A learned classifier slots in behind the same `classify()` signature — the scores -above are what it would be trained to reproduce. +> 아래는 프로젝트 개요입니다. 벤치마크, 아키텍처 설계 근거, API 레퍼런스, 패키징 등 +> 전체 문서는 [영문 README](README.en.md)를 참고하세요. --- -## Status +## 프로젝트 배경 -The engine is real and verified. On a laptop RTX 3050 Ti (4 GB), against Kodak -reference images degraded with blur + noise + JPEG-50 and upscaled 4x: +업스케일러는 모두 같은 갈림길을 만납니다. 사진과 스캔 문서와 셀 셰이딩 그림은 각각 +**다른** 모델이 필요하고, 잘못 고르면 결과가 조금 나빠지는 정도로 끝나지 않습니다. +선화에 사진용 모델을 돌리면 평평해야 할 면에 잡텍스처가 끼고, 인물 사진에 애니용 +모델을 돌리면 피부가 플라스틱처럼 변합니다. 대부분의 도구는 이 선택을 드롭다운으로 +사용자에게 넘기지만, 그건 사용자가 이미 정답을 알고 있을 때만 작동하는 방식입니다. -| | LPIPS ↓ (SR) | LPIPS ↓ (bicubic) | improvement | -|---|---|---|---| -| kodim04 | **0.391** | 0.685 | **43 %** | -| kodim05 | **0.359** | 0.724 | **50 %** | +VisionSR은 그 판단을 직접 합니다. 색상 통계, 엣지 구조, 무참조 노이즈·블러·JPEG +추정, 얼굴 검출로 이미지를 측정하고, 그 측정값이 가리키는 모델로 라우팅한 뒤 실제로 +존재하는 VRAM에 맞춰 타일을 자릅니다. 드롭다운은 그대로 있지만 전제 조건이 아니라 +재정의 수단입니다. -LPIPS is the gate, not PSNR. Real-ESRGAN is a *perceptual* model: it synthesises -texture that is right to a human eye and wrong to a pixel metric, so it loses ~1 dB of -PSNR while being dramatically closer perceptually. Details in -[`scripts/verify_quality.py`](scripts/verify_quality.py) — run it yourself. +나머지 설계를 결정한 세 가지 제약: -Face restoration works: from a 128×192 JPEG-50 crop, GFPGAN recovers individual hair -strands, eyelashes and iris detail that super resolution alone smooths away. Background -removal works: IS-Net keeps the spokes of a bicycle wheel. +- **이미지가 있는 곳에서 실행됩니다.** 업로드도, 장당 과금도, 대기열도 없습니다. + 설계 기준은 4GB 노트북 GPU이며, 그래서 타일링과 VRAM 추정이 선택 기능이 아니라 + 핵심입니다. +- **주장은 측정합니다.** `scripts/verify_quality.py`가 원본을 의도적으로 열화시키고 + 복원한 뒤 LPIPS로 채점합니다. README의 수치는 그 스크립트 출력이고 직접 재현할 수 + 있습니다. +- **결과는 정직해야 합니다.** 크기가 맞는 이미지는 아무것도 증명하지 못하므로, + 테스트는 형태가 아니라 픽셀과 메타데이터를 검증합니다. -**Built and verified end to end:** the AI engine, the pluggable model/backend registries, -streaming tiled inference, analysis and model selection, the face and matting pipelines, -the CLI, the HTTP API, the Next.js workspace, and an Electron desktop app that ships with -no PyTorch in it. - -**Not built:** the training pipeline and the cloud/multi-tenant layer. See -[Roadmap](#roadmap). - ---- - -## Install - -Requires Python 3.11+ and, for GPU, an NVIDIA card. +## 실행 방법 ```bash -python -m venv .venv && . .venv/Scripts/activate # Windows +python -m venv .venv && . .venv/Scripts/activate # Windows (Unix는 bin/activate) pip install -e ".[api,onnx,metrics,dev]" -# PyTorch, matched to your driver — check `nvidia-smi` first: -# driver >= 525 -> CUDA 12.x +# 드라이버에 맞춘 PyTorch — 먼저 `nvidia-smi` 확인 (>=525 -> cu126, 452~525 -> cu118) pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126 -# driver 452-525 -> CUDA 11.8 (older laptops; CUDA 11 minor-version compatibility) -pip install torch==2.7.1+cu118 torchvision==0.22.1+cu118 --index-url https://download.pytorch.org/whl/cu118 -python scripts/download_weights.py --all # ~485 MB -visionsr doctor # confirm it found your GPU +python scripts/download_weights.py --all # 체크포인트 약 485MB +visionsr doctor # GPU 인식 확인 ``` -`visionsr doctor` is the first thing to run when anything looks wrong. It reports every -backend it can see, the runtimes that are installed, and whether CUDA actually -initialised — which is not the same as whether you have an NVIDIA card. - ---- - -## Use - -### CLI +이후 원하는 인터페이스를 고르면 됩니다: ```bash -visionsr enhance photo.jpg # 4x, auto everything -visionsr enhance photo.jpg -s 2 -o out.png # 2x, explicit output -visionsr enhance ./album -r -o ./upscaled # batch, recursive -visionsr enhance scan.png --model realesr-general-x4v3 # pin a model -visionsr enhance photo.jpg --remove-bg # upscale, restore, and cut the subject out - -visionsr analyze photo.jpg # what it sees and which model it would choose -visionsr models # what is registered, and what can actually run here -visionsr doctor # what hardware it can use -``` - -### Python - -```python -from visionsr import enhance_file, EnhanceOptions - -result = enhance_file("photo.jpg", EnhanceOptions(scale=4)) - -result.image # HWC uint8 RGB numpy array -result.analysis # what the classifier and quality estimators found -result.runs # which models ran, on what backend, for how long +visionsr enhance photo.jpg --scale 4 # CLI +uvicorn backend.app.main:app --port 8000 # HTTP API (문서: /docs) +cd frontend && npm install && npm run dev # 웹 UI (:3000) +cd desktop && npm install && npm start # 데스크톱 앱 (엔진을 직접 기동) ``` -### HTTP API +주요 CLI 명령: ```bash -uvicorn backend.app.main:app --port 8000 # docs at /docs +visionsr enhance photo.jpg --remove-bg # 업스케일 + 얼굴 복원 + 배경 제거 +visionsr enhance ./album -r -o ./out # 폴더 일괄 처리 +visionsr analyze photo.jpg # 무엇으로 인식했고 어떤 모델을 고를지 +visionsr models # 등록된 모델과 실행 가능 여부 +visionsr doctor # 사용 가능한 하드웨어 ``` -| | | -|---|---| -| `GET /api/v1/health` | backends, device, resident models | -| `GET /api/v1/models` | the registry, with install status | -| `POST /api/v1/analyze` | profile an image; no enhancement | -| `POST /api/v1/jobs` | queue an enhancement → `202 {job_id}` | -| `GET /api/v1/jobs/{id}/events` | live progress over SSE | -| `GET /api/v1/jobs/{id}/result` | the enhanced image | +테스트와 E2E 실행은 영문 README의 [Tests](README.en.md#tests) 절을 참고하세요. -`POST /api/v1/jobs` takes `remove_background`, and refuses `output_format=jpeg` with it — -JPEG has no alpha channel, so the cutout would be silently flattened onto white. A request -that succeeds while throwing away what it was asked for is the worst possible outcome. +## 디렉토리 -### Desktop - -```bash -cd desktop && npm install && npm start ``` +ai/ 엔진. HTTP나 UI를 전혀 알지 못합니다. + visionsr/ + analysis/ 이 이미지는 무엇이고 얼마나 손상됐는가 — 분류기, 무참조 + 품질 추정, 얼굴 검출, 모델 선택. + backends/ 하나의 인터페이스, 여러 런타임: torch, ONNX, OpenVINO. + core/ 타입, 설정, 예외, 모델/아키텍처 레지스트리. + inference/ 엔진 루프와 타일러(오버랩, 페더링, VRAM 계산). + models/ 신경망 정의 — RRDBNet, SRVGG, GFPGAN, StyleGAN2. + pipelines/ 다중 모델 흐름: 얼굴 복원, 알파 매팅. + preprocessing/ 이미지 디코드/인코드. BGR/RGB와 EXIF를 다루는 유일한 곳. + postprocessing/ 샤프닝 등 출력 필터. + exporters/ Torch -> ONNX. 배포판이 PyTorch 없이 동작하기 위한 경로. + configs/models.yaml 모델 레지스트리. 유일한 진실 공급원이며, 코드에는 모델 + 이름이 등장하지 않습니다. + checkpoints/ 내려받은 가중치 (gitignore 대상). + +backend/app/ FastAPI 서비스: 라우트, 작업 큐, SSE 진행률, 스키마. +frontend/ Next.js 워크스페이스(정적 export)와 E2E 스크립트. +desktop/ Electron 셸: 엔진을 사이드카로 기동, 핫 폴더, 네이티브 열기/저장. +packaging/ PyTorch 없는 서버 바이너리를 만드는 PyInstaller 스펙. +scripts/ 가중치·에셋 다운로드, ONNX 익스포트, 품질 게이트, 픽스처, 인스톨러. +tests/ 두 계층: 무표식(가중치·GPU 불필요)과 @pytest.mark.weights. +.github/workflows/ CI(린트·타입·테스트·빌드)와 Security(CVE, CodeQL, 시크릿, 파일시스템). +``` + +## 사용 모델 + +레지스트리는 [`ai/configs/models.yaml`](ai/configs/models.yaml)입니다. 엔진은 모델을 +이름으로 부르지 않고 "콘텐츠 유형 Y에 대해 X를 하는 무언가"를 요청하므로, 이미 아는 +아키텍처의 체크포인트를 추가하는 데는 코드가 필요 없습니다. + +### 딥러닝 + +| 모델 | 작업 | 아키텍처 | 배율 | 특징 | 라이선스 | +|---|---|---|---|---|---| +| `realesrgan-x4plus` | 초해상도 | RRDBNet (16.7M) | 4x | 디테일 복원 최강, 가장 느림. 성능이 되는 GPU의 기본값. | BSD-3-Clause | +| `realesrgan-x2plus` | 초해상도 | RRDBNet | 2x | 네이티브 2x. 4x 결과를 축소하는 것보다 선명. | BSD-3-Clause | +| `realesr-general-x4v3` | 초해상도 | SRVGG (1.2M) | 4x | 약 8배 빠르고 VRAM 부담이 훨씬 적음. 4GB 카드의 적정 기본값. | BSD-3-Clause | +| `realesrgan-x4plus-anime` | 애니 / 일러스트 | RRDBNet 6B | 4x | 평평한 셀 셰이딩을 잡텍스처로 만들지 않고 보존. | BSD-3-Clause | +| `realesr-animevideov3` | 애니 / 일러스트 | SRVGG | 4x | 경량 애니 경로. | BSD-3-Clause | +| `gfpgan-v1.4` | 얼굴 복원 | GFPGAN + StyleGAN2 | 1x | 정렬된 512px 크롭에서 동작 후 원본에 블렌딩. 머리카락·속눈썹·홍채 복원. | Apache-2.0 | +| `isnet-general` | 배경 제거 | IS-Net (ONNX) | 1x | 알파 매팅. 자전거 바퀴살 같은 얇은 구조를 유지. | Apache-2.0 | +| `u2netp` | 배경 제거 | U²-Net lite (ONNX) | 1x | 4MB 폴백. | Apache-2.0 | +| YuNet | 얼굴 검출 | OpenCV DNN (ONNX) | — | 340KB. 얼굴 복원과 인물 분류의 관문. | Apache-2.0 | -Electron's job here is narrow, and stays that way: it starts the Python engine, serves -the same frontend the browser build uses, and adds the things a web page cannot do. -There is no second implementation of anything — the desktop and web apps run identical -UI code against an identical HTTP API. The API just happens to be a child process. - -- **Hot folder.** Point it at a directory; anything dropped in comes back enhanced - beside the original. This is what makes a desktop build worth having: a background - service with a window attached, rather than a page. It runs in the main process, so a - crashed renderer does not stop it, and it shares the backend's single-worker queue, so - it cannot starve an interactive job. -- **Native open and save.** The browser can only drop a file into Downloads. "Save this - 192 MP PNG next to the original" is the actual workflow. -- **The engine dies with the app.** An orphaned `uvicorn` holding a GPU is invisible, and - it makes the *next* launch fail for reasons that look nothing like the cause. It is - killed through the process tree, and there is a test that asserts the port is dead - after the window closes. - -The sidecar takes an ephemeral port (hard-coding 8000 means the app cannot start -alongside anything else, including a second copy of itself), which is why the frontend -resolves its API base URL at runtime rather than at build time. - -### Shipping: one .exe, and no PyTorch in it +초해상도 모델은 GAN 기반이라 PSNR은 **떨어지면서** 지각적으로는 훨씬 가까워지는 것이 +정상입니다. 자세한 내용은 영문 README의 LPIPS 설명을 참고하세요. -```bash -bash scripts/build_installer.sh # -> desktop/release/VisionSR-0.1.0-x64.exe -``` +### 고전 머신러닝 / 신호처리 -The shipped app does not contain PyTorch, and that is the central packaging decision. +전부가 신경망은 아니며, 신경망이 아닌 부분은 의도적인 선택입니다: -| | size | runs on | +| 구성 요소 | 방법 | 왜 신경망이 아닌가 | |---|---|---| -| torch + CUDA | ~2.5 GB | NVIDIA only, and only with a recent driver | -| **ONNX Runtime + DirectML** | **~250 MB** | **any DX12 GPU — NVIDIA, AMD, Intel — no CUDA at all** | - -Nobody ships a desktop image tool by freezing a 2.5 GB CUDA stack into it. So torch is a -*development* dependency — training, research, and exporting the graphs — and the -shipped runtime is ONNX. Verified on the development machine, whose NVIDIA driver is too -old for CUDA 12: the full chain (Real-ESRGAN → GFPGAN → IS-Net) runs on DirectML with no -torch present at all, and a packaged-app test asserts exactly that (`cuda` and `cpu` are -torch-only backends, so their *absence* from `/health` is the proof). - -What it costs is a build gate. An ONNX export can be silently wrong — the graph builds, -the session loads, the output is the right size and subtly degraded. So -`scripts/export_onnx.py` checks every graph against the torch model it came from, on real -input, **on every execution provider the app might use**, and fails the build if any -deviates by more than 2/255 (below the quantisation step of the 8-bit image either would -be written to, so anything under it is provably invisible). Measured: 0.001/255 for the SR -models, 0.005/255 for GFPGAN. - -"On every provider" is not belt-and-braces. Verifying on CPU alone nearly shipped a -catastrophically broken GFPGAN — see the bug list below. - -The installer bundles the graphs (652 MB) rather than the `.pth` checkpoints, which the -shipped runtime cannot use. Building it needs ~4 GB of free disk. - ---- - -## How it decides - -``` -upload → analyse → classify → select model → tiled inference - → face restore → post-process → remove background → export -``` - -**Analysis** measures what is actually wrong with the image, not what it looks like: -noise via Immerkær's estimator, JPEG blocking by comparing gradients across the 8×8 -grid against gradients inside it, blur as Laplacian variance normalised by the image's -own contrast (so a low-contrast but sharp photo is not misread as blurred). - -**Classification** is a documented heuristic ensemble — flatness, palette -concentration, axis-alignment of gradients, colourfulness — not a learned model. That -is a deliberate trade: it is inspectable (`analysis.scores` says exactly which signal -drove the decision), it needs no weights, and the boundary that matters most -(illustration vs photograph) is genuinely separable by these statistics. Getting it -wrong is the most *visible* failure this system can have — a photo model stipples fake -texture into flat cel-shaded regions — so the fallbacks are conservative and -`--model` always wins. - -**Selection** asks the registry for "a model that does X", never for a model by name. -It matches model *capacity* to how much repair the image needs: a pristine 24 MP photo -gets the 1.2 M-parameter model and a result in seconds; a ruined 0.2 MP JPEG earns the -16.7 M-parameter one. - -**Background removal** is last, and that ordering is not incidental. The segmenter -downsamples whatever it is handed to a fixed 1024 px square, so a 512 px upscaled canvas -resized to 1024 is a far better-defined input than a 128 px original resized to 1024 — -and the edge quality of a cutout is entirely a question of detail. Cutting first would -throw away the detail super resolution had just reconstructed. The alpha stays *soft*: -hair and fur are genuinely semi-transparent, and a hard threshold turns them into a -jagged silhouette. - -**Tiling** is what makes a 4 GB card work at all. Tiles overlap and are blended with a -raised-cosine feather (not cropped), so there is no seam; tile size is derived from -*live* free VRAM; on OOM the runner halves the tile and retries rather than failing. -Verified numerically: tiled output matches untiled output to within one 8-bit level. - -**Assembly streams.** The obvious way to blend overlapping tiles is a float32 -accumulator the size of the output plus a weight map — 16 bytes per output pixel. At -4x on a 12 MP photo that is a 2.9 GB allocation, and when it fails there is no -exception to catch: the process is killed. But tiles arrive in row-major order, so -once the next tile row starts, every row above it is final. Only the rows in play need -to exist in float. Memory drops from O(output pixels) to O(tile height × width), the -canvas stays `uint8`, and each tile is converted to float individually on its way into -the network — which is the only place float was ever needed. - -The result: a 12 MP photo upscales to **192 MP (16000 × 12000) in 36 s on a 4 GB laptop -GPU**. The old assembler could not have allocated the buffer at all. - ---- - -## Architecture - -``` -ai/visionsr/ - core/ types, errors, config, device probing, registries - backends/ torch (CPU/CUDA/TensorRT), ONNX Runtime, DirectML, OpenVINO - models/ RRDBNet, SRVGGNetCompact, GFPGAN — architectures only - inference/ tiled runner, engine - analysis/ quality estimators, classifier, face detection, model selector - pipelines/ face restoration - pre/postprocessing/ - exporters/ torch -> ONNX, on demand -ai/configs/models.yaml the model registry — the only place a model is named -backend/app/ FastAPI: jobs, SSE progress, models, analyze -frontend/ Next.js workspace -``` - -Two rules hold the whole thing together. - -**No model is named in engine code.** Models live in `ai/configs/models.yaml`; adding a -checkpoint of a known architecture is a YAML entry and no code at all. Adding a new -*architecture* is one file and one `@register_architecture` decorator. This is enforced -by tests that build a registry of entirely fictional models and assert the engine still -routes correctly. - -**Backends expose one contract: NHWC float32 in [0,1], in and out.** Networks disagree -about their value range — SR models want [0,1], anything with a StyleGAN decoder wants -[-1,1] — so each `ModelSpec` declares its `value_range` and the backend converts at the -boundary. Nothing above that layer knows or cares. - ---- - -## Things that were wrong, and are documented so they stay fixed - -Every one of these produced a correctly-sized image or a green test run. None raised. -They were found by putting the output on screen and by driving the real app in a real -browser. Each now has a regression test that asserts something about the *pixels* or the -*behaviour*, not the plumbing. - -- **GFPGAN's `[-1,1]` output was clamped to `[0,1]`** by the backend, deleting every - shadow and returning a flat pink smear where the face was. Fixed by making value range - a declared property of the model, not an assumption of the backend. -- **GFPGAN collapses in fp16.** Not a crash — the StyleGAN2 decoder's demodulation plus - a per-layer √2 gain exceeds half precision's dynamic range, and the output silently - degenerates to a near-constant dark field (measured: fp32 std 0.343 → fp16 std 0.123). - It is pinned to fp32 in the registry; the SR models keep fp16. -- **PyTorch's caching allocator hid free VRAM.** `mem_get_info` reports what the *driver* - has left, but torch does not return freed blocks to the driver — so after one inference - the tiler concluded it was out of memory and dropped to 64 px tiles for the rest of the - process. Correct output, several times slower. -- **The model selector could not do what its own comment claimed.** "Prefer the cheap - model on clean input" was unreachable: `priority` contributed up to 100 points and the - quality term at most 20. Rewritten around cost-in-proportion-to-work, and the policy - reversed once measured: a *small* image gets the best model, because there the expensive - one costs a second; a large one gets the fast model, because there it costs minutes. -- **`asyncio.create_task` without a strong reference.** The event loop holds only a weak - one, so a fire-and-forget job could be garbage-collected mid-flight — leaving the - client's progress bar frozen forever, and its abandoned coroutine still holding GPU - tensors. That leak starved VRAM until the tiler fell to 64 px tiles and a routine job - took **ten minutes**. Found by running the actual UI, not the tests. -- **`POST /analyze` blocked the event loop.** It is `async def`, but it decoded the image - and ran face detection inline: 100–500 ms during which every other request — including - the health check the frontend uses to decide the backend is alive — stalled behind it. - `async def` is a promise not to block; that function could not keep it. -- **Progress meant two different things.** The engine emitted a fraction *within the - current stage*, so a progress bar hit 100% four times. It is now a single monotonic - 0→100% across the whole job, with each stage weighted by how long it actually takes. -- **The ONNX export gate was measuring the wrong machine.** The shipped runtime is ONNX - Runtime on DirectML, and the export was verified on ONNX Runtime's *CPU* provider. Those - are not the same computer. In fp16, ONNX Runtime's CPU kernels quietly compute much of a - graph in fp32 — so the fp16 GFPGAN measured **1.5/255** from torch and passed. Run the - identical graph on DirectML and its StyleGAN2 decoder collapses exactly as torch's fp16 - did: output std 0.343 → 0.123, **179/255** from the truth, no error and no NaN — a dark - smear where the face was. The same CPU-only gate also hid that *every* fp16 SR graph - fails outright on DirectML, on its final `Conv`, with an HRESULT the Python binding - cannot even decode. The graphs were fine. The gate was wrong. It now verifies on every - provider that ships, and demotes a model to fp32 the moment fp16 misbehaves anywhere it - will run. - -- **A big scale killed the process.** The tiler assembled into a full-size float32 - accumulator plus weight map — 16 bytes per output pixel — so a 12 MP photo at 4x wanted - 2.9 GB of host RAM just to stitch. When that allocation fails there is nothing to catch: - the worker dies and takes every queued job with it. Fixed properly rather than papered - over: assembly now streams row-bands into a `uint8` canvas, so the float buffer is - bounded by tile height and does not grow with the image. Same output, to the byte — - the LPIPS figures above are unchanged after the refactor. - ---- - -## Tests - -```bash -pytest # 91 tests, 2 skipped (DirectML/OpenVINO not installed here) -pytest -m "not weights" # the subset needing no checkpoints and no GPU -python scripts/verify_quality.py # quality gate against ground truth - -python scripts/download_assets.py # ground-truth photos; without them ~19 tests skip -python scripts/make_e2e_fixtures.py # build the generated inputs the e2e scripts upload - -cd frontend && node e2e-smoke.mjs # drives the real UI in Chromium against the real backend -cd frontend && node e2e-orientation.mjs # a phone photo (EXIF-rotated) survives the round trip -cd desktop && npm run e2e # launches the real Electron app, runs a real job -cd desktop && npm run e2e:hotfolder # drops a file in a watched folder, waits for the result -``` - -The end-to-end tests are not decoration. Five of the bugs above were invisible to the -Python suite and only appeared when something actually drove the app: three when a -browser ran a job through a long-running server, one when a file was dropped into a -watched folder a moment too early, and one that only a browser could see at all — the -result was rotated by its own EXIF on the way to the screen, which nothing that -inspects the pixel array in memory can detect. - -The browser scripts prefer Playwright's pinned Chromium and fall back to an installed -Chrome or Edge if it will not start; `PW_CHANNEL=chrome|msedge|chromium` pins one. - ---- - -## Roadmap - -Built and verified end to end: the engine, the model and backend registries, streaming -tiled inference, analysis and selection, the face pipeline, the CLI, the HTTP API, the -Next.js workspace, and the Electron desktop shell with its hot folder. - -Next, in the order they unlock the most: - -1. **More architectures** — SwinIR, HAT, DAT, CodeFormer. Each is one file plus a YAML - entry; the engine does not change. -3. **Training pipeline** — losses, metrics, AMP, DDP, EMA, resume. -4. **Cloud** — Postgres, Redis, RabbitMQ workers, S3, multi-tenancy, billing. The job API - is already the shape a queue-backed deployment would expose, so the frontend does not - change. - ---- - -## Licence - -Apache-2.0. Model weights carry their own licences (Real-ESRGAN: BSD-3-Clause; -GFPGAN: Apache-2.0) and are downloaded, not redistributed. +| 콘텐츠 분류 | 색상 수·채도·엣지·주파수 통계에 대한 휴리스틱 앙상블 | 설명 가능 — `ImageAnalysis.scores`가 어떤 신호가 결정했는지 알려줌. 가중치도 GPU도 불필요하고 약 10ms. | +| 노이즈 추정 | Immerkær의 Laplacian-of-Laplacian 커널 | 무참조이고 해석적으로 닫힌 형태라 학습 데이터가 필요 없음. | +| 블러 추정 | Laplacian 분산 + 그래디언트 통계 | 위와 동일. | +| JPEG 아티팩트 추정 | 8x8 격자 경계의 불연속 측정 | 추론이 아니라 아티팩트를 직접 측정. | +| 모델 선택 | 위 지표에 대한 임계값 규칙 + 우선순위 | 잘못된 라우팅이 가장 눈에 띄는 실패이므로 설명 가능해야 함. | + +학습 기반 분류기는 동일한 `classify()` 시그니처 뒤로 그대로 대체할 수 있으며, 위 +점수들이 곧 그 모델이 재현하도록 학습될 목표입니다. diff --git a/README.zh.md b/README.zh.md index 717cc73..6f4a27c 100644 --- a/README.zh.md +++ b/README.zh.md @@ -1,8 +1,8 @@

VisionSR

- English · - 한국어 · + 한국어 · + English · 中文 · 日本語

@@ -31,7 +31,7 @@ visionsr enhance photo.jpg --scale 4 # 一条命令就够了 ``` > 以下为项目概览。基准测试、架构取舍、API 参考与打包等完整文档请见 -> [英文 README](README.md)。 +> [英文 README](README.en.md)。 --- @@ -87,7 +87,7 @@ visionsr models # 已注册模型及其可用性 visionsr doctor # 可用硬件 ``` -测试与端到端脚本的运行方式见英文 README 的 [Tests](README.md#tests) 一节。 +测试与端到端脚本的运行方式见英文 README 的 [Tests](README.en.md#tests) 一节。 ## 目录结构 diff --git a/pyproject.toml b/pyproject.toml index 2acb551..b5ae681 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -6,7 +6,10 @@ build-backend = "hatchling.build" name = "visionsr" version = "0.1.0" description = "Enterprise AI super resolution, restoration and enhancement engine" -readme = "README.md" +# The English file, not README.md. This becomes the package's long description on +# PyPI, whose audience is not the same as the repository's — the repository leads in +# Korean by choice, the package index should stay readable to everyone installing it. +readme = "README.en.md" requires-python = ">=3.11" license = { text = "Apache-2.0" }