Skip to content

Add CPU-first inference with automatic SDPA fallback#85

Open
Lakshya717 wants to merge 1 commit into
Robbyant:mainfrom
Lakshya717:cpu-support
Open

Add CPU-first inference with automatic SDPA fallback#85
Lakshya717 wants to merge 1 commit into
Robbyant:mainfrom
Lakshya717:cpu-support

Conversation

@Lakshya717

Copy link
Copy Markdown

Summary

Makes LingBot-Map run on CPU-first setups without FlashInfer, so inference works on machines with no capable GPU or missing CUDA wheels.

Why

The repo assumed CUDA + FlashInfer by default. On this dev machine (i5-13420H, 13 GB RAM, RTX 2050 4 GB), FlashInfer was unavailable and the ~4.4 GB checkpoint cannot fit in VRAM. The goal was reliability over speed — run anywhere, GPU optional.

What changed

  • lingbot_map/utils/device.py — device/dtype resolution, autocast helper, FlashInfer probe, SDPA vs FlashInfer backend selection
  • demo.py--device (cpu default), auto SDPA when FlashInfer missing, device-aware autocast, images stay on CPU, --offload_to_cpu default on
  • aggregator/stream.py — soft FlashInfer → SDPA fallback at build time (no crash)
  • gct_base.py — device-aware autocast in prediction heads
  • Benchmark — CPU/SDPA-safe defaults in method + YAML configs
  • geometry.py — replace hard .cuda() with .to(device)
  • README — appended CPU/low-spec install and usage notes (CUDA path unchanged)
  • Profiling scripts — clearer CUDA-only errors

FlashInfer remains an optional CUDA fast path; SDPA is the portable default.

Status

Successful — CPU inference path works without FlashInfer. Verified locally: FlashInfer import fails → SDPA selected; --device auto correctly falls back to CPU on low VRAM (~3.2 GiB free).

Local results (this machine)

Follow-up (not in this PR)

Further refinement is needed for hybrid / partial offload — e.g. weights on CPU with selective GPU compute when VRAM allows, or smarter auto-switching between CPU and CUDA within a single run. Current scope is CPU-only or explicit CUDA, not layered weight offload.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant