sitevac

A fast, concurrent website crawler that downloads complete websites for offline browsing.

What It Does

sitevac recursively crawls websites, downloads HTML pages and all assets (images, CSS, JS, fonts, media), rewrites links to work offline, and saves the result in a directory you can open directly in a browser. It seeds discovery from sitemap.xml for faster coverage and reaches pages with no inbound links.

Quick Start

# Build
go build -o sitevac .

# Crawl a website
./sitevac https://example.com

# Serve the saved mirror locally (avoids file:// CORS issues)
./sitevac serve

Output is saved to ./site/ by default, mirroring the site's path structure.

Commands

Crawl

sitevac [flags] <url>

Flag	Default	Description
-o	site	Output directory
-c	32	Number of concurrent workers
-depth	0	Maximum crawl depth (0 = unlimited)
-delay	0	Polite delay between requests per worker (e.g. 500ms)
-retries	1	Retries on network errors and 5xx responses
-exclude	—	Skip URLs matching regexp (repeatable)
-skip-existing	false	Resume an interrupted crawl
-ignore-robots	false	Ignore robots.txt rules
-ua	sitevac/VERSION	Custom user agent string
-v	false	Verbose output (per-URL status codes, robots/exclude hits)
-version	—	Print version and exit

Serve

Serve a saved mirror over HTTP so links and fonts work correctly (avoids file:// CORS restrictions).

sitevac serve [flags] [dir]

Flag	Default	Description
-port	8080	Port to listen on
-addr	localhost	Address to bind to

dir defaults to ./site. Use -addr 0.0.0.0 to expose on all interfaces.

Examples

# Crawl with more workers and a custom output directory
sitevac -c 16 -o ./mirror https://example.com

# Limit depth and add a polite delay
sitevac -depth 3 -delay 200ms https://example.com

# Exclude certain paths
sitevac -exclude '/tag/' -exclude '/page/[0-9]+' https://example.com

# Resume an interrupted crawl
sitevac -skip-existing -o ./mirror https://example.com

# Ignore robots.txt (use responsibly)
sitevac -ignore-robots https://example.com

# Serve the result locally
sitevac serve ./mirror

# Serve on a custom port
sitevac serve -port 3000 ./mirror

Features

• Fast — 32 concurrent workers by default with connection pooling and HTTP/2
• Complete — extracts assets from HTML tags, inline style attributes, <style> blocks, and CSS url() references
• Smart discovery — seeds from sitemap.xml (including sitemap indexes) before link-following, reaching orphaned pages
• Offline-ready — rewrites all internal links to relative paths so the saved copy works with file:// or sitevac serve
• SPA support — detects React/Vue/Next.js/Nuxt/Gatsby sites and re-renders with headless Chrome when built with -tags chromedp
• Resumable — skip already-downloaded files with -skip-existing
• Filterable — exclude paths with repeatable -exclude regexp flags
• Polite — respects robots.txt, configurable per-worker delay, retries with exponential backoff
• Progress display — live counters (pages, assets, active workers, MB/s) printed to stderr
• Local server — sitevac serve serves your mirror over HTTP, no separate tool needed

How It Works

1. Fetches and parses robots.txt
2. Fetches and parses sitemap.xml (and sitemap indexes) to seed the queue
3. Launches 32 concurrent workers sharing a connection pool
4. Workers fetch URLs, extract all links and assets, rewrite for offline use
5. CSS files are fully parsed for url() references (fonts, background images, etc.)
6. Saves HTML and assets mirroring the original site structure
7. Deduplicates by URL path — query-string variants of the same path are crawled once
8. Reports a statistics summary on completion

Building from Source

# Standard build (pure Go, no external dependencies)
go build -o sitevac .
make build

# Build with headless Chrome / JS rendering support
# Requires Chrome or Chromium installed at runtime
go get github.com/chromedp/chromedp
make build-chromedp

# Cross-compile for Linux, macOS, and Windows
make build-all

# Build with version embedded
go build -ldflags "-X main.version=v1.1.0" -o sitevac .

Pre-built Binaries

Download the latest release for your platform from the Releases page. Checksums are provided for verification.

Requirements

• Go 1.21 or higher
• Chrome or Chromium (only required for -tags chromedp builds)

License

See LICENSE file.

Author

Jacob David Alcock