Website Crawler

A powerful Node.js tool for downloading complete websites with all their assets (images, CSS, JavaScript, fonts, etc.). Supports authentication, custom configurations, and handles both static and dynamic websites with comprehensive resource extraction.

✨ Features

• Complete Website Downloads: Download entire websites with all assets
• Modern Resource Support: SVG, WebP images, responsive images (srcset), CSS fonts, and more
• Dynamic Content: Support for SPAs and dynamic websites using Puppeteer
• Authentication: Username/password login with customizable selectors
• Robust Resource Extraction: Handles complex CSS/JS embedded resources with relative paths
• Protocol-Relative URLs: Automatic handling of //example.com URLs
• Duplicate Detection: Smart handling of resources referenced in multiple locations
• Professional CLI: Comprehensive command-line interface with validation
• Proxy Support: HTTP/HTTPS/SOCKS proxy support with authentication
• Progress Tracking: Real-time download progress with statistics
• Error Handling: Retry mechanisms with exponential backoff
• Configurable: Extensive customization options

🚀 Installation

# Clone the repository
git clone <your-repo-url>
cd crawler

# Install dependencies
npm install

# Run tests to verify installation
npm test

📖 Usage

Basic Website Download

# Download a static website
npm start download --url https://example.com --output ./downloads/example

# Download a dynamic website (SPA) using Puppeteer
npm start download --url https://app.example.com --output ./downloads/app --live

# Preview what would be downloaded (dry run)
npm start download --url https://example.com --output ./downloads/example --dry-run

Advanced Usage

# Download with authentication
npm start download \
  --url https://members.example.com \
  --output ./downloads/members \
  --username "user@example.com" \
  --password "password123" \
  --live

# Download with custom settings
npm start download \
  --url https://example.com \
  --output ./downloads/example \
  --max-concurrent 10 \
  --timeout 60000 \
  --include-external \
  --user-agent "Custom Bot 1.0"

# Download with proxy
npm start download \
  --url https://example.com \
  --output ./downloads/example \
  --proxy-host proxy.example.com \
  --proxy-port 8080 \
  --proxy-username proxyuser \
  --proxy-password proxypass

CSS File Download

# Download a CSS file and all its dependencies
npm start css --css-url https://example.com/styles.css --output ./downloads/css

# With custom base URL for resolving relative paths
npm start css \
  --css-url https://example.com/assets/styles.css \
  --output ./downloads/css \
  --base-url https://example.com/assets

Configuration Validation

# Validate configuration without downloading
npm start validate --url https://example.com --output ./test

🛠️ CLI Options

Global Options

• -v, --verbose - Enable verbose logging
• -q, --quiet - Suppress non-error output
• --no-color - Disable colored output
• -V, --version - Show version number
• -h, --help - Show help

Download Command Options

Required

• -u, --url <url> - Website URL to download
• -o, --output <directory> - Output directory for downloaded files

Basic Options

• -c, --max-concurrent <number> - Maximum concurrent downloads (1-20, default: 5)
• -t, --timeout <ms> - Request timeout in milliseconds (default: 30000)
• --user-agent <string> - Custom user agent string
• --dry-run - Show what would be downloaded without actually downloading
• --live - Use Puppeteer for dynamic/SPA sites

Resource Options

• --include-external - Include external resources (default: false)
• --exclude <patterns...> - Exclude files matching these patterns
• --include <patterns...> - Only include files matching these patterns

Network Options

• --follow-redirects - Follow HTTP redirects (default: true)
• --max-redirects <number> - Maximum redirects to follow (default: 5)
• --delay <ms> - Delay between requests (default: 0)
• --retry-attempts <number> - Retry attempts for failed downloads (default: 3)
• --retry-delay <ms> - Delay between retry attempts (default: 1000)

Authentication Options

• --username <username> - Username for authentication
• --password <password> - Password for authentication
• --username-selector <selector> - CSS selector for username input
• --password-selector <selector> - CSS selector for password input
• --submit-selector <selector> - CSS selector for submit button
• --wait-for-selector <selector> - CSS selector to wait for after login
• --login-url <url> - Specific login URL (if different from main URL)

Proxy Options

• --proxy-host <host> - Proxy server hostname
• --proxy-port <port> - Proxy server port
• --proxy-protocol <protocol> - Proxy protocol (http|https|socks4|socks5)
• --proxy-username <username> - Proxy authentication username
• --proxy-password <password> - Proxy authentication password

Advanced Options

• --custom-headers <headers...> - Custom HTTP headers (format: "Header: Value")
• --cookies <cookies> - Custom cookies string

CSS Command Options

Required

• -u, --css-url <url> - CSS file URL to download
• -o, --output <directory> - Output directory for CSS and assets

Optional

• -b, --base-url <url> - Base URL for resolving relative paths
• -c, --max-concurrent <number> - Maximum concurrent downloads (default: 5)
• -t, --timeout <ms> - Request timeout (default: 30000)

💻 Programmatic Usage

import { download, downloadCssFile } from './src/index.js';

// Download website
const result = await download({
  url: 'https://example.com',
  output: './downloads/example',
  maxConcurrent: 5,
  timeout: 30000,
  live: false
});

// Download CSS file
const cssResult = await downloadCssFile({
  cssUrl: 'https://example.com/styles.css',
  output: './downloads/css',
  baseUrl: 'https://example.com'
});

🏗️ Project Structure

src/
├── cli.js              # CLI entry point
├── index.js            # Main API exports
├── services/           # Core services
│   ├── Crawler.js      # Website crawling and resource extraction
│   ├── Downloader.js   # File downloading with retry logic
│   └── HtmlFetcher.js  # HTML fetching (axios + Puppeteer)
├── utils/              # Utility functions
│   ├── logger.js       # Winston-based logging
│   ├── ProgressTracker.js # Download progress tracking
│   └── fileUtils.js    # File system utilities
├── validation/         # Input validation
│   └── schemas.js      # Joi validation schemas
├── errors/             # Custom error classes
│   └── index.js        # Error definitions
└── __tests__/          # Test suites
    ├── Crawler.test.js
    ├── Downloader.test.js
    ├── HtmlFetcher.test.js
    └── validation.test.js

🧪 Testing

# Run all tests
npm test

# Run tests in watch mode
npm run test:watch

# Run tests with coverage
npm run test:coverage

# Lint code
npm run lint

# Format code
npm run format

🔧 Development

# Install dependencies
npm install

# Run in development mode with debugging
npm run dev

# Validate code quality
npm run validate

# Prepare for commit (runs linting, tests, formatting)
npm run prepare

📝 Examples

Download a Blog

npm start download \
  --url https://blog.example.com \
  --output ./downloads/blog \
  --include-external \
  --max-concurrent 8

Download Protected Content

npm start download \
  --url https://members.example.com/dashboard \
  --output ./downloads/members \
  --username "user@example.com" \
  --password "secure-password" \
  --live \
  --wait-for-selector ".dashboard-content"

Download SPA with Custom Headers

npm start download \
  --url https://app.example.com \
  --output ./downloads/app \
  --live \
  --custom-headers "Authorization: Bearer token123" "X-API-Key: key456"

🚨 Requirements

• Node.js: >= 16.0.0
• npm: >= 8.0.0

📄 License

ISC License

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests: npm test
5. Submit a pull request

🐛 Troubleshooting

Common Issues

Dynamic content not loading:

• Use the --live flag to enable Puppeteer for SPAs and dynamic websites

Authentication not working:

• Verify selectors with browser dev tools
• Use --wait-for-selector to wait for content after login
• Check if login requires a specific --login-url

Downloads failing:

• Increase --timeout for slow connections
• Adjust --retry-attempts and --retry-delay
• Use --verbose for detailed logging

External resources missing:

• Add --include-external flag to download external assets
• Check if resources are blocked by CORS or authentication