fix(robots): fetch robots.txt through the spider's configured session - #107

fix(robots): fetch robots.txt through the spider's configured session#107

claude[bot] merged 1 commit into

masterLiohtml/RUSTScrapling:masterfrom

claude/robots-via-session-1sub94Liohtml/RUSTScrapling:claude/robots-via-session-1sub94Copy head branch name to clipboard