GitHub - commandrine/autobots: Autobots is a small Python utility that retrieves and prints the contents of a website’s robots.txt file, making it easy to quickly inspect crawling rules and disallowed paths for a given site.

Run Python script from Terminal.

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
README.md		README.md
autobots.py		autobots.py

Navigation Menu