Mycelium

An experimental parallelized web crawler written in Golang.

Name		Name	Last commit message	Last commit date
Latest commit History 44 Commits
main		main
test_site		test_site
.gitignore		.gitignore
.travis.yml		.travis.yml
DEPENDENCIES		DEPENDENCIES
LICENSE		LICENSE
README.md		README.md
data_store.go		data_store.go
infinite_crawl.go		infinite_crawl.go
page.go		page.go
rank_url.go		rank_url.go
redis_data_store.go		redis_data_store.go
redis_task_queue.go		redis_task_queue.go
robots.go		robots.go
staged_crawl.go		staged_crawl.go
staged_crawl_test.go		staged_crawl_test.go
task_queue.go		task_queue.go
worker.go		worker.go
worker_test.go		worker_test.go

Provide feedback