The ToS;DR Crawler is important to the functionally of Phoenix.
By crawling a service we ensure that the documents are mirrored and cannot be altered until a further crawl (Verified using CRC)
We do not index websites on our own, all websites are crawled manually by curators or staff on our site.
Identifying the ToS;DR Crawler
All ToS;DR Crawlers send a respective user agent with all their requests
Check for the following user agent:
If you want to forbid the crawling for some reason you can include the following directive into the robots.txt
User-Agent: TosDRCrawler Disallow: YOUR_PATH
184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 126.96.36.199
If you are the provider of the website, common crawling issues are
- IPTables based restriction (See Crawler Clusters)
- User-Agent based blocking
To fix this, add our servers or user agents to their respective whitelist.