Robots.txt Checker

Check the publicly available robots.txt resource for a website.