Parse and validate robots.txt files for crawler rules, disallow directives, crawl delays and sitemap declarations.