Validate robots.txt rules (User-agent, Allow root, Disallow admin paths, HTTPS Sitemap XML) for search and AI agent crawlers.