Fetches a domain's robots.txt and parses its Content-Signal and AI-preference directives (search, ai-input, ai-train) into structured JSON plus a plain-English summary of what's allowed. The pre-crawl check for any agent scraping content or building a RAG index, ahead of the industry's move toward stricter bot-gating defaults later this year.

Payment Options