Checks whether a URL's path may be crawled under robots.txt (RFC 9309): the verdict for a user agent (default: *, Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot), the rule that decided it and its line, crawl-delay, sitemaps and the matching group's lines. Longest rule wins, allow wins a tie, * and $ work. No robots.txt means everything is allowed; a server error is a 503, not charged.

Payment Options