Is a URL crawlable? Fetches the site's robots.txt and evaluates the given path for a specific user-agent (Googlebot, GPTBot, ClaudeBot, Bingbot or any token; default *), returning allowed/disallowed, the matching rule, the group used, crawl-delay and declared sitemaps. Pre-flight for crawlers and SEO checks. $0.01 per check.
| Network | Scheme | Amount | Pay To |
|---|---|---|---|
| Base | exact | $0.01 USDC | 0x91D5...0abd |
| Base Sepolia | exact | $0.01 USDC | 0x91D5...0abd |