A site's robots.txt rules per user agent, whether AI crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot) are blocked, and the URLs listed in its sitemap. Use before crawling a site or for SEO audits.
| Network | Scheme | Amount | Pay To |
|---|---|---|---|
| Base | exact | $0.003000 USDC | 0x1129...1871 |
| Polygon | exact | $0.003000 USDC | 0x1129...1871 |
| Arbitrum One | exact | $0.003000 USDC | 0x1129...1871 |