robots.txt and AI crawler policy check. Tells an agent whether a website permits crawling and whether it blocks AI bots such as GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot, before scraping, RAG ingestion, training data collection or archiving. Parses robots.txt and returns a crawl-policy tier with per-bot allow and disallow rules and sitemap URLs.
| Network | Scheme | Amount | Pay To |
|---|---|---|---|
| Base | exact | $0.002000 USDC | 0xC1FE...e874 |
Loading activity for 1 address...
| Network | Address | USDC Balance | ETH Balance | Tx Count |
|---|---|---|---|---|
| Base | 0xC1FE...e874 | n/a | 0.000000 ETH | 1 |