Check robots.txt before crawling or scraping a page. Decide whether GPTBot, ClaudeBot, Googlebot, or another crawler may fetch the exact URL and return ALLOW/DENY, winning rule, crawl-delay, sitemap URLs, and live provenance.
| Network | Scheme | Amount | Pay To |
|---|---|---|---|
| Base | exact | $0.001000 USDC | 0x5dea...ef0a |