robots.txt fetched and parsed: every user-agent group with its allow/disallow rules and crawl-delay, declared sitemaps, and whether the file exists at all. Understand a site's crawl policy before fetching it. $0.01 per site.
| Network | Scheme | Amount | Pay To |
|---|---|---|---|
| Base | exact | $0.01 USDC | 0x91D5...0abd |
| Base Sepolia | exact | $0.01 USDC | 0x91D5...0abd |