Read a site's robots.txt: send a domain or URL and get allow/disallow rules per user agent and the sitemap URLs. Use it to check whether a crawl is permitted before fetching pages. $0.003 USDC per call on Base.
| Network | Scheme | Amount | Pay To |
|---|---|---|---|
| Base | exact | $0.003000 USDC | 0x6c10...dfff |