Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com.
| Network | Scheme | Amount | Pay To |
|---|---|---|---|
| Base | exact | n/a | 0x4466...125e |
Loading activity for 1 address...
| Network | Address | USDC Balance | ETH Balance | Tx Count |
|---|---|---|---|---|
| Base | 0x4466...125e | $17.92 | 0.005488 ETH | 1 |