Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com. Accepts payment on Base or Solana — either network works.
| Network | Scheme | Amount | Pay To |
|---|---|---|---|
| Base | exact | $0.005000 USDC | 0x4466...125e |
| solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp | exact | 0.01 tokens | 9urRvU...o1up |