Robots.txt check. Read a site's robots.txt and say whether a crawler may fetch a URL, by RFC 9309 rules as Google applies them (most specific user-agent group, longest matching Allow or Disallow, Allow on ties, * and $ wildcards). Returns the deciding rule, the group used, the rule count and the sitemaps listed. A missing robots.txt (4xx) allows everything. Use before scraping or crawling a site politely; for the sitemap URLs themselves use sitemap.
| Network | Scheme | Amount | Pay To |
|---|---|---|---|
| Base | exact | $0.01 USDC | 0x2074...77f5 |
| Polygon | exact | $0.01 USDC | 0x2074...77f5 |
| Arbitrum One | exact | $0.01 USDC | 0x2074...77f5 |
| solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp | exact | 0.01 tokens | 82wfSC...a9Q6 |