Robots.txt & AI Crawler Checker
v1.0.0
dev.workers.quietworklabs.robots-check/robots-check
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
“Crawl” 共 130 个结果
v1.0.0
dev.workers.quietworklabs.robots-check/robots-check
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
v0.4.2
io.github.clearcrawl/clearcrawl
Any URL to clean markdown/JSON; PDF, XLSX, CSV, DOCX to text and tables. Free tier for agents.
v1.0.0
xyz.crawlhive/crawlhive
Memecoin safety scanner for Solana, BNB and Robinhood Chain with an onchain track record
v1.0.0
io.github.realtycrawl/realtycrawl
Search source-linked real estate observations with capability discovery and bounded credits.
v1.0.0
com.crawlerdata/google-shopping-serp
Google Shopping products and seller prices, plus Google Search, Images, News and Local results.
v1.0.0
com.jobcrawls/connector
Search jobs in Finland and calculate net salary. Read-only, no login. English, Finnish, Swedish.
v1.0.1791044090
com.saastemly/crawl4agent
crawl4ai-compatible web crawler API: POST a URL.
v1.0.3
io.github.prismcrawl/prismcrawl
Live Google, Bing, Amazon, Maps, Yelp & app store data. As low as $0.15/1K successes; failures free.
v0.1.1
com.ranovam/flarum-scraper
搜索或监控任意 Flarum 论坛的讨论、回复、参与者。
v1.0.0
com.monocrawl/mcp
Public social media, search and e-commerce data from 70+ platforms, on one API key.
v1.0.0
io.github.sadri-dridi/agent-crawl-readiness
Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.
v0.1.0
io.github.newmediamadman/nm-signals
Audit webpage access for AI crawlers. API key and Premium or Partner plan required.
v1.0.0
app.rowb.ai-visibility/audit
One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.
v1.0.0
com.lumnika/ai-crawler-readiness
Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index
v1.0.0
io.github.Ansarii/firecrawl-stealth-mcp
WAF-bypass web scraper and passive URL mapper for AI agents with zero pay-on-fail.
v1.7.0
dev.workers.pathwren.www/crawler-log-triage
对访问日志中的爬虫流量做归类与诊断,免费无需密钥(原文功能描述为未填写的占位符)。
v1.6.0
dev.workers.pathwren.www/ai-crawler-index
查询与统计抓取本站的 AI 爬虫索引情况,免费无需密钥(原文功能描述为未填写的占位符)。
v1.4.0
dev.workers.pathwren.www/crawler-ip-verifier
校验来访 IP 是真实的 Googlebot 等爬虫还是假冒者,免费无需密钥(原文功能描述为未填写的占位符)。
v0.5.0
net.fastcrawl/fastcrawl
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
v1.0.0
io.github.taylorsmithgg/crawl-census
Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?
v4.1.2
com.crimsoncrawler/mcp-server
通过 80 余个类型化 MCP 工具对威胁情报进行富化、检索、评估与管理。
v1.0.0
com.crawlraven/mcp
读取你站点的 Search Console 表现、关键词机会与批注。
v1.4.2
com.actablesite/ai-crawler-robots-monitor
检查 AI 爬虫的 robots.txt 策略,并监控公开站点的策略、sitemap 与 llms.txt 变化。
v1.4.1
com.actablesite/readiness
检查 AI 爬虫的 robots.txt 策略,并监控公开站点的策略、sitemap 与 llms.txt 变化。