搜索「Crawl」的结果

io.github.prismcrawl/prismcrawl

Live Google, Bing, Amazon, Maps, Yelp & app store data. As low as $0.15/1K successes; failures free.

io.github.nickjlamb/pubcrawl

PubMed 与 Europe PMC 文献检索、美英药品说明书查询与临床试验检索，供 AI 助手使用。

com.ranovam/flarum-scraper

搜索或监控任意 Flarum 论坛的讨论、回复、参与者。

io.github.pipeworx-io/firecrawl

Firecrawl MCP，封装 Firecrawl API（firecrawl.dev）。

com.monocrawl/mcp

Public social media, search and e-commerce data from 70+ platforms, on one API key.

io.github.pipeworx-io/crawlbase

Crawlbase MCP — wraps the Crawlbase Crawling API (crawlbase.com, formerly

io.github.pipeworx-io/commoncrawl

Common Crawl's public web archive — find every time a URL was crawled since 2008 and read.

io.github.pipeworx-io/crawlgraph

CrawlGraph MCP — backlink intelligence on the public Common Crawl webgraph

io.github.sadri-dridi/agent-crawl-readiness

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

io.github.newmediamadman/nm-signals

Audit webpage access for AI crawlers. API key and Premium or Partner plan required.

app.rowb.ai-visibility/audit

One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.

io.github.mrfentmen/commoncrawl-mcp

Use this MCP server to Common Crawl index discovery for historical web captures.

io.github.SanggonBoy/PyreCrawl

Web scraping + research toolkit for AI agents: scrape, crawl, monitor URLs with anti-bot bypass.

com.lumnika/ai-crawler-readiness

Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index

io.github.jmshinhwa/robots-txt-ai-crawler-audit

Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

io.github.Ansarii/firecrawl-stealth-mcp

WAF-bypass web scraper and passive URL mapper for AI agents with zero pay-on-fail.

dev.workers.pathwren.www/crawler-log-triage

对访问日志中的爬虫流量做归类与诊断，免费无需密钥（原文功能描述为未填写的占位符）。

io.github.RidioDevelopment/socialcrawl

一个 API 覆盖 65 个平台、572 个端点：社交、电商、零售、招聘、金融与网页抓取。

dev.workers.pathwren.www/ai-crawler-index

查询与统计抓取本站的 AI 爬虫索引情况，免费无需密钥（原文功能描述为未填写的占位符）。

dev.workers.pathwren.www/crawler-ip-verifier

校验来访 IP 是真实的 Googlebot 等爬虫还是假冒者，免费无需密钥（原文功能描述为未填写的占位符）。

io.github.waiky-github/CrawlEyes

Web scraping & search for AI agents: extract, SearXNG search, rerank, robots compliance. No API key.

net.fastcrawl/fastcrawl

Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.

io.github.taylorsmithgg/crawl-census

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

io.github.amati032-dev/crawlbit

Four free tools to check if AI engines can find, read and recognise a site. No API key.