搜索「Crawl」的结果

Firecrawl

io.mcp-cn.firecrawl-mcp

基于Firecrawl的智能网页爬取和内容提取MCP工具

io.github.imfurkana/unrendered-ai-crawler-checker

AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.

io.github.mysleekdesigns/crawlforge-mcp-server

网页抓取、遍历、深度调研与自主信息抽取，31 个 MCP 工具，输出干净的 Markdown/JSON。

io.github.xberg-io/crawlberg

通过本地 CLI 抓取、爬取并映射网站，输出 Markdown 或 JSON。

com.saastemly/crawl4agent

crawl4ai-compatible web crawler API: POST a URL.

io.github.tidytools/tidytools-apify-tools

Enrich Google Maps leads, crawl docs to RAG Markdown, transcribe podcasts, check AI crawler access.

io.github.firecrawl/firecrawl-mcp-server

Firecrawl 的 MCP server——网页搜索、网页抓取，以及生物医学/arXiv 论文检索。

site.crawlwise/crawlwise

SEO audits and AI crawler checks. Honest findings only; no invented scores.

io.github.Mingye-Lu/acrawl

自主网页爬虫：17 个浏览器工具，外加目标驱动的 run_goal 智能体；单二进制、隐身模式。

io.github.Grubbomatic/crawl-readiness

Audit any site for 50+ AI crawlers, generate llms.txt, robots.txt and schema, track AI mentions.

io.github.schemacrawler/schemacrawler-ai

支持用自然语言查询数据库模式——快速浏览表、键、存储过程并获取 SQL 帮助。

dev.oassis/scraper-crawler

Scrape to markdown, map and crawl sites, drive a real browser. Pay per call, no API key, no signup.

net.crawlora/crawlora-mcp

托管 MCP：1873 个结构化网页数据工具，覆盖搜索、地图、电商、社交、游戏与金融。

io.github.prismcrawl/prismcrawl

Live Google, Bing, Amazon, Maps, Yelp & app store data. As low as $0.15/1K successes; failures free.

io.github.nickjlamb/pubcrawl

PubMed 与 Europe PMC 文献检索、美英药品说明书查询与临床试验检索，供 AI 助手使用。

com.ranovam/flarum-scraper

搜索或监控任意 Flarum 论坛的讨论、回复、参与者。

io.github.pipeworx-io/firecrawl

Firecrawl MCP，封装 Firecrawl API（firecrawl.dev）。

com.monocrawl/mcp

Public social media, search and e-commerce data from 70+ platforms, on one API key.

io.github.pipeworx-io/crawlbase

Crawlbase MCP — wraps the Crawlbase Crawling API (crawlbase.com, formerly

io.github.pipeworx-io/commoncrawl

Common Crawl's public web archive — find every time a URL was crawled since 2008 and read.

io.github.pipeworx-io/crawlgraph

CrawlGraph MCP — backlink intelligence on the public Common Crawl webgraph

io.github.sadri-dridi/agent-crawl-readiness

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

io.github.newmediamadman/nm-signals

Audit webpage access for AI crawlers. API key and Premium or Partner plan required.

app.rowb.ai-visibility/audit

One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.