Crawlgraph
v0.2.1
io.github.pipeworx-io/crawlgraph
CrawlGraph MCP — backlink intelligence on the public Common Crawl webgraph (4.4B edges.
“Crawl” 共 84 个结果
v0.2.1
io.github.pipeworx-io/crawlgraph
CrawlGraph MCP — backlink intelligence on the public Common Crawl webgraph (4.4B edges.
v0.2.1
io.github.pipeworx-io/commoncrawl
Common Crawl's public web archive — find every time a URL was crawled since 2008 and read.
v0.2.9
io.github.Grubbomatic/crawl-readiness
Check what AI crawlers may read and what a page permits, generate llms.txt and schema.
v0.3.1
io.github.crawlerpeek/crawlerpeek
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.
v0.33.1
io.github.DasDigitaleMomentum/searxncrawl
Web search via SearXNG and/or You.com, plus clean Markdown crawling via Crawl4AI. Self-hosted.
v1.1.1
com.soflution/deepcrawl
Clone websites, scrape pages to markdown and crawl whole sites. A free Firecrawl alternative.
v0.17.0
io.github.Mingye-Lu/acrawl
自主网页爬虫:17 个浏览器工具,外加目标驱动的 run_goal 智能体;单二进制、隐身模式。
v1.12.3
io.github.xberg-io/crawlberg
通过本地 CLI 抓取、爬取并映射网站,输出 Markdown 或 JSON。
v1.1.0
io.github.Soflutionltd/deepcrawl
Clone websites, scrape pages to markdown and crawl whole sites. A free Firecrawl alternative.
v1.0.1
io.github.alphaparkinc/genpark-bloom-filter-url-dedup-crawler-skill
In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge
v0.1.4
io.github.missiongrowth/unrendered-ai-crawler-checker
AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
v0.1.2
com.crawlwisedata/crawlwise-mcp
Per-URL tech-stack, feed and sitemap change events for AI agents, delivered as signed webhooks.
v1.0.0
io.github.puneetindersingh/open-seo-crawler
Self-hosted SEO crawler: run site audits and AI crawler checks, then read the issues and pages.
v1.0.0
io.github.IliasSami/scrawly
MCP server for the Scrawly SEO crawler: query local audits and propose confirmation-gated fixes.
v0.1.3
io.github.imfurkana/unrendered-ai-crawler-checker
AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
v1.0.1
io.github.mrfentmen/commoncrawl-mcp
Use this MCP server to Common Crawl index discovery for historical web captures.
v0.8.1
io.github.SanggonBoy/PyreCrawl
Web scraping + research toolkit for AI agents: scrape, crawl, monitor URLs with anti-bot bypass.
v0.1.3
io.github.jmshinhwa/robots-txt-ai-crawler-audit
Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI
v0.2.0
io.github.waiky-github/CrawlEyes
Web scraping & search for AI agents: extract, SearXNG search, rerank, robots compliance. No API key.
v1.0.1
io.github.amati032-dev/crawlbit
Four free tools to check if AI engines can find, read and recognise a site. No API key.
v1.11.0
app.crawlio/crawlio-browser
把实时 Chrome 浏览器接入你的 Agent:150 个抓取、抽取与观测工具。
v0.2.2
io.github.pucilpet/crawlgraph-mcp
基于 Common Crawl 开放网页图谱(44 亿条链接)提供外链查询与竞品差距分析。
v0.5.8
io.github.manchittlab/thecrawler
通用网页抓取器,输出适配 LLM 的 Markdown,支持 RAG 分块与 PDF/DOCX 解析。
v0.3.0
io.github.kreuzberg-dev/kreuzcrawl
通过本地 CLI 抓取、爬取并映射网站,输出 Markdown 或 JSON。