Commoncrawl
v0.2.1
io.github.pipeworx-io/commoncrawl
Common Crawl's public web archive — find every time a URL was crawled since 2008 and read.
“Crawl” 共 236 个结果
v0.2.1
io.github.pipeworx-io/commoncrawl
Common Crawl's public web archive — find every time a URL was crawled since 2008 and read.
v0.2.9
io.github.Grubbomatic/crawl-readiness
Check what AI crawlers may read and what a page permits, generate llms.txt and schema.
v0.3.1
io.github.crawlerpeek/crawlerpeek
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.
v0.33.1
io.github.DasDigitaleMomentum/searxncrawl
Web search via SearXNG and/or You.com, plus clean Markdown crawling via Crawl4AI. Self-hosted.
v1.1.1
com.soflution/deepcrawl
Clone websites, scrape pages to markdown and crawl whole sites. A free Firecrawl alternative.
v0.17.0
io.github.Mingye-Lu/acrawl
自主网页爬虫:17 个浏览器工具,外加目标驱动的 run_goal 智能体;单二进制、隐身模式。
v1.12.3
io.github.xberg-io/crawlberg
通过本地 CLI 抓取、爬取并映射网站,输出 Markdown 或 JSON。
v0.1.1
io.github.CYAN-ECLIPSE-TECHNOLOGIES-LLC/eclipse-crawlway-mcp
Pay-per-crawl licensing receipts: sealed identity+terms+fingerprint binding. LOCAL/fixture only.
v1.1.0
io.github.Soflutionltd/deepcrawl
Clone websites, scrape pages to markdown and crawl whole sites. A free Firecrawl alternative.
v1.0.0
com.crawlerzoo/arena
Games for AI agents: negotiation, social deduction, racing and puzzles, vs bots and agents.
v1.0.2
dev.oassis/scraper-crawler
Web Tools and Web Tasks for AI agents. 100-credit MCP trial; API-key credits or x402.
v1.8.0
app.openkrill/site-check
Check public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.
v1.0.1
io.github.alphaparkinc/genpark-bloom-filter-url-dedup-crawler-skill
In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge
v0.1.4
io.github.missiongrowth/unrendered-ai-crawler-checker
AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
v1.0.0
com.insightfulpipe/enrichment
Hosted Enrichment Crawlers MCP server, with each action switched on or off per account.
v1.0.0
com.insightfulpipe/crawler
Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
v1.0.0
dev.workers.quietworklabs.robots-check/robots-check
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
v0.4.2
io.github.clearcrawl/clearcrawl
Any URL to clean markdown/JSON; PDF, XLSX, CSV, DOCX to text and tables. Free tier for agents.
v0.1.2
com.crawlwisedata/crawlwise-mcp
Per-URL tech-stack, feed and sitemap change events for AI agents, delivered as signed webhooks.
v1.0.0
io.github.puneetindersingh/open-seo-crawler
Self-hosted SEO crawler: run site audits and AI crawler checks, then read the issues and pages.
v1.0.0
xyz.crawlhive/crawlhive
Memecoin safety scanner for Solana, BNB and Robinhood Chain with an onchain track record
v1.0.0
io.github.realtycrawl/realtycrawl
Search source-linked real estate observations with capability discovery and bounded credits.
v1.0.0
com.crawlerdata/google-shopping-serp
Google Shopping products and seller prices, plus Google Search, Images, News and Local results.
v1.0.0
io.github.IliasSami/scrawly
MCP server for the Scrawly SEO crawler: query local audits and propose confirmation-gated fixes.