OpenCrawl
v0.1.0
io.github.getopencrawl/opencrawl
Search shared agent history and workspace sources, read evidence, and coordinate on an agent board.
“Crawl” 共 116 个结果
v0.1.0
io.github.getopencrawl/opencrawl
Search shared agent history and workspace sources, read evidence, and coordinate on an agent board.
v1.4.8
io.github.brainfish-ai/webcrawl
Scrape, map, crawl, search and monitor the web. Runs locally with no API key, cites every source.
v1.2.2
io.github.emmanuelorta/crawlcheck
Verification layer for the agentic web: a signed answer about any site before an agent acts.
v1.0.0
app.pandacrawl/pandacrawl
SEO and AI visibility: site audits, Search Console, Analytics, Bing, rank and ChatGPT tracking.
v1.0.3
io.nstdata/crawl
Scrape webpages, run crawls, and read Markdown and other results with Nstdata Crawl.
v0.1.1
io.github.stellalytics/seo-site-audit
Technical SEO audit crawler: broken links, 404s, redirects, titles, canonicals. Runs on Apify.
v1.1.0
io.github.hamzawamiqhq/ai-crawler-check
Can ChatGPT, Claude and Perplexity crawlers reach a site? Real per-bot fetch, firewall/CDN blocks.
v1.0.0
io.github.bepolabs/web-to-markdown
Turn a website into Markdown with headings, links and word counts.
v3.40.0
io.github.firecrawl/firecrawl-mcp-server
Firecrawl 的 MCP server——网页搜索、网页抓取,以及生物医学/arXiv 论文检索。
v0.8.13
io.github.77777R7/octocrawl
Pages, crawls and URL batches as clean Markdown and JSON for AI agents, with an Evidence Record.
v1.0.0
com.profullstack/crawlproof
AI-crawler audits + ads (crawlproof.com)
v0.2.0
io.github.OfficialSpicrawl/mcp
Web scraping MCP server: any URL to Markdown or JSON, with JavaScript rendering and batch jobs.
v1.17.20
net.crawlora/crawlora-mcp
托管 MCP:1873 个结构化网页数据工具,覆盖搜索、地图、电商、社交、游戏与金融。
v2.2.0
io.github.RidioDevelopment/socialcrawl
一个 API 覆盖 65 个平台、572 个端点:社交、电商、零售、招聘、金融与网页抓取。
v1.0.0
io.github.ByteBlisk/sitereader
Website to clean Markdown or JSON for AI and RAG: read pages, crawl sites, extract described fields.
v1.0.0
io.github.prometheus-maker/website-to-markdown
Crawl a website into clean Markdown for LLMs/RAG: main content only, linked docs converted.
v1.1.1
site.crawlwise/crawlwise
SEO audits and AI crawler checks. Honest findings only; no invented scores.
v1.0.2
io.github.CrawlKit560/crawlkit-scrapers
58 scrapers via Apify MCP: Booking, Airbnb, Amazon, Google, Zillow, Indeed, Yelp, Reddit, Vinted
v1.0.0
io.github.AdsCrawl/ads-crawl
Cloud browser, web extraction, screenshots, and remote CDP tools for AI agents.
v1.1.0
com.atalaya-ai/ai-crawler-check
Which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.
v1.0.2
io.github.kriton-dev/ai-crawler-access-check
Check declared robots.txt permissions for six AI crawlers at URL paths
v1.0.1
io.github.outcrawlai/outcrawl
Web infrastructure for AI agents: scrape, crawl and search the web, run browsers and agents.
v1.0.0
co.civai.nova/mcp-firecrawl
Scrape, crawl, and search the web via Firecrawl’s official hosted MCP.
v1.0.4
io.github.kriton-dev/website-to-markdown
Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay