Atalaya AI crawler check
v1.1.0
com.atalaya-ai/ai-crawler-check
Which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.
“Crawl” 共 202 个结果
v1.1.0
com.atalaya-ai/ai-crawler-check
Which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.
v1.0.2
io.github.kriton-dev/ai-crawler-access-check
Check declared robots.txt permissions for six AI crawlers at URL paths
v2.8.0
io.github.nickjlamb/pubcrawl
PubMed 与 Europe PMC 文献检索、美英药品说明书查询与临床试验检索,供 AI 助手使用。
v1.0.1
io.github.outcrawlai/outcrawl
Web infrastructure for AI agents: scrape, crawl and search the web, run browsers and agents.
v1.0.0
co.civai.nova/mcp-firecrawl
Scrape, crawl, and search the web via Firecrawl’s official hosted MCP.
v1.0.4
io.github.kriton-dev/website-to-markdown
Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay
v1.0.11
io.github.kriton-dev/broken-links
Find broken links on a website: crawls its pages and reports 404s, errors and redirect chains
v1.0.52
com.crawlwitness/oracle
AI consent & licensing check for any domain, with signed on-chain receipts. Paid via x402.
v0.1.0
io.github.Nero-Engine/ai-crawler-access-checker
Shows if GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can read each site in a list.
v0.1.3
com.webcrawlerapi/mcp-server
Turn any web page into clean Markdown for your AI assistant. OAuth sign-in, no API key.
v0.1.0
io.github.CloudCorpRecords/crawlport
Firecrawl web data: scrape, map, search, crawl, extract, research, and llms.txt.
v0.2.1
io.github.pipeworx-io/crawlgraph
CrawlGraph MCP — backlink intelligence on the public Common Crawl webgraph (4.4B edges.
v0.2.1
io.github.pipeworx-io/commoncrawl
Common Crawl's public web archive — find every time a URL was crawled since 2008 and read.
v0.2.9
io.github.Grubbomatic/crawl-readiness
Check what AI crawlers may read and what a page permits, generate llms.txt and schema.
v0.3.1
io.github.crawlerpeek/crawlerpeek
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.
v0.33.1
io.github.DasDigitaleMomentum/searxncrawl
Web search via SearXNG and/or You.com, plus clean Markdown crawling via Crawl4AI. Self-hosted.
v1.1.1
com.soflution/deepcrawl
Clone websites, scrape pages to markdown and crawl whole sites. A free Firecrawl alternative.
v0.17.0
io.github.Mingye-Lu/acrawl
自主网页爬虫:17 个浏览器工具,外加目标驱动的 run_goal 智能体;单二进制、隐身模式。
v1.12.3
io.github.xberg-io/crawlberg
通过本地 CLI 抓取、爬取并映射网站,输出 Markdown 或 JSON。
v0.1.1
io.github.CYAN-ECLIPSE-TECHNOLOGIES-LLC/eclipse-crawlway-mcp
Pay-per-crawl licensing receipts: sealed identity+terms+fingerprint binding. LOCAL/fixture only.
v1.1.0
io.github.Soflutionltd/deepcrawl
Clone websites, scrape pages to markdown and crawl whole sites. A free Firecrawl alternative.
v1.0.0
com.crawlerzoo/arena
Games for AI agents: negotiation, social deduction, racing and puzzles, vs bots and agents.
v1.0.2
dev.oassis/scraper-crawler
Web Tools and Web Tasks for AI agents. 100-credit MCP trial; API-key credits or x402.
v1.8.0
app.openkrill/site-check
Check public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.