GenPark Bloom Filter URL Dedup Crawler
v1.0.1
io.github.alphaparkinc/genpark-bloom-filter-url-dedup-crawler-skill
In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge
“Crawler” 共 117 个结果
v1.0.1
io.github.alphaparkinc/genpark-bloom-filter-url-dedup-crawler-skill
In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge
v0.1.4
io.github.missiongrowth/unrendered-ai-crawler-checker
AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
v1.0.0
com.insightfulpipe/enrichment
Hosted Enrichment Crawlers MCP server, with each action switched on or off per account.
v1.0.0
com.insightfulpipe/crawler
Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
v1.0.0
dev.workers.quietworklabs.robots-check/robots-check
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
v1.0.0
io.github.puneetindersingh/open-seo-crawler
Self-hosted SEO crawler: run site audits and AI crawler checks, then read the issues and pages.
v1.0.0
com.crawlerdata/google-shopping-serp
Google Shopping products and seller prices, plus Google Search, Images, News and Local results.
v0.1.3
io.github.imfurkana/unrendered-ai-crawler-checker
AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
v0.1.1
com.ranovam/flarum-scraper
搜索或监控任意 Flarum 论坛的讨论、回复、参与者。
v0.1.0
io.github.newmediamadman/nm-signals
Audit webpage access for AI crawlers. API key and Premium or Partner plan required.
v1.0.0
app.rowb.ai-visibility/audit
One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.
v1.0.0
com.lumnika/ai-crawler-readiness
Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index
v0.1.3
io.github.jmshinhwa/robots-txt-ai-crawler-audit
Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI
v1.7.0
dev.workers.pathwren.www/crawler-log-triage
对访问日志中的爬虫流量做归类与诊断,免费无需密钥(原文功能描述为未填写的占位符)。
v1.6.0
dev.workers.pathwren.www/ai-crawler-index
查询与统计抓取本站的 AI 爬虫索引情况,免费无需密钥(原文功能描述为未填写的占位符)。
v1.4.0
dev.workers.pathwren.www/crawler-ip-verifier
校验来访 IP 是真实的 Googlebot 等爬虫还是假冒者,免费无需密钥(原文功能描述为未填写的占位符)。
v4.1.2
com.crimsoncrawler/mcp-server
通过 80 余个类型化 MCP 工具对威胁情报进行富化、检索、评估与管理。
v1.4.2
com.actablesite/ai-crawler-robots-monitor
检查 AI 爬虫的 robots.txt 策略,并监控公开站点的策略、sitemap 与 llms.txt 变化。
v1.4.1
com.actablesite/readiness
检查 AI 爬虫的 robots.txt 策略,并监控公开站点的策略、sitemap 与 llms.txt 变化。
v0.5.8
io.github.manchittlab/thecrawler
通用网页抓取器,输出适配 LLM 的 Markdown,支持 RAG 分块与 PDF/DOCX 解析。
v0.2.1
io.github.shulex-voc/smart-crawler
电商竞品情报、VOC 情感分析、Amazon ASIN 分析与 Reddit 增长方案。
v1.0.0
io.github.lazymac2x/ai-crawler-policy
部署在 Cloudflare Workers 上的 MCP 服务器:ai-crawler-policy。
v2.0.1
io.github.houtini-ai/seo-crawler-mcp
使用 Crawlee 抓取并分析网站的 SEO 错误,结果存储于 SQLite。
io.github.opensanctions/opensanctions/refactor-crawler
将杂乱或 AI 生成的爬虫代码重写为符合 zavod 最佳实践的整洁、生产级代码。当用户要求清理、重构、整理爬虫代码、“达到生产可用”或对齐最佳实践时使用。