Code Crawler
v0.1.96
io.github.nithin554/code-crawler
Static and dynamic call tracing MCP server backed by a SQLite property graph of your codebase
“Crawler” 共 32 个结果
v0.1.96
io.github.nithin554/code-crawler
Static and dynamic call tracing MCP server backed by a SQLite property graph of your codebase
v0.7.10
io.github.mark1russell7/polite-crawler
A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
vv18.0.3-1
io.github.schemacrawler/schemacrawler-ai
支持用自然语言查询数据库模式——快速浏览表、键、存储过程并获取 SQL 帮助。
v1.0.2
com.mambabuilt/mcp-site-crawler-links-tables
Crawl a website for links, file download links, HTML tables as rows, and lists, with no AI
v0.3.1
io.github.crawlerpeek/crawlerpeek
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.
v1.0.1
io.github.alphaparkinc/genpark-bloom-filter-url-dedup-crawler-skill
In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge
v0.1.4
io.github.missiongrowth/unrendered-ai-crawler-checker
AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
v1.0.0
io.github.puneetindersingh/open-seo-crawler
Self-hosted SEO crawler: run site audits and AI crawler checks, then read the issues and pages.
v0.1.3
io.github.imfurkana/unrendered-ai-crawler-checker
AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
v0.1.3
io.github.jmshinhwa/robots-txt-ai-crawler-audit
Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI
v0.5.8
io.github.manchittlab/thecrawler
通用网页抓取器,输出适配 LLM 的 Markdown,支持 RAG 分块与 PDF/DOCX 解析。
v2.0.1
io.github.houtini-ai/seo-crawler-mcp
使用 Crawlee 抓取并分析网站的 SEO 错误,结果存储于 SQLite。
v0.4.0
io.github.kopachlager/prerenderbuddy-mcp
Crawler diagnostics, AI visibility evidence and article drafts for review in Prerender Buddy.
v1.0.5
com.atlas-compiler/atlas
Deterministic Zero-LLM Web Compiler, Scraper, Crawler, and Site Mapper for AI Agents.
v1.0.1
io.github.shufflethis/geo-tool-check
Is this page readable for AI search? Crawler access, structure and citability checks, runs locally.
v0.1.0
io.github.CloudCorpRecords/crawlsight
Check a site's AI visibility: crawler access, llms.txt generate and validate, robots.txt lint.
v2.1.1
io.github.mazprotech/mtscraper
Free local crawler: any site, JavaScript pages and linked PDFs, into clean text.
v0.1.0
io.github.CloudCorpRecords/domaintrack
Look up domain reputation snapshots and AI crawler policy from Domduck's public store.
v0.1.0
io.github.CloudCorpRecords/geoscore
AI-crawler readiness checks and fixes: GEO score, robots.txt, schema.org, FAQ and llms.txt.
v0.2.0
io.github.heyyviv/web-shooter
Fast, low-token web reader and focused crawler. Returns ranked passages, not whole pages.
v0.1.0
io.github.CloudCorpRecords/visprobe
AI-visibility audit: AI crawler access, identity files, page signals, hosted analysis.
v0.17.0
io.github.Mingye-Lu/acrawl
自主网页爬虫:17 个浏览器工具,外加目标驱动的 run_goal 智能体;单二进制、隐身模式。
v1.0.0
com.queryrev/ai-readiness
Checks a site for AI crawler access, robots.txt and llms.txt. Runs locally, no telemetry.
v0.5.1
com.causabi/geo-optimizer-mcp
Audit a site for AI readiness: AI-crawler access, schema, FAQ, content structure. Generates fixes.