搜索「Crawler」的结果

io.github.imfurkana/unrendered-ai-crawler-checker

AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.

io.github.schemacrawler/schemacrawler-ai

支持用自然语言查询数据库模式——快速浏览表、键、存储过程并获取 SQL 帮助。

io.github.jmshinhwa/robots-txt-ai-crawler-audit

Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

io.github.manchittlab/thecrawler

通用网页抓取器，输出适配 LLM 的 Markdown，支持 RAG 分块与 PDF/DOCX 解析。

io.github.houtini-ai/seo-crawler-mcp

使用 Crawlee 抓取并分析网站的 SEO 错误，结果存储于 SQLite。

io.github.shufflethis/geo-tool-check

Is this page readable for AI search? Crawler access, structure and citability checks, runs locally.

io.github.Mingye-Lu/acrawl

自主网页爬虫：17 个浏览器工具，外加目标驱动的 run_goal 智能体；单二进制、隐身模式。

io.github.jmshinhwa/tdm-reservation-lint

Finds the robots.txt lines that reserve nothing — retired crawler tokens, shadowed groups, rules abo

io.github.iliasabk/geolint

ESLint for AI search: audits AI-crawler access, llms.txt, schema and citability.

com.see-geo/seegeo

Audit any website for AI visibility: graded report, findings with fixes, AI crawler access check.

io.github.maxaeo/maxaeo-ai-visibility-mcp

本地优先的 MCP 服务器，用于 AI 可见度、GEO/AEO、llms.txt 与 AI 抓取友好度审计。

io.github.kopachlager/prerenderbuddy-mcp

Local MCP tools for crawler-readability and discovery-file diagnostics.

io.github.Bigsupe55/geo-inspector-mcp

检查站点的 AI 搜索就绪度：AI 爬虫访问、llms.txt、schema 结构化标记与 meta 指令。

io.github.arthurcarlsonn/aeo-mcp

面向 AI 智能体的 AEO 工具：爬虫权限、llms.txt、结构化数据与 AI 就绪度审计。

io.github.sharozdawa/robotstxt-ai

分析与生成 robots.txt，可检测 GPTBot、ClaudeBot 等 20+ AI 爬虫。