Site SEO & AI Crawler Audit
v1.8.0
app.openkrill/site-check
Check public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.
“Crawl” 共 203 个结果
v1.8.0
app.openkrill/site-check
Check public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.
v1.0.1
io.github.alphaparkinc/genpark-bloom-filter-url-dedup-crawler-skill
In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge
v0.1.4
io.github.missiongrowth/unrendered-ai-crawler-checker
AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
v1.0.0
com.insightfulpipe/enrichment
Hosted Enrichment Crawlers MCP server, with each action switched on or off per account.
v1.0.0
com.insightfulpipe/crawler
Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
v1.0.0
dev.workers.quietworklabs.robots-check/robots-check
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
v0.4.2
io.github.clearcrawl/clearcrawl
Any URL to clean markdown/JSON; PDF, XLSX, CSV, DOCX to text and tables. Free tier for agents.
v0.1.2
com.crawlwisedata/crawlwise-mcp
Per-URL tech-stack, feed and sitemap change events for AI agents, delivered as signed webhooks.
v1.0.0
io.github.puneetindersingh/open-seo-crawler
Self-hosted SEO crawler: run site audits and AI crawler checks, then read the issues and pages.
v1.0.0
xyz.crawlhive/crawlhive
Memecoin safety scanner for Solana, BNB and Robinhood Chain with an onchain track record
v1.0.0
io.github.realtycrawl/realtycrawl
Search source-linked real estate observations with capability discovery and bounded credits.
v1.0.0
com.crawlerdata/google-shopping-serp
Google Shopping products and seller prices, plus Google Search, Images, News and Local results.
v1.0.0
io.github.IliasSami/scrawly
MCP server for the Scrawly SEO crawler: query local audits and propose confirmation-gated fixes.
v1.0.0
com.jobcrawls/connector
Search jobs in Finland and calculate net salary. Read-only, no login. English, Finnish, Swedish.
v0.1.3
io.github.imfurkana/unrendered-ai-crawler-checker
AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
v1.0.1791044090
com.saastemly/crawl4agent
crawl4ai-compatible web crawler API: POST a URL.
v1.0.3
io.github.prismcrawl/prismcrawl
Live Google, Bing, Amazon, Maps, Yelp & app store data. As low as $0.15/1K successes; failures free.
v0.1.1
com.ranovam/flarum-scraper
搜索或监控任意 Flarum 论坛的讨论、回复、参与者。
v1.0.0
com.monocrawl/mcp
Public social media, search and e-commerce data from 70+ platforms, on one API key.
v1.0.0
io.github.sadri-dridi/agent-crawl-readiness
Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.
v0.1.0
io.github.newmediamadman/nm-signals
Audit webpage access for AI crawlers. API key and Premium or Partner plan required.
v1.0.0
app.rowb.ai-visibility/audit
One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.
v1.0.1
io.github.mrfentmen/commoncrawl-mcp
Use this MCP server to Common Crawl index discovery for historical web captures.
v0.8.1
io.github.SanggonBoy/PyreCrawl
Web scraping + research toolkit for AI agents: scrape, crawl, monitor URLs with anti-bot bypass.