data-scraper-agent
为任意公开来源(招聘网站、价格、新闻、GitHub、体育赛事等)构建全自动的 AI 数据收集代理。按计划抓取,用免费 LLM(Gemini Flash)丰富数据,将结果存入 Notion/Sheets/Supabase,并从用户反馈中学习。完全免费地在 GitHub Actions 上运行。当用户希望自动监控、收集或追踪公开数据时使用。
使用场景/网页抓取与采集
从网页提取结构化内容、爬取文档、监控变更。适合调研、内容聚合、知识库构建。
共匹配 1,401 个资源 · 第 9 / 30 页
网页抓取类 MCP Server 把互联网内容变成 AI 可直接消费的结构化文本:Firecrawl 类服务负责整站爬取与 Markdown 转换,Fetch 类服务负责单页拉取与重定向处理。相比浏览器自动化,它们不渲染交互、速度快、token 消耗低,是构建 RAG 语料与知识库的首选采集层。
典型工作流:用 Firecrawl MCP 批量爬取文档站 → 清洗为 Markdown → 送入向量库(如 Qdrant / Chroma MCP)→ 再用检索 MCP 让 AI 基于自有语料回答。AgentHub 上每个抓取类资源都附带安装命令与客户端配置,可直接复制到 Cursor 或 Claude Code。
合规边界:只抓取公开页面,遵守目标站点 robots.txt 与服务条款;控制并发与频率,避免给对方服务造成压力;需要登录才能访问的内容不要自动化批量拉取。
为任意公开来源(招聘网站、价格、新闻、GitHub、体育赛事等)构建全自动的 AI 数据收集代理。按计划抓取,用免费 LLM(Gemini Flash)丰富数据,将结果存入 Notion/Sheets/Supabase,并从用户反馈中学习。完全免费地在 GitHub Actions 上运行。当用户希望自动监控、收集或追踪公开数据时使用。
v0.2.0 起弃用——请改用 browser-extract;本技能仅为向后兼容的薄壳,将在 v0.3.0 移除。
借助 55+ Actor 在所有主流平台上进行 AI 驱动的数据提取。该技能会自动为你的任务挑选最合适的 Actor。
从 FRED、世界银行等 API 获取经济数据。
提供 Next.js App Router 数据获取模式,包括 SWR 与 React Query 集成、并行数据获取、增量静态再生(ISR)、再验证策略与错误边界。在 Next.js 应用中实现数据获取、在服务端与客户端获取之间取舍、设置缓存策略或处理加载与错误状态时使用。
构建一个全自动化的AI驱动数据收集代理,适用于任何公共来源——招聘网站、价格信息、新闻、GitHub、体育赛事等任何内容。按计划进行抓取,使用免费LLM(Gemini Flash)丰富数据,将结果存储在Notion/Sheets/Supabase中,并从用户反馈中学习。完全免费在GitHub Actions上运行。适用于用户希望自动监控、收集或跟踪任何公共数据的场景。
面向 AI 编码智能体的 X API 与 Twitter 抓取技能。基于 Xquik REST API、MCP server 与 webhook 构建集成:推文搜索、用户查询、粉丝提取、互动指标、抽奖开奖、热门话题、账号监控、回复/转推/引用抽取、社群与 Space 数据、互相关注检查。支持 Claude Code、Cursor、Codex、Copilot、Windsurf 等 40+ 智能体。
在使用 Next.js 数据获取模式(包括 SSG、SSR 与 ISR)时使用。适合构建数据驱动的 Next.js 应用。
在 Next.js 中基于 URL 参数拉取数据的专注模式讲解。涵盖动态路由([id]、[slug])的创建,以及在服务端组件中读取路由参数并调用 API 取数。适用于构建按 URL 参数展示单条内容的页面(商品详情、博客文章、用户主页)。与 nextjs-dynamic-routes-params 互补,提供简化后的常见场景写法。
v1.0.0
面向 OpenClaw 的零机器人检测网页抓取。绕过 Cloudflare、处理重 JavaScript 网站,并自动适应网站改版。当你需要……时使用。
v0.1.0
A Google Search API alternative and SERP API alternative on fetcher.sh — pay-per-call in USDC via x402, or prepaid credits with a Bearer key, no API key application. Use when the user wants programmatic Google search results as clean JSON, including Google's own operators — site:, filetype:, intitle:, and quoted exact phrases — plus pagination, language (hl), and country/region scoping. Also covers rank tracking input, competitive research, and search-result monitoring without Google's own Custom Search API quota and billing.
v0.1.0
An Apple App Store API alternative on fetcher.sh — pay-per-call in USDC via x402, or prepaid credits with a Bearer key, no Apple Developer Program membership. Use when the user wants to search iOS apps by keyword in any country's storefront, fetch an app's or app bundle's full details, reviews sorted by recent/helpful, apps similar to a given app, search or fetch app bundles, or fetch a developer's app catalog. Also covers iOS app store optimization (ASO) research, competitor app monitoring, localized storefront comparison, and app discovery without Apple Developer Program access.
v0.1.0
A Yelp API alternative on fetcher.sh — pay-per-call in USDC via x402, or prepaid credits with a Bearer key, no Yelp Fusion API app approval. Use when the user wants to search local businesses by query and location sorted by rating or review count, fetch a business's full details by ID or by its Yelp URL handle/slug, or fetch a business's reviews. Also covers local business discovery, restaurant/service research, review sentiment input, and competitor monitoring for local businesses without Yelp Fusion API's app-approval process and daily call caps.
v0.1.0
A Google News API alternative on fetcher.sh — pay-per-call in USDC via x402, or prepaid credits with a Bearer key, no RSS scraping. Use when the user wants keyword search across Google News scoped to a language edition, section headlines (world, business, technology, entertainment, sport, science, health, or a specific topic ID), the latest headlines, a list of supported language-region codes, or to decode a Google News redirect URL into the real article URL. Also covers news monitoring, headline aggregation, media tracking, and press-mention alerts without scraping Google News' RSS feeds directly.
v0.1.0
A Google Play Store API alternative on fetcher.sh — pay-per-call in USDC via x402, or prepaid credits with a Bearer key, no Google Play Console access. Use when the user wants to search Android apps by keyword with price (free/paid) and country storefront filters, fetch an app's full details, reviews sorted by newest/rating/helpfulness, permissions, or data safety disclosure, list apps similar to a given app, or fetch a developer's app catalog. Also covers Android app store optimization (ASO) research, competitor app monitoring, review sentiment input, and app discovery without Google Play Console developer access.
v0.1.0
A Google Maps API alternative and Google Places API alternative on fetcher.sh — pay-per-call in USDC via x402, or prepaid credits with a Bearer key, no Google Cloud billing account. Use when the user wants to search places by text query or anchor the search to a latitude/longitude coordinate, fetch a place's full details by its feature ID (fid), pull a place's reviews sorted by relevance/newest/rating, or look up a single review by ID. Also covers local business discovery, points-of-interest data, store-locator input, and review monitoring without Google Cloud Platform's API key setup, billing, and per-request pricing.
v0.1.0
A YouTube API alternative on fetcher.sh — pay-per-call in USDC via x402, or prepaid credits with a Bearer key, no OAuth and no daily quota. Use when the user wants to search YouTube videos with filters for upload date, duration, or sort order, search channels or playlists, look up a channel by ID, @handle, or custom URL path, list a channel's videos, shorts, or live streams, fetch a video's or short's details and comments, list a playlist's videos, pull posts under a hashtag, or check trending videos by region. Also covers YouTube data pipelines, channel monitoring, video analytics input, and trend tracking without the official YouTube Data API's daily quota limits.
v1.0.0
使用无头浏览器通过多个搜索引擎搜索内容。用于当用户需要搜索最新新闻、时事、股票信息或其他需要实时网络数据的内容时。支持百度、Bing、360、Sogou、微信、今日头条、谷歌等国内外搜索引擎。
v0.1.0
An X API alternative and Twitter API alternative on fetcher.sh — pay-per-call in USDC via x402, or prepaid credits with a Bearer key, no OAuth and no developer application. Use when the user wants to search X posts by keyword, hashtag, or advanced operators (from:, to:, since:, until:, min_faves:, filter:), scrape an X/Twitter profile by handle, pull a user's posts, replies, followers, or followings, fetch a single post with its replies or reposters, read an X List's members or posts, check trending topics by country, or search for X accounts by name. Also covers building an X data pipeline, social listening, competitor monitoring, hashtag tracking, or follower export without the official X API's pricing tiers or app-review process.
v0.1.0
A Reddit API alternative on fetcher.sh — pay-per-call in USDC via x402, or prepaid credits with a Bearer key, no OAuth app registration. Use when the user wants to search Reddit posts across every subreddit by keyword and sort by top, hot, new, or most-discussed, search subreddits or users by keyword, fetch a subreddit's info or its hot/new/top post feed, fetch a single post with its comment tree and comment replies, pull the sitewide best/hot/new/top feeds, or fetch a user's profile, posts, and comments. Also covers Reddit sentiment analysis input, subreddit monitoring, keyword tracking, and Reddit data pipelines without Reddit's own API app registration or rate-limit tiers.
AI 会话记忆:每次会话前由你的 AI 阅读的简报,避免任何会话从零开始。跨 Claude、ChatGPT、Cursor 与 Codex 加载项目当前任务、近期决策与未决问题。
v1.1.0
EU-native web scraping for AI agents. scrape pages, map sites, run background jobs, check usage.
v0.6.0
将 HTML 发布为带追踪的链接,查看谁打开过以及读了哪些章节。
v0.3.2
面向 AI 智能体的网页情报:抓取、渲染、抽取与研究。使用 x402 微支付,无需 API 密钥。
v2.3.1
免费发现 BitBooth API,再购买限额的 x402 抓取、PageDelta 与钱包检查。
v1.0.0
BCP 47 lang attribute shape
v1.0.5
MCP server for HTML2PDF Converter
v1.1.0
付费前先检查 x402 端点:可用性、价格历史与刷量农场检测。另提供俄语网页访问。
v0.1.1
YouTube transcripts for AI agents: videos, channels, playlists, search. Timestamps, SRT, VTT.
v0.3.5
从任意 MCP 客户端把 HTML 或 Markdown 发布为 htmldrop.app 上可分享的真实链接。
v1.0.0
4 Yelp endpoints. Pay per call in USDC via x402.
v1.0.0
9 App Store endpoints. Pay per call in USDC via x402.
v1.0.0
7 Google Play endpoints. Pay per call in USDC via x402.
v1.0.0
12 Google News endpoints. Pay per call in USDC via x402.
v1.0.0
4 Google Maps endpoints. Pay per call in USDC via x402.
v1.0.0
1 Google Search endpoints. Pay per call in USDC via x402.
v1.0.0
15 Reddit endpoints. Pay per call in USDC via x402.
v1.0.0
16 Instagram endpoints. Pay per call in USDC via x402.
v1.0.0
13 TikTok endpoints. Pay per call in USDC via x402.
v1.0.0
15 YouTube endpoints. Pay per call in USDC via x402.
v1.0.0
15 Twitter / X endpoints. Pay per call in USDC via x402.
v1.9.0
SpiderIQ Leads 获客 MCP:招聘岗位、营销活动、IDAP、地图、人物信息、核验、企业背景与 spiderPR。
v0.1.2
See and kill the dev servers your coding agents leave running, and get collision-free ports.
v0.1.1
Scrape Google Maps business data from any MCP client — names, phones, emails, websites, ratings.
v0.1.1
Curated SISTRIX SEO tools: visibility, rankings, keywords, backlinks, AI visibility, Amazon data.
v0.1.3
Curated read-only Matomo Analytics tools: traffic, pages, referrers, e-commerce, real-time & more.
v1.1.0
SpiderIQ Media:SpiderMedia 文件、视频与按租户划分的媒体目录(只读)。
v1.3.0
SpiderIQ Gate:SpiderGate LLM 网关(补全、模型、用量、追踪)。
静态或服务端渲染的页面、整站文档爬取用抓取类(Firecrawl/Fetch);需要点击、滚动、登录交互或抓渲染后截图,用浏览器自动化(Playwright MCP)。
托管版需要;也可以自部署开源版,或用免费的 Fetch MCP 做单页抓取。资源详情页会标注分发方式与所需环境变量。
常见做法是 Markdown 清洗后写入向量库 MCP(Qdrant、Chroma、pgvector 等),再配合检索 Skill 或 Context7 类知识库 MCP 完成问答。