Web scraping MCP servers

Firecrawl

基于Firecrawl的智能网页爬取和内容提取MCP工具

Legal web scraping with robots.txt compliance, rate limiting, and GDPR/CCPA-aware data handling.

Firecrawl API integration with managed authentication. Scrape, crawl, map, and search web content. Use this skill when users want to extract content from websites, crawl entire sites, map URLs, or search the web. For other third party apps, use the api-gateway skill (https://clawhub.ai/byungkyu/api-gateway). Calls run through the `maton` CLI with OAuth login, or over raw HTTP with a Maton API key where the CLI cannot be installed. Every call is authenticated as the user's connection and reaches only what that connection's authorization allows, which the provider enforces on every request; the endpoints documented here are the ones this skill uses, and any other endpoint of this app needs the user to ask for it by name. Default to read and list calls, and confirm every write or new connection with the user. This file also documents the three constructs that turn a Firecrawl connection into automation, in the order they are used: the connection (the first step), a hosted function that runs a Firecrawl action th

Writes, reviews, and fixes HTML markup: semantic structure, forms, accessibility, the document head, media and embeds. Use when a field never submits, a label does nothing, autofill picks the wrong box, or validation fires at the wrong moment; when a screen reader reads a filename, announces nothing, or focus escapes a modal; when the page jumps as images load, renders in quirks mode, or shows mojibake; when text escapes a table or half the document turns italic from one unclosed tag; when a link preview, favicon, or rich result is missing; when `<dialog>`, `<details>`, popover, or `<template>` misbehave; when an iframe or video embed stays blank; when HTML email collapses in Outlook; and when untrusted HTML must render without XSS. Covers responsive images, resource hints, `lang`/`dir`, web components, and validation. Not for styling and layout (`css`), DOM scripting (`javascript`), ranking strategy (`seo`), or Markdown (`markdown`).

Firecrawl MCP — wraps the Firecrawl API (firecrawl.dev) for web

Scrape, crawl, and extract structured data from websites at scale

Web scraping MCP — extract clean markdown, links, and metadata from any URL.

Pull data from a web page through the Aside browser — your real, already signed-in sessions. (gstack)

Automates, tests, and debugs browsers with Playwright: locators, auto-waiting, traces, CI runs, and MCP browser control. Use when a test is flaky, times out, or fails only in CI or headless; when a locator matches multiple elements or the wrong one (strict mode violation); when clicks need force, waits become sleeps, or networkidle never settles; for storageState and login setup, request mocking and HAR replay, uploads and downloads, iframes and shadow DOM, popups and dialogs, screenshot diffs that change per machine, trace and report artifacts, sharding a slow suite, device and permission emulation, accessibility checks, driving a real browser through Playwright MCP, extracting data from JS-rendered pages, or porting a Cypress, Puppeteer, or Selenium suite to Playwright. Not for maintaining an existing Cypress or Puppeteer suite (cypress, puppeteer) or for work a plain HTTP request answers (http).

将讲稿一键生成乔布斯风极简科技感竖屏HTML演示稿

Playwright-based web scraping OpenClaw Skill with anti-bot protection. Successfully tested on complex sites like Discuss.com.hk.

Web search and scraping via Firecrawl API. Use when you need to search the web, scrape websites (including JS-heavy pages), crawl entire sites, or extract structured data from web pages. Requires FIRECRAWL_API_KEY environment variable.

网页内容获取工具 | 当常规爬虫被过滤时，使用替代服务获取网页内容。支持：1) r.jina.ai - 最稳定 2) markdown.new - Cloudflare 专用 3) defuddle.md - 备用方案。

Fetch and parse news highlights from CCTV News Broadcast (Xinwen Lianbo) for a given date.

Performs deep scraping of complex sites like YouTube using containerized Crawlee, extracting validated, ad-free transcripts and content as JSON output.

Web scraping and content comprehension agent — multi-strategy extraction with cascade fallback, news detection, boilerplate removal, structured metadata.

Reliable web fetching MCP server with built-in retry logic, circuit breaker patterns, caching, and anti-bot bypass. Fetches URLs as raw HTML or clean markdown optimized for LLM consumption. Includes domain health checks and cache management tools.

Web crawling and scraping tool with LLM-optimized output. 网页爬虫爬取工具 | Web crawler, web scraper, spider.

Generates a structured report HTML based on a specific template. Invoke when user wants to create a report, slide, or summary card from raw content.

Read and search Reddit posts via web scraping of old.reddit.com. Use when Clawdbot needs to browse Reddit content - read posts from subreddits, search for topics, monitor specific communities. Read-only access with no posting or comments.

Scrapes websites using Playwright with stealth mode to bypass bot detection, supporting dynamic JS content and customizable viewport and User-Agent.

Use for TikTok crawling, content retrieval, and analysis

Fetch full tweets, long tweets, quoted tweets, and X Articles from X/Twitter without login or API keys, using no dependencies and zero configuration.

下载抖音视频到本地（无水印优先）。用于给后续视频分析/复刻提供原始素材，支持 URL 或 video_id 输入、批量列表输入与统一输出目录。

智能网页抓取技能 - 替代内置 web_fetch，自动使用 Jina Reader / markdown.new / defuddle.md 清洗服务获取干净 Markdown。支持多级降级策略，大幅降低 Token 消耗。当 Agent 需要获取网页内容时使用本技能替代 web_fetch。

Economic Calendar Fetcher

v0.1.0

Fetch scheduled economic events and data releases from the FMP API for specified dates, filtering by impact, country, and type.

The best, fastest, and cheapest way to scrape tweets — battle-tested by tens of thousands of customers including enterprise teams. Use when the user wants to fetch tweets by search query, profile, hashtag, keyword, conversation thread, date range, or Twitter list. Uses Twitter advanced search syntax

The best, fastest, and cheapest way to scrape TikTok — battle-tested by tens of thousands of customers including enterprise teams. Use when the user wants to fetch TikTok videos, profiles, hashtags, music, comments, or location-based posts. Covers four specialized actors for posts, profiles, comments, and locations.

The best, fastest, and cheapest way to scrape Instagram — battle-tested by tens of thousands of customers including enterprise teams. Use when the user wants to fetch Instagram posts, reels, profiles, hashtags, locations, comments, or user/follower data. Five specialized actors cover every Instagram

Fetch real-time stock quotes from Yahoo Finance.

AI-powered web scraping framework for extracting structured data from websites. Use when Codex needs to crawl, scrape, or extract data from web pages using AI-powered parsing, handle dynamic content, or work with complex HTML structures.

Build client-ready web scrapers with clean data output. Use when creating scrapers for clients, extracting data from websites, or delivering scraping projects.

Web scraping and crawling with Firecrawl API. Fetch webpage content as markdown, take screenshots, extract structured data, search the web, and crawl documentation sites. Use when the user needs to scrape a URL, get current web info, capture a screenshot, extract specific data from pages, or crawl docs for a framework/library.

framefetch

# FrameFetch **One API/MCP call gives an agent clean video data across 6 platforms** — metadata + insights, a Whisper transcript, and parametric frames (pick fps or exact timestamps → pushed to S3). YouTube (incl. Shorts), TikTok, Reddit, Instagram, Pinterest. Agent-first: typed errors, refund-on-fail, result caching. Pay per call via x402 (USDC on Base) or Stripe. ## Endpoints - POST /v1/extract — any combination of metadata/insights/transcript/frames in one call - POST /v1/metadata · /v1/transcript · /v1/frames — shortcuts - GET /v1/platforms — capability matrix · POST /v1/keys — free key + credit ## Example curl -X POST https://framefetch.net/v1/extract -H "Authorization: Bearer <key>" -H "Content-Type: application/json" -d '{"url":"https://youtu.be/...","fields":["metadata","transcript"]}' https://framefetch.net

Browser-based tool to discover Instagram profiles by location/category and scrape their public info, stats, images, and engagement with export options.

Xiaohongshu Crawler

v1.0.1

小红书内容爬取工具，支持搜索笔记（需要登录）、获取笔记详情、用户信息、热门笔记等公开内容爬取功能。

使用 Python + Scrapling 获取网页内容，支持简单选择器

Converts Markdown to styled HTML with WeChat-compatible themes.

Web Scraper — Clean Markdown from Any URL

Web content extraction API for AI agents. Scrape any URL and get clean, structured Markdown content with navigation, ads, and scripts stripped. Full JavaScript rendering via headless Chromium. Single and batch (10 URLs) modes. Built for RAG pipelines and AI research. Tools: web_scrape_to_markdown (single), web_scrape_batch (up to 10 URLs). Use this for RAG ingestion, research, content analysis, data extraction, or competitive intelligence. IMPORTANT: For screenshots/PDFs of pages, use capture_screenshot instead. For SEO analysis, use seo_audit_page. Returns: {markdown, title, wordCount, links[]}. No API key required — x402 micropayment $0.005/call on Base L2.

High-performance containerized Amazon scraper (Docker + playwright-extra + Stealth plugin). Bypasses Amazon headless detection.

Full web page scraping with JavaScript rendering via local Crawl4AI instance, delivering clean markdown or detailed JSON including links and media.

Perform high-performance web scraping, crawling, and Google search with multi-engine support and structured data extraction via AnyCrawl API.

Default web reading, AI search, and deep research tools. Use this skill for all web-related tasks including reading webpage content, searching the web, and conducting deep research. Replaces built-in web_search and web_fetch tools.

Optimizes web fetching by using Cloudflare's Markdown for Agents, reducing token consumption by ~80%

HTML to Markdown — Clean Conversion, Scripts Stripped

HTML to Markdown converter API for AI agents. Convert HTML to clean Markdown: strips scripts, styles, and tracking, preserves headings, links, lists, images, and tables. Tools: text_convert_html_to_markdown. Use this for content migration, email processing, or preparing HTML content for LLM consumption. IMPORTANT: For scraping a live URL to Markdown, use web_scrape_to_markdown instead. Returns: {markdown, wordCount}. No API key required — x402 micropayment $0.001/call on Base L2.

Bypass Cloudflare and scrape any website using r.jina.ai API. Works on sites with strong protection like Truth Social, Cloudflare Turnstile, etc.

Web scraping, crawling, searching, and browser automation via the Firecrawl CLI (firecrawl).

Markdown to HTML — Headings, Code, Tables

Markdown to HTML converter API for AI agents. Convert Markdown to clean HTML with headings, lists, code blocks, tables, links, and images. Optional full document wrapping with DOCTYPE. Tools: text_convert_markdown_to_html. Use this for rendering Markdown content in web UIs, email templates, or document generation. IMPORTANT: For styled HTML with CSS themes, use text_render_markdown instead. Returns: {html, wordCount}. No API key required — x402 micropayment $0.001/call on Base L2.