What is Crawl4ai good for?

Crawl4ai is a fit for: Web scraping. See AgentHub scenario pages for pairings.

Crawl4ai Skill Install & Guide

Q: How do I install Crawl4ai?

SkillClawHub

io.clawhub.codylrn804/crawl4ai · v1.0.0

View on ClawHub ↗

Copy the install config on this page first, then verify docs and permissions upstream.

Popularity

Stars2

Downloads5k

Installs121

Overview

AI-powered web scraping framework for extracting structured data from websites. Use when Codex needs to crawl, scrape, or extract data from web pages using AI-powered parsing, handle dynamic content, or work with complex HTML structures. Crawl4ai is a Agent Skill listed from ClawHub. This page includes an overview, setup tutorial, install commands, and use cases for Trae, Tongyi Lingma, Cursor, Claude Code, and VS Code.

Web Scraping Scrape

Use cases

Web scraping

AgentHub Verified AvailabilityTested & Ready

Automated pipeline validated install commands, protocol & client compatibility

Verified At2026-09-30

Install Snippets TestedCLI & JSON config syntax verified

Protocol Handshake ReadyComplies with JSON-RPC 2.0 specifications

Origin Registry ActiveSourced from clawhub

Verified Clientsopenclaw、Claude Code、Claude Desktop 等

Security Tier: A+ 级 · 零权限本地沙盒 (A+)·Scanned for high-risk vulnerabilities. Recommended to run with project-scoped permissions.

Score 70/100

Send this prompt to your AI to install the Skill

Recommended

Target client (auto-optimizes prompt):

PROMPT >

Follow the install guide at https://myagenthub.cn/install/skill.md?lang=en to automatically detect the current IDE and install the skill "codylrn804/crawl4ai" without asking, and tell me how to verify it afterwards.

💡Paste into your AI coding assistant (Cursor, Trae, Claude Code) — the agent will handle download and configuration automatically.

Copy by platformAlternative

Choose your platform

Choose install method

Click to copy snippet

# 改 --ai 切换客户端：trae | lingma | workbuddy | kimi | qwen | cursor | claude-code | windsurf | cline | vscode | codex | openclaw
curl -fsSL "https://myagenthub.cn/cli/agenthub-skill.mjs" | node --input-type=module - init codylrn804/crawl4ai --ai cursor -y

Setup tutorial

Open the Crawl4ai page and confirm the source is ClawHub.
Copy the install command for Trae, Lingma, Cursor, Claude Code, or universal CLI.
Run it, or place SKILL.md under .trae/skills/, .lingma/skills/, or .cursor/skills/.
Reload the client, then describe your task so the agent can match this skill.

Install commands

Install commands and setup steps are in the HTML so search engines and no-JS browsers can read them without running client JavaScript.

Universal — Skills CLI (recommended)

AgentHub CLI (change --ai)

# 改 --ai 切换客户端：trae | lingma | workbuddy | kimi | qwen | cursor | claude-code | windsurf | cline | vscode | codex | openclaw
curl -fsSL "https://myagenthub.cn/cli/agenthub-skill.mjs" | node --input-type=module - init codylrn804/crawl4ai --ai cursor -y

One-click API download

# 手动下载（Cursor 项目级示例）
mkdir -p .cursor/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d .cursor/skills/crawl4ai

Cursor

AgentHub CLI (change --ai)

curl -fsSL "https://myagenthub.cn/cli/agenthub-skill.mjs" | node --input-type=module - init codylrn804/crawl4ai --ai cursor -y

One-click API download

mkdir -p .cursor/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d .cursor/skills/crawl4ai

ClawHub CLI, then copy

# 在项目根目录执行 install
clawhub install codylrn804/crawl4ai
mkdir -p .cursor/skills/crawl4ai
cp -R ./skills/crawl4ai/. .cursor/skills/crawl4ai/

# 源：./skills/crawl4ai/SKILL.md
# 目标：.cursor/skills/crawl4ai/SKILL.md

Trae (ByteDance)

Trae natively supports Skill specs: project-level in .trae/skills/<name>/, global in ~/.trae/skills/
Copy and run the command in your project terminal to download SKILL.md
Reload Trae window, then describe your task in AI chat to activate automatically

AgentHub CLI (change --ai)

curl -fsSL "https://myagenthub.cn/cli/agenthub-skill.mjs" | node --input-type=module - init codylrn804/crawl4ai --ai trae -y

One-click API download

mkdir -p .trae/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d .trae/skills/crawl4ai

User directory

mkdir -p ~/.trae/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d ~/.trae/skills/crawl4ai

Tongyi Lingma (Alibaba)

Tongyi Lingma supports skills in project .lingma/skills/<name>/ or global ~/.lingma/skills/
Run the command to install SKILL.md
In Lingma Agent mode, prompt your task and the skill will be automatically loaded

AgentHub CLI (change --ai)

curl -fsSL "https://myagenthub.cn/cli/agenthub-skill.mjs" | node --input-type=module - init codylrn804/crawl4ai --ai lingma -y

One-click API download

mkdir -p .lingma/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d .lingma/skills/crawl4ai

User directory

mkdir -p ~/.lingma/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d ~/.lingma/skills/crawl4ai

WorkBuddy / CodeBuddy (Tencent)

WorkBuddy reads skills only from .workbuddy/skills/<name>/ (user-level ~/.workbuddy/skills/); .codebuddy/skills/ belongs to CodeBuddy
The Skills CLI command installs with -a codebuddy and appends a copy step into .workbuddy/skills
AgentHub CLI (--ai workbuddy) syncs it for you; reload the chat panel so the skill gets indexed

AgentHub CLI (change --ai)

curl -fsSL "https://myagenthub.cn/cli/agenthub-skill.mjs" | node --input-type=module - init codylrn804/crawl4ai --ai workbuddy -y

One-click API download

mkdir -p .workbuddy/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d .workbuddy/skills/crawl4ai

User directory

mkdir -p ~/.workbuddy/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d ~/.workbuddy/skills/crawl4ai

Claude Code

AgentHub CLI (change --ai)

curl -fsSL "https://myagenthub.cn/cli/agenthub-skill.mjs" | node --input-type=module - init codylrn804/crawl4ai --ai claude-code --global -y

One-click API download

mkdir -p ~/.claude/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d ~/.claude/skills/crawl4ai

ClawHub CLI, then copy

clawhub install codylrn804/crawl4ai
mkdir -p ~/.claude/skills/crawl4ai
cp -R ./skills/crawl4ai/. ~/.claude/skills/crawl4ai/

# 源：./skills/crawl4ai/SKILL.md
# 目标：~/.claude/skills/crawl4ai/SKILL.md
# https://clawhub.ai/codylrn804/crawl4ai

Kimi Code (Moonshot AI)

Kimi Code supports standard Agent Skills: project-level in .agents/skills/<name>/, global in ~/.agents/skills/
Run the command to install
In Kimi Code CLI, describe your task to trigger the skill

AgentHub CLI (change --ai)

curl -fsSL "https://myagenthub.cn/cli/agenthub-skill.mjs" | node --input-type=module - init codylrn804/crawl4ai --ai kimi -y

One-click API download

mkdir -p .agents/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d .agents/skills/crawl4ai

User directory

mkdir -p ~/.agents/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d ~/.agents/skills/crawl4ai

Qwen Code (Alibaba)

Qwen Code supports project .qwen/skills/<name>/ and global ~/.qwen/skills/
Run the command to deploy SKILL.md
Prompt the agent to trigger the skill

AgentHub CLI (change --ai)

curl -fsSL "https://myagenthub.cn/cli/agenthub-skill.mjs" | node --input-type=module - init codylrn804/crawl4ai --ai qwen -y

One-click API download

mkdir -p .qwen/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d .qwen/skills/crawl4ai

User directory

mkdir -p ~/.qwen/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d ~/.qwen/skills/crawl4ai

Windsurf

Windsurf supports project-level .windsurf/skills/<name>/ or global ~/.codeium/windsurf/skills/
Run the install command and refresh Cascade
Cascade will auto-activate the skill based on its description

AgentHub CLI (change --ai)

curl -fsSL "https://myagenthub.cn/cli/agenthub-skill.mjs" | node --input-type=module - init codylrn804/crawl4ai --ai windsurf -y

One-click API download

mkdir -p .windsurf/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d .windsurf/skills/crawl4ai

User directory

mkdir -p ~/.codeium/windsurf/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d ~/.codeium/windsurf/skills/crawl4ai

Cline

Cline supports standard .agents/skills/<name>/ structure
Run the command to install the skill
Ask questions in the Cline panel to trigger the skill

AgentHub CLI (change --ai)

curl -fsSL "https://myagenthub.cn/cli/agenthub-skill.mjs" | node --input-type=module - init codylrn804/crawl4ai --ai cline -y

One-click API download

mkdir -p .agents/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d .agents/skills/crawl4ai

Cherry Studio

Use this skill as an Assistant system prompt in Cherry Studio
Copy the command to view SKILL.md text
Paste the contents into Cherry Studio Assistant system prompt

# 下载并提取 SKILL.md 指令内容
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -p /tmp/crawl4ai.zip SKILL.md
# 复制上述输出，粘贴至 Cherry Studio → 助手设置 →「系统提示词」

VS Code / GitHub Copilot

AgentHub CLI (change --ai)

curl -fsSL "https://myagenthub.cn/cli/agenthub-skill.mjs" | node --input-type=module - init codylrn804/crawl4ai --ai vscode -y

One-click API download

curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
mkdir -p .github/skills/crawl4ai
unzip -o /tmp/crawl4ai.zip -d .github/skills/crawl4ai

OpenAI Codex

AgentHub CLI (change --ai)

curl -fsSL "https://myagenthub.cn/cli/agenthub-skill.mjs" | node --input-type=module - init codylrn804/crawl4ai --ai codex --global -y

One-click API download

mkdir -p ~/.codex/skills/crawl4ai
curl -sL "https://clawhub.ai/api/v1/download?slug=crawl4ai&version=1.0.0" -o /tmp/crawl4ai.zip
unzip -o /tmp/crawl4ai.zip -d ~/.codex/skills/crawl4ai

ClawHub CLI, then copy

clawhub install codylrn804/crawl4ai
mkdir -p ~/.codex/skills/crawl4ai
cp -R ./skills/crawl4ai/. ~/.codex/skills/crawl4ai/

# 源：./skills/crawl4ai/SKILL.md
# 目标：~/.codex/skills/crawl4ai/SKILL.md

OpenClaw — ClawHub CLI

# OpenClaw / ClawHub CLI（推荐）
# 在项目根或 OpenClaw 工作区目录执行
clawhub install codylrn804/crawl4ai

# 安装后 SKILL.md 位于：
#   ./skills/crawl4ai/SKILL.md
# （若已配置 OpenClaw 工作区，则在 <workspace>/skills/crawl4ai/）

# 仅查看不安装
clawhub inspect codylrn804/crawl4ai

Troubleshooting & Common ErrorsFAQ

Common connection errors and verified fixes for Crawl4ai

Getting 'connection closed' or exit code 1 in Cursor / Claude Code for Crawl4ai?

Usually caused by missing runtime paths or IDE environment inheritance. Fix steps: 1. Verify Node.js 18+ (npx) or Python 3.10+ (uvx) is installed; 2. Run 'which npx' or 'which uvx' in terminal, and replace 'command' with absolute path; 3. Reload or restart the client window after modifying config.

Fix Snippet

# Check binary path in terminal:
which npx
node -v

Getting 'spawn npx ENOENT' or 'command not found'?

The editor background process does not inherit your full terminal PATH. Solution: Globally install the package, or set the absolute binary path (e.g. C:\Program Files\nodejs\npx.cmd on Windows, or /usr/local/bin/npx on macOS).

Tool calls timing out or failing to download packages?

First-time package downloading might be slow or timeout due to network limits. Configure a reliable mirror or check outbound proxy settings.

Agent does not match or apply Crawl4ai during conversations?

Verify SKILL.md is located at .cursor/skills/Crawl4ai/SKILL.md (Cursor) or ~/.claude/skills/Crawl4ai/SKILL.md (Claude Code). You can explicitly mention the skill name in your prompt to boost match confidence.

Still not connecting? Read the MCP troubleshooting guide →

Tool Mock Playground

Sandbox

Crawl4ai Rule Injection & Agent Behavior Simulation · Preview tool schema & outputs without local runtime

Sandbox Ready

fn: agent_skill_runtimeInject system prompt guidelines & workflow constraints from Crawl4ai

Request ArgumentsJSON Schema

{
  "skill": "Crawl4ai",
  "rulePath": ".cursor/skills/crawl4ai/SKILL.md",
  "userGoal": "请使用 Crawl4ai 的方法与标准帮我完成当前任务",
  "client": "Cursor / Claude Code"
}

💡Parameters generated dynamically by Agent runtime

Agent Tool Output

Click 'Run Mock' above

to preview the raw response returned to the LLM

Env: AgentHub Virtual SandboxJSON-RPC 2.0

Decision Guide: Why & When to Use

Assess suitability before installing to save trial-and-error time

Best Suited For

Doc/news harvesting
Competitor page monitoring
RAG corpora

When NOT to Use

Authenticated sensitive scraping
Violating robots.txt or site ToS

Recommended Workflow Pairing:View Scenario →

Firecrawl / Fetch MCP + vector store → ingest and retrieve

Skill hands-on: install to visible results

Follow the full lab (expected UI/output + contrast checks). After installing this item, verify with the tutorial prompts.

Open tutorial →

An Agent Skill is a folder with SKILL.md instructions. After install, the AI loads it automatically when your task matches its description.

How to use

After installing, describe your task in chat (or mention the skill name). The agent reads the skill description to decide when to activate it, then follows the instructions in SKILL.md. Check loaded skills via /skills in Claude Code or in your client settings.

Related resources

API Gateway

v1.2.25

SkillClawHub87.2k

io.clawhub.byungkyu/api-gateway

Call third-party APIs through the Maton gateway, which injects the credential for an app the user has already connected. Use this skill when the user names a connected app and a concrete action in it - read a mailbox, query a CRM, file an issue, update a spreadsheet, etc.

source

View on ClawHub

v1.2.6

SkillClawHub13.5k

io.clawhub.byungkyu/linkedin-api

LinkedIn API integration with managed OAuth. Share posts, manage profile, and access LinkedIn features. Use this skill when users want to share content on LinkedIn, get profile/organization information, or interact with LinkedIn's platform.

source

View on ClawHub

WhatsApp Business

v1.2.8

SkillClawHub24.1k

io.clawhub.byungkyu/whatsapp-business

WhatsApp Business API integration with managed OAuth. Send messages, manage templates, and handle conversations. Use this skill when users want to interact with WhatsApp Business. For other third party apps, use the api-gateway skill (https://clawhub.ai/byungkyu/api-gateway).

source

View on ClawHub

Salesforce

v1.2.7

SkillClawHub20.1k

io.clawhub.byungkyu/salesforce-api

Salesforce CRM API integration with managed OAuth. Install only if you need Salesforce CRM administration. Connect with the narrowest Salesforce permissions available, prefer sandbox orgs for destructive or batch work, verify the intended connection ID before each request, etc.

source

View on ClawHub

腾讯云 CloudBase · Tencent CloudBase

v1.92.120

SkillClawHub9k

io.clawhub.binggg/cloudbase

Use this skill when you develop, design, build, deploy, debug, migrate, or troubleshoot CloudBase (腾讯云开发, 云开发, TCB, 微信云开发) projects — Web, 微信小程序, 小程序, uni-app, mobile (iOS, Android, Flutter, React Native). Covers UI (页面, 界面, 表单, dashboard, prototype, 原型); auth (登录, 注册, OAuth, publishable key); databases (NoSQL 文档数据库, MySQL 关系型数据库, PostgreSQL/CloudBase PG, app.rdb(), queryPgDatabase/managePgDatabase, CRUD, security rules); 云函数/cloud functions (serverless, scf_bootstrap); CloudRun (云托管, Dockerfile); 云存储; built-in AI (内置大模型, AI 对话, streaming, 流式输出, 图片生成, generateText, streamText, createModel, generateImage, TokenHub, Hunyuan, DeepSeek, GLM, Kimi, Token Credits 资源包, 小程序成长计划); third-party/custom model onboarding (第三方大模型接入, 大模型调用, LLM API); AI agent (智能体, AG-UI, LangGraph); ops troubleshooting (巡检, 诊断, 日志); spec workflow (需求文档, 技术方案, requirements, tasks.md). Do NOT use for non-CloudBase projects, pure frontend without CloudBase, or self-hosted backends without CloudBase.

source

View on ClawHub

ai-insurance-advisor

v2.0.125

SkillClawHub7.9k

io.clawhub.mnetfairy/ai-insurance-advisor

中国大陆保险顾问。本 skill 仅覆盖 6 项实际实现的能力：保险需求分析（needs_analyzer.py）、产品对比（premium_calculator.py + 本地 products.json）、保费计算（premium_calculator.py）、方案设计（plan_designer.py）、保险知识问答（insurance-knowledge.md）、合规要点提示（compliance.md）。不提供核保预审、理赔代办、朋友圈/营销文案生成、培训话术、代理人展业工具等能力——这些场景请转人工或调用专业服务。

source

View on ClawHub

Often paired with

GlobalGov — Government Contracts, Tenders & Procurement

v1.0.8

MCP ServerMCP Registry

io.globalgov/mcp

Find government contracts, tenders and RFPs you can still bid on, in 193 countries. Free, no key.

streamable-httpregistry listed

Fillo

v0.10.0

MCP ServerMCP Registry

io.github.jacobfunch/usefillo

Fillo MCP server — provision, scaffold, publish, and query forms from your coding agent.

stdiostreamable-httpnpmregistry listed

securityscorecard-mcp

v2.0.0

MCP ServerMCP Registry

io.github.CallMarcus/securityscorecard-mcp

Community-built, comprehensive MCP server for the SecurityScorecard API (unofficial).

stdionpmregistry listed

taxapp.nz

v1.0.0

MCP ServerMCP Registry

nz.taxapp/taxapp

Your taxapp.nz income, expenses, rental properties and NZ tax figures, and adding new records.

streamable-httpregistry listed

Keep exploring AgentHub

Most people compare similar tools or check scenario guides before installing—start here.

Browse more like this Search similar Pick by scenario Prompt playbooks Skill hands-on Reviews & comparisons

Listing badge: put AgentHub on your site

Copy either snippet into your project homepage, docs, or GitHub README. The badge is a hotlinked SVG — nothing to host — and it links back to this page so visitors can find the install steps.

Preview

HTML

<a href="https://myagenthub.cn/p/io.clawhub.codylrn804/crawl4ai" title="Listed on AgentHub: Crawl4ai" target="_blank" rel="noopener">
  <img src="https://myagenthub.cn/badge/io.clawhub.codylrn804/crawl4ai?lang=en" alt="Listed on AgentHub: Crawl4ai" height="20" style="border:0"/>
</a>

Markdown (GitHub README)

[![Listed on AgentHub: Crawl4ai](https://myagenthub.cn/badge/io.clawhub.codylrn804/crawl4ai?lang=en)](https://myagenthub.cn/p/io.clawhub.codylrn804/crawl4ai)

Badges are generated on the fly from /badge/<package-id>, so name and listing changes propagate automatically. Keep the link target unchanged — it is what counts as the referral.

Unified Manifest

{
  "id": "io.clawhub.codylrn804/crawl4ai",
  "type": "skill",
  "version": "1.0.0",
  "displayName": "Crawl4ai",
  "description": "AI-powered web scraping framework for extracting structured data from websites. Use when Codex needs to crawl, scrape, or extract data from web pages using AI-powered parsing, handle dynamic content, or work with complex HTML structures.",
  "author": {
    "name": "codylrn804",
    "url": "https://clawhub.ai/codylrn804"
  },
  "homepage": "https://clawhub.ai/codylrn804/crawl4ai",
  "distribution": {
    "packages": [
      {
        "registryType": "source",
        "identifier": "codylrn804/crawl4ai",
        "version": "1.0.0",
        "runtimeHint": "clawhub install"
      }
    ],
    "remotes": []
  },
  "dependencies": [],
  "installTargets": [
    "openclaw",
    "claude-code",
    "claude-desktop",
    "cursor",
    "codex",
    "vscode"
  ],
  "keywords": [
    "downloads:5015",
    "stars:2",
    "installs:121",
    "Web Scraping",
    "Scrape"
  ],
  "provenance": {
    "origin": "clawhub",
    "originalId": "codylrn804/crawl4ai",
    "originalUrl": "https://clawhub.ai/codylrn804/crawl4ai",
    "isOfficial": false,
    "status": "active"
  }
}