Agent Plugins Marketplace
← All plugins

crawlberg

v0.2.3

Web crawling and scraping with HTML→Markdown and headless-Chrome fallback.

Codex7 Skills1 MCP serverstdio

By Kreuzberg, Inc.License: MIT1.1k GitHub starsUpdated 13 hours ago

Directory evidence

Runtimes
Codex
Parsed components
8 skill or MCP entries
Source updated
Sep 23, 2026
Manifest status
Canonical path parsed

The directory validates manifest shape and source location. It does not execute the plugin or provide a security endorsement. Review the indexing methodology →

Install crawlberg for Codex

Installs for the current user
codex plugin marketplace add hashgraph-online/awesome-codex-plugins
codex plugin marketplace upgrade awesome-codex-plugins
codex plugin add crawlberg@awesome-codex-plugins

Paste and run these commands in a terminal with Codex. They add and refresh the awesome-codex-plugins catalog, then install this plugin.

Compatibility: the page URL and API slug “crawlberg” remain stable.

  • Codex: crawlberg@agent-plugin-marketplace → crawlberg@awesome-codex-plugins

The installer fetches third-party code from the source repository shown on this page. This directory validates manifest structure and source location, but does not perform a security audit; review the manifest, components, and source before installing.

Get the source manually
git clone https://github.com/hashgraph-online/awesome-codex-plugins

Clone the source repository, then follow its setup instructions to add the plugin to a compatible client. The plugin root is plugins/kreuzberg-dev/plugins/plugins/crawlberg/.

Plugin files

plugins/kreuzberg-dev/plugins/plugins/crawlberg/
├── .codex-plugin/plugin.json
├── skills/automating-the-browser/SKILL.md
├── skills/crawlberg/SKILL.md
├── skills/crawling-a-site/SKILL.md
├── skills/headless-fallback/SKILL.md
├── skills/mapping-urls/SKILL.md
├── skills/scraping-html-to-markdown/SKILL.md
├── skills/serving-the-api/SKILL.md
└── .mcp.json

Included Skills7

automating-the-browserskills/automating-the-browser/SKILL.md

Use when extracting a page needs scripted interaction first — click, type, press a key, scroll, wait, screenshot, or run JS before capturing the DOM. Covers `crawlberg interact URL --actions` with the real action schema, result shape, limits, and external-CDP options.

crawlbergskills/crawlberg/SKILL.md

Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, batch-scrape, batch-crawl, download, citations, version, mcp, serve), output formats (JSON + Markdown), browser fallback, and when to prefer the MCP server over shelling out.

crawling-a-siteskills/crawling-a-site/SKILL.md

Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.

headless-fallbackskills/headless-fallback/SKILL.md

Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.

mapping-urlsskills/mapping-urls/SKILL.md

Use when the user wants the list of URLs on a site rather than the page content — sitemap analysis, link planning, or seeding another tool. Covers `crawlberg map URL` with `--limit`, `--search`, robots, output, and how it differs from a full crawl.

scraping-html-to-markdownskills/scraping-html-to-markdown/SKILL.md

Use when the user wants a single page rendered as clean Markdown plus structured metadata. Covers `crawlberg scrape URL`, JSON vs Markdown output, what metadata is returned, and how to handle JS-heavy pages.

serving-the-apiskills/serving-the-api/SKILL.md

Use when the user wants a long-running HTTP service for scrape/crawl/map instead of one-shot CLI calls or the MCP server — for example wiring crawlberg into other apps over REST. Covers `crawlberg serve`, the Firecrawl-v1-compatible endpoints, `--host`/`--port`, and when to prefer it.

MCP servers1

crawlbergstdio
command
./scripts/mcp-launch.sh
args
mcp

Plugin manifests1

plugins/kreuzberg-dev/plugins/plugins/crawlberg/.codex-plugin/plugin.json
{
  "name": "crawlberg",
  "version": "0.2.3",
  "description": "Web crawling and scraping with HTML→Markdown and headless-Chrome fallback.",
  "author": {
    "name": "Kreuzberg, Inc.",
    "email": "[email protected]",
    "url": "https://xberg.io"
  },
  "homepage": "https://xberg.io",
  "repository": "https://github.com/xberg-io/plugins",
  "license": "MIT",
  "keywords": [
    "web-scraping",
    "crawling",
    "html-to-markdown",
    "headless-chrome"
  ],
  "skills": "./skills/",
  "mcpServers": "./.mcp.json",
  "interface": {
    "displayName": "Crawlberg",
    "shortDescription": "Crawl and scrape sites to clean markdown",
    "longDescription": "Scrape single pages, crawl entire sites, or map a domain's URLs through the local crawlberg CLI. Every page is converted to clean Markdown via the built-in HTML→Markdown engine. WAF-blocked or JS-heavy pages transparently fall back to a headless Chrome backend.",
    "developerName": "Kreuzberg, Inc.",
    "category": "web-scraping",
    "capabilities": [
      "Read",
      "Write"
    ],
    "defaultPrompt": [
      "Crawl this site to markdown",
      "Scrape a single page",
      "Fall back to headless mode"
    ],
    "websiteURL": "https://xberg.io",
    "privacyPolicyURL": "https://xberg.io/privacy",
    "termsOfServiceURL": "https://xberg.io/terms",
    "brandColor": "#7C3AED",
    "composerIcon": "./assets/icon.svg",
    "logo": "./assets/logo.png"
  }
}

For maintainers

If you maintain this plugin, link to this source-backed listing from your README so users can review its manifest and indexed components.

[crawlberg on Agent Plugins Marketplace](https://pluginsmp.com/plugins/crawlberg)