data-pipeline plugins

by ScaleVector GmbH

Deploy a GitHub issues pipeline to dltHub Platform in one shot

by ScaleVector GmbH

Build REST API pipelines with dlt: scope, debug and validate data

by ScaleVector GmbH

Build SQL database pipelines with dlt: connect to any SQL source, load tables to a destination, tune performance with backends

by Matt Corbett

Data-orchestration team — agents (orchestration-architect, pipeline-orchestration-engineer) for the layer that RUNS and SCHEDULES data pipelines: orchestrator selection (Airflow, Dagster, Prefect, Mage, Temporal-for-data; cloud-native MWAA/Step Functions, Azure Data Factory, Cloud Composer/Workflows), DAG and software-defined-asset design, scheduling/triggers (cron, sensor, data-aware), backfills and catchup, idempotency and retries with exponential backoff, partitioning, freshness SLAs and alerting, and lineage. skills (choose-orchestrator, design-dag-and-dependencies, handle-backfills-and-retries), a knowledge bank (a Mermaid selection tree + a 2026 patterns reference), and templates. Distinct from data-platform (ELT/warehouse/BI), analytics-engineering (dbt), and data-streaming-engineering (real-time) — this is the scheduling/run layer they plug into. Requires ravenclaude-core@>=0.7.0.

by Matt Corbett

Web-scraping & data-extraction engineering team — agents (extraction-architect, scraper-implementation-engineer) answering 'how do we acquire this web data legally, robustly, and at the right cost?': API-vs-scrape source choice, a legal/ethical gate (robots.txt, ToS, public-vs-authenticated, rate, PII/copyright) BEFORE any code, fetch strategy (HTTP vs headless, JSON-endpoint-first), resilient parsing (structured data over brittle selectors, validate-to-schema), change-detection & re-crawl scheduling, and the extract→validate→dedup→store pipeline with provenance. skills, a knowledge bank (a Mermaid decision tree + a dated 2026 tooling/legal reference). Distinct from data-orchestration/data-streaming (move data that ALREADY exists) and backend-engineering — this team owns ACQUIRING web data. Legality-first: anti-bot is a stop-signal, never evasion for abuse; ToS/robots specifics volatile + jurisdictional, retrieval-dated + verify-at-use. Needs ravenclaude-core.