Agent Plugins Marketplace
← All plugins

eval-runner

v0.1.0

Scaffolding, running, documenting, and publishing AI evaluations. Ships skills and commands for setting up eval workspaces, creating custom evals (or adapting existing frameworks/benchmarks), running them, and publishing evals or datasets. Bundles a curated ground-truth list of open-source eval tools and benchmarks as a reference data source.

Claude Code6 Skills

By Daniel RosehillLicense: MIT4 GitHub starsUpdated 5 months ago

Directory evidence

Runtimes
Claude Code
Parsed components
6 skill or MCP entries
Source updated
Apr 24, 2026
Manifest status
Canonical path parsed

The directory validates manifest shape and source location. It does not execute the plugin or provide a security endorsement. Review the indexing methodology →

Install eval-runner for Claude Code

Installs for the current user
claude plugin marketplace add IchenDEV/agent-plugin-mkt
claude plugin marketplace update agent-plugin-marketplace
claude plugin install eval-runner@agent-plugin-marketplace

Paste and run these commands in a terminal with Claude Code. They add and refresh the PluginsMP catalog, then install this plugin.

The installer fetches third-party code from the source repository shown on this page. This directory validates manifest structure and source location, but does not perform a security audit; review the manifest, components, and source before installing.

Get the source manually
git clone https://github.com/danielrosehill/Claude-Eval-Runner-Plugin

Clone the source repository, then follow its setup instructions to add the plugin to a compatible client. The repository root is the plugin root.

Plugin files

eval-runner/
├── .claude-plugin/plugin.json
├── skills/create-eval/SKILL.md
├── skills/new-workspace/SKILL.md
├── skills/publish-dataset/SKILL.md
├── skills/publish-eval/SKILL.md
├── skills/run-eval/SKILL.md
└── skills/setup-eval/SKILL.md

Included Skills6

create-evalskills/create-eval/SKILL.md

Design a custom eval from scratch, or remix an existing benchmark. Use when the user wants to define the eval itself — task framing, dataset composition, scoring rubric, and reporting format — rather than simply wiring up a framework. Produces a fully specified eval definition ready to be run.

new-workspaceskills/new-workspace/SKILL.md

Provision a new eval-runner workspace on disk. Use when the user wants to start a new evaluation project — scaffolds evals/, datasets/, results/, and docs/ directories, personalises CLAUDE.md, and (by default) creates a GitHub repo.

publish-datasetskills/publish-dataset/SKILL.md

Publish an eval dataset to Hugging Face Hub (or GitHub as a fallback). Use when the user wants to share the inputs/labels used by an eval — with a dataset card, licensing, splits, and a content hash so downstream runs can verify integrity.

publish-evalskills/publish-eval/SKILL.md

Publish an eval (definition + results) so others can reproduce it. Use when the user wants to share an eval publicly — as a GitHub repo, Hugging Face space, or a standalone writeup. Produces a clean, self-contained bundle with README, task spec, rubric, dataset pointer, and a run report.

run-evalskills/run-eval/SKILL.md

Execute an eval defined in the current workspace and capture results with full metadata. Use when the user wants to actually run an eval (one or many SUTs), collect scored outputs under results/, and produce a run manifest so findings are reproducible and comparable over time.

setup-evalskills/setup-eval/SKILL.md

Set up an evaluation in the current workspace. Use when the user wants to scaffold a single eval — choosing an existing framework (DeepEval, Inspect AI, OpenAI Evals, lm-evaluation-harness, LightEval, OLMES, Promptfoo, etc.), adapting an existing benchmark, or preparing a clean slate for a custom eval. Writes a brief, installs deps, and stubs the eval directory.

Plugin manifests1

.claude-plugin/plugin.json
{
  "name": "eval-runner",
  "version": "0.1.0",
  "description": "Scaffolding, running, documenting, and publishing AI evaluations. Ships skills and commands for setting up eval workspaces, creating custom evals (or adapting existing frameworks/benchmarks), running them, and publishing evals or datasets. Bundles a curated ground-truth list of open-source eval tools and benchmarks as a reference data source.",
  "author": {
    "name": "Daniel Rosehill",
    "url": "https://github.com/danielrosehill"
  },
  "repository": "https://github.com/danielrosehill/Claude-Eval-Runner-Plugin",
  "license": "MIT",
  "keywords": [
    "evaluation",
    "evals",
    "benchmark",
    "llm",
    "rag",
    "agent-eval",
    "dataset",
    "inspect-ai",
    "deepeval",
    "openai-evals"
  ]
}

For maintainers

If you maintain this plugin, link to this source-backed listing from your README so users can review its manifest and indexed components.

[eval-runner on Agent Plugins Marketplace](https://pluginsmp.com/plugins/eval-runner)