llm-evaluation-engineering

v0.1.2

LLM-evaluation engineering team — agents (eval-strategy-lead, eval-harness-engineer) for the question every team shipping an AI feature has to answer: is it getting better or worse? Eval design (task-grounded metrics, offline-vs-online split, sample size, a ship-gate decided as a number before the run), and the harness that proves it (frozen golden sets with provenance, LLM-as-judge rubrics with a bias audit and human calibration, CI regression gates, guardrail/red-team suites). skills, a decision-tree knowledge bank (an eval-method Mermaid tree + a dated 2026 tooling/method map), best-practices. Engineering decision-support, not a safety certification; model/judge/tooling specifics are volatile — retrieval-dated + verify-at-use; no eval-data PII. Seams: retrieval quality -> ai-rag-engineering; model selection -> ai-coding-model-guidance; classical-ML training/metrics -> ml-engineering; product A/B -> experimentation-growth-engineering. Requires ravenclaude-core@>=0.7.0.

Claude Code3 Skills

By Matt CorbettLicense: MIT7 GitHub starsUpdated last week

Directory evidence

Runtimes: Claude Code
Parsed components: 3 skill or MCP entries
Source updated: Sep 15, 2026
Manifest status: Canonical path parsed

The directory validates manifest shape and source location. It does not execute the plugin or provide a security endorsement. Review the indexing methodology →

Install llm-evaluation-engineering for Claude Code

Installs for the current user

claude plugin marketplace add IchenDEV/agent-plugin-mkt
claude plugin marketplace update agent-plugin-marketplace
claude plugin install llm-evaluation-engineering@agent-plugin-marketplace

Paste and run these commands in a terminal with Claude Code. They add and refresh the PluginsMP catalog, then install this plugin.

The installer fetches third-party code from the source repository shown on this page. This directory validates manifest structure and source location, but does not perform a security audit; review the manifest, components, and source before installing.

Get the source manually

$ git clone https://github.com/mcorbett51090/RavenClaude

Clone the source repository, then follow its setup instructions to add the plugin to a compatible client. The plugin root is plugins/llm-evaluation-engineering/.

Plugin files

plugins/llm-evaluation-engineering/

├── .claude-plugin/plugin.json
├── skills/build-llm-judge/SKILL.md
├── skills/design-eval-suite/SKILL.md
└── skills/gate-releases-with-evals/SKILL.md

Included Skills3

build-llm-judgeskills/build-llm-judge/SKILL.md

Design, calibrate, and bias-audit an LLM-as-judge so its scores track human judgment: named criteria with anchored levels (not 'rate 1-10'), a structured verdict + reason, a pinned judge model/version, a human-agreement calibration on a sample, and checks for position / verbosity / self-preference / leniency bias. Reach for it whenever a judge grades outputs. Used by `eval-harness-engineer` (primary).

design-eval-suiteskills/design-eval-suite/SKILL.md

Turn a fuzzy 'is the AI good enough?' into a concrete measurement plan: task spec, the eval method from the decision tree (exact / rule-based / model-graded / human), the metric + what it misses, the offline-vs-online split, a power-adequate sample size, and a numeric ship-gate decided before the run. Reach for it at the START of any LLM-feature eval. Used by `eval-strategy-lead` (primary).

gate-releases-with-evalsskills/gate-releases-with-evals/SKILL.md

Wire an offline eval suite into a CI ship-gate so a prompt/model change is scored against a frozen baseline and blocked on a regression past a preset threshold, with guardrail/red-team checks as a zero-tolerance blocker and cost/latency tracked alongside quality. Reach for it to make evals a merge gate instead of a report. Used by both agents.

Plugin manifests1

plugins/llm-evaluation-engineering/.claude-plugin/plugin.json

{
  "name": "llm-evaluation-engineering",
  "version": "0.1.2",
  "description": "LLM-evaluation engineering team — agents (eval-strategy-lead, eval-harness-engineer) for the question every team shipping an AI feature has to answer: is it getting better or worse? Eval design (task-grounded metrics, offline-vs-online split, sample size, a ship-gate decided as a number before the run), and the harness that proves it (frozen golden sets with provenance, LLM-as-judge rubrics with a bias audit and human calibration, CI regression gates, guardrail/red-team suites). skills, a decision-tree knowledge bank (an eval-method Mermaid tree + a dated 2026 tooling/method map), best-practices. Engineering decision-support, not a safety certification; model/judge/tooling specifics are volatile — retrieval-dated + verify-at-use; no eval-data PII. Seams: retrieval quality -> ai-rag-engineering; model selection -> ai-coding-model-guidance; classical-ML training/metrics -> ml-engineering; product A/B -> experimentation-growth-engineering. Requires ravenclaude-core@>=0.7.0.",
  "author": {
    "name": "Matt Corbett"
  },
  "homepage": "https://github.com/mcorbett51090/RavenClaude",
  "license": "MIT",
  "keywords": [
    "llm-evaluation",
    "llm-as-judge",
    "evals",
    "eval-harness",
    "guardrails",
    "red-teaming",
    "golden-set",
    "regression-testing",
    "offline-eval",
    "online-eval",
    "prompt-testing",
    "ship-gate",
    "llmops"
  ],
  "requires": {
    "plugins": [
      "ravenclaude-core@>=0.7.0"
    ]
  }
}

For maintainers

If you maintain this plugin, link to this source-backed listing from your README so users can review its manifest and indexed components.

[llm-evaluation-engineering on Agent Plugins Marketplace](https://pluginsmp.com/plugins/llm-evaluation-engineering)