plugin-eval

v0.1.2

Evaluate Codex skills and plugins from chat with a beginner-friendly start command, local-first reports, token budget explanations, and guided benchmarking.

Codex5 skills

by OpenAI CodexMIT5kupdated 3 weeks ago

Source

$ git clone https://github.com/openai/plugins

Clone the source, then follow the repository's marketplace instructions for your runtime. The plugin root is plugins/plugin-eval/ inside the repository.

Layout

plugins/plugin-eval/

├── .codex-plugin/plugin.json
├── skills/evaluate-plugin/SKILL.md
├── skills/evaluate-skill/SKILL.md
├── skills/improve-skill/SKILL.md
├── skills/metric-pack-designer/SKILL.md
└── skills/plugin-eval/SKILL.md

Skills5

evaluate-pluginskills/evaluate-plugin/SKILL.md

Evaluate a local Codex plugin in engineer-friendly language. Use when the user says "evaluate this plugin", "audit this plugin", "why did this score that way", "what should I fix first", "help me benchmark this plugin", or asks for a plugin-wide report before comparing versions.

evaluate-skillskills/evaluate-skill/SKILL.md

Evaluate a local Codex skill in engineer-friendly terms. Use when the user says "evaluate this skill", "give me an analysis of the game dev skill", "audit this skill", "why did this score that way", "what should I fix first", or asks for a skill-specific report before benchmarking it.

improve-skillskills/improve-skill/SKILL.md

Turn plugin-eval findings into a concrete rewrite brief for a Codex skill. Use when the user already evaluated a skill and now wants Codex to improve it, especially after asking what to fix first.

metric-pack-designerskills/metric-pack-designer/SKILL.md

Design custom metric packs for plugin-eval so teams can add local evaluation rubrics that emit schema-compatible checks and metrics. Use when the user wants their own evaluation criteria or visualizations.

plugin-evalskills/plugin-eval/SKILL.md

Help engineers evaluate a local skill or plugin, explain why it scored that way, show what to fix first, measure real token usage, benchmark starter scenarios, or decide what to run next. Use when the user says things like "evaluate this skill", "give me an analysis of the game dev skill", "why did this score that way", "what should I fix first", "measure the real token usage of this skill", or "what should I run next?".

Manifests1

plugins/plugin-eval/.codex-plugin/plugin.json

{
  "name": "plugin-eval",
  "version": "0.1.2",
  "description": "Evaluate Codex skills and plugins from chat with a beginner-friendly start command, local-first reports, token budget explanations, and guided benchmarking.",
  "author": {
    "name": "OpenAI Codex",
    "email": "support@openai.com",
    "url": "https://openai.com/"
  },
  "homepage": "https://openai.com/",
  "repository": "https://github.com/openai/plugins",
  "license": "MIT",
  "keywords": [
    "codex",
    "plugin",
    "skill",
    "evaluation",
    "quality",
    "budget"
  ],
  "skills": "./skills/",
  "interface": {
    "displayName": "Plugin Eval",
    "shortDescription": "Start from chat, then evaluate or benchmark locally",
    "longDescription": "Ask Codex to evaluate a plugin or skill, give you a full analysis of a named plugin such as game-studio, explain why it scored that way, show what to fix first, explain its token budget, measure real token usage, benchmark a plugin, or tell you what to run next. Plugin Eval keeps the path engineer-friendly: start with a natural chat request, then use the local `plugin-eval start` entrypoint or the routed workflow command it recommends.",
    "developerName": "OpenAI",
    "category": "Developer Tools",
    "capabilities": [
      "Interactive",
      "Write"
    ],
    "websiteURL": "https://openai.com/",
    "privacyPolicyURL": "https://openai.com/policies/privacy-policy/",
    "termsOfServiceURL": "https://openai.com/policies/terms-of-use/",
    "defaultPrompt": [
      "Give me an analysis of the game studio plugin.",
      "Evaluate this plugin.",
      "Why did this score that way?"
    ],
    "brandColor": "#0F766E",
    "composerIcon": "./assets/plugin-eval.svg",
    "logo": "./assets/plugin-eval.svg",
    "screenshots": []
  }
}