llmops plugins

by Matt Corbett

LLM-evaluation engineering team — agents (eval-strategy-lead, eval-harness-engineer) for the question every team shipping an AI feature has to answer: is it getting better or worse? Eval design (task-grounded metrics, offline-vs-online split, sample size, a ship-gate decided as a number before the run), and the harness that proves it (frozen golden sets with provenance, LLM-as-judge rubrics with a bias audit and human calibration, CI regression gates, guardrail/red-team suites). skills, a decision-tree knowledge bank (an eval-method Mermaid tree + a dated 2026 tooling/method map), best-practices. Engineering decision-support, not a safety certification; model/judge/tooling specifics are volatile — retrieval-dated + verify-at-use; no eval-data PII. Seams: retrieval quality -> ai-rag-engineering; model selection -> ai-coding-model-guidance; classical-ML training/metrics -> ml-engineering; product A/B -> experimentation-growth-engineering. Requires ravenclaude-core@>=0.7.0.

by yukineko

External-benchmark (SWE-bench Verified) runner for Claude Code plugins: a typed SWE-bench Instance model, a deterministic fixture-based JSONL loader, and a gated `download` subcommand that shells out to `curl` (no HTTP crate) to fetch the Princeton NLP dataset only on explicit invocation. Subscription-native (one bundled Rust binary).

by yukineko

Real-time cost budget gate for Claude Code: on Stop, read the session transcript, compute USD cost, and block the turn when session or daily limits are exceeded — feeding the overage back so the agent winds down gracefully. Subscription-native (one Stop hook + bundled Rust binary, no API key).

by yukineko

Promote fugu-router playbooks into versioned golden eval datasets for evalkit: turn the append-only routing log into a curated, deduplicated holdout set. Mechanical acceptance criteria (`cargo test`, a backticked command) become runnable golden cases; the rest become drafts for a human to fill. The supply side of the offline eval loop. Subscription-native (one bundled Rust binary, no API key).

by yukineko

Offline golden-regression eval harness for Claude Code plugins: read golden *.jsonl cases (file-content or CLI-output assertions) and exit non-zero when a prompt edit or CLI contract regresses. The offline sibling of condukt's online verifier — runnable as a CI gate with no API key. Subscription-native (one bundled Rust binary).

by yukineko

Local LLMOps telemetry for Claude Code: a Stop hook that reads each session's transcript and records token usage, cache hits, tool calls, latency, and estimated cost to a local store — then `gauge report` rolls it up by project, model, and day. `gauge subagents` attributes cost to each individual sub-agent (Task) live from its transcript, so callers like condukt can record true per-task cost instead of a lumped session total. v0.3.9: `gauge subagents --json` now also emits `tokens_input`/`tokens_output` per sub-agent, so callers can record real token usage alongside cost. Observability for your own agent runs, subscription-native (one hook + bundled Rust binary, no API key, nothing leaves the machine).

by yukineko

Trace→golden replay regression harness: convert tracekit-recorded condukt run traces into evalkit golden replay cases. The sibling of curate (playbook→golden) — three subcommands (extract/verify/promote) pin a run's phase set, error count, and cost as a portable, self-verifying snapshot so regressions surface as a failing golden. Subscription-native (one bundled Rust binary, no API key).

by yukineko

Schema-validation gate for LLM structured outputs at source→executor boundaries: validates named declared schemas, emits structured errors for one re-ask, and counts rejects to metrics so silent drops become observable. Subscription-native (one bundled Rust binary, no API key).

by yukineko

Span-tree tracer for condukt runs: record one run's interpreter→worker→verifier phases as a parent-linked span tree (phase, model, ms, cost, status), render it with `tracekit trace <RID>`, and export OpenTelemetry GenAI-semconv JSON. Answers which phase of a failed run was slow/expensive/broke. Subscription-native (one bundled Rust binary, file-only, no network, no API key).

by yukineko

Trajectory-match verifier sibling: checks the PATH a worker took (its ordered tool-call sequence) against an expected trajectory spec, in strict/unordered/subsequence modes. v0.1.7: adds a tolerance-based `fuzzy_hash` diff strategy — `TierConfig::threshold_permille` config plus a pure `fuzzy_diff()` computing a deterministic permille drift distance (differing leaves / total leaves) that tolerates drift at or under the threshold and escalates to `DiffOutcome::DriftedBeyondThreshold` above it; distance is stored as `u32` to keep the `Eq` derives on `DiffOutcome`/`TierVerdict` intact (no `f64`). v0.1.6: the seeded 1-in-N sampling hash now delegates to `harness_core::hash::fnv1a64` (the single canonical FNV-1a implementation) instead of a private reimplementation, closing a code-duplication finding; hash values are unchanged (same algorithm/constants), so sampling behavior is bit-for-bit identical. v0.1.5: `tier` subcommand is now e2e-tested end-to-end via a committed example core-flow allowlist (examples/tier-config.json); seeded 1-in-N non-core sampling has a rate-pinning test (catches subtler mutations than mere non-constancy); a core flow using the unimplemented `screenshot` diff strategy now yields an explicit tri-state `needs-human` verdict (distinct exit code 3) instead of silently masquerading as a hard diff failure. v0.1.4: adds risk-tiered e2e verification (`tier` subcommand) — a config-driven core allowlist where core flows get an every-run deterministic structured-data snapshot diff (perceptual-hash/screenshot behind a documented stub boundary) and non-core flows get an existence check or seeded low-frequency sampling. Subscription-native (one bundled Rust binary, no API key).