pm-ai

Review an LLM agent design and find where it will be unreliable, expensive, or unsafe. Use when asked to review an agent architecture, critique a multi-step/tool-using agent, debug an agent that loops or goes off-task, or harden an agent before launch. Produces a structured review — task fit, control flow, tools, memory/context, failure handling, cost, and safety — with prioritised findings and fixes.

Specify an autonomous or tool-using AI agent before building it. Use when asked to design an AI agent, define an agent's tools and guardrails, scope what an agent is allowed to do, or write an agent spec/PRD. Produces an agent spec — goal & scope, tools with permissions, the control loop, guardrails & approval gates, memory, escalation/handoff, evaluation, and failure handling.

Design an evaluation plan for an LLM or AI feature before shipping it. Use when asked how to evaluate a prompt/model/agent, set up an eval harness, define quality metrics for an AI feature, or build a regression gate. Produces an eval plan — task definition, datasets, metrics & rubrics, baselines, automated + human evals, a pass bar, and a regression gate.

Write a PRD for an AI-powered feature, covering the things normal PRDs miss. Use when asked to spec an AI/LLM feature, write a PRD for a feature that uses a model, or plan an AI capability (assistant, summarizer, generator, classifier). Produces an AI feature PRD — problem & UX of uncertainty, model approach, eval criteria, guardrails, fallback behaviour, the data flywheel, and cost/latency budget.

Document a dataset so others know what it is, how it was made, and when not to use it. Use when asked to write a datasheet for a dataset, document training/eval data, or assess whether a dataset is fit for a use. Produces a datasheet — motivation, composition, collection process, preprocessing, recommended uses & limits, distribution, and maintenance.

Design a scoring rubric and LLM-as-judge prompt to evaluate the quality of an AI feature's output. Use when asked to create an eval rubric, define quality dimensions, build an LLM judge, or decide how to measure whether AI output is good. Produces a rubric with weighted dimensions and concrete 1–5 anchors, a ready-to-run judge prompt, a labelling guide, and notes on judge reliability.

Model the cost and latency of an LLM feature before it ships and surprises the bill. Use when asked to estimate LLM API costs, set a latency/token budget, decide which model tier to use, or bring down the cost of an AI feature. Produces a cost & latency budget — token math per request, monthly cost projection, model tiering, caching/streaming levers, p95 latency targets, and a guardrail/alert plan.

Specify the safety and reliability guardrails for an LLM feature before it ships. Use when asked to define LLM guardrails, add safety controls to an AI feature, prevent prompt injection or jailbreaks, or harden a chatbot/agent against misuse. Produces a guardrails spec — threats, input/output controls, refusal and escalation policy, logging, and a red-team test set — mapped to where each control runs.

Document a deployed ML/AI model so others can use it responsibly. Use when asked to write a model card, document a model's intended use and limitations, or prepare an AI model for review/launch. Produces a complete model card — intended use, training data, evaluation metrics across slices, limitations, ethical considerations, and a deployment checklist.

Choose the right LLM for a task by trading off quality, cost, latency, and constraints. Use when asked which model to use, whether to upgrade/downgrade a model, how to cut LLM costs without hurting quality, or to justify a model choice. Produces a recommendation with the decision criteria, a per-option comparison, a routing strategy (cheap-by-default, escalate when needed), and how to validate the choice with an eval.

Diagnose and rewrite an underperforming LLM prompt so it produces reliable, well-structured output. Use when asked to improve a prompt, fix a prompt that gives inconsistent or wrong results, reduce hallucination/refusals, or make output follow a format. Produces a rewritten prompt with a diagnosis of what was failing, the specific changes and why, and a small test set to verify the fix.

Review an existing Retrieval-Augmented Generation system and find why it underperforms. Use when asked to review or audit a RAG pipeline, diagnose wrong/ungrounded answers from a 'chat with your docs' feature, or improve an already-built knowledge assistant. Produces a staged review — ingestion, chunking, retrieval, reranking, generation, evaluation — with prioritised findings, root causes, and concrete fixes.

Design a Retrieval-Augmented Generation system end to end. Use when asked to design a RAG pipeline, a 'chat with your docs' feature, a knowledge assistant, or to debug why a RAG system gives wrong/ungrounded answers. Produces a RAG design doc — ingestion & chunking, embeddings & index, retrieval & reranking, the generation prompt, grounding/citations, evaluation, and failure modes with mitigations.