PM Cookbook: Foundations of a Repeatable Evals Process скачать в хорошем качестве

PM Cookbook: Foundations of a Repeatable Evals Process

In this cookbook, we’ll help you lay the foundation for measuring and improving the quality of your AI agents while defining a repeatable, collaborative evaluation process across the AI development lifecycle. Using a healthcare scribing agent as an example, we demonstrate how to compare prompt versions, run automated LLM evaluations on test datasets, trace agent workflows, and monitor production logs. With these processes, product teams can measure AI quality with clear metrics, identify failure modes early, track latency and cost, and continuously evaluate production performance. If you're building LLM-powered features or agent workflows, this cookbook helps you ship reliable AI systems with confidence and control. 00:00 - Intro 01:58 - Prompt Engineering 07:42 - Offline Evals 13:30 - Analyze an Evaluation run report 18:37 - Observability 20:50 - Online Evals