Lessons from building evals in Healthcare at Scale with Clara Matos скачать в хорошем качестве

Lessons from building evals in Healthcare at Scale with Clara Matos

AI is transforming healthcare delivery by enabling more personalized, effective and efficient care at scale. However, deploying these models in a highly regulated, safety-critical environment introduces unique challenges, especially when it comes to ensuring consistency, reliability, and alignment with clinical standards. In this practical, example-driven talk, Clara shares how to evaluate Health AI products throughout their lifecycle: from development to deployment and continuous improvement in production. 00:15 Why evals matter in healthcare (safety, reliability, usefulness) 02:52 Why evals are crucial for deploying LLMs in production 03:19 Benefits: iteration, regression detection, model comparison, cost savings 04:11 Offline eval loop: dataset → evaluators → V-check → release 05:01 Building eval datasets: not just random samples (rare + hard + regressions + tool/RAG branches) 06:05 What an eval item looks like (context + transcript) + continuous dataset updates 06:36 Defining criteria: binary outputs (pass/fail) to reduce ambiguity 07:09 Human annotation for expert alignment (clinical/product reviewers) 08:02 Building evaluators: human vs code-based vs LLM-as-judge 08:47 Human pairwise comparisons + internal tooling for release decisions 09:15 Code-based evaluators: scalable checks (example: character limits) 09:48 LLM-as-judge: prompt design + train/dev/test split for alignment 11:21 Iterating on judge prompts: disagreement analysis → refine → validate on test set 12:41 Why manual V-check still matters (metrics can improve but quality can regress) 14:08 Online evals: continuous evaluation in production 14:29 Guardrails as real-time safety checks (code + LLM judge for critical cases) 15:42 A/B tests in LLM products: metrics, lift, stakeholder patience, statistical constraints 17:03 Manual audits: high-ROI pattern detection + root cause analysis 17:38 Observability: log inputs/metadata/outputs/RAG docs/tools/feedback/evals 18:16 Running evaluators on production traces + alerting on pass-rate drops 19:10 Closing: evals are continuous — systems must improve with every interaction Thank you to all our partner to make this happen! A big thanks to our gold sponsors for believing in us: Uphold: Founded in 2013, Uphold is a digital wallet and trading platform that makes cryptocurrencies and other assets affordable and accessible to everyone. With coverage of 300+ assets, Uphold allows users to move seamlessly between digital and traditional currencies, enabling borderless access to financial services you can’t get through your bank. Their Anything-to-Anything interface lets anyone fund, trade, or send money globally in just one tap. Check it here: https://uphold.com Datalinks: DataLinks recognizes the importance of nurturing the AI ecosystem. They bring ontologies and knowledge graphs to Lisbon AI, redefining data engineering and shaping the future of Agentic Workflows and Vertical Search. Discover how scattered data can be unified and linked to power your agents and backends, all with a single click and some prompts: https://datalinks.com/ Follow Clara: https://x.com/clarafrmatos Follow ‪@sword_health‬: https://x.com/SwordHealth Follow us on X: https://x.com/lisbonai_ Follow us on LinkedIn:   / lisbon-ai   Opening music: ‪@NIN‬​