A Deep Dive Into Automated RAG Evaluation with open-rag-eval скачать в хорошем качестве

A Deep Dive Into Automated RAG Evaluation with open-rag-eval

Confidence in RAG output is non-negotiable, but manually providing golden chunks and golden answers is slow and subjective. This talk explores `open-rag-eval`, an open-source framework designed to automate RAG evaluation and build trust at scale. Building on pioneering research from the University of Waterloo, this framework integrates innovative tools like UMBRELA for reference-free relevance scoring, AutoNuggetizer for automated fact-checking, and HHEM for hallucination detection. Designed with a flexible connectors architecture, it seamlessly plugs into any RAG pipeline while delivering fast, transparent, and interpretable metrics on retrieval, generation, and hallucination in RAG. Join us for a live demo to discover how `open-rag-eval` can help you enhance RAG response quality, and accelerate your RAG development efforts. ⏱️ Chapters 00:00 – Intro: Why RAG Evaluation Is So Hard 00:50 – The Golden Answers Problem 01:45 – What Is open-rag-eval? 02:50 – Live Demo: Side-by-Side RAG Comparison Dashboard 04:20 – UMBRELA: Reference-Free Relevance Scoring 05:30 – AutoNuggetizer: Atomic Fact Detection Without Labels 06:40 – Metrics: Citation Faithfulness & Hallucination Detection 08:10 – How It Works Without Golden Chunks or Answers 09:10 – Flexible Connectors: LangChain, LlamaIndex & More 10:15 – In-Progress: Response Consistency Metric (Preview) 11:30 – How To Get Started With open-rag-eval 12:30 – Q&A: Chunking Strategies & Multimodal RAG 🔗 More on open-rag-eval: https://github.com/vectara/open-rag-eval 📌 Learn more about Arize community events: https://arize.com/community/