Stanford Seminar - Lessons from evaluating and debugging healthcare AI in deployment скачать в хорошем качестве

Stanford Seminar - Lessons from evaluating and debugging healthcare AI in deployment

James Zou, Stanford University May 25, 2022 Dive into ai in healthcare and translating trustworthy AI from research into healthcare deployment is a major - and exciting challenge. I will discuss insights that we learned from conducting the first real-time AI trials at Stanford and analyzing data from 100 FDA-approved medical AI systems. We will explore challenges and new opportunities in each step of translation: 1. Data curation, quantifying how different data contribute to model’s success or biases 2. Model testing and monitoring, continuous real-time testing and explaining model’s mistakes 3. Human-AI interactions, designing AI for optimizing clinician’s performance. For more information about Stanford’s Artificial Intelligence professional and graduate programs, visit: https://stanford.io/ai 0:00 Introduction 2:57 Al make clinical trials more efficient 8:27 Why did the Derm Al performance crater? 14:50 Language model captures ethnic stereotypes 16:56 Two Muslims walked into... 19:50 Data used to train dermatology Al 28:56 Data Shapley Value 29:07 Dermatology classification 30:55 Shapley value identifies mis-annotations 32:23 Data Shapley improves fairness 33:24 Auditing ML data w/ data Shapley 36:20 Understanding what the network is doing 39:19 Sparse neurons responsible for prediction 42:04 Neuron Shapley identifies dataset bias 44:26 Model repair by removing bias neurons 46:30 Why did the model make this mistake? 49:50 Conceptual explanation of mistakes Mistakes made by the model 51:44 Natural language model editing reduces bias 52:01 Takeaways: challenge shifts from model training to evaluation and monitoring #artificialintelligence