How to Run Rigorous Multi-Party LLM Experiments with Crystal Qian | The Frontier Series: E1 Part 2 скачать в хорошем качестве

How to Run Rigorous Multi-Party LLM Experiments with Crystal Qian | The Frontier Series: E1 Part 2

Google DeepMind's PAIR (People + AI Research) team returns for the second half of their conversation on Deliberate Lab, the open-source platform for large-scale, real-time behavioral research on human and LLM group dynamics. Jerome Wynne, Senior AI Research Engineer at Prolific, continues his discussion with Crystal Qian, Senior Research Scientist and PAIR team lead at Google DeepMind, covering the decision to open-source Deliberate Lab and the mixed rollout strategy they landed on: a public GitHub repository alongside a Google-hosted closed allow list. We get into the practical realities of running synchronous web studies with live cohorts: why paying participants the full amount upfront outperforms staged bonuses, how the first message in a group chat sets the tone for everything that follows, and the five-minute engagement cliff that changes how you think about lobby wait times. We also discuss what's coming next for the platform, including robust simulated participants, multimodal capabilities, and the still-unsolved challenge of longitudinal studies. We zoom out to the bigger research questions Deliberate Lab could help answer, including whether AI models should be evaluated dynamically in group settings rather than static single-user benchmarks. Crystal closes with her advice for early-career researchers: Don't chase the AI frontier. Focus on the human part, because that's the part that changes slowly enough to study. This is a conversation about how to do rigorous science on a moving target, and what it actually takes to build tools that let other people do the same. 0:00 - Recap of Part 1 1:39 - The decision to open-source Deliberate Lab 3:57 - Mixed rollout: GitHub fork vs. Google-hosted allow list 6:11 - What the open-source community gave back 7:17 - What contributions would matter most 10:06 - Advice for first-time users of the platform 12:23 - What bad data looks like in live cohort studies 15:33 - Why paying upfront beats staged bonuses 17:34 - The five-minute engagement cliff 18:31 - What participants taught the team about study design 19:36 - Collective dynamics as a research frontier 22:53 - What's next: simulated participants, multimodal, longitudinal 27:26 - How Deliberate Lab could reshape model evaluation 29:03 - The case for human-centric benchmarks 30:57 - What we still don't know about LLM social capabilities 32:50 - Moving AI facilitation upstream of the conversation 35:14 - Advice for early-career researchers 37:01 - Peer review in a field that moves faster than publishing About the guest: Crystal Qian is a Senior Research Scientist at Google DeepMind, within the People + AI Research Group (PAIR). She leads a team investigating how LLMs can shape and improve social dynamics. Recent work includes simulating voting patterns in group elections, evaluating how LLM assistance can improve bargaining outcomes and group consensus, and developing scalable evaluation methods. Her current research interests involve human-AI interaction, agentic simulations, and societal impact, grounded through the analytical lens of game mechanics and behavioral experimentation. Read more of the studies from the PAIR team covered in this episode: Deliberate Lab paper on ArXiv: ⁠https://arxiv.org/pdf/2510.13011v1 To Mask or to Mirror: Human-AI Alignment in Collective Reasoning: ⁠https://aclanthology.org/2025.emnlp-m... Strategic Tradeoffs Between Humans and AI in Multi-Agent Bargaining: ⁠https://arxiv.org/abs/2509.09071 Learn more about Deliberate Lab: ⁠https://deliberate-lab.appspot.com/#/ Get the quality human data you need for AI research and development: https://www.prolific.com/ai?utm_sourc... Follow The Frontier Series: 🟣 Spotify: https://open.spotify.com/show/3adqmfo... 🟣 TikTok:   / thefrontierseries   🟣 Instagram:   / thefrontierseries   Connect with Prolific: 🔵 X:   / prolific   🔵 LinkedIn:   / prolific-com   🔵 Facebook:   / joinprolific   🔵 Instagram:   / joinprolific   🔵 Bluesky: https://bsky.app/profile/joinprolific... #ai #deepmind #prolific