Chain of Thought: Introducing Remote Labor Index (RLI) скачать в хорошем качестве

Chain of Thought: Introducing Remote Labor Index (RLI)

Introducing the Remote Labor Index, RLI. Brad Kenstler, Head of Agent Capabilities and Environments, discusses RLI with Bing Liu, Head of Research, Madhu Sehwag, Research Scientist, and Mantas Mazeika, Research Scientist at the Center for AI Safety. The Remote Labor Index (RLI) is a benchmark that empirically measures the capability of AI agents to perform real-world, economically valuable remote work. 0:00 Introduction 1:00 Overview of RLI 5:11 Benchmarking Freelance work 10:32 Comparing RLI to other professional domain benchmarks 12:18 Deep dive on RLI tasks 17:27 Making tasks representative of real-world work 22:50 Rubrics vs judge-based evaluation 26:15 Bottlenecks on agentic capabilities 29:30 Which agents does RLI evaluate 34:04 Failure modes of RLI 37:30 Implications on the future of remote labor 42:10 Unlocking performance improvements on RLI Learn more about the benchmark at: https://scale.com/leaderboard/rli