SelfAsk: Measuring and Narrowing the Compositionality Gap in Language Models скачать в хорошем качестве

SelfAsk: Measuring and Narrowing the Compositionality Gap in Language Models

This research explores the compositionality gap in large language models, which is the failure of a system to answer a multi-step question even when it knows the individual facts required. The authors introduce two new datasets, Bamboogle and Compositional Celebrities, to prove that this performance deficit does not naturally disappear as models increase in size. To address this, they propose a novel prompting strategy called Self-Ask, which encourages the model to explicitly break down complex queries into follow-up questions. This method improves accuracy by separating the logic of decomposing a problem from the retrieval of specific facts. Furthermore, the researchers demonstrate that integrating a search engine into this "Self-Ask" framework significantly boosts a model's ability to provide correct, verifiable answers. Evidence suggests that a model is more likely to solve a compositional challenge only when it possesses high confidence in each underlying sub-fact.