LLM-Based Literature Review Techniques by Gaurav Sahu, PostDoc@Mila - Quebec AI Institute скачать в хорошем качестве

LLM-Based Literature Review Techniques by Gaurav Sahu, PostDoc@Mila - Quebec AI Institute

Bio: Gaurav Sahu is a postdoctoral researcher at Mila - Quebec AI Institute. He holds a Ph.D. in Computer Science from the University of Waterloo. His research focuses on natural language processing, deep learning, and large language models, with applications in AI-driven scientific tasks such as automated literature reviews. His broader expertise includes prompting-based data augmentation, text classification, text summarization, and evaluation frameworks for LLM-based agents. Beyond research, Gaurav is passionate about making art and learning new languages. Abstract: Literature reviews are an essential component of scientific research, but they remain time-intensive and challenging to write, especially due to the recent influx of research papers. This paper explores the zero-shot abilities of recent Large Language Models (LLMs) in assisting with the writing of literature reviews based on an abstract. We decompose the task into two components: 1. Retrieving related works given a query abstract, and 2. Writing a literature review based on the retrieved results. We analyze how effective LLMs are for both components. For retrieval, we introduce a novel two-step search strategy that first uses an LLM to extract meaningful keywords from the abstract of a paper and then retrieves potentially relevant papers by querying an external knowledge base. Additionally, we study a prompting-based re-ranking mechanism with attribution and show that re-ranking doubles the normalized recall compared to naive search methods, while providing insights into the LLM's decision-making process. In the generation phase, we propose a two-step approach that first outlines a plan for the review and then executes steps in the plan to generate the actual review. To evaluate different LLM-based literature review methods, we create test sets from arXiv papers using a protocol designed for rolling use with newly released LLMs to avoid test set contamination in zero-shot evaluations. We release this evaluation protocol to promote additional research and development in this regard. Our empirical results suggest that LLMs show promising potential for writing literature reviews when the task is decomposed into smaller components of retrieval and planning. Our project page including a demonstration system and toolkit can be accessed here: https://litllm.github.io/