Training Dynamics of Parametric and In-Context Knowledge Utilization in Language Models скачать в хорошем качестве

Training Dynamics of Parametric and In-Context Knowledge Utilization in Language Models

Paper: https://arxiv.org/abs/2510.02370 Title: Training Dynamics of Parametric and In-Context Knowledge Utilization in Language Models Authors: Minsung Kim, Dong-Kyum Kim, Jea Kwon, Nakyeong Yang, Kyomin Jung, Meeyoung Cha Abstract: Large language models often encounter conflicts between in-context knowledge retrieved at inference time and parametric knowledge acquired during pretraining. Models that accept external knowledge uncritically are vulnerable to misinformation, whereas models that adhere rigidly to parametric knowledge fail to benefit from retrieval. Despite the widespread adoption of retrieval-augmented generation, we still lack a systematic understanding of what shapes knowledge-arbitration strategies during training. This gap risks producing pretrained models with undesirable arbitration behaviors and, consequently, wasting substantial computational resources after the pretraining budget has already been spent. To address this problem, we present the first controlled study of how training conditions influence models' use of in-context and parametric knowledge, and how they arbitrate between them. We train transformer-based language models on a synthetic biographies corpus while systematically controlling various conditions. Our experiments reveal that intra-document repetition of facts fosters the development of both parametric and in-context capabilities. Moreover, training on a corpus that contains inconsistent information or distributional skew encourages models to develop robust strategies for leveraging parametric and in-context knowledge. Rather than viewing these non-ideal properties as artifacts to remove, our results indicate that they are important for learning robust arbitration. These insights offer concrete, empirical guidance for pretraining models that harmoniously integrate parametric and in-context knowledge. Tags: Machine Learning, Natural Language Processing, Research, gan, transformer, unsupervised, supervised, few-shot, gru, attention, search, training, dynamics, parametric, context, knowledge, research paper, academic, study, analysis, tutorial, explained, breakdown, paper review, research summary, AI research, scientific paper, methodology, results, findings, innovation, technology, computing, algorithm, model, dataset, evaluation, performance, accuracy, efficiency Welcome to the Mayuresh Shilotri's Youtube . Maintained by Mayuresh Shilotri You can follow me at Blog - https://shilotri.com/ LinkedIn -   / mayureshshilotri   Twitter -   / mshilotri   Note: I only claim to have read the research paper and created a Video using AI tool. I am not the author. All intellectual heavy lifting was performed by the respective authors. 🙏