Crosslingual Generalization through Multitask Finetuning (BLOOMZ & mT0) скачать в хорошем качестве

Crosslingual Generalization through Multitask Finetuning (BLOOMZ & mT0)

A description of the work "Crosslingual Generalization through Multitask Finetuning" by N. Muennighoff et al. which appeared on arxiv in November 2022. This work was produced as part of the BigScience Workshop. Timestamps: 00:00 - Crosslingual Generalization through Multitask Finetuning (BLOOMZ & mT0) 01:56 - Finetuning data 03:55 - Dataset variants 05:11 - Finetuned models 06:46 - Evaluation details 07:35 - Results: Zero-shot multilingual task generalization 09:31 - Results: Language generalization 10:11 - Results: Language generalization (on "never intentionally seen" languages) 11:36 - Results: Multilingual prompting 12:44 - Results: Scaling multitask finetuning 13:31 - Influence of multitask finetuning on generative tasks 15:40 - Code generation 17:06 - Influence of language proportions 18:12 - Qualitative results 21:18 - Related Work 23:25 - Takeaways and released assets 24:24 - Nuts and bolts of finetuning Detailed description: This video describes studies "Crosslingual Generalization through Multitask Finetuning", a work that studies multilingual multitask finetuning for zero-shot task generalization on non-English tasks. We start by describing the motivation for the work: previous studies have shown that finetuning large language models (LLMs) brings benefits, but has mostly focused on English models and tasks. While multilingual LLMs demonstrate promise, their zero-shot performance still trails their performance with task and language-specific finetuning. Collecting such finetuning is difficult for low-resource languages and tasks. We summarise the key findings of this work: (1) English multitask finetuning helps non-English tasks, (2) multilingual finetuning data further helps, (3) larger models benefit more from multitask finetuning, (4) finetuning helps tasks on rarely seen languages. We outline the finetuning data used in this work, xP3, which expands the P3 dataset used in prior work by adding 28 multilingual datasets. A second variant dubbed xP3mt is also employed for finetuning. This uses machine translation to generate non-English prompts from those found in xP3. We then describe the families of models finetuned in the study: BLOOM and mT5, and how finetuning on xP3 is used to produce BLOOMZ and mT0. Next, we describe evaluation details - the three held-out task clusters (coreference resolution, sentence completion and natural language inference) and the additional assessment conducted on program synthesis (on HumanEval). We then turn to results, first describing how mT0-13B outperforms BLOOMZ for zero-shot multilingual task generalization, then describing how xP3 finetuning yields better results than P3 finetuning (which in turn yields a boost even on non-English tasks). We describe how performance improves on languages "never intentionally seen" during pretraining, and how training on machine-translated prompts can boost performance on non-English prompts (though not always). The video describes a scaling study that suggests that multitask finetuning benefits larger models more, and flags how multitask finetuning can harm generative task performance. This is illustrated both qualitatively and quantitatively with code generation. The influence of pretraining language proportions is also discussed. Qualitative results are provided, together with a brief summary of related work. The video wraps up with a list of takeaways and released assets and some low-level nuts and bolts details about fine-tuning. Answer to the question "Why is the sky blue?" https://xkcd.com/1818/ Full author list: Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, Colin Raffel Note: as a co-author of this work, my description of this work may be favourably biased in its assessment. Topics: #BLOOMZ #mT0 #bigscience #foundationmodels #languagemodels Slides (pdf): https://samuelalbanie.com/files/diges... References for papers mentioned in the video can be found at http://samuelalbanie.com/digests/2022... Arxiv paper: https://arxiv.org/abs/2211.01786 Code and models: https://github.com/bigscience-worksho... BigScience Workshop: https://bigscience.huggingface.co/ Lastly, I'd like to express my thanks to the mystery donor of the Spanish voiceover. (Optional) If you'd like to support future videos: https://www.buymeacoffee.com/samuelal...