Codex: Evaluating Large Language Models Trained on Code скачать в хорошем качестве

Codex: Evaluating Large Language Models Trained on Code

A slow description of the paper "Evaluating Large Language Models Trained on Code" by M. Chen et al. posted on arxiv in July 2021. Timestamps: 00:00 - Evaluating Large Language Models Trained on Code (Codex) 00:21 - Outline 01:02 - Background 02:15 - A language model for code 05:10 - Task and approach 08:36 - Evaluation framework 13:07 - Nuances of pass@k estimation 20:32 - Implementing the pass@k estimator 23:25 - Evaluation details 27:01 - Code Fine-tuning 30:39 - Prompting for evaluation 33:51 - Loss scaling and temperature 36:15 - Model scaling at optimal temperatures 37:12 - Sampling heuristics and BLEU score 40:00 - Comparative Analysis of Related Models 42:32 - Results on the APPS dataset 47:27 - Code generation examples 53:38 - Supervised Fine-tuning 01:01:48 - Supervised Fine-tuning: Results 01:03:41 - Comparing Codex and Codex-S 01:05:08 - Docstring generation 01:10:21 - Limitation: sample efficiency 01:11:21 - Limitation: generation flaws 01:15:45 - Limitation: degradation with docstring length 01:17:10 - Docstring complexity 01:20:30 - Broader Impacts and Hazard Analysis 01:23:58 - Misalignment 01:26:54 - Analysis of Alignment Problems 01:31:03 - Misalignment Results 01:36:51 - Experiment Details 01:40:35 - Bias Analysis 01:43:42 - Bias probes 01:47:38 - Economic Impact 01:51:38 - Economic Impact Analysis 01:56:10 - Economic Impact Analysis: Future directions 01:57:47 - Security implications 02:04:07 - Insecure code generation 02:05:35 - Environmental Impact and Legal Implications 02:08:47 - Risk Mitigation 02:09:53 - Related Work 02:14:32 - Summary Detailed description: We start with the background for Codex, which draws inspiration from the successes of scalable sequence prediction in other domains and seeks to reproduce these gains for program synthesis. We describe the task of generating Python functions from docstrings considered by Codex, and how it compares to other models such as GPT-J-6B. We then discuss evaluation and the nuances of the pass@k metric, together with the HumanEval dataset of hand-written coding problems. Next, we describe how code fine-tuning is performed from GPT-3, and how prompting is used for evaluation. We discuss how the loss scales with model capacity and the influence of temperature. We illustrate the issues with BLEU score as a metric for evaluating code correctness and compare Codex to other code generation models on HumanEval and APPS. We then turn to code generation examples, where we see a creative solution to prime number checking. We see how supervised fine-tuning on standalone functions (producing Codex-S) yields further gains over Codex, and how the same training data can be inverted to train Codex-D, which maps from functions to docstrings. We talk about the limitations of the model, which include sample efficiency and difficulty with generating code for complex docstrings. Broader impacts are also discussed, looking at hazard analysis, over-reliance, misalignment, bias, economic impact, security implications, carbon emissions and legal implications under a "fair use" interpretation. We discuss risk mitigations such as content controls and rate limiting. Finally, we conclude with a discussion of previous work on code generation. Particular thanks are due to Almut Sohpia Koepke for her help with decoding some of the code produced by Codex and other technical details. Topics: #codex #ai #machinelearning #coding Slides (pdf): https://samuelalbanie.com/files/diges... The paper can be found on arxiv here: https://arxiv.org/abs/2107.03374 References for papers mentioned in the video can be found at http://samuelalbanie.com/digests/2022... For related content: Twitter:   / samuelalbanie   Research lab: https://caml-lab.com/ YouTube:    / @samuelalbanie1   (Optional) if you'd like to support the channel: https://www.buymeacoffee.com/samuelal...   / samuel_albanie