An Extensive Evaluation of Automatic Metrics for Machine Translation [ LingMon скачать в хорошем качестве

An Extensive Evaluation of Automatic Metrics for Machine Translation [ LingMon

📅 October 18, 2021 | 🗣 Tom Kocmi (Microsoft, Munich) | 📝 To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation 🔖 ABOUT THE LECTURE ▔▔▔▔▔▔▔▔▔▔▔▔▔ ▸Automatic metrics are commonly used as the exclusive tool for declaring the superiority of one machine translation system's quality over another. The community choice of automatic metric guides research directions and industrial developments by deciding which models are deemed better. Evaluating metrics correlations with sets of human judgements has been limited by the size of these sets. In this paper, we corroborate how reliable metrics are in contrast to human judgements on -- to the best of our knowledge -- the largest collection of judgements reported in the literature. Arguably, pairwise rankings of two systems are the most common evaluation tasks in research or deployment scenarios. Taking human judgement as a gold standard, we investigate which metrics have the highest accuracy in predicting translation quality rankings for such system pairs. Furthermore, we evaluate the performance of various metrics across different language pairs and domains. Lastly, we show that the sole use of BLEU impeded the development of improved models leading to bad deployment decisions. We release the collection of 2.3M sentence-level human judgements for 4380 systems for further analysis and replication of our work. 📽️ ABOUT THE SERIES ▔▔▔▔▔▔▔▔▔▔▔▔ ▸The history of Linguistic Mondays dates back to the 1980s and their original aim was to make both the students and the faculty members as well as the wider research community aware of the field of computational linguistics in general and of the results achieved by the members of our team in particular. During the years, with the growing awareness of the domain and with new trends appearing on the scene and with more master and doctoral students coming in, the scope of the topics introduced has broadened correspondingly, covering all aspects of the field from the basics of computational linguistics, its linguistic and formal background through corpora case studies and natural language processing applications such as machine translation and information retrieval up to the most modern trends including machine learning. It is also offers an excellent opportunity for PhD students to present their results and to receive a relevant response from leading experts in the field. ❔ SOCIAL MEDIA ▔▔▔▔▔▔▔▔▔▔ ▸web: https://ufal.mff.cuni.cz ▸facebook:   / ufalmffuk   ▸twitter:   / ufal_cuni