Volume 11, Issue 2
  • ISSN 2211-3711
  • E-ISSN: 2211-372X
Buy:$35.00 + Taxes



This article intends to contribute to the current debate on the quality of neural machine translation (NMT) vs. (professional) human translation quality, where recently claims concerning (super)human performance of NMT systems have emerged. The article will critically analyse some current machine translation (MT) quality evaluation methodologies employed in studies claiming such performance of their MT systems. This analysis aims to identify areas where these methodologies are potentially biased in favour of MT and hence may overvalue MT performance while undervaluing human translation performance. Then, the article provides some Translation Studies informed suggestions for improving or debiasing these methodologies in order to arrive at a more balanced picture of MT vs. (professional) human translation quality.


Article metrics loading...

Loading full text...

Full text loading...


  1. ELIS
    ELIS 2021European Language Industry Survey. AccessedJune 9, 2021. https://www.gala-global.org/sites/default/files/gala/ELIS%202021%20Results_0.pdf
    [Google Scholar]
  2. ErgoTrans
    ErgoTrans 2015Final Report: Cognitive and Physical Ergonomics of Translation (ErgoTrans). AccessedJune 24 2021. https://www.zhaw.ch/storage/linguistik/forschung/uebersetzungswissenschaft/ergotrans_final_report.pdf
    [Google Scholar]
  3. Freitag, Markus, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey
    2021 “Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation.” arXiv. AccessedJune 9, 2021. https://arxiv.org/abs/2104.14478. 10.1162/tacl_a_00437
    https://doi.org/10.1162/tacl_a_00437 [Google Scholar]
  4. Grice, Herbert P.
    1975 “Logic and Conversation.” InSyntax and Semantics. Volume31, edited byPeter Cole, and Jerry L. Morgan. 41–58. New York: Academic Press.
    [Google Scholar]
  5. Hassan, Hany, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou
    2018 “Achieving Human Parity on Automatic Chinese to English News Translation.” arXiv. AccessedJune 9, 2021. https://arxiv.org/abs/1803.05567
    [Google Scholar]
  6. Horn-Helf, Brigitte
    1999Technisches Übersetzen in Theorie und Praxis. [The Theory and Practice of Technical Translation]. Tübingen/Basel: Francke.
    [Google Scholar]
  7. House, Juliane
    2006 “Communicative Styles in English and German.” European Journal of English Studies10(3), 249–267. 10.1080/13825570600967721
    https://doi.org/10.1080/13825570600967721 [Google Scholar]
  8. Kade, Otto
    1968Zufall und Gesetzmäßigkeit in der Übersetzung [Coincidence and Regularities in Translation]. Leipzig: Verlag Enzyklopädie.
    [Google Scholar]
  9. Koehn, Philipp
    2020Neural Machine Translation. Cambridge: University Press. 10.1017/9781108608480
    https://doi.org/10.1017/9781108608480 [Google Scholar]
  10. Krüger, Ralph
    2015The Interface between Scientific and Technical Translation Studies and Cognitive Linguistics. With Particular Emphasis on Explicitation and Implicitation as Indicators of Translational Text-Context Interaction. Berlin: Frank & Timme.
    [Google Scholar]
  11. 2016 “Situated LSP Translation from a Cognitive Translational Perspective.” Lebende Sprachen61(2), 297–332. 10.1515/les‑2016‑0014
    https://doi.org/10.1515/les-2016-0014 [Google Scholar]
  12. 2020 “Explicitation in Neural Machine Translation.” Across Languages and Cultures21(2), 195–216. 10.1556/084.2020.00012
    https://doi.org/10.1556/084.2020.00012 [Google Scholar]
  13. Läubli, Samuel, Rico Sennrich, and Martin Volk
    2018 “Has Machine Translation Achieved Human Parity? A Case for Document-Level Evaluation.” InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, edited byEllen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii. 4791–4796. Association for Computational Linguistics. AccessedJune 9, 2021.   10.18653/v1/D18‑1512
    https://doi.org/10.18653/v1/D18-1512 [Google Scholar]
  14. Läubli, Samuel, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen, and Antonio Toral
    2020 “A Set of Recommendations for Assessing Human-Machine Parity in Language Translation.” Journal of Artificial Intelligence Research671, 653–672. AccessedJune 9, 2021.   10.1613/jair.1.11371
    https://doi.org/10.1613/jair.1.11371 [Google Scholar]
  15. Lommel, Arle
    2018 “Metrics for Translation Quality Assessment: A Case for Standardising Error Typologies.” InTranslation Quality Assessment. From Principles to Practice, edited byJoss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty. 109–127. Springer. 10.1007/978‑3‑319‑91241‑7_6
    https://doi.org/10.1007/978-3-319-91241-7_6 [Google Scholar]
  16. 2020 “At Human Parity? A Skeptical Response to MT Quality Claims” InMaschinelle Übersetzung für Übersetzungsprofis, edited byJörg Porsiel. 185–197. BDÜ Fachverlag.
    [Google Scholar]
  17. Macken, Lieve, Daniel Prou, and Arda Tezcan
    2020 “Quantifying the Effect of Machine Translation in a High-Quality Human Translation Production Process.” Informatics7(2), 1–19. AccessedJune 25, 2021. https://www.mdpi.com/2227-9709/7/2/12
    [Google Scholar]
  18. Maruf, Sameen, Fahimeh Saleh, and Gholamreza Haffari
    2021 A Survey on Document-Level Neural Machine Translation: Methods and Evaluation. ACM Computing Surveys54(2), 1–36. AccessedNovember 1, 2021.   10.1145/3441691
    https://doi.org/10.1145/3441691 [Google Scholar]
  19. Melby, Alan
    2019 “Bells MT (Machine Translation) Does Not Yet Ring.” Presentation atAPTIF 9: Reality vs. Illusion: From Morse Code to Machine Translation.
    [Google Scholar]
  20. Muzii, Luigi
    2021 “Close Call – Observations on Productivity, Talent Shortages, & Human Parity MT.” eMpTy Pages. AccessedJune 12, 2021. kv-emptypages.blogspot.com/2021/06/close-call-observations-on-productivity.html
    [Google Scholar]
  21. Nord, Christiane
    1997Translating as a Purposeful Activity. Functionalist Approaches Explained. Manchester: St. Jerome.
    [Google Scholar]
  22. 2009Textanalyse und Übersetzen. Theoretische Grundlagen, Methode und didaktische Anwendung einer übersetzungsrelevanten Textanalyse [Text Analysis and Translation. Theoretical Foundations, Method and Didactic Application of a Translation-Relevant Text Analysis]. 4th edition. Tübingen: Gross.
    [Google Scholar]
  23. Popel, Martin, Marketa Tomkova, Jakub Tomek, Łukasz Kaiser, Jakob Uszkoreit, Ondřej Bojar, and Zdeněk Žabokrtský
    2020 “Transforming Machine Translation: a Deep Learning System Reaches News Translation Quality Comparable to Human Professionals.” Nature Communications111, 1–15. AccessedJune 9, 2021.   10.1038/s41467‑020‑18073‑9
    https://doi.org/10.1038/s41467-020-18073-9 [Google Scholar]
  24. Pym, Anthony
    2020 “Translation, Risk Management and Cognition.” InThe Routledge Handbook of Translation and Cognition, edited byFavio Alves and Arnt Lykke Jakobsen. 445–458. New York: Routledge. 10.4324/9781315178127‑29
    https://doi.org/10.4324/9781315178127-29 [Google Scholar]
  25. Reiß, Katharina, Hans J. Vermeer
    1991Grundlegung einer allgemeinen Translationstheorie [Laying the Foundations for a General Theory of Translation and Interpreting]. 2nd edition. Tübingen: Niemeyer.
    [Google Scholar]
  26. Risku, Hanna
    2004Translationsmanagement. Interkulturelle Fachkommunikation im Kommunikationszeitalter [Translation Management. Intercultural LSP Communication in the Communication Age]. Tübingen: Narr.
    [Google Scholar]
  27. Schmitt, Peter A.
    2015 “Who Is Afraid of MT?” Lebende Sprachen60(2), 234–258. 10.1515/les‑2015‑0010
    https://doi.org/10.1515/les-2015-0010 [Google Scholar]
  28. Sulubacak, Umut, Ozan Caglayan, Stig-Arne Grönroos, Aku Rouhe, Desmond Elliott, Lucia Specia, and Jörg Tiedemann
    2020 “Multimodal Machine Translation through Visuals and Speech.” Machine Translation34(2–3), 97–147. 10.1007/s10590‑020‑09250‑0
    https://doi.org/10.1007/s10590-020-09250-0 [Google Scholar]
  29. Toral, Antonio, Sheila Castilho, Ken Hu, and Andy Way
    2018 “Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation.” InProceedings of the Third Conference on Machine Translation: Research Papers, edited byOndřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor. 113–123. AccessedJune 9, 2021.   10.18653/v1/W18‑6312
    https://doi.org/10.18653/v1/W18-6312 [Google Scholar]
  30. Vashee, Kirti
    2021a “The Quest for Human Parity Machine Translation.” eMpTy Pages. AccessedNovember 6, 2021. kv-emptypages.blogspot.com/2021/03/the-quest-for-human-parity-machine.html
    [Google Scholar]
  31. [Google Scholar]
  32. 2021c “The Human-in-the-Loop Driving MT Progress.” eMpTy Pages. AccessedNovember 6, 2021. kv-emptypages.blogspot.com/2021/11/the-human-in-loop-driving-mt-progress.html
    [Google Scholar]
  33. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jacob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin
    2017 “Attention Is All You Need.” InAdvances in Neural Information Processing Systems 30 (NIPS 2017), edited byIsabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett. 1–11. AccessedJune 9, 2021. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
    [Google Scholar]
  34. Vieira, Lucas Nunes
    2020 “Machine Translation in the News. A Framing Analysis of the Written Press.” Translation Spaces9(1), 98–122.   10.1075/ts.00023.nun
    https://doi.org/10.1075/ts.00023.nun [Google Scholar]
  35. Way, Andy
    2019 “Machine Translation: Where Are We at Today?InThe Bloomsbury Companion to Language Industry Studies, edited byErik Angelone, Maureen Ehrensberger-Dow, and Gary Massey. 311–332. Bloomsbury Academic.
    [Google Scholar]
  36. Wu, Yonghui, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean
    2016 “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation.” arXiv. AccessedJune 9, 2021. https://arxiv.org/abs/1609.08144
    [Google Scholar]

Data & Media loading...

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error