Volume 21, Issue 2
  • ISSN 1384-6647
  • E-ISSN: 1569-982X
Buy:$35.00 + Taxes



The article aims to show how interpreter trainers holistically grade student performances. For this purpose, experimental rating sessions were held for four undergraduate interpreter trainers. The raters were asked to think aloud their quality judgments while holistically assessing six recordings of consecutive interpretation. Their concurrent verbal reports, along with reflective reports, interview transcripts, and video recordings of computer screen activity, were collected and analysed in detail. Findings revealed various facets of interpreting performance assessment, including what procedures the raters followed, what aspects of the performance they focused on, what criteria they depended on for their judgment decisions, and why two ratings of the same performance were divergent. This article also presents a tentative model for holistic rating of consecutive interpretation.


Article metrics loading...

Loading full text...

Full text loading...


  1. Bartłomiejczyk, M.
    (2007) Interpreting quality as perceived by trainee interpreters: Self-evaluation. The Interpreter and Translator Trainer1 (2), 247–267. 10.1080/1750399X.2007.10798760
    https://doi.org/10.1080/1750399X.2007.10798760 [Google Scholar]
  2. Bejar, I. I.
    (2012) Rater cognition: Implications for validity. Educational Measurement: Issues and Practice31 (3), 2–9. 10.1111/j.1745‑3992.2012.00238.x
    https://doi.org/10.1111/j.1745-3992.2012.00238.x [Google Scholar]
  3. Bernardini, S.
    (2001) Think-aloud protocols in translation research: Achievements, limits, future prospects. Target13 (2), 241–263. 10.1075/target.13.2.03ber
    https://doi.org/10.1075/target.13.2.03ber [Google Scholar]
  4. Bontempo, K. & Napier, J.
    (2011) Evaluating emotional stability as a predictor of interpreter competence and aptitude for interpreting. Interpreting13 (1), 85–105. 10.1075/intp.13.1.06bon
    https://doi.org/10.1075/intp.13.1.06bon [Google Scholar]
  5. Bowles, M. A.
    (2010) The think-aloud controversy in second language research. London: Routledge. 10.4324/9780203856338
    https://doi.org/10.4324/9780203856338 [Google Scholar]
  6. Bühler, H.
    (1986) Linguistic (semantic) and extralinguistic (pragmatic) criteria for the evaluation of conference interpretation and interpreters. Multilingua5 (4), 231–235.
    [Google Scholar]
  7. Chabasse, C. & Kader, S.
    (2014) Putting interpreting admissions exams to the test. Interpreting16 (1), 19–33. 10.1075/intp.16.1.02cha
    https://doi.org/10.1075/intp.16.1.02cha [Google Scholar]
  8. Chiaro, D. & Nocella, G.
    (2004) Interpreters’ perception of linguistic and non-linguistic factors affecting quality: A survey through the World Wide Web. Meta49 (2), 279–293. 10.7202/009351ar
    https://doi.org/10.7202/009351ar [Google Scholar]
  9. Choi, J. Y.
    (2013) Assessing the impact of text length on consecutive interpreting. InD. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt am Main: Peter Lang, 85–96.
    [Google Scholar]
  10. Cifuentes-Férez, P. & Rojo, A.
    (2015) Thinking for translating: A think-aloud protocol on the translation of manner-of-motion verbs. Target27 (2), 273–300. 10.1075/target.27.2.05cif
    https://doi.org/10.1075/target.27.2.05cif [Google Scholar]
  11. Clifford, A.
    (2005) Putting the exam to the test: Psychometric validation and interpreter certification. Interpreting7 (1), 97–131. 10.1075/intp.7.1.06cli
    https://doi.org/10.1075/intp.7.1.06cli [Google Scholar]
  12. Coffey, A. & Atkinson, P.
    (1996) Making sense of qualitative data: Complementary research strategies. Thousand Oaks, CA: Sage.
    [Google Scholar]
  13. Cohen, A. D.
    (2000) Exploring strategies in test-taking: Fine-tuning verbal reports from respondents. InG. Ekbatani & H. Pierson (Eds.), Learner-directed assessment in ESL. Mahwah, NJ: Lawrence Erlbaum, 127–150.
    [Google Scholar]
  14. De Gregoris, G.
    (2014) The limits of expectations vs. assessment questionnaire-based surveys on simultaneous interpreting quality: The need for a gestaltic model of perception. Rivista internazionale di tecnica della traduzione16, 57–87.
    [Google Scholar]
  15. DeRember, M. L.
    (1998) Writing assessment: Raters’ elaboration of the rating task. Assessing Writing5 (1), 7–29. 10.1016/S1075‑2935(99)80003‑8
    https://doi.org/10.1016/S1075-2935(99)80003-8 [Google Scholar]
  16. Englund Dimitrova, B. & Tiselius, E.
    (2014) Retrospection in interpreting and translation: Explaining the process?MonTISpecial Issue1, 177–200. 10.6035/MonTI.2014.ne1.5
    https://doi.org/10.6035/MonTI.2014.ne1.5 [Google Scholar]
  17. Ericsson, K. A.
    (2000/2001) Expertise in interpreting: An expert-performance perspective. Interpreting5 (2), 187–220. 10.1075/intp.5.2.08eri
    https://doi.org/10.1075/intp.5.2.08eri [Google Scholar]
  18. Ericsson, K. A. & Simon, H. A.
    (1993) Protocol analysis: Verbal reports as data (Rev. ed.). Cambridge, MA: MIT Press.
    [Google Scholar]
  19. (1998) How to study thinking in everyday life: Contrasting think-aloud protocols with descriptions and explanations of thinking. Mind, Culture and Activity5 (3), 178–186. 10.1207/s15327884mca0503_3
    https://doi.org/10.1207/s15327884mca0503_3 [Google Scholar]
  20. Eyckmans, J., Anckaert, P. & Segers, W.
    (2016) Translation and interpretation skills. InD. Tsagari & J. Banerjee (Eds.), Handbook of second language assessment. Berlin: Walter de Gruyter, 219–235. 10.1515/9781614513827‑016
    https://doi.org/10.1515/9781614513827-016 [Google Scholar]
  21. Garzone, G.
    (2003) Reliability of quality criteria evaluation in survey research. InA. Collados Aís, M. M. Fernández Sánchez & D. Gile (Eds.), La evaluación de la calidad en interpretación: Investigación. Granada: Comares.
    [Google Scholar]
  22. Green, A.
    (1998) Verbal protocol analysis in language testing research: A handbook. Cambridge: Cambridge University Press.
    [Google Scholar]
  23. Han, C.
    (2015) Investigating rater severity/leniency in interpreter performance testing: A multifaceted Rasch measurement approach. Interpreting17 (2), 255–283. 10.1075/intp.17.2.05han
    https://doi.org/10.1075/intp.17.2.05han [Google Scholar]
  24. Iglesias Fernández, E.
    (2013) Unpacking delivery criteria in interpreting quality assessment. InD. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt am Main: Peter Lang, 51–66.
    [Google Scholar]
  25. Ivanova, A.
    (2000) The use of retrospection in research on simultaneous interpreting. InS. Tirkkonen-Condit & R. Jääskeläinen (Eds.), Tapping and mapping the processes of translation and interpreting: Outlooks on empirical research. Amsterdam: John Benjamins, 27–52. 10.1075/btl.37.05iva
    https://doi.org/10.1075/btl.37.05iva [Google Scholar]
  26. Jakobsen, A. L.
    (2003) Effects of think aloud on translation speed, revision, and segmentation. InF. Alves (Ed.), Triangulating translation: Perspectives in process-oriented research. Amsterdam: John Benjamins, 69–95. 10.1075/btl.45.08jak
    https://doi.org/10.1075/btl.45.08jak [Google Scholar]
  27. Jarvella, R. J., Jensen, A., Jensen, E. H. & Anderson, M. S.
    (2002) Towards characterizing translator expertise, knowledge and know-how: Some findings using TAPs and experimental methods. InA. Riccardi (Ed.), Translation studies: Perspectives on an emerging discipline. Cambridge: Cambridge University Press, 172–197.
    [Google Scholar]
  28. Jourdenais, R.
    (2001) Cognition, instruction and protocol analysis. InP. Robinson (Ed.), Cognition and second language instruction. Cambridge: Cambridge University Press, 354–375. 10.1017/CBO9781139524780.014
    https://doi.org/10.1017/CBO9781139524780.014 [Google Scholar]
  29. Künzli, A.
    (2007) Translation revision: A study of the performance of ten professional translators revising a legal text. InY. Gambier, M. Shlesinger & R. Stolze (Eds.), Doubts and directions in translation studies. Amsterdam: John Benjamins, 115–126. 10.1075/btl.72.14kun
    https://doi.org/10.1075/btl.72.14kun [Google Scholar]
  30. Lee, J.
    (2009) Toward more reliable assessment of interpreting performance. InS. Hale, U. Ozolins & L. Stern (Eds.), The Critical Link 5: Quality in interpreting – a shared responsibility. Amsterdam: John Benjamins, 171–185. 10.1075/btl.87.14lee
    https://doi.org/10.1075/btl.87.14lee [Google Scholar]
  31. Lee, S.-B.
    (2014) An interpreting self-efficacy (ISE) scale for undergraduate students majoring in consecutive interpreting: Construction and preliminary validation. The Interpreter and Translator Trainer8 (2), 183–203. 10.1080/1750399X.2014.929372
    https://doi.org/10.1080/1750399X.2014.929372 [Google Scholar]
  32. Li, D.
    (2004) Trustworthiness of think-aloud protocols in the study of translation processes. International Journal of Applied Linguistics14 (3), 301–313. 10.1111/j.1473‑4192.2004.00067.x
    https://doi.org/10.1111/j.1473-4192.2004.00067.x [Google Scholar]
  33. Liu, M.
    (2013) Design and analysis of Taiwan’s interpretation certification examination. InD. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt am Main: Peter Lang, 163–178.
    [Google Scholar]
  34. (2015) Assessment. InF. Pöchhacker (Ed.), Routledge encyclopedia of interpreting studies. London: Routledge, 20–22.
    [Google Scholar]
  35. Lörscher, W.
    (2005) The translation process: Methods and problems of its investigation. Meta50 (2), 597–608. 10.7202/011003ar
    https://doi.org/10.7202/011003ar [Google Scholar]
  36. Miles, M. B., Huberman, A. M. & Saldaña, J.
    (2014) Qualitative data analysis: A methods sourcebook (3rd ed.). Thousand Oaks, CA: Sage.
    [Google Scholar]
  37. Naumenko, O.
    (2015) Improving performance assessment score validation practices: An instructional module on generalizability theory. Working Papers on Language and Diversity in EducationI (1), 1–17.
    [Google Scholar]
  38. O’Hagan, S.
    (2014) Variability in assessor responses to undergraduate essays: An issue for assessment quality in higher education. Bern: Peter Lang.
    [Google Scholar]
  39. Pöchhacker, F.
    (2001) Quality assessment in conference and community interpreting. Meta46 (2), 410–425. 10.7202/003847ar
    https://doi.org/10.7202/003847ar [Google Scholar]
  40. (2011) Assessing aptitude for interpreting: The SynCloze test. Interpreting13 (1), 106–120. 10.1075/intp.13.1.07poc
    https://doi.org/10.1075/intp.13.1.07poc [Google Scholar]
  41. (2016) Introducing interpreting studies (2nd ed.). London: Routledge. 10.4324/9781315649573
    https://doi.org/10.4324/9781315649573 [Google Scholar]
  42. Pradas Macías, M.
    (2006) Probing quality criteria in simultaneous interpreting: The role of silent pauses in fluency. Interpreting8 (1), 25–43. 10.1075/intp.8.1.03pra
    https://doi.org/10.1075/intp.8.1.03pra [Google Scholar]
  43. Pressley, M. & Afflerbach, P.
    (1995) Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Lawrence Erlbaum.
    [Google Scholar]
  44. Rallis, S. F. & Rossman, G. B.
    (2003) Mixed methods in evaluation contexts: A pragmatic framework. InA. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research. Thousand Oaks, CA: Sage, 491–512.
    [Google Scholar]
  45. Russell, D. & Winston, B.
    (2014) Tapping into the interpreting process: Using participant reports to inform the interpreting process in educational settings. Translation & Interpreting6 (1), 102–127.
    [Google Scholar]
  46. Saldaña, J.
    (2016) The coding manual for qualitative researchers (3rd ed.). Thousand Oaks, CA: Sage.
    [Google Scholar]
  47. Saldanha, G. & O’Brien, S.
    (2013) Research methodologies in translation studies. London: Routledge.
    [Google Scholar]
  48. Sawyer, D. B.
    (2004) Fundamental aspects of interpreter education: Curriculum and assessment. Amsterdam: John Benjamins. 10.1075/btl.47
    https://doi.org/10.1075/btl.47 [Google Scholar]
  49. Sun, S.
    (2011) Think-aloud-based translation process research: Some methodological considerations. Meta56 (4), 928–951. 10.7202/1011261ar
    https://doi.org/10.7202/1011261ar [Google Scholar]
  50. Tiselius, E.
    (2009) Revisiting Carroll’s scales. InC. Angelelli & H. E. Jacobson (Eds.), Testing and assessment in translation and interpreting studies: A call for dialogue between research and practice. Amsterdam: John Benjamins. 10.1075/ata.xiv.07tis
    https://doi.org/10.1075/ata.xiv.07tis [Google Scholar]
  51. Van Someren, M. W., Barnard, Y. F. & Sandberg, J. A. C.
    (1994) The think-aloud method: A practical guide to modeling cognitive processes. London: Academic Press.
    [Google Scholar]
  52. Vik-Tuovinen, G. V.
    (2002) Retrospection as a method of studying the process of simultaneous interpreting. InG. Garzone & M. Viezzi (Eds.), Interpreting in the 21st century: Challenges and opportunities. Amsterdam: John Benjamins, 63–72. 10.1075/btl.43.07vik
    https://doi.org/10.1075/btl.43.07vik [Google Scholar]
  53. Wallace, M.
    (2013) Rethinking bifurcated testing models in the court interpreter certification process. InD. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt am Main: Peter Lang, 67–83.
    [Google Scholar]
  54. Wang, J., Napier, J., Goswell, D. & Carmichael, A.
    (2015) The design and application of rubrics to assess signed language interpreting performance. The Interpreter and Translator Trainer9 (1), 83–103. 10.1080/1750399X.2015.1009261
    https://doi.org/10.1080/1750399X.2015.1009261 [Google Scholar]
  55. Wigglesworth, G.
    (2005) Current approaches to researching second language learner processes. Annual Review of Applied Linguistics25, 90–111. 10.1017/S026719050500005X
    https://doi.org/10.1017/S026719050500005X [Google Scholar]
  56. Wu, F. S.
    (2013) How do we assess students in the interpreting examinations?InD. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt: Peter Lang, 15–33.
    [Google Scholar]
  57. Zwischenberger, C.
    (2010) Quality criteria in simultaneous interpreting: An international vs. a national view. The Interpreters’ Newsletter15, 127–142.
    [Google Scholar]

Data & Media loading...

  • Article Type: Research Article
Keyword(s): coding; quality judgment; raters; rating process; think-aloud protocol (TAP)
This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error