Volume 6, Issue 3
  • ISSN 2215-1931
  • E-ISSN: 2215-194X
Buy:$35.00 + Taxes



This study investigates how Mandarin and Slavic language speakers’ comprehensibility, accentedness, and fluency ratings, as assigned by experienced teacher-raters and novice raters, align with discrete linguistic measures, and raters’ accounts of influences on their scoring. In addition to examining mean ratings in relation to rater experience and speaker first language background, we correlated ratings with segmental, prosodic, and temporal measures. Introspective reports were segmented, coded, enumerated, and submitted to loglinear analysis to elucidate influences on ratings. Results showed that ratings were strongly correlated with prosodic goodness and moderately correlated with segmental errors, implying the importance of both segmentals and prosody in L2 speech ratings. Experienced teacher-raters provided lengthier reports than novice raters, producing more comments for all coded categories where an error was identified except for pausing (a dysfluency marker). This may be because novice raters observed little else about the speech or struggled to pinpoint or articulate other features.


Article metrics loading...

Loading full text...

Full text loading...


  1. Bannigan, K. , & Watson, R.
    (2009) Reliability and validity in a nutshell. Journal of Clinical Nursing, 18(23), 3237–3243. 10.1111/j.1365‑2702.2009.02939.x
    https://doi.org/10.1111/j.1365-2702.2009.02939.x [Google Scholar]
  2. Bongaerts, T. , Mennen, S. , & van der Slik, F.
    (2000) Authenticity of pronunciation in naturalistic second language acquisition: The case of very advanced late learners of Dutch as a second language. Studia Linguistica, 54(2), 298–308. 10.1111/1467‑9582.00069
    https://doi.org/10.1111/1467-9582.00069 [Google Scholar]
  3. Bongaerts, T. , van Summeren, C. , Planken, B. , & Schils, E.
    (1997) Age and ultimate attainment in the pronunciation of a foreign language. Studies in Second Language Acquisition, 19(4), 447–465. 10.1017/S0272263197004026
    https://doi.org/10.1017/S0272263197004026 [Google Scholar]
  4. Bradlow, A. , Clopper, C. , Smiljanic, R. , & Walter, M. A.
    (2010) A perceptual phonetic similarity space for languages: Evidence from five native language listener groups. Speech Communication, 52(11–12), 930–942. 10.1016/j.specom.2010.06.003
    https://doi.org/10.1016/j.specom.2010.06.003 [Google Scholar]
  5. Browne, K. , & Fulcher, G.
    (2017) Pronunciation and intelligibility in assessing spoken fluency. In T. Isaacs & P. Trofimovich (Eds.), Second language pronunciation: Interdisciplinary perspectives (pp.37–53). Bristol, UK: Multilingual Matters.
    [Google Scholar]
  6. Chalhoub-Deville, M.
    (1995) Deriving oral assessment scales across different tests and rater groups. Language Testing, 12(1), 62–70. 10.1177/026553229501200102
    https://doi.org/10.1177/026553229501200102 [Google Scholar]
  7. Creswell, J. W. , & Plano Clark, V. L.
    (2017) Designing and conducting mixed methods research (3rd ed.). Thousand Oaks, CA: SAGE.
    [Google Scholar]
  8. Crowther, D. , Trofimovich, P. , Saito, K. , & Isaacs, T.
    (2015) Second language comprehensibility revisited: Investigating the effects of learner background. TESOL Quarterly, 49(4), 814–837. 10.1002/tesq.203
    https://doi.org/10.1002/tesq.203 [Google Scholar]
  9. de Boer, M. , & Heeren, W.
    (2019) The speaker-specificity of filled pauses: A cross-linguistic study. Proceedings of the International Congress of Phonetic Sciences (ICPhS) 2019 (pp.607–611). Melbourne, Australia: Australasian Speech Science and Technology Association.
    [Google Scholar]
  10. Derwing, T. M. , & Munro, M. J.
    (1997) Accent, intelligibility, and comprehensibility: Evidence from four L1s. Studies in Second Language Acquisition, 19(1), 1–16. 10.1017/S0272263197001010
    https://doi.org/10.1017/S0272263197001010 [Google Scholar]
  11. (2009) Comprehensibility as a factor in listener interaction preferences: Implications for the workplace. Canadian Modern Language Review, 66(2), 181–202. 10.3138/cmlr.66.2.181
    https://doi.org/10.3138/cmlr.66.2.181 [Google Scholar]
  12. (2013) The development of L2 oral language skills in two L1 groups: A 7-year study. Language Learning, 63(2), 163–185. 10.1111/lang.12000
    https://doi.org/10.1111/lang.12000 [Google Scholar]
  13. Derwing, T. M. , Rossiter, M. J. , Munro, M. J. , & Thomson, R. I.
    (2004) Second language fluency: Judgments on different tasks. Language Learning, 54(4), 665–679. 10.1111/j.1467‑9922.2004.00282.x
    https://doi.org/10.1111/j.1467-9922.2004.00282.x [Google Scholar]
  14. Derwing, T. M. , Thomson, R. I. , & Munro, M. J.
    (2006) English pronunciation and fluency development in Mandarin and Slavic speakers. System, 34(2), 183–193. 10.1016/j.system.2006.01.005
    https://doi.org/10.1016/j.system.2006.01.005 [Google Scholar]
  15. Douglas, D.
    (1994) Quantity and quality in speaking test performance. Language Testing, 11(1), 125–144. 10.1177/026553229401100203
    https://doi.org/10.1177/026553229401100203 [Google Scholar]
  16. Ericsson, K. A. , & Simon, H. A.
    (1993) Protocol analysis: Verbal reports as data (Rev. ed.). Cambridge, MA: MIT Press. 10.7551/mitpress/5657.001.0001
    https://doi.org/10.7551/mitpress/5657.001.0001 [Google Scholar]
  17. Foote, J. A. , Isaacs, T. , & Trofimovich, P.
    (2013, June3–5). Developing a teacher-friendly assessment tool for L2 comprehensibility. Canadian Association of Applied Linguistics (ACLA/CAAL) conference, Calgary, AB.
    [Google Scholar]
  18. Foote, J. A. , & Trofimovich, P.
    (2018) Is it because of my language background? A study of language background influence on comprehensibility judgments. Canadian Modern Language Review, 74(2), 253–278. 10.3138/cmlr.2017‑0011
    https://doi.org/10.3138/cmlr.2017-0011 [Google Scholar]
  19. Gass, S. M. , & Mackey, A.
    (2000) Stimulated recall methodology in second language research. Mahwah, NJ: Lawrence Erlbaum.
    [Google Scholar]
  20. Hahn, L. D.
    (2004) Primary stress and intelligibility: Research to motivate the teaching of suprasegmentals. TESOL Quarterly, 38(2), 201–233. 10.2307/3588378
    https://doi.org/10.2307/3588378 [Google Scholar]
  21. Isaacs, T. , & Harding, L.
    (2017) Research timeline: Pronunciation assessment. Language Teaching, 50(3), 347–366. 10.1017/S0261444817000118
    https://doi.org/10.1017/S0261444817000118 [Google Scholar]
  22. Isaacs, T. , & Thomson, R. I.
    (2013) Rater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions. Language Assessment Quarterly, 10(2), 135–159. 10.1080/15434303.2013.769545
    https://doi.org/10.1080/15434303.2013.769545 [Google Scholar]
  23. Isaacs, T. , & Trofimovich, P.
    (2012) Deconstructing comprehensibility: Identifying the linguistic influences on listeners’ L2 comprehensibility ratings. Studies in Second Language Acquisition, 34(3), 475–505. 10.1017/S0272263112000150
    https://doi.org/10.1017/S0272263112000150 [Google Scholar]
  24. Isaacs, T. , Trofimovich, P. , Yu, G. , & Chereau, B. M.
    (2015) Examining the linguistic aspects of speech that most efficiently discriminate between upper levels of the revised IELTS pronunciation scale. IELTS research reports online series, 4.
    [Google Scholar]
  25. Kang, O. , & Moran, M.
    (2014) Functional loads of pronunciation features in nonnative speakers’ oral assessment. TESOL Quarterly, 48(1), 176–187. 10.1002/tesq.152
    https://doi.org/10.1002/tesq.152 [Google Scholar]
  26. Lennon, P.
    (1990) Investigating fluency in EFL: A quantitative approach. Language Learning, 40(3), 387–417. 10.1111/j.1467‑1770.1990.tb00669.x
    https://doi.org/10.1111/j.1467-1770.1990.tb00669.x [Google Scholar]
  27. Lumley, T.
    (2005) Assessing second language writing: The rater’s perspective. Frankfurt: Peter Lang.
    [Google Scholar]
  28. McAndrews, M. M. , & Thomson, R. I.
    (2017) Establishing an empirical basis for priorities in pronunciation teaching. Journal of Second Language Pronunciation, 3(2), 267–287. 10.1075/jslp.3.2.05mca
    https://doi.org/10.1075/jslp.3.2.05mca [Google Scholar]
  29. Munro, M. J.
    (2018) Dimensions of pronunciation. In O. Kang , R. Thomson , & J. Murphy . The Routledge handbook of contemporary English pronunciation (pp.413–431). New York: Routledge.
    [Google Scholar]
  30. Munro, M. J. , & Derwing, T. M.
    (2006) The functional load principle in ESL pronunciation instruction: An exploratory study. System, 34(4), 520–531. 10.1016/j.system.2006.09.004
    https://doi.org/10.1016/j.system.2006.09.004 [Google Scholar]
  31. O’Brien, M. G.
    (2016) Methodological choices in rating speech samples. Studies in Second Language Acquisition, 38(3), 587–605. 10.1017/S0272263115000418
    https://doi.org/10.1017/S0272263115000418 [Google Scholar]
  32. Pawlikowska-Smith, G.
    (2000) Canadian Language Benchmarks 2000: Theoretical framework. Ottawa, ON: Centre for Canadian Language Benchmarks.
    [Google Scholar]
  33. Rajadurai, J.
    (2007) Intelligibility studies: A consideration of empirical and ideological issues. World Englishes, 26(1), 87–98. 10.1111/j.1467‑971X.2007.00490.x
    https://doi.org/10.1111/j.1467-971X.2007.00490.x [Google Scholar]
  34. Riggenbach, H.
    (1991) Toward an understanding of fluency: A microanalysis of non-native speaker conversations. Discourse Processes, 14(4), 423–441. 10.1080/01638539109544795
    https://doi.org/10.1080/01638539109544795 [Google Scholar]
  35. Rose, H. , & Galloway, N.
    (2019) Global Englishes for language teaching. Cambridge: Cambridge University Press. 10.1017/9781316678343
    https://doi.org/10.1017/9781316678343 [Google Scholar]
  36. Rossiter, M. J.
    (2009) Perceptions of L2 fluency by native and non-native speakers of English. Canadian Modern Language Review, 65(3), 395–412. 10.3138/cmlr.65.3.395
    https://doi.org/10.3138/cmlr.65.3.395 [Google Scholar]
  37. Saito, K. , Trofimovich, P. , & Isaacs, T.
    (2016) Second language speech production: Investigating linguistic correlates of comprehensibility and accentedness for learners at different ability levels. Applied Psycholinguistics, 37(2), 217–240. 10.1017/S0142716414000502
    https://doi.org/10.1017/S0142716414000502 [Google Scholar]
  38. Saito, K. , Trofimovich, P. , Isaacs, T. , & Webb, S.
    (2017) Re-examining phonological and lexical correlates of second language comprehensibility: The role of rater experience. In T. Isaacs & P. Trofimovich (Eds.), Second language pronunciation assessment: Interdisciplinary perspectives (pp.131–146). Bristol, UK: Multilingual Matters.
    [Google Scholar]
  39. Saito, K. , Webb, S. , Trofimovich, P. , & Isaacs, T.
    (2016) Lexical correlates of comprehensibility versus accentedness in second language speech. Bilingualism: Language and Cognition, 19(3), 597–609. 10.1017/S1366728915000255
    https://doi.org/10.1017/S1366728915000255 [Google Scholar]
  40. Schiavetti, N.
    (1992) Scaling procedures for the measurement of speech intelligibility. In R. D. Kent (Ed.), Intelligibility in speech disorders (pp.11–34). Amsterdam: John Benjamins. 10.1075/sspcl.1.02sch
    https://doi.org/10.1075/sspcl.1.02sch [Google Scholar]
  41. Stevens, J. P.
    (2009) Applied multivariate statistics for the social sciences (5th ed.). New York: Taylor & Francis.
    [Google Scholar]
  42. Suzukida, Y. , & Saito, K.
    (2019) Which segmental features matter for successful L2 comprehensibility? Revisiting and generalizing the pedagogical value of the Functional Load principle. Language Teaching Research. Advance online publication. doi:  10.1177/1362168819858246
    https://doi.org/10.1177/1362168819858246 [Google Scholar]
  43. Thomson, R. I. , & Isaacs, T.
    (2009) Within-category variation in L2 English vowel learning. Canadian Acoustics, 37, 138–139.
    [Google Scholar]
  44. Thompson, I.
    (1991) Foreign accents revisited: The English pronunciation of Russian immigrants. Language Learning, 41(2), 177–204. 10.1111/j.1467‑1770.1991.tb00683.x
    https://doi.org/10.1111/j.1467-1770.1991.tb00683.x [Google Scholar]
  45. Upshur, J. A. , & Turner, C. E.
    (1999) Systematic effects in the rating of second-language speaking ability: Test method and learner discourse. Language Testing, 16(1), 82–111. 10.1177/026553229901600105
    https://doi.org/10.1177/026553229901600105 [Google Scholar]
  46. Winke, P. , Gass, S. , & Myford, C.
    (2013) Raters’ L2 background as a potential source of bias in rating oral performance. Language Testing, 30(2), 231–252. 10.1177/0265532212456968
    https://doi.org/10.1177/0265532212456968 [Google Scholar]
  47. Zielinski, B. W.
    (2008) The listener: No longer the silent partner in reduced intelligibility. System, 36(1), 69–84. 10.1016/j.system.2007.11.004
    https://doi.org/10.1016/j.system.2007.11.004 [Google Scholar]

Data & Media loading...

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error