Volume 21, Issue 3
  • ISSN 1018-2101
  • E-ISSN: 2406-4238


This study addresses variability among native speaker raters who evaluated pragmatic performance of learners of English as a foreign language. Using a five-point rating scale, four native English speakers of mixed cultural background (one African American, one Asian American, and two Australians) assessed the appropriateness of two types of speech acts (requests and opinions) produced by 48 Japanese EFL students. To explore norms and the reasoning behind the raters’ assessment practice, individual introspective verbal interviews were conducted. Eight students’ speech act productions (64 speech acts in total) were selected randomly, and the raters were asked to rate each speech act and then explain their rating decision. Interview data revealed similarities and differences in their use of pragmatic norms and social rules in evaluating appropriateness.


Article metrics loading...

Loading full text...

Full text loading...


  1. Austin, J.L
    (1962) How to do things with words. Cambridge: Harvard University Press.
    [Google Scholar]
  2. Bachman, Lyle , and Adrian Palmer
    (1996) Language Testing in Practice: Designing and Developing Useful Language Tests. Oxford: Oxford University Press.
    [Google Scholar]
  3. Brown, Annie
    (2000) An investigation of rater's orientation in awarding scores in the IELTS interview. In R. Tulloch (ed.), IELTS Research Report3: 49-84.
    [Google Scholar]
  4. (2003) Interviewer variation and the co-construction of speaking proficiency. Language Testing20: 1-25. doi: 10.1191/0265532203lt242oa
    https://doi.org/10.1191/0265532203lt242oa [Google Scholar]
  5. (2005) Interviewer Variability in Oral Proficiency Interviews. Frankfurt am Main: Peter Lang.
    [Google Scholar]
  6. Brown, James D
    (2001) Six types of pragmatics tests in two different contexts. In K. Rose & G. Kasper (eds.), Pragmatics in Language Teaching. Cambridge: Cambridge University Press, pp. 301-325. doi: 10.1017/CBO9781139524797.020
    https://doi.org/10.1017/CBO9781139524797.020 [Google Scholar]
  7. Clark, H.H
    (1979) Responding to indirect speech acts. Cognitive Psychology11: 430-477. doi: 10.1016/0010‑0285(79)90020‑3
    https://doi.org/10.1016/0010-0285(79)90020-3 [Google Scholar]
  8. Cohen, Andrew
    (1994) Assessing Language Ability in the Classroom. Rowley, MS: Newbury House.
    [Google Scholar]
  9. Cohen, Andrew , and Elite Olshtain
    (1981) Developing a measure of socio-cultural competence: The case of apology. Language Learning31: 113-134. doi: 10.1111/j.1467‑1770.1981.tb01375.x
    https://doi.org/10.1111/j.1467-1770.1981.tb01375.x [Google Scholar]
  10. Davies, Alan
    (2003) The native speaker in applied linguistics. In A. Davies & K. Elder (eds.), Handbook of Applied Linguistics. New York: Blackwell, pp. 431-450.
    [Google Scholar]
  11. Ducassee, Ana M
    (2009) Assessing paired orals: Raters' orientation to interaction. Language Testing26: 423-443. doi: 10.1177/0265532209104669
    https://doi.org/10.1177/0265532209104669 [Google Scholar]
  12. Green, A
    (1998) Verbal Protocol Analysis in Language Testing Research: A Handbook (vol.5). Cambridge: Cambridge University Press.
    [Google Scholar]
  13. Hoffman-Hicks, Sheila
    (1992) Linguistic and pragmatic competence: Their relationship in the overall competence of the language learner. In L.F. Bouton & Y. Kachru (eds.), Pragmatics and Language Learning Monograph SeriesVol.3. Urbana-Champaign, IL: University of Illinois, pp. 66-80.
    [Google Scholar]
  14. Hudson, Thom , Emily Detmer , and James D. Brown
    (1995) Developing Prototypic Measures of Cross- Cultural Pragmatics (Technical Report No.7). Honolulu, HI: University of Hawai'I at Manoa, Second Language Teaching & Curriculum Center.
    [Google Scholar]
  15. Johnson, Jeff , and Gad Lim
    (2009) The influence of rater language background on writing performance assessment. Language Testing26: 485-505. doi: 10.1177/0265532209340186
    https://doi.org/10.1177/0265532209340186 [Google Scholar]
  16. Levinson, Stephen
    (1983) Pragmatics. Cambridge: Cambridge University Press.
    [Google Scholar]
  17. Liu, Jianda
    (2006) Measuring Interlanguage Pragmatic Knowledge of EFL Learners. Frankfurt: Peter Lang.
    [Google Scholar]
  18. May, Lyn
    (2007) Interaction in a paired speaking test: The rater's perspective. Ph.D. dissertation: The University of Melbourne.
  19. (2009) Co-constructed interaction in a paired speaking test: The rater's perspective. Language Testing26: 397-421. doi: 10.1177/0265532209104668
    https://doi.org/10.1177/0265532209104668 [Google Scholar]
  20. McNamara, Tim , and Tom Lumley
    (1997) The effect of interlocutor and assessment mode variables in overseas assessments of speaking skills in occupational settings. Language Testing14: 140-156. doi: 10.1177/026553229701400202
    https://doi.org/10.1177/026553229701400202 [Google Scholar]
  21. Mey, Jacob
    (1993) Pragmatics: An Introduction. Oxford: Blackwell.
    [Google Scholar]
  22. Roever, Carston
    (2005) Testing EFL Pragmatics. Frankfurt: Peter Lang.
    [Google Scholar]
  23. O'Loughlin, Kieran
    (2002) The impact of gender in oral proficiency testing. Language Testing19: 169-192. doi: 10.1191/0265532202lt226oa
    https://doi.org/10.1191/0265532202lt226oa [Google Scholar]
  24. Poliitt, Alastair , and Neil Murray
    (1996) What do raters really pay attention to?In M. Milanovic & N. Saville (eds.), Performance Testing Cognition and Assessment: Selected papers from the 15th Language Testing Research Colloquium. Cambridge: Cambridge University Press, pp. 74-91.
    [Google Scholar]
  25. Taguchi, Naoko
    (2006) Analysis of appropriateness in a speech act of request in L2 English. Pragmatics16: 513-535.
    [Google Scholar]
  26. Thomas, Jenny
    (1995) Meaning in Interaction: An Introduction to Pragmatics. London: Longman.
    [Google Scholar]
  27. Sasaki, Miyuki
    (1998) Investigating EFL students' production of speech acts: A comparison of production questionnaires and role-plays. Journal of Pragmatics30: 457-484. doi: 10.1016/S0378‑2166(98)00013‑7
    https://doi.org/10.1016/S0378-2166(98)00013-7 [Google Scholar]
  28. Walter, Scott
    (2007) A conversation-analytic hermeneutic rating protocol to assess L2 oral pragmatic competence. Language Testing24: 155-183. doi: 10.1177/0265532207076362
    https://doi.org/10.1177/0265532207076362 [Google Scholar]
  29. Yamashita, Sayoko
    (1996) Six Measures of JSL Pragmatics (Technical report #14). Honolulu, HI: University of Hawai'i at Manoa, Second Language Teaching & Curriculum Center.
    [Google Scholar]
  30. (2001) Using pictures for research in pragmatics-eliciting pragmatic strategies by picture response tests. In T. Hudson & J.D. Brown (eds.), A Focus on Language Test Development: Expanding the Language Proficiency Construct Across a Variety of Tests. (Technical Report #21). University of Hawaii, Second language Teaching and Curriculum Center, pp. 35-56.
    [Google Scholar]
  31. Yoshitake, S
    (1997) Measuring interlanguage pragmatic competence of Japanese students of English as a foreign language: A multi-test framework evaluation. Unpublished Ph.D. thesis, Columbia Pacific University, Novata, CA.
  • Article Type: Research Article
Keyword(s): Assessment; Pragmatic competence; Rater variation; Speech acts
This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error