1887
  1. Home
  2. e-Journals
  3. Applied Pragmatics
  4. Volume 7, Issue 1
  5. Article
Volume 7, Issue 1
  • ISSN 2589-109X
  • E-ISSN: 2589-1103
USD
Buy:$35.00 + Taxes

Abstract

Abstract

Strategic competence, conceptualized as the ability to put semantic, grammatical, and pragmatic knowledge into use, is a key element in models of communicative language proficiency but remains a difficult construct to assess in language tests. In the oral proficiency interview (OPI), strategic competence is typically assessed through the use of role-plays with a complication. Assessment of test-taker performance on the role-play is subjective and is contingent on raters accurately identifying interactional evidence of strategic competence. Accordingly, validation of the strategic competence exhibited in role-plays has been mostly interpretive. To obtain evidential support for an interpretive argument that role-plays can indeed isolate and provide assessment evidence of strategic competence, the criterion of ontological realism is applied in this study. Towards that end, eleven samples of English-as-a-foreign-language OPI role-plays with a complication were judged by 52 untrained English native speakers. Evidence in support of the ontological validity of assessing strategic competence via role-plays is presented through analyses of the untrained raters’ judgments, augmented by quantitative analyses that identify sources of variation among the raters, including a post-study additional round of coding in which the notion of “success” in the role-plays was examined more granularly than can be done with dichotomous decisions.

© 2025 John Benjamins Publishing Company
Loading

Article metrics loading...

/content/journals/10.1075/ap.21010.oco
2025-08-05
2026-04-13

  • From This Site

    /content/journals/10.1075/ap.21010.oco
    dcterms_title,dcterms_subject,pub_keyword
    -contentType:Journal -contentType:Contributor -contentType:Concept -contentType:Institution
    10
    5
Loading full text...

Full text loading...

References

  1. Astington, J. W. & Dack, L. A.
    (2008) Theory of mind. InM. M. Haith & J. B. Benson (Eds.), Encyclopedia of infant and early childhood development (pp. 343–356). Academic Press. 10.1016/B978‑012370877‑9.00163‑8
     https://doi.org/10.1016/B978-012370877-9.00163-8 [Google Scholar]
  2. Bachman, L. F.
    (1990) Fundamental considerations in language testing. Oxford University Press.
     [Google Scholar]
  3. Bachman, L. F., & Palmer, A. S.
    (1996) Language testing in practice: Designing and developing useful language tests. Oxford University Press.
     [Google Scholar]
  4. Bachman, L. F., & Palmer, A.
    (2010) Language assessment in practice. Oxford University Press.
     [Google Scholar]
  5. Bachman, L. F. & Savignon, S. J.
    (1986) The evaluation of communicative language proficiency: A critique of the ACTFL oral interview. The Modern Language Journal, 70(4), 380–390. 10.1111/j.1540‑4781.1986.tb05294.x
     https://doi.org/10.1111/j.1540-4781.1986.tb05294.x [Google Scholar]
  6. Baron-Cohen, S.
    (1991) Precursors to a theory of mind: Understanding attention in others. InA. Whiten (Ed.), Natural theories of mind: Evolution, development, and simulation of everyday mindreading (pp. 233–251). Blackwell.
     [Google Scholar]
  7. Bejar, I. I.
    (2012) Rater cognition: Implications for validity. Educational Measurement: Issues and Practice, 31(3), 2–9. 10.1111/j.1745‑3992.2012.00238.x
     https://doi.org/10.1111/j.1745-3992.2012.00238.x [Google Scholar]
  8. Borsboom, D.
    (2006) The attack of the psychometricians. Psychometrika, 71(3), 425–440. 10.1007/s11336‑006‑1447‑6
     https://doi.org/10.1007/s11336-006-1447-6 [Google Scholar]
  9. Borsboom, D., Mellenbergh, G. J., & van Heerden, J.
    (2004) The concept of validity. Psychological Review, 111(4), 1061–1071. 10.1037/0033‑295X.111.4.1061
     https://doi.org/10.1037/0033-295X.111.4.1061 [Google Scholar]
  10. Canale, M.
    (1983) On some dimensions of language proficiency. InJ. W. Oller (Ed.), Issues in language testing research (pp. 333–342). Newbury House.
     [Google Scholar]
  11. Canale, M., & Swain, M.
    (1981) A theoretical framework for communicative competence. InA. S. Palmer, P. J. M. Groot, & G. A. Trosper (Eds.), The construct validation of tests of communicative competence (pp. 31–36). Teachers of English to Speakers of Other Languages (TESOL).
     [Google Scholar]
  12. Chapelle, C. A.
    (2018, Mar.21). Validity arguments in language assessment: Contributions from applied linguistics [Opening address]. Language Assessment Research Conference, Iowa State University, Ames, IA, United States.
     [Google Scholar]
  13. Frost, K., Elder, C., & Wigglesworth, G.
    (2012) Investigating the validity of an integrated listening-speaking task: A discourse-based analysis of test takers’ oral performances. Language Testing, 29(3), 345–369. 10.1177/0265532211424479
     https://doi.org/10.1177/0265532211424479 [Google Scholar]
  14. Fulcher, G.
    (2003) Testing second language speaking. Pearson Longman.
     [Google Scholar]
  15. Galaczi, E. & Ockey, G.
    (2022, Mar.10). Why is interaction so difficult to assess and what can we do about it? [Symposium]. Language Testing Research Colloquium, online. https://cdn.ymaws.com/www.iltaonline.com/resource/resmgr/images/ltrc2022/ltrc_2022_program.pdf
     [Google Scholar]
  16. Harding, L.
    (2014) Communicative language testing: Current issues and future research. Language Assessment Quarterly, 11(2), 186–197. 10.1080/15434303.2014.895829
     https://doi.org/10.1080/15434303.2014.895829 [Google Scholar]
  17. Herzog, M.
    (n.d.). History of the ILR scale. Interagency Language Roundtable. https://govtilr.org/Skills/IRL%20Scale%20History.htm
     [Google Scholar]
  18. Hinkle, D. E., Wiersma, W., & Jurs, S. G.
    (2003) Applied statistics for the behavioral sciences. Houghton Mifflin Company.
     [Google Scholar]
  19. Hymes, D. H.
    (1972) On communicative competence. InJ. B. Pride & J. Holmes (Eds.), Sociolinguistics (pp. 269–293). Penguin.
     [Google Scholar]
  20. Jacoby, S., & McNamara, T.
    (1999) Locating competence. English for Specific Purposes, 18(4), 213–241. 10.1016/S0889‑4906(97)00053‑7
     https://doi.org/10.1016/S0889-4906(97)00053-7 [Google Scholar]
  21. Kane, M.
    (1992) An argument-based approach to validity. Psychological Bulletin, 112(3), 527–535. 10.1037/0033‑2909.112.3.527
     https://doi.org/10.1037/0033-2909.112.3.527 [Google Scholar]
  22. (2006) Validation. InR. Brennen (Ed.), Educational measurement (pp. 17–64). Rowan and Littlefield Publishers.
     [Google Scholar]
  23. (2011) Validating score interpretations and uses: Messick lecture, Language Testing Research Colloquium, Cambridge, April 2010. Language Testing, 29(1), 3–17. 10.1177/0265532211417210
     https://doi.org/10.1177/0265532211417210 [Google Scholar]
  24. Kasper, G., & Rose, K.
    (2002) Pragmatic development in a second language. Blackwell Publishing.
     [Google Scholar]
  25. Kasper, G., & Ross, S. J.
    (2007) Multiple questions in oral proficiency interviews. Journal of Pragmatics, 371, 2045–2070. 10.1016/j.pragma.2007.07.011
     https://doi.org/10.1016/j.pragma.2007.07.011 [Google Scholar]
  26. (2013) Assessing second language pragmatics: An overview and introductions. InS. J. Ross & G. Kasper (Eds.), Assessing second language pragmatics (pp. 1–40). Palgrave Macmillan. 10.1057/9781137003522_1
     https://doi.org/10.1057/9781137003522_1 [Google Scholar]
  27. Knoch, U. & Chapelle, C. A.
    (2018) Validation of rating processes within an argument-based framework. Language Testing, 35(4), 477–499. 10.1177/0265532217710049
     https://doi.org/10.1177/0265532217710049 [Google Scholar]
  28. Kormos, J.
    (1999) Simulating conversations in oral-proficiency assessment: A conversation analysis of role plays and non-scripted interviews in language exams. Language Testing, 16(2), 163–188. 10.1177/026553229901600203
     https://doi.org/10.1177/026553229901600203 [Google Scholar]
  29. Kramsch, C.
    (1986) From language proficiency to interactional competence. The Modern Language Journal, 70(4), 366–372. 10.1111/j.1540‑4781.1986.tb05291.x
     https://doi.org/10.1111/j.1540-4781.1986.tb05291.x [Google Scholar]
  30. McNamara, T.
    (1996) Measuring second language performance. Longman.
     [Google Scholar]
  31. (1997) ‘Interaction’ in second language performance assessment: Whose performance?Applied Linguistics, 18(4), 446–466. 10.1093/applin/18.4.446
     https://doi.org/10.1093/applin/18.4.446 [Google Scholar]
  32. Okada, Y.
    (2010) Role-play in oral proficiency interviews: Interactive footing and interactional competencies. Journal of Pragmatics, 42(6), 1647–1668. 10.1016/j.pragma.2009.11.002
     https://doi.org/10.1016/j.pragma.2009.11.002 [Google Scholar]
  33. Plough, I., Banerjee, J., & Iwashita, N.
    (2018) Interactional competence: Genie out of the bottle. Language Testing, 35(3), 427–455. 10.1177/0265532218772325
     https://doi.org/10.1177/0265532218772325 [Google Scholar]
  34. Roever, C., & Kasper, G.
    (2018) Speaking in turns and sequences: Interactional competence as a target construct in testing speaking. Language Testing, 35(2), 331–355. 10.1177/0265532218758128
     https://doi.org/10.1177/0265532218758128 [Google Scholar]
  35. Ross, S. J.
    (1992) Accommodative questions in oral proficiency interviews. Language Testing, 9(2), 173–185. 10.1177/026553229200900205
     https://doi.org/10.1177/026553229200900205 [Google Scholar]
  36. (2007) A comparative task-in-interaction analysis of OPI backsliding. Journal of Pragmatics, 39(11), 2017–2044. 10.1016/j.pragma.2007.07.010
     https://doi.org/10.1016/j.pragma.2007.07.010 [Google Scholar]
  37. (2017) Interviewing for language proficiency: Interaction and interpretation. Palgrave MacMillan. 10.1007/978‑3‑319‑60528‑9
     https://doi.org/10.1007/978-3-319-60528-9 [Google Scholar]
  38. Ross, S. J., & O’Connell, S. P.
    (2013) The situation with complication as a site for strategic competence. InS. J. Ross & G. Kasper (Eds.), Assessing second language pragmatics (pp. 311–326). Palgrave Macmillan. 10.1057/9781137003522_12
     https://doi.org/10.1057/9781137003522_12 [Google Scholar]
  39. Salaberry, R.
    (2000) Revising the revised format of the ACTFL Oral Proficiency Interview. Language Testing, 17(3), 289–310. 10.1177/026553220001700301
     https://doi.org/10.1177/026553220001700301 [Google Scholar]
  40. Sato, T., & McNamara, T.
    (2019) What counts in second language oral communication ability? The perspective of linguistic laypersons. Applied Linguistics, 40(4), 894–916. 10.1093/applin/amy032
     https://doi.org/10.1093/applin/amy032 [Google Scholar]
  41. Schmidgall, J., & Powers
    (2020) TOEIC Writing test scores as indicators of the functional adequacy of writing in the international workplace: Evaluation by linguistic laypersons. Assessing Writing461, 10049210.1016/j.asw.2020.100492
     https://doi.org/10.1016/j.asw.2020.100492 [Google Scholar]
  42. Stansfield, C. W., & Kenyon, D. M.
    (1992) Research on the comparability of the oral proficiency interview and the simulated oral proficiency interview. System, 20(3), 347–364. 10.1016/0346‑251X(92)90045‑5
     https://doi.org/10.1016/0346-251X(92)90045-5 [Google Scholar]
  43. Stokoe, E.
    (2013) The (in)authenticity of simulated talk: Comparing role-played and actual interaction and the implications for communication training. Research on Language & Social Interaction, 46(2), 165–185. 10.1080/08351813.2013.780341
     https://doi.org/10.1080/08351813.2013.780341 [Google Scholar]
  44. Surface, E. A., & Dierdorff, E. C.
    (2003) Reliability and the ACTFL Oral Proficiency Interview: Reporting indices of interrater consistency and agreement for 19 languages. Foreign Language Annals, 36(4), 507–519. 10.1111/j.1944‑9720.2003.tb02140.x
     https://doi.org/10.1111/j.1944-9720.2003.tb02140.x [Google Scholar]
  45. Taguchi, N., & Roever, C.
    (2017) Second language pragmatics. Oxford University Press.
     [Google Scholar]
  46. Trim, J. L. M.
    (2012) Preface. InA. Green (Ed.), Language functions revisited: Theoretical and empirical bases for language construct definition across the ability range, (pp. xxi–xli). Cambridge University Press.
     [Google Scholar]
  47. Youn, S. J.
    (2015) Validity argument for assessing L2 pragmatics in interaction using mixed methods. Language Testing, 32(2), 199–225. 10.1177/0265532214557113
     https://doi.org/10.1177/0265532214557113 [Google Scholar]
  48. Youn, S. J., & Bi, N. Z.
    (2019) Investigating test-takers’ strategy use in task-based L2 pragmatic speaking assessment. Intercultural Pragmatics, 16(2), 185–218. 10.1515/ip‑2019‑0009
     https://doi.org/10.1515/ip-2019-0009 [Google Scholar]
/content/journals/10.1075/ap.21010.oco
Loading
Ontological realism as a validity criterion in second-language strategic competence assessment
Applied Pragmatics 7, 56 (2025); https://doi.org/10.1075/ap.21010.oco
/content/journals/10.1075/ap.21010.oco
/content/journals/10.1075/ap.21010.oco
Loading

Data & Media loading...

  • Article Type: Research Article
Keyword(s): interpretive argument; ontological realism; oral proficiency interview; strategic competence; strategic competence assessment; validity criterion

Most Cited

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error