1887
  1. Home
  2. e-Journals
  3. Applied Pragmatics
  4. Volume 5, Issue 1
  5. Article
Volume 5, Issue 1
  • ISSN 2589-109X
  • E-ISSN: 2589-1103
USD
Buy:$35.00 + Taxes

Abstract

Abstract

There has long been interest in the effect of stay abroad on pragmatic development in second language (L2) pragmatics research. However, few testing instruments exist to assess pragmatics learning in stay abroad and provide information about learners’ development to help institutions evaluate the success of stay abroad programs. In this study, we describe the adaptation and validation of Roever’s (2005) test of second language routine formulae to be used large-scale with Japanese stay-abroad learners. We follow Kane’s (2006) argument-based approach to validation, evaluating the first four inferences: Domain description, Evaluation, Generalization, and Explanation. We also investigate the test’s practicality within the institutional setting of intended use. We find evidence supporting the use of the test for providing information on pragmatics learning during stay abroad. The test can differentiate between low- and high-ability learners, shows adequate reliability, and reflects the roles of proficiency and exposure in pragmatics learning in stay abroad contexts. The test is also practical and requires few resources. Further steps, such as the investigation of the Extrapolation and Decision inference, are discussed.

© John Benjamins Publishing Company
Loading

Article metrics loading...

/content/journals/10.1075/ap.20015.roe
2022-03-04
2025-02-09

  • From This Site

    /content/journals/10.1075/ap.20015.roe
    dcterms_title,dcterms_subject,pub_keyword
    -contentType:Journal -contentType:Contributor -contentType:Concept -contentType:Institution
    10
    5
Loading full text...

Full text loading...

References

  1. Bachman, L. F.
    (2004) Statistical analyses for language assessment. Cambridge University Press. 10.1017/CBO9780511667350
     https://doi.org/10.1017/CBO9780511667350 [Google Scholar]
  2. Bachman, L. F., & Palmer, A. S.
    (2010) Language assessment in practice: Developing language tests and justifying their use in the real world. Oxford University Press.
     [Google Scholar]
  3. Baker-Smemoe, W., Dewey, D. P., Bown, J., & Martinsen, R. A.
    (2014) Variables affecting L2 gains during study abroad. Foreign Language Annals, 47(3), 464–486. 10.1111/flan.12093
     https://doi.org/10.1111/flan.12093 [Google Scholar]
  4. Bardovi-Harlig, K.
    (2009) Conventional expressions as a pragmalinguistic resource: Recognition and production of conventional expressions in L2 pragmatics. Language Learning, 59(4), 755–795. 10.1111/j.1467‑9922.2009.00525.x
     https://doi.org/10.1111/j.1467-9922.2009.00525.x [Google Scholar]
  5. Bardovi-Harlig, K., & Bastos, M. T.
    (2011) Proficiency, length of stay, and intensity of interaction and the acquisition of conventional expressions in L2 pragmatics. Intercultural Pragmatics, 81, 347–384. 10.1515/iprg.2011.017
     https://doi.org/10.1515/iprg.2011.017 [Google Scholar]
  6. Bardovi-Harlig, K., & Dörnyei, Z.
    (1998) Do language learners recognize pragmatic violations? Pragmatic vs. grammatical awareness in instructed L2 learning. TESOL Quarterly, 321, 233–259. 10.2307/3587583
     https://doi.org/10.2307/3587583 [Google Scholar]
  7. Bardovi-Harlig, K., Mossman, S., & Vellenga, H. E.
    (2015) The effect of instruction on pragmatic routines in academic discussion. Language Teaching Research, 19(3), 324–350. 10.1177/1362168814541739
     https://doi.org/10.1177/1362168814541739 [Google Scholar]
  8. Barron, A.
    (2019) Pragmatic development and stay abroad. Journal of Pragmatics, 1461, 43–53. 10.1016/j.pragma.2019.05.003
     https://doi.org/10.1016/j.pragma.2019.05.003 [Google Scholar]
  9. Brecht, R., Davidson, D., & Ginsberg, R.
    (1995) Predictors of foreign language gain during study abroad. InB. F. Freed (Ed.), Second language acquisition in a study abroad context (pp.37–66). John Benjamins. 10.1075/sibil.9.05bre
     https://doi.org/10.1075/sibil.9.05bre [Google Scholar]
  10. Chapelle, C. A.
    (2021) Argument-based validation in testing and assessment. Sage. 10.4135/9781071878811
     https://doi.org/10.4135/9781071878811 [Google Scholar]
  11. Chapelle, C. A., Enright, M. K., & Jamieson, J.
    (Eds.) (2008) Building a validity argument for the Test of English as a Foreign Language. Routledge.
     [Google Scholar]
  12. Coulmas, F.
    (1979) On the sociolinguistic relevance of routine formulae. Journal of Pragmatics, 31, 239–266. 10.1016/0378‑2166(79)90033‑X
     https://doi.org/10.1016/0378-2166(79)90033-X [Google Scholar]
  13. Davidson, D. E.
    (2010) Study abroad: When, how long, and with what results? New data from the Russian front. Foreign Language Annals, 43(1), 6–26. 10.1111/j.1944‑9720.2010.01057.x
     https://doi.org/10.1111/j.1944-9720.2010.01057.x [Google Scholar]
  14. Di Silvio, F., Donovan, A., & Malone, M. E.
    (2014) The effect of study abroad homestay placements: Participant perspectives and oral proficiency gains. Foreign Language Annals, 47(1), 168–188. 10.1111/flan.12064
     https://doi.org/10.1111/flan.12064 [Google Scholar]
  15. Ebel, R. L.
    (1964) The social consequences of educational testing. InProceedings of the 1963 invitational conference on testing problems (pp.130–143). Educational Testing Service.
     [Google Scholar]
  16. ETS
    ETS (2010) Linking TOEFL iBT scores to IELTS scores – A research report. ETS.
     [Google Scholar]
  17. Geeslin, K. L., & Schmidt, L. B.
    (2018) Study abroad and L2 learner attitudes. InC. Sanz & A. Morales-Font (Eds.), The Routledge handbook of study abroad research and practice (pp.385–405). Routledge. 10.4324/9781315639970‑26
     https://doi.org/10.4324/9781315639970-26 [Google Scholar]
  18. Hulstijn, J. H.
    (2011) Language proficiency in native and nonnative speakers: An agenda for research and suggestions for second-language assessment. Language Assessment Quarterly, 8(3), 229–249. 10.1080/15434303.2011.565844
     https://doi.org/10.1080/15434303.2011.565844 [Google Scholar]
  19. (2015) Language proficiency in native and non-native speakers: Theory and research (Vol.411). John Benjamins Publishing Company. 10.1075/lllt.41
     https://doi.org/10.1075/lllt.41 [Google Scholar]
  20. Ikeda, N.
    (2017) Measuring L2 oral pragmatic abilities for use in social contexts: Development and validation of an assessment instrument for L2 pragmatics performance in university settings (Unpublished doctoral dissertation). The University of Melbourne, Australia.
  21. Kane, M. T.
    (2006) Validation. InR. L. Brennan (Ed.), Educational measurement (4th ed.) (pp.17–64). American Council on Education/Praeger Publishers.
     [Google Scholar]
  22. Matsumura, S.
    (2001) Learning the rules for offering advice: A quantitative approach to second language socialization. Language Learning, 51(4), 635–679. 10.1111/0023‑8333.00170
     https://doi.org/10.1111/0023-8333.00170 [Google Scholar]
  23. (2003) Modelling the relationships among interlanguage pragmatic development, L2 proficiency, and exposure to L2. Applied Linguistics, 24(4), 465–491. 10.1093/applin/24.4.465
     https://doi.org/10.1093/applin/24.4.465 [Google Scholar]
  24. Messick, S.
    (1989) Validity. InR. L. Linn (Ed.), Educational Measurement (3rd ed.) (pp.13–103). American Council on Education & Macmillan.
     [Google Scholar]
  25. Myles, F., & Cordier, C.
    (2017) Formulaic sequence (FS) cannot be an umbrella term in SLA: Focusing on psycholinguistic FSs and their identification. Studies in Second Language Acquisition, 39(1), 3–28. 10.1017/S027226311600036X
     https://doi.org/10.1017/S027226311600036X [Google Scholar]
  26. Pawley, A., & Syder, F. H.
    (1983) Two puzzles for linguistic theory: Nativelike selection and nativelike fluency. InJ. C. Richards & R. W. Schmidt (Eds.), Language and communication (pp.191–226). Longman.
     [Google Scholar]
  27. Powers, D. E., Roever, C., Huff, K., & Trapani, C.
    (2003) Validating LanguEdge Courseware Scores Against Faculty Ratings and Student Self-assessments. (ETS Research Report RR-03-11). Educational Testing Service. 10.1002/j.2333‑8504.2003.tb01903.x
     https://doi.org/10.1002/j.2333-8504.2003.tb01903.x [Google Scholar]
  28. Ren, W.
    (2013) The effect of study abroad on the pragmatic development of the internal modification of refusals. Pragmatics, 231, 715–741. 10.1075/prag.23.4.06ren
     https://doi.org/10.1075/prag.23.4.06ren [Google Scholar]
  29. (2015) L2 pragmatic development in study abroad contexts. Peter Lang. 10.3726/978‑3‑0351‑0734‑0
     https://doi.org/10.3726/978-3-0351-0734-0 [Google Scholar]
  30. (2019) Pragmatic development of Chinese during study abroad: A cross-sectional study of learner requests. Journal of Pragmatics, 1461, 137–149. 10.1016/j.pragma.2019.01.017
     https://doi.org/10.1016/j.pragma.2019.01.017 [Google Scholar]
  31. Roever, C.
    (2005) Testing ESL pragmatics. Peter Lang. 10.3726/978‑3‑653‑04780‑6
     https://doi.org/10.3726/978-3-653-04780-6 [Google Scholar]
  32. (2012) What learners get for free (and when): Learning of routine formulae in ESL and EFL environments. ELT Journal, 66(1), 10–21. 10.1093/elt/ccq090
     https://doi.org/10.1093/elt/ccq090 [Google Scholar]
  33. Roever, C., Fraser, C., & Elder, C.
    (2014) Testing ESL sociopragmatics. Peter Lang. 10.3726/978‑3‑653‑04598‑7
     https://doi.org/10.3726/978-3-653-04598-7 [Google Scholar]
  34. Roever, C., & Phakiti, A.
    (2018) Quantitative methods in language assessment and acquisition research. Routledge. 10.4324/9780203067659
     https://doi.org/10.4324/9780203067659 [Google Scholar]
  35. Roever, C., Wang, S., & Brophy, S.
    (2014) Learner background factors and learning of second language pragmatics. International Review of Applied Linguistics, 52(4), 377–401. 10.1515/iral‑2014‑0016
     https://doi.org/10.1515/iral-2014-0016 [Google Scholar]
  36. Sánchez-Hernández, A., & Alcón-Soler, E.
    (2019) Pragmatic gains in the study abroad context: Learners’ experiences and recognition of pragmatic routines. Journal of Pragmatics, 1461, 54–71. 10.1016/j.pragma.2018.08.006
     https://doi.org/10.1016/j.pragma.2018.08.006 [Google Scholar]
  37. Schauer, G. A.
    (2006) Pragmatic awareness in ESL and EFL contexts: Contrast and development. Language Learning, 56(2), 269–318. 10.1111/j.0023‑8333.2006.00348.x
     https://doi.org/10.1111/j.0023-8333.2006.00348.x [Google Scholar]
  38. Schauer, G.
    (2009) Interlanguage pragmatic development: The study abroad context. Bloomsbury Publishing.
     [Google Scholar]
  39. Schenker, T.
    (2018) Making short-term study abroad count – Effects on German language skills. Foreign Language Annals, 51(2), 411–429. 10.1111/flan.12339
     https://doi.org/10.1111/flan.12339 [Google Scholar]
  40. Sell, F., Renkwitz, K., Sickinger, P., & Schneider, K. P.
    (2019) Measuring pragmatic competence on the functional and lexical level: The development of German high-school students’ requests during a stay abroad in Canada. Journal of Pragmatics, 1461, 106–120. 10.1016/j.pragma.2018.11.005
     https://doi.org/10.1016/j.pragma.2018.11.005 [Google Scholar]
  41. Shively, R. L.
    (2013) Learning to be funny in Spanish during study abroad: L2 humor development. Modern Language Journal, 971, 930–946. 10.1111/j.1540‑4781.2013.12043.x
     https://doi.org/10.1111/j.1540-4781.2013.12043.x [Google Scholar]
  42. (2015) Developing interactional competence during study abroad: Listener responses in L2 Spanish. System, 481, 86–98. 10.1016/j.system.2014.09.007
     https://doi.org/10.1016/j.system.2014.09.007 [Google Scholar]
  43. Sireci, S., & Faulkner-Bond, M.
    (2014) Validity evidence based on test content. Psicothema, 26(1), 100–107. 10.7334/psicothema2013.256
     https://doi.org/10.7334/psicothema2013.256 [Google Scholar]
  44. Sinclair, J.
    (1991) Corpus, concordance and collocation. Oxford University Press.
     [Google Scholar]
  45. Taguchi, N.
    (2014) Development of interactional competence in Japanese as a second language: Use of incomplete sentences as interactional resources. Modern Language Journal, 98(2), 518–535. 10.1111/modl.12087
     https://doi.org/10.1111/modl.12087 [Google Scholar]
  46. Taguchi, N., Li, S., & Xiao, F.
    (2013) Production of formulaic expressions in L2 Chinese: A developmental investigation in a study abroad context. Chinese as a Second Language Research, 2(1), 23–58. 10.1515/caslar‑2013‑0021
     https://doi.org/10.1515/caslar-2013-0021 [Google Scholar]
  47. Taguchi, N., Xiao, F., & Li, S.
    (2016) Effects of intercultural competence and social contact on speech act production in a Chinese study abroad context. Modern Language Journal, 1001, 775–796. 10.1111/modl.12349
     https://doi.org/10.1111/modl.12349 [Google Scholar]
  48. Tannenbaum, R. J., & Baron, P. A.
    (2011) Mapping TOEFL® ITP scores onto the Common European Framework of Reference. ETS.
     [Google Scholar]
  49. Tare, M., Golonka, E., Lancaster, A. K., Bonilla, C., Doughty, C. J., Belnap, R. K., & Jackson, S. R.
    (2018) The role of cognitive aptitudes in a study abroad language-learning environment. InC. Sanz & A. Morales-Font (Eds.), The Routledge handbook of study abroad research and practice (pp.406–420). Routledge. 10.4324/9781315639970‑27
     https://doi.org/10.4324/9781315639970-27 [Google Scholar]
  50. Tullock, B., & Ortega, L.
    (2017) Fluency and multilingualism in study abroad: Lessons from a scoping review. System, 711, 7–21. 10.1016/j.system.2017.09.019
     https://doi.org/10.1016/j.system.2017.09.019 [Google Scholar]
  51. Xiao, F.
    (2015) Adult second language learners’ pragmatic development in the study-abroad context: A review. Frontiers: The Interdisciplinary Journal of Study Abroad, 251, 132–149. 10.36366/frontiers.v25i1.349
     https://doi.org/10.36366/frontiers.v25i1.349 [Google Scholar]
  52. Xu, Y.
    (2019) Changes in interlanguage complexity during study abroad: A meta-analysis. System, 801, 199–211. 10.1016/j.system.2018.11.008
     https://doi.org/10.1016/j.system.2018.11.008 [Google Scholar]
  53. Yang, J.
    (2016) CFL learners’ recognition and production of pragmatic routine formulae. Chinese as a Second Language. The Journal of the Chinese Language Teachers Association, USA, 51(1), 29–61. 10.1075/csl.51.1.02yan
     https://doi.org/10.1075/csl.51.1.02yan [Google Scholar]
  54. Youn, S. J.
    (2015) Validity argument for assessing L2 pragmatics in interaction using mixed methods. Language Testing, 32(2), 199–225. 10.1177/0265532214557113
     https://doi.org/10.1177/0265532214557113 [Google Scholar]
/content/journals/10.1075/ap.20015.roe
Loading
Validating a test of L2 routine formulae to detect pragmatics learning in stay abroad
Applied Pragmatics 5, 41 (2023); https://doi.org/10.1075/ap.20015.roe
/content/journals/10.1075/ap.20015.roe
/content/journals/10.1075/ap.20015.roe
Loading

Data & Media loading...

  • Article Type: Research Article
Keyword(s): argument-based validation; conventional expressions; L2 pragmatics; routine formulae; stay abroad; study abroad

Most Cited

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error