1887
image of The effect of lexical complexity on grading of Swedish EFL learners’ texts during high-stakes
exams
USD
Buy:$35.00 + Taxes

Abstract

Abstract

The present study concerns the effect of lexical complexity on grading of Swedish EFL learners’ texts during high-stakes exams. A learner corpus consisting of 142 texts graded by expert raters and 175 texts graded by teachers was analysed to establish if the latter graded in agreement with the former as intended by the Swedish National Agency for Education (SNAE). Four indices of lexical complexity available in TAALED and TAALES were chosen to explore if this is the case. The method includes conducting ordinal regression with interactions to determine the effect of the independent variables on grade and if these variables have the same effect in texts graded by teachers and expert raters. The findings reveal a discrepancy between expert raters and teachers as they appear to consider lexical complexity to a different extent. It was also found that expert raters and teachers graded more in agreement during source-based writing tasks compared to independent writing tasks.

Loading

Article metrics loading...

/content/journals/10.1075/ijlcr.23038.hol
2025-01-13
2025-02-15
Loading full text...

Full text loading...

References

  1. Bergström, D., Norberg, C., & Nordlund, M.
    (2021) “Words are picked up along the way”– Swedish EFL teachers’ conceptualizations of vocabulary knowledge and learning. Language Awareness, , –. 10.1080/09658416.2021.1893326
    https://doi.org/10.1080/09658416.2021.1893326 [Google Scholar]
  2. Bottini, R., & Le Foll, E.
    (in press). The more proficient the learners, the less sophisticated their L2 vocabulary? The curious effect of the reference corpus on mean-frequency measures of lexical sophistication in written and spoken production. International Journal of Learner Corpus Research. Preprint available on: https://osf.io/mne6w/
    [Google Scholar]
  3. Brysbaert, M., New, B., & Keuleers, E.
    (2012) Adding part-of-speech information to the SUBTLEX-US word frequencies. Behavior Research Methods, , –. 10.3758/s13428‑012‑0190‑4
    https://doi.org/10.3758/s13428-012-0190-4 [Google Scholar]
  4. Christensen, R.
    (2023) ordinal — Regression Models for Ordinal Data. R package version, 2023.12-4.1. https://cran.r-project.org/package=ordinal
    [Google Scholar]
  5. Covington, M. A., & McFall, J. D.
    (2010) Cutting the gordian knot: The moving-average type–token ratio (MATTR). Journal of Quantitative Linguistics, (), –. 10.1080/09296171003643098
    https://doi.org/10.1080/09296171003643098 [Google Scholar]
  6. Crossley, S. A., Roscoe, R., & McNamara, D. S.
    (2014) What is successful writing? An investigation into the multiple ways writers can write successful essays. Written Communication, (), –. 10.1177/0741088314526354
    https://doi.org/10.1177/0741088314526354 [Google Scholar]
  7. Council of Europe
    Council of Europe (2001) Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge University Press. https://rm.coe.int/1680459f97
    [Google Scholar]
  8. Cumming, A., Kantor, R., Baba, K., Erdosy, U., Eouanzoui, K., & James, M.
    (2005) Differences in written discourse in independent and integrated prototype tasks for next generation TOEFL. Assessing Writing, (), –. 10.1016/j.asw.2005.02.001
    https://doi.org/10.1016/j.asw.2005.02.001 [Google Scholar]
  9. Educational Testing Service
    Educational Testing Service (2023–05–30). Writing Scoring Guide. TOEFL iBT® Writing Scoring Guide Flyer (ets.org)
    [Google Scholar]
  10. Egbert, J., Larsson, T., & Biber, D.
    (2020) Doing linguistics with a corpus: Methodological considerations for the everyday user. Cambridge University Press. 10.1017/9781108888790
    https://doi.org/10.1017/9781108888790 [Google Scholar]
  11. Egbert, J., Burch, B., & Biber, D.
    (2020) Lexical dispersion and corpus design. International Journal of Corpus Linguistics, (), –. 10.1075/ijcl.18010.egb
    https://doi.org/10.1075/ijcl.18010.egb [Google Scholar]
  12. Egbert, J., & Burch, B.
    (2023) Which words matter most? Operationalizing lexical prevalence for rank-ordered word lists. Applied Linguistics, (), –. 10.1093/applin/amac030
    https://doi.org/10.1093/applin/amac030 [Google Scholar]
  13. Eguchi, M., & Kyle, K.
    (2020) Continuing to explore the multidimensional nature of lexical sophistication: The case of oral proficiency interviews. The Modern Language Journal, (), –. 10.1111/modl.12637
    https://doi.org/10.1111/modl.12637 [Google Scholar]
  14. Engber, C. A.
    (1995) The relationship of lexical proficiency to the quality of ESL compositions. Journal of Second Language Writing, (), –. 10.1016/1060‑3743(95)90004‑7
    https://doi.org/10.1016/1060-3743(95)90004-7 [Google Scholar]
  15. Erickson, G., Kao, T., & Lin, Y.
    (2010) Good practice in language testing and assessment: A matter of responsibility and respect. A new look at teaching and testing: English as subject and vehicle. Selected papers from the 2009 LTTC International Conference on English Language Teaching and Testing (pp.–). University of Gothenburg.
    [Google Scholar]
  16. Erickson, G.
    (2020) National assessment of foreign languages in Sweden. University of Gothenburg. https://www.gu.se/sites/default/files/2020-04/Nat_Assesment_of_Foreign_Lang_in_Swe2020.pdf
    [Google Scholar]
  17. Erickson, G., & Tholin, J.
    (2022) Overall, a good test, but… — Swedish lower secondary teachers’ perceptions and use of national test results of English. Languages, (), . 10.3390/languages7010064
    https://doi.org/10.3390/languages7010064 [Google Scholar]
  18. Fox, J., Weisberg, S., Price, B., Adler, D., Bates, D., Baud-Bovy, G., Bolker, B., Ellison, S., Firth, D., Friendly, M., Gorjanc, G., Graves, S., Heidelberger, R., Krivitsky, P., Laboissiere, R., Maechler, M., Monette, G., Murdoch, D., Nilsson, H., … & R-Core
    (2012) Package ‘car’ version, 2024 3.1–3. Vienna: R Foundation for Statistical Computing, (), . https://cran.uni-muenster.de/web/packages/car/car.pdf
    [Google Scholar]
  19. Guo, L., Crossley, S. A., & McNamara, D. S.
    (2013) Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, (), –. 10.1016/j.asw.2013.05.002
    https://doi.org/10.1016/j.asw.2013.05.002 [Google Scholar]
  20. Hashimoto, B. J., & Egbert, J.
    (2019) More than frequency? Exploring predictors of word difficulty for second language learners. Language Learning, (), –. 10.1111/lang.12353
    https://doi.org/10.1111/lang.12353 [Google Scholar]
  21. IELTS
    IELTS (2023–05–30). Writing Band Descriptors. Writing Band Descriptors (ielts.org)
    [Google Scholar]
  22. Jarvis, S.
    (2002) Short texts, best-fitting curves and new measures of lexical diversity. Language Testing, (), –. 10.1191/0265532202lt220oa
    https://doi.org/10.1191/0265532202lt220oa [Google Scholar]
  23. Kaatari, H., Wang, Y., & Larsson, T.
    (2024) Introducing the Swedish Learner English Corpus: A corpus that enables investigations of the impact of extramural activities on L2 writing. Corpora, (), –. 10.3366/cor.2024.0296
    https://doi.org/10.3366/cor.2024.0296 [Google Scholar]
  24. Kim, M., Crossley, S. A., & Kyle, K.
    (2018) Lexical sophistication as a multidimensional phenomenon: Relations to second language lexical proficiency, development, and writing quality. The Modern Language Journal, (), –. 10.1111/modl.12447
    https://doi.org/10.1111/modl.12447 [Google Scholar]
  25. Kleiber, C., Zeileis, A., & Zeileis, M. A.
    (2020) R package ‘aer’ version, 1.2–14. AER: Applied Econometrics with R (r-project.org)
    [Google Scholar]
  26. Kuperman, V., Stadthagen-Gonzalez, H., & Brysbaert, M.
    (2012) Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods, , –. 10.3758/s13428‑012‑0210‑4
    https://doi.org/10.3758/s13428-012-0210-4 [Google Scholar]
  27. Kyle, K.
    (2022) Measuring lexical richness. InS. Webb (Ed.), The Routledge Handbook of Vocabulary Studies (pp.–). Routledge.
    [Google Scholar]
  28. Kyle, K., & Crossley, S. A.
    (2015) Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Quarterly, (), –. 10.1002/tesq.194
    https://doi.org/10.1002/tesq.194 [Google Scholar]
  29. Kyle, K., & Crossley, S.
    (2016) The relationship between lexical sophistication and independent and source-based writing. Journal of Second Language Writing, , –. 10.1016/j.jslw.2016.10.003
    https://doi.org/10.1016/j.jslw.2016.10.003 [Google Scholar]
  30. Kyle, K., Crossley, S., & Berger, C.
    (2018) The tool for the automatic analysis of lexical sophistication (TAALES): version 2.0. Behavior Research Methods, , –. 10.3758/s13428‑017‑0924‑4
    https://doi.org/10.3758/s13428-017-0924-4 [Google Scholar]
  31. Kyle, K., Crossley, S. A., & Jarvis, S.
    (2021) Assessing the validity of lexical diversity indices using direct judgements. Language Assessment Quarterly, (), –. 10.1080/15434303.2020.1844205
    https://doi.org/10.1080/15434303.2020.1844205 [Google Scholar]
  32. Laufer, B., & Nation, P.
    (1995) Vocabulary size and use: Lexical richness in L2 written production. Applied Linguistics, (), –. 10.1093/applin/16.3.307
    https://doi.org/10.1093/applin/16.3.307 [Google Scholar]
  33. Mangiafico, S.
    (2024) rcompanion: Functions to support extension education program evaluation. Rutgers cooperative xxtension, New Brunswick, New Jersey. version 2.4.36, https://CRAN.R-project.org/package=rcompanion/
    [Google Scholar]
  34. Plakans, L., & Gebril, A.
    (2013) Using multiple texts in an integrated writing assessment: Source text use as a predictor of score. Journal of Second Language Writing, (), –. 10.1016/j.jslw.2013.02.003
    https://doi.org/10.1016/j.jslw.2013.02.003 [Google Scholar]
  35. R Development Core Team
    R Development Core Team (2023) R: A language and environment for statistical computing. [Computer software]. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/
  36. Read, J.
    (2000) Assessing Vocabulary. Cambridge University Press. 10.1017/CBO9780511732942
    https://doi.org/10.1017/CBO9780511732942 [Google Scholar]
  37. Schlegel, B. E., Steenbergen, M., Schlegel, M. B., & Imports, M. A. S. S.
    (2018) R Package brant, version 0.3–0. brant: Test for Parallel Regression Assumption (r-project.org)
    [Google Scholar]
  38. Schoonen, R., Van Gelderen, A., Stoel, R. D., Hulstijn, J., & De Glopper, K.
    (2011) Modeling the development of L1 and EFL writing proficiency of secondary school students. Language Learning, (), –. 10.1111/j.1467‑9922.2010.00590.x
    https://doi.org/10.1111/j.1467-9922.2010.00590.x [Google Scholar]
  39. Skolverket [Google Scholar]
  40. University of Gothenburg
    University of Gothenburg (2023–05–30a). Översikt — Skriftlig produktion och interaktion, Engelska 5. Exempel på uppgiftstyper för Engelska 5 | Projektet Nationella prov i främmande språk — Nafs, Göteborgs universitet (gu.se)
    [Google Scholar]
  41. University of Gothenburg
    University of Gothenburg (2023–05–30b). Exempel på texter bedömda som E, Engelska 5. Exempel på uppgiftstyper för Engelska 5 | Projektet Nationella prov i främmande språk — Nafs, Göteborgs universitet (gu.se)
    [Google Scholar]
  42. Venables, W. N., & Ripley, B. D.
    (2002) Modern Applied Statistics with S. (4th ed.) Springer. 10.1007/978‑0‑387‑21706‑2
    https://doi.org/10.1007/978-0-387-21706-2 [Google Scholar]
  43. Vögelin, C., Jansen, T., Keller, S. D., Machts, N., & Möller, J.
    (2019) The influence of lexical features on teacher judgements of ESL argumentative essays. Assessing Writing, , –. 10.1016/j.asw.2018.12.003
    https://doi.org/10.1016/j.asw.2018.12.003 [Google Scholar]
  44. Warnby, M.
    (2022) Receptive academic vocabulary knowledge and extramural English involvement — is there a correlation?. ITL-International Journal of Applied Linguistics, (), –. 10.1075/itl.21021.war
    https://doi.org/10.1075/itl.21021.war [Google Scholar]
  45. Yang, W.
    (2014) Mapping the relationships among the cognitive complexity of independent writing tasks, L2 writing quality, and complexity, accuracy and fluency of L2 writing [Doctoral dissertation, Georgia State University]. ScholarWorks@Georgia State University. 10.57709/5687326
    https://doi.org/10.57709/5687326
  46. Yang, W., Lu, X., & Weigle, S. C.
    (2015) Different topics, different discourse: Relationships among writing topic, measures of syntactic complexity, and judgments of writing quality. Journal of Second Language Writing, , –. 10.1016/j.jslw.2015.02.002
    https://doi.org/10.1016/j.jslw.2015.02.002 [Google Scholar]
  47. Yang, W., & Kim, Y.
    (2020) The effect of topic familiarity on the complexity, accuracy, and fluency of second language writing. Applied Linguistics Review, (), –. 10.1515/applirev‑2017‑0017
    https://doi.org/10.1515/applirev-2017-0017 [Google Scholar]
  48. Yoon, H. J.
    (2017) Linguistic complexity in L2 writing revisited: Issues of topic, proficiency, and construct multidimensionality. System, , –. 10.1016/j.system.2017.03.007
    https://doi.org/10.1016/j.system.2017.03.007 [Google Scholar]
  49. Yu, G.
    (2010) Lexical diversity in writing and speaking task performances. Applied Linguistics, (), –. 10.1093/applin/amp024
    https://doi.org/10.1093/applin/amp024 [Google Scholar]
  50. Zenker, F., & Kyle, K.
    (2021) Investigating minimum text lengths for lexical diversity indices. Assessing Writing, , 100505. 10.1016/j.asw.2020.100505
    https://doi.org/10.1016/j.asw.2020.100505 [Google Scholar]
/content/journals/10.1075/ijlcr.23038.hol
Loading
/content/journals/10.1075/ijlcr.23038.hol
Loading

Data & Media loading...

  • Article Type: Research Article
Keywords: writing assessment ; lexical complexity ; inter-rater reliability ; high-stakes exams
This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error