1887
Volume 175, Issue 1
  • ISSN 0019-0829
  • E-ISSN: 1783-1490

Abstract

Abstract

The study aims to demonstrate the procedure for constructing the CEFR-based Sentence Profile (CEFR-SP), a dataset with the CEFR levels assigned for sentences, and to identify the characteristics at each level. Basic statistics such as word length and sentence length are presented for each CEFR level for 7,511 carefully selected sentences, and statistical tests are conducted between adjacent levels to identify criterial features. The findings reveal significant differences in word length between adjacent levels, while word difficulty is not significant in discriminating levels at either end (A1–A2, C1–C2). Sentence length and depth are also not significant discriminators for higher levels (B2–C1, C1–C2). Notably, sentence-level data generally exhibit discriminative values compared to text-level statistics, indicating their direct capture of characteristics at each CEFR level.

Available under the CC BY 4.0 license.
Loading

Article metrics loading...

/content/journals/10.1075/itl.22018.uch
2024-03-22
2024-12-06
Loading full text...

Full text loading...

/deliver/fulltext/itl.22018.uch.html?itemId=/content/journals/10.1075/itl.22018.uch&mimeType=html&fmt=ahah

References

  1. Alfter, D., Tiedemann, T. L., & Volodina, E.
    (2020) Expert judgments versus crowdsourcing in ordering multi-word expressions. Eighth Swedish Language Technology Conference (SLTC). https://gubox.app.box.com/v/SLTC-2020-paper-16
    [Google Scholar]
  2. Arase, Y., Uchida, S., & Kajiwara, T.
    (2022) CEFR-based sentence difficulty annotation and assessment. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 6206–6219. 10.18653/v1/2022.emnlp‑main.416
    https://doi.org/10.18653/v1/2022.emnlp-main.416 [Google Scholar]
  3. Bax, S.
    (2012) Text inspector: Online text analysis tool. https://textinspector.com/
    [Google Scholar]
  4. Capel, A.
    (2015) The English vocabulary profile. InJ. Harrison & F. Barker (Eds.), English profile in practice51 (pp.9–27). Cambridge University Press.
    [Google Scholar]
  5. Chen, Y. H., & Baker, P.
    (2016) Investigating criterial discourse features across second language development: Lexical bundles in rated learner essays, CEFR B1, B2 and C1. Applied Linguistics, 37(6), 849–880. 10.1093/applin/amu065
    https://doi.org/10.1093/applin/amu065 [Google Scholar]
  6. Chujo, K., Oghigian, K., & Akasegawa, S.
    (2015) A corpus and grammatical browsing system for remedial EFL learners. InA. Leńko-Szymańska & A. Boulton (Eds.), Multiple affordances of language corpora for data-driven learning (pp.109–130). John Benjamins. 10.1075/scl.69.06chu
    https://doi.org/10.1075/scl.69.06chu [Google Scholar]
  7. Collins-Thompson, K.
    (2014) Computational assessment of text readability: A survey of current and future research. ITL-International Journal of Applied Linguistics, 165(2), 97–135. 10.1075/itl.165.2.01col
    https://doi.org/10.1075/itl.165.2.01col [Google Scholar]
  8. Douglas, C. E., & Fligner, A. M.
    (1991) On distribution-free multiple comparisons in the one-way analysis of variance. Communications in Statistics: Theory and Methods, 201, 127–139. 10.1080/03610929108830487
    https://doi.org/10.1080/03610929108830487 [Google Scholar]
  9. Dürlich, L., & François, T.
    (2018) EFLLex: A graded lexical resource for learners of English as a foreign language. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 873–879.
    [Google Scholar]
  10. Dwass, M.
    (1960) Some k-sample rank-order tests. InI. Olkin, S. G. Ghurye, W. Hoeffding, W. G. Madow, & B. H. Mann (Eds.), Contributions to Probability and Statistics. Essays in Honor of H. Hotelling (pp.198–202). Stanford University Press.
    [Google Scholar]
  11. Ehara, Y.
    (2018) Building an English vocabulary knowledge dataset of Japanese English-as-a-second-language learners using crowdsourcing. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 485–488.
    [Google Scholar]
  12. Flesch, R.
    (1948) A new readability yardstick. Journal of Applied Psychology, 32(3), 221–233. 10.1037/h0057532
    https://doi.org/10.1037/h0057532 [Google Scholar]
  13. François, T.
    (2015) When readability meets computational linguistics: A new paradigm in readability. Revue française de linguistique appliquée, 20(2), 79–97. hdl.handle.net/2078.1/193855. 10.3917/rfla.202.0079
    https://doi.org/10.3917/rfla.202.0079 [Google Scholar]
  14. François, T., & Fairon, C.
    (2012) An “AI readability” formula for French as a foreign language. Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, 466–477.
    [Google Scholar]
  15. Harrison, J.
    (2015) The English grammar profile. InJ. Harrison & F. Barker (Eds.), English profile in practice51 (pp.28–48). Cambridge University Press.
    [Google Scholar]
  16. Hawkins, J. A., & Filipović, L.
    (2012) Criterial features in L2 English: Specifying the reference levels of the Common European Framework (Vol.11). Cambridge University Press.
    [Google Scholar]
  17. Ishii, Y., & Tono, Y.
    (2018) Investigating Japanese EFL learners’ overuse/underuse of English grammar categories and their relevance to CEFR levels. Proceedings of the 4th Asia Pacific Corpus Linguistics Conference, 160–165.
    [Google Scholar]
  18. Jiang, C., Maddela, M., Lan, W., Zhong, Y., & Xu, W.
    (2020) Neural CRF model for sentence alignment in text simplification. arXiv preprintarXiv:2005.02324. 10.18653/v1/2020.acl‑main.709
    https://doi.org/10.18653/v1/2020.acl-main.709 [Google Scholar]
  19. Khallaf, N., & Sharoff, S.
    (2021) Automatic difficulty classification of Arabic sentences. Proceedings of the Sixth Arabic Natural Language Processing Workshop, 105–114.
    [Google Scholar]
  20. Kilgarriff, A., Husák, M., McAdam, K., Rundell, M., & Rychlý, P.
    (2008) GDEX: Automatically finding good dictionary examples in a corpus. Proceedings of the XIII EURALEX international congress, 11, 425–432.
    [Google Scholar]
  21. Kim, S.
    (2021) Generalizability of CEFR criterial grammatical features in a Korean EFL corpus across A1, A2, B1, and B2 levels. Language Assessment Quarterly, 18(3), 273–295. 10.1080/15434303.2020.1855647
    https://doi.org/10.1080/15434303.2020.1855647 [Google Scholar]
  22. Klare, G. R.
    (1968) The role of word frequency in readability. Elementary English, 45(1), 12–22.
    [Google Scholar]
  23. Marcus, M. P., Santorini, B., & Marcinkiewicz, M. A.
    (1993) Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2), 313–330.
    [Google Scholar]
  24. Nation, I.
    (2006) How large a vocabulary is needed for reading and listening?Canadian Modern Language Review, 63(1), 59–82. 10.3138/cmlr.63.1.59
    https://doi.org/10.3138/cmlr.63.1.59 [Google Scholar]
  25. Nishihara, D., Kajiwara, T., & Arase, Y.
    (2019) Controllable text simplification with lexical constraint loss. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, 260–266. 10.18653/v1/P19‑2036
    https://doi.org/10.18653/v1/P19-2036 [Google Scholar]
  26. Ozasa, T., Weir, G., & Fukui, M.
    (2007) Measuring readability for Japanese learners of English. Proceedings of the 12th Conference of Pan-Pacific Association of Applied Linguistics, 122–125.
    [Google Scholar]
  27. Pilán, I., Vajjala, S., & Volodina, E.
    (2016) A readable read: Automatic assessment of language learning materials based on linguistic complexity. arXiv preprintarXiv:1603.08868.
    [Google Scholar]
  28. Pilán, I., Volodina, E., & Johansson, R.
    (2014) Rule-based and machine learning approaches for second language sentence-level readability. Proceedings of the ninth workshop on innovative use of NLP for building educational applications, 174–184. 10.3115/v1/W14‑1821
    https://doi.org/10.3115/v1/W14-1821 [Google Scholar]
  29. Qi, P., Zhang, Y., Zhang, Y., Bolton, J., & Manning, C. D.
    (2020) Stanza: A Python natural language processing toolkit for many human languages. arXiv preprintarXiv:2003.07082. 10.18653/v1/2020.acl‑demos.14
    https://doi.org/10.18653/v1/2020.acl-demos.14 [Google Scholar]
  30. Salamoura, A., & Saville, N.
    (2010) Exemplifying the CEFR: Criterial features of written learner English from the English Profile Programme. InI. Bartning, M. Maisa, & I. Vedder (Eds.), Communicative proficiency and linguistic development: Intersections between SLA and language testing research (Vol.11, pp.101–132). European Second Language Association.
    [Google Scholar]
  31. Steel, R. G. D.
    (1960) A rank sum test for comparing all pairs of treatments, Technometrics, 21, 197–207. 10.1080/00401706.1960.10489894
    https://doi.org/10.1080/00401706.1960.10489894 [Google Scholar]
  32. Uchida, S., & Negishi, M.
    (2018) Assigning CEFR-J levels to English texts based on textual features. Proceedings of the 4th Asia Pacific Corpus Linguistics Conference, 463–467.
    [Google Scholar]
  33. Vajjala, S., & Meurers, D.
    (2014) Assessing the relative reading level of sentence pairs for text simplification. Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, 288–297. 10.3115/v1/E14‑1031
    https://doi.org/10.3115/v1/E14-1031 [Google Scholar]
/content/journals/10.1075/itl.22018.uch
Loading
/content/journals/10.1075/itl.22018.uch
Loading

Data & Media loading...

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error