1887
  1. Home
  2. e-Journals
  3. The Mental Lexicon
  4. Online First Articles
  5. Online First Article
image of NLP and education
USD
Buy:$35.00 + Taxes

Abstract

Abstract

This study examines the applicability of the Cloze test, a widely used tool for assessing text comprehension proficiency, while highlighting its challenges in large-scale implementation. To address these limitations, an automated correction approach was proposed, utilizing Natural Language Processing (NLP) techniques, particularly word embeddings (WE) models, to assess semantic similarity between expected and provided answers. Using data from Cloze tests administered to students in Brazil, WE models for Brazilian Portuguese (PT-BR) were employed to measure the semantic similarity of the responses. The results were validated through an experimental setup involving twelve judges who classified the students’ answers. A comparative analysis between the WE models’ scores and the judges’ evaluations revealed that GloVe was the most effective model, demonstrating the highest correlation with the judges’ assessments. This study underscores the utility of WE models in evaluating semantic similarity and their potential to enhance large-scale Cloze test assessments. Furthermore, it contributes to educational assessment methodologies by offering a more efficient approach to evaluating reading proficiency.

© John Benjamins Publishing Company
Loading

Article metrics loading...

/content/journals/10.1075/ml.24027.deg
2025-01-10
2025-01-20

  • From This Site

    /content/journals/10.1075/ml.24027.deg
    dcterms_title,dcterms_subject,pub_keyword
    -contentType:Journal -contentType:Contributor -contentType:Concept -contentType:Institution
    10
    5
Loading full text...

Full text loading...

References

  1. Bickley, A. C., Ellington, B. J., & Bickley, R. T.
    (1970) The cloze procedure: A conspectus. Journal of Reading Behavior, (), –. 10.1080/10862967009546900
     https://doi.org/10.1080/10862967009546900 [Google Scholar]
  2. Brown, J. D.
    (2002) Do cloze tests work? Or is it just an illusion?. Second Language Studies, (), –.
     [Google Scholar]
  3. (1980) Relative merits of four methods for scoring cloze tests. The Modern Language Journal, (), –. 10.1111/j.1540‑4781.1980.tb05198.x
     https://doi.org/10.1111/j.1540-4781.1980.tb05198.x [Google Scholar]
  4. Cardoso, P. B., Menezes, K. V., Freitas, F. O., & Freitag, R. M. K.
    (2024) Eficiência na leitura: medidas de precisão e velocidade entre alunos do Colégio de Aplicação da Universidade Federal de Sergipe. Revista Científica Sigma, (), –.
     [Google Scholar]
  5. Chandrasekaran, D., & Mago, V.
    (2021) Evolution of semantic similarity — a survey. ACM Computing Surveys (CSUR), (), –. 10.1145/3440755
     https://doi.org/10.1145/3440755 [Google Scholar]
  6. Cunha, N. D. B., & Santos, A. A. A. D.
    (2010) Estudos de validade entre instrumentos que avaliam habilidades linguísticas. Estudos de Psicologia (Campinas), , –. 10.1590/S0103‑166X2010000300003
     https://doi.org/10.1590/S0103-166X2010000300003 [Google Scholar]
  7. Darnell, D. K.
    (1968) The Development of an English Language Proficiency Test of Foreign Students, Using a Clozentropy Procedure. Final Report.
     [Google Scholar]
  8. Gorman, J., & Curran, J. R.
    (2006, July). Scaling distributional similarity to large corpora. InProceedings of the 21 International Conference on Computational Linguistics and 44 Annual Meeting of the Association for Computational Linguistics (pp.–). 10.3115/1220175.1220221
     https://doi.org/10.3115/1220175.1220221 [Google Scholar]
  9. Hartmann, N., Fonseca, E., Shulby, C., Treviso, M., Rodrigues, J., & Aluisio, S.
    (2017) Portuguese word embeddings: Evaluating on word analogies and natural language tasks. arXiv preprint arXiv:1708.06025.
     [Google Scholar]
  10. Lange, K., Kühn, S., & Filevich, E.
    (2015) “Just another tool for online studies” (JATOS): An easy solution for setup and management of web servers supporting online studies. PloS one, (), e0130834. 10.1371/journal.pone.0130834
     https://doi.org/10.1371/journal.pone.0130834 [Google Scholar]
  11. Levy, O., & Goldberg, Y.
    (2014) Neural word embedding as implicit matrix factorization. Advances in neural information processing systems, .
     [Google Scholar]
  12. Ling, W., Dyer, C., Black, A. W., & Trancoso, I.
    (2015) Two/too simple adaptations of word2vec for syntax problems. InProceedings of the 2015 conference of the North American chapter of Association for Computational Linguistics: human language technologies (pp.–). 10.3115/v1/N15‑1142
     https://doi.org/10.3115/v1/N15-1142 [Google Scholar]
  13. Lowry, D. T., & Marr, T. J.
    (1975) Clozentropy as a measure of international communication comprehension. Public Opinion Quarterly, (), –. 10.1086/268230
     https://doi.org/10.1086/268230 [Google Scholar]
  14. Mikolov, T., Chen, K., Corrado, G., & Dean, J.
    (2013) Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
     [Google Scholar]
  15. Mohammad, S. M., & Hirst, G.
    (2012) Distributional measures of semantic distance: A survey. arXiv preprint arXiv:1203.1858.
     [Google Scholar]
  16. Oller Jr, J. W., & Conrad, C. A.
    (1971) The Cloze technique and ESL proficiency. Language Learning, (), –. 10.1111/j.1467‑1770.1971.tb00057.x
     https://doi.org/10.1111/j.1467-1770.1971.tb00057.x [Google Scholar]
  17. Pennington, J., Socher, R., & Manning, C. D.
    (2014, October). Glove: Global vectors for word representation. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp.–). 10.3115/v1/D14‑1162
     https://doi.org/10.3115/v1/D14-1162 [Google Scholar]
  18. Taylor, W. L.
    (1953) “Cloze procedure”: A new tool for measuring readability. Journalism Quarterly, (), –. 10.1177/107769905303000401
     https://doi.org/10.1177/107769905303000401 [Google Scholar]
  19. Wobbrock, J. O., Findlater, L., Gergle, D., & Higgins, J. J.
    (2011, May). The aligned rank transforms for nonparametric factorial analyses using ANOVAanova procedures. InProceedings of the SIGCHI conference on human factors in computing systems (pp.–). 10.1145/1978942.1978963
     https://doi.org/10.1145/1978942.1978963 [Google Scholar]
/content/journals/10.1075/ml.24027.deg
Loading
NLP and education
John Benjamins ; https://doi.org/10.1075/ml.24027.deg
/content/journals/10.1075/ml.24027.deg
/content/journals/10.1075/ml.24027.deg
Loading

Data & Media loading...

  • Article Type: Research Article
Keywords: word embeddings ; semantic similarity ; Cloze test

Most Cited

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error