1887
  1. Home
  2. e-Journals
  3. Journal of Second Language Pronunciation
  4. Volume 10, Issue 1
  5. Article
Volume 10, Issue 1
  • ISSN 2215-1931
  • E-ISSN: 2215-194X
USD
Buy:$35.00 + Taxes

Abstract

Abstract

Language institutions need efficient and reliable placement tests to ensure students are placed in appropriate classes. This can be achieved by automating the scoring of pronunciation tests via the use of speech recognition, as its reliability has been shown to be comparable to that of human raters. However, this technology can be costly as it requires development and maintenance, placing it beyond the means of many institutions. This study investigates the feasibility of assessing English second language pronunciation in placement tests through the use of a free automatic speech recognition tool, Google Voice Typing (GVT). We compared human-rated and GVT-rated scores of 56 pronunciation placement tests. Our results indicate a strong correlation between scores for the final rating and for each criterion on the rubric used by human raters. We conclude that leveraging this free speech technology could increase the test usefulness of language placement tests.

© John Benjamins Publishing Company
Loading

Article metrics loading...

/content/journals/10.1075/jslp.23033.joh
2024-06-06
2024-12-12

  • From This Site

    /content/journals/10.1075/jslp.23033.joh
    dcterms_title,dcterms_subject,pub_keyword
    -contentType:Journal -contentType:Contributor -contentType:Concept -contentType:Institution
    10
    5
Loading full text...

Full text loading...

References

  1. Ashwell, T., & Elam, J.
    (2017) How accurately can the Google Web Speech API recognize and transcribe Japanese L2 English learners’ oral production?The JALT CALL Journal, 13(1), 59–76. 10.29140/jaltcall.v13n1.212
     https://doi.org/10.29140/jaltcall.v13n1.212 [Google Scholar]
  2. Bachman, L., & Palmer, A.
    (1996) Language testing in practice. Oxford University Press.
     [Google Scholar]
  3. Bernstein, J., Van Moere, A., & Cheng, J.
    (2010) Validating automated speaking tests. Language Testing, 27(3), 355–377. 10.1177/0265532210364404
     https://doi.org/10.1177/0265532210364404 [Google Scholar]
  4. Browne, K., & Fulcher, G.
    (2016) Pronunciation and intelligibility in assessing spoken fluency. InT. Isaacs, & P. Trofimovich (Eds.), Second language pronunciation assessment: Interdisciplinary perspectives (pp.37–53). Multilingual Matters. 10.21832/ISAACS6848
     https://doi.org/10.21832/ISAACS6848 [Google Scholar]
  5. Cámara-Arenas, E., Tejedor-García, C., Tomas-Vázquez, C., & Escudero-Mancebo, D.
    (2023) Automatic pronunciation assessment vs. automatic speech recognition: A study of conflicting conditions for L2-English. Language Learning & Technology, 27(1), 1–19. https://hdl.handle.net/10125/73512
     [Google Scholar]
  6. Carey, M., Mannell, R., & Dunn, P.
    (2011) Does a rater’s familiarity with a candidate’s pronunciation affect the rating in oral proficiency interviews?Language Testing, 28(2), 201–219. 10.1177/0265532210393704
     https://doi.org/10.1177/0265532210393704 [Google Scholar]
  7. Chen, L., Zechner, K., Yoon, S-Y., Evanini, K., Wang, X., Loukina, A., Tao, J., Davis, L., Lee, M., Mundkowsky, R., Lu, C., Leong, C., & Gyawali, B.
    (2018) Automated scoring of nonnative speech using the SpeechRaterSM v. 5.0 engine. ETS Research Report Series, 11, 1–31. 10.1002/ets2.12198
     https://doi.org/10.1002/ets2.12198 [Google Scholar]
  8. Cobb, T.
    (n.d.). Web Vocabprofile. Lextutor. https://www.lextutor.ca/vp/eng/
     [Google Scholar]
  9. Cox, T., & Davies, R. S.
    (2012) Using automatic speech recognition technology with elicited oral response testing. CALICO Journal, 29(4), 601–618. 10.11139/cj.29.4.601‑618
     https://doi.org/10.11139/cj.29.4.601-618 [Google Scholar]
  10. Derwing, T., Munro, M., & Carbonaro, M.
    (2000) Does popular speech recognition software work with ESL speech?TESOL Quarterly, 34(3), 592–603. 10.2307/3587748
     https://doi.org/10.2307/3587748 [Google Scholar]
  11. Derwing, T., & Munro, M.
    (2015) Pronunciation fundamentals: Evidence-based perspectives for L2 teaching and research. John Benjamins. 10.1075/lllt.42
     https://doi.org/10.1075/lllt.42 [Google Scholar]
  12. Dillon, T., & Wells, D.
    (2021) Student perceptions of mobile automated speech recognition for pronunciation study and testing. English Teaching, 76(4), 101–122. 10.15858/engtea.76.4.202112.101
     https://doi.org/10.15858/engtea.76.4.202112.101 [Google Scholar]
  13. Evers, K., & Chen, S.
    (2021) Effects of automatic speech recognition software on pronunciation for adults with different learning styles. Journal of Educational Computing Research, 59(4), 669–685. 10.1177/0735633120972011
     https://doi.org/10.1177/0735633120972011 [Google Scholar]
  14. Feng, S., Kudina, O., Halpern, B., & Scharenborg, O.
    (2021) Quantifying bias in automatic speech recognition. In22nd Annual Conference of the International Speech Communication Association (INTERSPEECH 2021). International Speech Communication Association. 10.48550/arXiv.2103.15122
     https://doi.org/10.48550/arXiv.2103.15122 [Google Scholar]
  15. Field, J.
    (2005) Intelligibility and the listener: The role of lexical stress. TESOL Quarterly, 39(3), 399–423. 10.2307/3588487
     https://doi.org/10.2307/3588487 [Google Scholar]
  16. Filippidou, F., & Moussiades, L.
    (2020) Α benchmarking of IBM, Google and Wit automatic speech recognition systems. Artificial Intelligence Applications and Innovations, 5831, 73–82. 10.1007/978‑3‑030‑49161‑1_7
     https://doi.org/10.1007/978-3-030-49161-1_7 [Google Scholar]
  17. Graham, C., Lonsdale, D., Kennington, C., Johnson, A., & Mcghee, J.
    (2008) Elicited imitation as an oral proficiency measure with ASR scoring. InN. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, & D. Tapias (Eds.), Proceedings of the Sixth International Conference on Language Resources and Evaluation (pp.1604–1610). European Language Resources Association. www.lrec-conf.org/proceedings/lrec2008/pdf/409_paper.pdf
     [Google Scholar]
  18. Guskaroska, A.
    (2020) ASR-dictation on smartphones for vowel pronunciation practice. Journal of Contemporary Philology, 3(2), 45–61. 10.37834/JCP2020045g
     https://doi.org/10.37834/JCP2020045g [Google Scholar]
  19. Hahn, L.
    (2004) Primary stress and intelligibility: Research to motivate the teaching of suprasegmentals. TESOL Quarterly, (38)2, 201–223. 10.2307/3588378
     https://doi.org/10.2307/3588378 [Google Scholar]
  20. Inceoglu, S., Lim, H., & Chen, W-H.
    (2020) ASR for EFL pronunciation practice: Segmental development and learners’ beliefs. The Journal of Asia TEFL, 17(3), 824–840. 10.18823/asiatefl.2020.17.3.5.824
     https://doi.org/10.18823/asiatefl.2020.17.3.5.824 [Google Scholar]
  21. Inceoglu, S., Chen, W-H., & Lim, H.
    (2023) Assessment of L2 intelligibility: Comparing L1 listeners and automatic speech recognition. ReCALL, 35(1), 89–104; 10.1017/S0958344022000192
     https://doi.org/10.1017/S0958344022000192 [Google Scholar]
  22. Isaacs, T.
    (2018a) Fully automated speaking assessment: Changes to proficiency testing and the role of pronunciation. InO. Kang, R. Thomson, and J. Murphy (Eds.), The Routledge handbook of contemporary English pronunciation (pp.570–584). Routledge.
     [Google Scholar]
  23. (2018b) Shifting sands in second language pronunciation teaching and assessment research and practice. Language Assessment Quarterly, 15(3), 273–293. 10.1080/15434303.2018.1472264
     https://doi.org/10.1080/15434303.2018.1472264 [Google Scholar]
  24. John, P., Cardoso, W., & Johnson, C.
    (2022) Evaluating automatic speech recognition for L2 pronunciation feedback: A focus on Google Translate. InN. Zoghlami, C. Brudermann, C. Sarré, M. Grosbois, L. Bradley, & S. Thouësny (Eds.), Intelligent CALL, granular systems and learner data: Short papers from EUROCALL 2022. 10.14705/rpnet.2022.61.1458
     https://doi.org/10.14705/rpnet.2022.61.1458 [Google Scholar]
  25. John, P., Johnson, C., & Cardoso, W.
    (2024) Assessing Google Translate ASR for feedback on L2 pronunciation errors in unpredictable sentence contexts. InY. Choubsaz, B. Bédi, K. Friðriksdóttir, A. Gimeno-Sanz, S. Björg Vilhjálmsdóttir, S. Zahova (Eds.), CALL for all Languages – EUROCALL 2023 Short Papers. 10.4995/EuroCALL2023.2023.16987
     https://doi.org/10.4995/EuroCALL2023.2023.16987 [Google Scholar]
  26. Johnson, C. & Cardoso, W.
    (in press). Hey Google, let’s write: Examining L2 learners’ acceptance of automatic speech recognition as a writing tool. CALICO Journal.
     [Google Scholar]
  27. Kennedy, S., & Trofimovich, P.
    (2008) Intelligibility, comprehensibility, and accentedness of L2 speech: The role of listener experience and semantic context. Modern Language Review, 64(3), 459–489. 10.3138/cmlr.64.3.459
     https://doi.org/10.3138/cmlr.64.3.459 [Google Scholar]
  28. Khabbazbashi, N., Xu, J., & Galaczi, E.
    (2021) Opening the black box: Exploring Automated speaking evaluation. InB. Lanteigne, C. Coombe, & J. D. Brown (Eds.), Challenges in Language Testing Around the World (pp.333–343). Springer. 10.1007/978‑981‑33‑4232‑3_25
     https://doi.org/10.1007/978-981-33-4232-3_25 [Google Scholar]
  29. Levis, J., & Suvorov, R.
    (2012) Automatic speech recognition. InC. Chapelle (Ed.), The encyclopedia of applied linguistics. John Wiley & Sons. 10.1002/9781405198431.wbeal0066
     https://doi.org/10.1002/9781405198431.wbeal0066 [Google Scholar]
  30. Liakin, D., Cardoso, W., & Liakina, N.
    (2014) Learning L2 pronunciation with a mobile speech recognizer: French/y/. CALICO Journal, 32(1), 1–25. 10.1558/cj.v32i1.25962
     https://doi.org/10.1558/cj.v32i1.25962 [Google Scholar]
  31. Liakin, D., Cardoso, W., Liakina, N.
    (2017) Mobilizing instruction in a second-language context: learners’ perceptions of two speech technologies. Languages. 2017. 2 (3), 1–21. 10.3390/languages2030011
     https://doi.org/10.3390/languages2030011 [Google Scholar]
  32. Ling, G., Mollaun, P., & Xiaoming, X.
    (2014) A study on the impact of fatigue on human raters when scoring speaking responses. Language Testing, 31(4), 479–499. 10.1177/0265532214530699
     https://doi.org/10.1177/0265532214530699 [Google Scholar]
  33. Loukina, A., Davis, L., & Xi, X.
    (2018) Automated assessment of pronunciation in spontaneous speech. InO. Kang & A. Ginther (Eds.), Assessment in second language pronunciation (pp.153–171). Routledge.
     [Google Scholar]
  34. Luo, D., Minematsu, N., Yamauchi, Y., & Hiroshi, K.
    (2009) Analysis and comparison of automatic language proficiency assessment between shadowed sentences and read sentences. InProceedings of SLaTE. ISCA. www.eee.bham.ac.uk/SLaTE2009/papers/SLaTE2009-20-v2.pdf. 10.21437/SLaTE.2009‑10
     https://doi.org/10.21437/SLaTE.2009-10 [Google Scholar]
  35. McCrocklin, S.
    (2016) Pronunciation learner autonomy: The potential of automatic speech recognition. System, 571, 25–42. 10.1016/j.system.2015.12.013
     https://doi.org/10.1016/j.system.2015.12.013 [Google Scholar]
  36. (2018) Learners’ feedback regarding ASR-based dictation practice for pronunciation learning. CALICO Journal, 36(2), 119–137. 10.1558/cj.34738
     https://doi.org/10.1558/cj.34738 [Google Scholar]
  37. (2019) ASR-based dictation practice for second language pronunciation improvement. Journal of Second Language Pronunciation, 5(1), 98–118. 10.1075/jslp.16034.mcc
     https://doi.org/10.1075/jslp.16034.mcc [Google Scholar]
  38. McCrocklin, S., Humaidan, A., & Edalatishams, E.
    (2019) ASR dictation program accuracy: Have current programs improved?InJ. Levis, C. Nagle, & E. Todey (Eds.), Proceedings of the 10th Pronunciation in Second language learning and Teaching Conference, (pp.191–200). Iowa State University.
     [Google Scholar]
  39. McCrocklin, S., & Edalatishams, I.
    (2020) Revisiting popular speech recognition software for ESL speech. TESOL Quarterly, 54(4), 1086–1097. 1–13. 10.1002/tesq.3006
     https://doi.org/10.1002/tesq.3006 [Google Scholar]
  40. Messick, S.
    (1989) Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5–11. 10.2307/1175249
     https://doi.org/10.2307/1175249 [Google Scholar]
  41. Mroz, A.
    (2018) Seeing how people hear you: French learners experiencing intelligibility through automatic speech recognition. Foreign Language Annals, 51(3), 617–637. 10.1111/flan.12348
     https://doi.org/10.1111/flan.12348 [Google Scholar]
  42. (2020) Aiming for advanced intelligibility and proficiency using mobile ASR. Journal of Second Language Pronunciation, 6(1), 12–38. 10.1075/jslp.18030.mro
     https://doi.org/10.1075/jslp.18030.mro [Google Scholar]
  43. Saito, K., Trofimovich, P., Isaacs, T., & Webb, S.
    (2016) Re-examining phonological and lexical correlates of second language comprehensibility: The role of rater experience. InT. Isaacs, & P. Trofimovich (Eds.), Second language pronunciation assessment: Interdisciplinary perspectives (pp.141–156). Multilingual Matters. 10.21832/ISAACS6848
     https://doi.org/10.21832/ISAACS6848 [Google Scholar]
  44. Saito, K., Macmillan, K., Kachlicka, M., Kunihara, T., & Minematsu, N.
    (2023) Automated assessment of second language comprehensibility: Review, training, validation, and generalization studies. Studies in Second Language Acquisition, 1–30. 10.1017/S027226312300044X
     https://doi.org/10.1017/S027226312300044X [Google Scholar]
  45. Shahnawazuddin, S., Adiga, N., Kumar, K., Poddar, A., & Ahmad, W.
    (2020) Voice conversion based data augmentation to improve children’s speech recognition in limited data scenario. InProceedings from INTERSPEECH, 2020 (pp.4382–4386). 10.21437/Interspeech.2020‑1112
     https://doi.org/10.21437/Interspeech.2020-1112 [Google Scholar]
  46. Tatman, R.
    (2017) Gender and dialect bias in Youtube’s automatic captions. InProceedings of the First ACL Workshop on Ethics in Natural Language Processing (pp.53–59). Association for Computational Linguistics. 10.18653/v1/W17‑1606
     https://doi.org/10.18653/v1/W17-1606 [Google Scholar]
  47. Tatman, R., & Kasten, C.
    (2017) Effects of talker dialect, gender, and race on accuracy of Bing speech and Youtube automatic captions. InProceedings from INTERSPEECH, 2017 (pp.934–938). International Speech Communication Association. 10.21437/Interspeech.2017‑1746
     https://doi.org/10.21437/Interspeech.2017-1746 [Google Scholar]
  48. Turner, C.
    (2014) Ratings Scales for Language Tests. InC. Chapelle (Ed.), The encyclopedia of applied linguistics. John Wiley & Sons.
     [Google Scholar]
  49. van der Walt, C., de Wet, F., & Niesler, T.
    (2008) Oral proficiency assessment: The use of automatic speech recognition systems. Southern African Linguistics and Applied Language Studies, 26(1), 135–146. 10.2989/SALALS.2008.26.1.11.426
     https://doi.org/10.2989/SALALS.2008.26.1.11.426 [Google Scholar]
  50. Van Moere, A., & Suzuki, M.
    (2018) Using speech processing technology in assessing pronunciation. InO. Kang & A. Ginther (Eds.), Assessment in second language pronunciation (pp.137–152). Routledge.
     [Google Scholar]
  51. Wallace, L.
    (2015) Reflexive photography, attitudes, behavior, and CALL: ITAs improving spoken English intelligibility. CALICO Journal, 32(3), 449–479. 10.1558/cj.v32i3.26384
     https://doi.org/10.1558/cj.v32i3.26384 [Google Scholar]
  52. (2016) Using Google web speech as a springboard for identifying personal pronunciation problems. InJ. Levis, H. Le, I. Lucic, E. Simpson, & S. Vo (Eds.). Proceedings of the 7th Pronunciation in Second Language Learning and Teaching Conference (pp.180–186). Iowa State University. https://apling.engl.iastate.edu/wp-content/uploads/sites/221/2016/06/PSLLT-7-Proceedings.pdf
     [Google Scholar]
  53. Williamson, D. M., Xi, X., & Breyer, F. J.
    (2012) A Framework for Evaluation and Use of Automated Scoring. Educational Measurement: Issues and Practice, 31(1), 2–13. 10.1111/j.1745‑3992.2011.00223.x
     https://doi.org/10.1111/j.1745-3992.2011.00223.x [Google Scholar]
  54. Winke, P., Gass, S., & Myford, C.
    (2012) Raters’ L2 background as a potential source of bias in rating oral performance. Language Testing. 30(2) 231–252. 10.1177/0265532212456968
     https://doi.org/10.1177/0265532212456968 [Google Scholar]
  55. Winke, P., & Gass, S.
    (2013) The influence of second language experience and accent familiarity on oral proficiency rating: A qualitative investigation. TESOL Quarterly, 47(4), 762–789. 10.1002/tesq.73
     https://doi.org/10.1002/tesq.73 [Google Scholar]
  56. Yan, X., & Ginther, A.
    (2018) Listeners and raters: Similarities and differences in evaluation of accented speech. InO. Kang & A. Ginther (Eds.), Assessment in second language pronunciation (pp.67–88). Routledge.
     [Google Scholar]
  57. Zechner, K., Higgins, D., Xi, X., & Williamson, D.
    (2009) Automatic scoring of non-native spontaneous speech in tests of spoken English. Speech Communication, 51(10), 883–895. 10.1016/j.specom.2009.04.009
     https://doi.org/10.1016/j.specom.2009.04.009 [Google Scholar]
/content/journals/10.1075/jslp.23033.joh
Loading
Assessing pronunciation using dictation tools
Journal of Second Language Pronunciation 10, 10 (2024); https://doi.org/10.1075/jslp.23033.joh
/content/journals/10.1075/jslp.23033.joh
/content/journals/10.1075/jslp.23033.joh
Loading

Data & Media loading...

  • Article Type: Research Article
Keyword(s): automatic speech recognition; Google Voice Typing; L2 pronunciation; placement tests

Most Cited

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error