1887
image of A systematic review of reliability in corpus-based metadiscourse studies
USD
Buy:$35.00 + Taxes

Abstract

Metadiscourse has been a major focus of research over the last twenty-five years, attracting methodological approaches from the areas of textual pragmatics and discourse studies, many of which are supported by corpus linguistics. A major challenge in corpus-based discourse studies, however, is subjectivity, which may affect their quality and undermine their methodological rigor. To reduce subjectivity and guarantee consistency, assessing reliability of coding is essential. This study advocates combining quantitative with qualitative approaches to reliability. We argue that this mixed-method approach will provide a better assessment of reliability. To this aim, this methodological synthesis surveyed research covering empirical corpus-based studies on metadiscourse published in indexed and peer-reviewed journals. One major finding is that most studies did not report conducting any reliability measure. Issues in reliability accounts were also identified for those that did. Another major finding is a pervasive lack of transparency and comprehensiveness in reliability reports. Recommendations for enhancing reliability are listed.

Loading

Article metrics loading...

/content/journals/10.1075/prag.25004.bou
2026-05-05
2026-05-11
Loading full text...

Full text loading...

References

  1. Ahmed, Abdelhamid M., and Xiao Zhang
    2023 “Students’ Voice in L2 English Writing: A Systematic Review of Literature.” Ampersand. 10.1016/j.amper.2023.100114
    https://doi.org/10.1016/j.amper.2023.100114 [Google Scholar]
  2. Belur, Jyoti, Lisa Tompson, Amy Thornton, and Miranda Simon
    2021 “Interrater Reliability in Systematic Review Methodology: Exploring Variation in Coder Decision-Making.” Sociological Methods & Research (): –. 10.1177/0049124118799372
    https://doi.org/10.1177/0049124118799372 [Google Scholar]
  3. Bouziri, Basma
    2019 “A Corpus-Assisted Genre Analysis of the Tunisian Lecture Corpus: Focus on Metadiscourse.” PhD dissertation. Université Catholique de Louvain.
  4. 2020 “Topic Signaling Metadiscourse Devices in the Tunisian Lecture Corpus.” ESP Today (): –. 10.18485/esptoday.2020.8.2.3
    https://doi.org/10.18485/esptoday.2020.8.2.3 [Google Scholar]
  5. 2021 “A Tripartite Interpersonal Model for Investigating Metadiscourse in Academic Lectures.” Applied Linguistics (): –. WOS:000727808600008. 10.1093/applin/amab001
    https://doi.org/10.1093/applin/amab001 [Google Scholar]
  6. Callies, Marcus
    2015 “Learner Corpus Methodology.” InThe Cambridge Handbook of Learner Corpus Research, edited bySylviane Granger, Gaetanelle Gilquin, and Fanny Meunier, –. Cambridge University Press. 10.1017/CBO9781139649414.003
    https://doi.org/10.1017/CBO9781139649414.003 [Google Scholar]
  7. Chong, Sin Wang, and Luke Plonsky
    2023 “A Typology of Secondary Research in Applied Linguistics.” Applied Linguistics Review (): –. 10.1515/applirev‑2022‑0189
    https://doi.org/10.1515/applirev-2022-0189 [Google Scholar]
  8. Chong, Sin Wang, and Luke Plonsky
    2024 “Research Synthesis in Language Education.” InReference Module in Social Sciences. Elsevier. 10.1016/B978‑0‑323‑95504‑1.00010‑7
    https://doi.org/10.1016/B978-0-323-95504-1.00010-7 [Google Scholar]
  9. Crismore, Avon, Raija Markkanen, and Margaret S. Steffensen
    1993 “Metadiscourse in Persuasive Writing: A Study of Texts Written by American and Finnish University Students.” Written Communication (): –. 10.1177/0741088393010001002
    https://doi.org/10.1177/0741088393010001002 [Google Scholar]
  10. Feng, Guangchao Charles
    2015 “Mistakes and How to Avoid Mistakes in Using Intercoder Reliability Indices.” Methodology (): –. 10.1027/1614‑2241/a000086
    https://doi.org/10.1027/1614-2241/a000086 [Google Scholar]
  11. Freelon, Deen
    2010 “ReCal: Intercoder Reliability Calculation as a Web Service.” International Journal of Internet Science (): –. dfreelon.org/utils/recalfront/recal2/
    [Google Scholar]
  12. Grisot, Cristina
    2017 “A Quantitative Approach to Conceptual, Procedural and Pragmatic Meaning: Evidence from Inter-Annotator Agreement.” Journal of Pragmatics (): –. 10.1016/j.pragma.2017.06.020
    https://doi.org/10.1016/j.pragma.2017.06.020 [Google Scholar]
  13. Hober, Nicole, Tülay Dixon, and Tove Larsson
    2023 “Towards Increased Reliability and Transparency in Projects with Manual Linguistic Coding.” Corpora (): –. 10.3366/cor.2023.0284
    https://doi.org/10.3366/cor.2023.0284 [Google Scholar]
  14. Hovy, E., and Julia Lavid
    2010 “Towards a ‘Science’ of Corpus Annotation: A New Methodological Challenge for Corpus Linguistics.” International Journal of Translation ().
    [Google Scholar]
  15. Hunston, Susan
    1999 “Evaluation and the Planes of Discourse: Status and Value in Persuasive Texts.” InEvaluation in Text: Authorial Stance and the Construction of Discourse, edited byGeoff Thompson, and Susan Hunston, –. Oxford University Press.
    [Google Scholar]
  16. Hyland, Ken
    2005Metadiscourse: Exploring Interaction in Writing. Continuum Discourse Series.
    [Google Scholar]
  17. Landis, J. Richard, and Gary G. Koch
    1977 “An Application of Hierarchical Kappa-type Statistics in the Assessment of Majority Agreement among Multiple Observers.” Biometrics (): –.
    [Google Scholar]
  18. Larsson, Tove, Magali Paquot, and Luke Plonsky
    2020 “Inter-Rater Reliability in Learner Corpus Research: Insights from a Collaborative Study on Adverb Placement.” International Journal of Learner Corpus Research (): –. 10.1075/ijlcr.20001.lar
    https://doi.org/10.1075/ijlcr.20001.lar [Google Scholar]
  19. Liu, Yanhua, and Guangwei Hu
    2021 “Mapping the Field of English for Specific Purposes (1980–2018): A Co-Citation Analysis.” English for Specific Purposes: –. 10.1016/j.esp.2020.10.003
    https://doi.org/10.1016/j.esp.2020.10.003 [Google Scholar]
  20. Norouzian, Reza
    2021 “Interrater Reliability in Second Language Meta-Analyses: The Case of Categorical Moderators.” Studies in Second Language Acquisition (): –. 10.1017/S0272263121000061
    https://doi.org/10.1017/S0272263121000061 [Google Scholar]
  21. Paltridge, Brian, and Aek Phakiti
    eds. 2015Research Methods in Applied Linguistics: A Practical Resource. Bloomsbury Academic.
    [Google Scholar]
  22. Pearson, William S., and Esmaeel Abdollahzadeh
    2023 “Metadiscourse in Academic Writing: A Systematic Review.” Lingua. 10.1016/j.lingua.2023.103561
    https://doi.org/10.1016/j.lingua.2023.103561 [Google Scholar]
  23. Perez Aguilar, Marta, and Elisabeth Macia Arno
    2002 “Metadiscourse in Lecture Comprehension: Does It Really Help Foreign Language Learners?” Atlantis (): –.
    [Google Scholar]
  24. Plonsky, Luke, and Frederick Oswald
    2015 “Meta-Analyzing Second Language Research.” InMethodological Synthesis in Quantitative L2 Research: A Review of Reviews and a Case Study of Exploratory Factor Analysis. Routledge, –. 10.4324/9781315870908‑6
    https://doi.org/10.4324/9781315870908-6 [Google Scholar]
  25. Plonsky, Luke, and Deirdre J. Derrick
    2016 “A Meta-Analysis of Reliability Coefficients in Second Language Research.” The Modern Language Journal (): –. 10.1111/modl.12335
    https://doi.org/10.1111/modl.12335 [Google Scholar]
  26. Rodwell, Mary K.
    1998Social Work Constructivist Research. Garland Publishing.
    [Google Scholar]
  27. Schröter, Thorsten, and Agnieszka Jablonska Eklöf
    2020 “Assessment in English for Young Learners in Sweden: Guidelines, Challenges and Coping Strategies.” Educare (): –. 10.24834/educare.2020.3.1
    https://doi.org/10.24834/educare.2020.3.1 [Google Scholar]
  28. Spooren, Wilbert, and Liesbeth Degand
    2010 “Coding Coherence Relations: Reliability and Validity.” Corpus Linguistics and Linguistic Theory (): –. 10.1515/cllt.2010.009
    https://doi.org/10.1515/cllt.2010.009 [Google Scholar]
  29. Swales, John M.
    2019 “The Futures of EAP Genre Studies: A Personal Viewpoint.” Journal of English for Academic Purposes (): –. 10.1016/j.jeap.2019.01.003
    https://doi.org/10.1016/j.jeap.2019.01.003 [Google Scholar]
  30. Wood, Margaret, Tove Larsson, Luke Plonsky, Scott Sterling, Merja Kytö, and Katherine Yaw
    2024Addressing Questionable Research Practices in Applied Linguistics: A Practical Guide. Applied Linguistics Press.
    [Google Scholar]
/content/journals/10.1075/prag.25004.bou
Loading
/content/journals/10.1075/prag.25004.bou
Loading

Data & Media loading...

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error