1887
  1. Home
  2. e-Journals
  3. International Journal of Corpus Linguistics
  4. Volume 29, Issue 4
  5. Article
Volume 29, Issue 4
  • ISSN 1384-6655
  • E-ISSN: 1569-9811
USD
Buy:$35.00 + Taxes

Abstract

Abstract

Certain forms of linguistic annotation, like part of speech and semantic tagging, can be automated with high accuracy. However, manual annotation is still necessary for complex pragmatic and discursive features that lack a direct mapping to lexical forms. This manual process is time-consuming and error-prone, limiting the scalability of function-to-form approaches in corpus linguistics. To address this, our study explores the possibility of using large language models (LLMs) to automate pragma-discursive corpus annotation. We compare GPT-3.5 (the model behind the free-to-use version of ChatGPT), GPT-4 (the model underpinning the precise mode of Bing chatbot), and a human coder in annotating apology components in English based on the local grammar framework. We find that GPT-4 outperformed GPT-3.5, with accuracy approaching that of a human coder. These results suggest that LLMs can be successfully deployed to aid pragma-discursive corpus annotation, making the process more efficient, scalable, and accessible.

© John Benjamins Publishing Company
Loading

Article metrics loading...

/content/journals/10.1075/ijcl.23087.yu
2024-06-03
2025-05-15

  • From This Site

    /content/journals/10.1075/ijcl.23087.yu
    dcterms_title,dcterms_subject,pub_keyword
    -contentType:Journal -contentType:Contributor -contentType:Concept -contentType:Institution
    10
    5
Loading full text...

Full text loading...

References

  1. Baker, P., Brookes, G., & Evans, C.
    (2019) The language of patient feedback: A corpus linguistic study of online health communication. Routledge. 10.4324/9780429259265
     https://doi.org/10.4324/9780429259265 [Google Scholar]
  2. Blum-Kulka, S., House, J., & Kasper, G.
    (1989) (Eds.). Cross-cultural pragmatics: Requests and apologies. Ablex Publishing Corporation.
     [Google Scholar]
  3. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., … & Amodei, D.
    (2020) Language models are few-shot learners. InH. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, & H. Lin (Eds.). Advances in neural information processing systems 33: 34th conference on neural information processing systems (pp.1877–1901). Neural Information Processing Systems Foundation, Inc.
     [Google Scholar]
  4. Cavasso, L., & Taboada, M.
    (2021) A corpus analysis of online news comments using the Appraisal framework. Journal of Corpora and Discourse Studies, (4), 1–38. 10.18573/jcads.61
     https://doi.org/10.18573/jcads.61 [Google Scholar]
  5. Cheng, W., & Ching, T.
    (2018) ‘Not a guarantee of future performance’: The local grammar of disclaimers. Applied Linguistics, 39(3), 263–301.
     [Google Scholar]
  6. Ding, B., Qin, C., Liu, L., Chia, Y. K., Joty, S., Li, B., & Bing, L.
    (2023) Is GPT-3 a good data annotator?arXiv. 10.18653/v1/2023.acl‑long.626
     https://doi.org/10.18653/v1/2023.acl-long.626 [Google Scholar]
  7. Frei, J., & Kramer, F.
    (2023) Annotated dataset creation through large language models for non-English medical NLP. Journal of Biomedical Informatics, (145). 10.1016/j.jbi.2023.104478
     https://doi.org/10.1016/j.jbi.2023.104478 [Google Scholar]
  8. Fuoli, M., & Hommerberg, C.
    (2015) Optimising transparency, reliability and replicability: Annotation principles and inter-coder agreement in the quantification of evaluative expressions. Corpora, 10(3), 315–349. 10.3366/cor.2015.0080
     https://doi.org/10.3366/cor.2015.0080 [Google Scholar]
  9. Fuoli, M., Littlemore, J., & Turner, S.
    (2022) Sunken ships and screaming banshees: Metaphor and evaluation in film reviews. English Language & Linguistics, 26(1), 75–103. 10.1017/S1360674321000046
     https://doi.org/10.1017/S1360674321000046 [Google Scholar]
  10. Garside, R., Leech, G., & McEnery, T.
    (1997) Corpus annotation: Linguistic information from computer text corpora. Routledge. 10.4324/9781315841366
     https://doi.org/10.4324/9781315841366 [Google Scholar]
  11. Garside, R., & Smith, N.
    (1997) A hybrid grammatical tagger: CLAWS4. InR. Garside, G. Leech, & T. McEnery (Eds.), Corpus annotation: Linguistic information from computer text corpora (pp.102–121). Routledge. 10.4324/9781315841366‑13
     https://doi.org/10.4324/9781315841366-13 [Google Scholar]
  12. Gilardi, F., Alizadeh, M., & Kubli, M.
    (2023) ChatGPT outperforms crowd-workers for text-annotation tasks. arXiv. 10.48550/arXiv.2303.15056
     https://doi.org/10.48550/arXiv.2303.15056 [Google Scholar]
  13. He, X., Lin, Z., Gong, Y., Jin, A., Zhang, H., Lin, C., Jiao, J., Yiu, S. M., Duan, N., & Chen, W.
    (2023) AnnoLLM: Making large language models to be better crowdsourced annotators. arXiv. 10.48550/arXiv.2303.16854
     https://doi.org/10.48550/arXiv.2303.16854 [Google Scholar]
  14. Hunston, S.
    (2002) Pattern grammar, language teaching, and linguistic variation: Applications of a corpus-driven grammar. InR. Reppen, S. Fitzmaurice, & D. Biber (Eds.), Using corpora to explore linguistic variation (pp.167–183). John Benjamins. 10.1075/scl.9.11hun
     https://doi.org/10.1075/scl.9.11hun [Google Scholar]
  15. (2011) Corpus approaches to evaluation: Phraseology and evaluative language. Routledge.
     [Google Scholar]
  16. Hunston, S., & Sinclair, J.
    (2001) A local grammar of evaluation. InS. Hunston & G. Thompson (Eds.), Evaluation in text: Authorial stance and the construction of discourse. Oxford University Press.
     [Google Scholar]
  17. Hunston, S., & Su, H.
    (2019) Patterns, constructions, and local grammar: A case study of ‘evaluation.’Applied Linguistics, 40(4), 567–593. 10.1093/applin/amx046
     https://doi.org/10.1093/applin/amx046 [Google Scholar]
  18. Kirk, J. M.
    (2016) The pragmatic annotation scheme of the SPICE-Ireland corpus. International Journal of Corpus Linguistics, 21(3), 299–322. 10.1075/ijcl.21.3.01kir
     https://doi.org/10.1075/ijcl.21.3.01kir [Google Scholar]
  19. Kolhatkar, V., Wu, H., Cavasso, L., Francis, E., Shukla, K., & Taboada, M.
    (2020) The SFU opinion and comments corpus: A corpus for the analysis of online news comments. Corpus Pragmatics, (4), 155–190. 10.1007/s41701‑019‑00065‑w
     https://doi.org/10.1007/s41701-019-00065-w [Google Scholar]
  20. Leech, G.
    (1993) Corpus annotation schemes. Literary and Linguistic Computing, 8(4), 275–281. 10.1093/llc/8.4.275
     https://doi.org/10.1093/llc/8.4.275 [Google Scholar]
  21. (1997) Introducing corpus annotation. InR. Garside, G. Leech, & T. McEnery (Eds.), Corpus annotation: Linguistic information from computer text corpora (pp.1–18) Routledge.
     [Google Scholar]
  22. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G.
    (2023) Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language Processing. ACM Computing Surveys, 55(9), 1–35. 10.1145/3560815
     https://doi.org/10.1145/3560815 [Google Scholar]
  23. Love, R., Dembry, C., Hardie, A., Brezina, V., & McEnery, T.
    (2017) The Spoken BNC2014: Designing and building a spoken corpus of everyday conversations. International Journal of Corpus Linguistics, 22(3), 319–344.
     [Google Scholar]
  24. Lutzky, U., & Kehoe, A.
    (2017a) “Oops, I didn’t mean to be so flippant”. A corpus pragmatic analysis of apologies in blog data. Journal of Pragmatics, (116), 27–36. 10.1016/j.pragma.2016.12.007
     https://doi.org/10.1016/j.pragma.2016.12.007 [Google Scholar]
  25. (2017b) “I apologise for my poor blogging”: Searching for apologies in the Birmingham Blog Corpus. Corpus Pragmatics, (1), 37–56. 10.1007/s41701‑017‑0004‑0
     https://doi.org/10.1007/s41701-017-0004-0 [Google Scholar]
  26. Martin, J. R., & White, P. R. R.
    (2005) The language of evaluation: Appraisal in English. Palgrave Macmillan. 10.1057/9780230511910
     https://doi.org/10.1057/9780230511910 [Google Scholar]
  27. McEnery, T., & Hardie, A.
    (2012) Corpus linguistics. Cambridge University Press.
     [Google Scholar]
  28. McEnery, T., & Wilson, A.
    (2001) Corpus linguistics: An introduction. Edinburgh University Press.
     [Google Scholar]
  29. Microsoft & OpenAI
    Microsoft & OpenAI (2023) Bing Chat(Apr-11-28-2023 version). [GPT-4 language model]. https://www.bing.com/search
     [Google Scholar]
  30. Milà-Garcia, A.
    (2018) Pragmatic annotation for a multi-layered analysis of speech acts: A methodological proposal. Corpus Pragmatics, (2), 265–287. 10.1007/s41701‑018‑0037‑z
     https://doi.org/10.1007/s41701-018-0037-z [Google Scholar]
  31. O’Keeffe, A.
    (2018) “Corpus-based function-to-form approaches”. InA. H. Jucker, K. P. Schneider & W. Bublitz (Eds.), Methods in pragmatics (pp.587–618). Mouton de Gruyter. 10.1515/9783110424928‑023
     https://doi.org/10.1515/9783110424928-023 [Google Scholar]
  32. OpenAI
    OpenAI (2023) ChatGPT (Apr11-28-2023version). [Large language model]. https://chat.openai.com/chat
     [Google Scholar]
  33. Page, R.
    (2014) Saying ‘sorry’: Corporate apologies posted on Twitter. Journal of Pragmatics, (62), 30–45. 10.1016/j.pragma.2013.12.003
     https://doi.org/10.1016/j.pragma.2013.12.003 [Google Scholar]
  34. Põldvere, N., De Felice, R., & Paradis, C.
    (2022) Advice in conversation: Corpus pragmatics meets mixed methods. Cambridge University Press. 10.1017/9781009053617
     https://doi.org/10.1017/9781009053617 [Google Scholar]
  35. Rayson, P., Archer, D., Piao, S., & McEnery, T.
    (2004) The UCREL semantic analysis system. InProceedings of the Workshop on Beyond Named Entity Recognition: Semantic Labelling for NLP Tasks in Association with the LREC 2004 (pp.7–12).
     [Google Scholar]
  36. Rühlemann, C., & Aijmer, K.
    (2014) Corpus pragmatics: Laying the foundations. InCorpus pragmatics: A handbook (pp.1–26). Cambridge University Press. 10.1017/CBO9781139057493.001
     https://doi.org/10.1017/CBO9781139057493.001 [Google Scholar]
  37. Simaki, V., Paradis, C., Skeppstedt, M., Sahlgren, M., Kucher, K., & Kerren, A.
    (2020) Annotating speaker stance in discourse: The Brexit Blog Corpus. Corpus Linguistics and Linguistic Theory, 16(2), 215–248.
     [Google Scholar]
  38. Su, H.
    (2017) Local grammars of speech acts: An exploratory study. Journal of Pragmatics, (111), 72–83. 10.1016/j.pragma.2017.02.008
     https://doi.org/10.1016/j.pragma.2017.02.008 [Google Scholar]
  39. (2021) Changing patterns of apology in spoken British English: A local grammar based diachronic investigation. Pragmatics and Society, 12(3), 410–436. 10.1075/ps.18031.su
     https://doi.org/10.1075/ps.18031.su [Google Scholar]
  40. Su, H., & Wei, N.
    (2018) “I’m really sorry about what I said”: A local grammar of apology. Pragmatics, 28(3), 439–462. 10.1075/prag.17005.su
     https://doi.org/10.1075/prag.17005.su [Google Scholar]
  41. Su, H., & Zhang, L.
    (2020) Local grammars and discourse acts in academic writing: A case study of exemplification in Linguistics research articles. Journal of English for Academic Purposes, (43), Article 100805. 10.1016/j.jeap.2019.100805
     https://doi.org/10.1016/j.jeap.2019.100805 [Google Scholar]
  42. Taylor, C.
    (2016) Mock politeness in English and Italian. John Benjamins. 10.1075/pbns.267
     https://doi.org/10.1075/pbns.267 [Google Scholar]
  43. Wei, X., Cui, X., Cheng, N., Wang, X., Zhang, X., Huang, S., Xie, P., Xu, J., Chen, Y., Zhang, M., Jiang, Y., & Han, W.
    (2023) Zero-shot information extraction via chatting with ChatGPT. arXiv. 10.48550/arXiv.2302.10205
     https://doi.org/10.48550/arXiv.2302.10205 [Google Scholar]
  44. Weisser, M.
    (2014) Speech act annotation. InK. Aijmer & C. Rühlemann (Eds.), Corpus pragmatics: A handbook (pp.84–110). Cambridge University Press. 10.1017/CBO9781139057493.005
     https://doi.org/10.1017/CBO9781139057493.005 [Google Scholar]
  45. (2016) DART – The dialogue annotation and research tool. Corpus Linguistics and Linguistic Theory, 12(2), 355–388. 10.1515/cllt‑2014‑0051
     https://doi.org/10.1515/cllt-2014-0051 [Google Scholar]
  46. Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., & Hu, X.
    (2023) Harnessing the power of LLMs in practice: A survey on ChatGPT and beyond. arXiv. 10.48550/arXiv.2304.13712
     https://doi.org/10.48550/arXiv.2304.13712 [Google Scholar]
  47. Yu, D.
    (2022) Cross-cultural genre analysis: Investigating Chinese, Italian and English CSR reports. Routledge.
     [Google Scholar]
  48. Zhao, T., & Kawahara, T.
    (2019) Joint dialog act segmentation and recognition in human conversations using attention to dialog context. Computer Speech & Language, (57), 108–127. 10.1016/j.csl.2019.03.001
     https://doi.org/10.1016/j.csl.2019.03.001 [Google Scholar]
/content/journals/10.1075/ijcl.23087.yu
Loading
Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis
International Journal of Corpus Linguistics 29, 534 (2024); https://doi.org/10.1075/ijcl.23087.yu
/content/journals/10.1075/ijcl.23087.yu
/content/journals/10.1075/ijcl.23087.yu
Loading

Data & Media loading...

  • Article Type: Research Article
Keyword(s): ChatGPT; corpus pragmatics; large language models; local grammar; pragma-discursive corpus annotation

Most Cited

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error