1887
  1. Home
  2. e-Journals
  3. Journal of Historical Pragmatics
  4. Volume 26, Issue 3
  5. Article
Volume 26, Issue 3
  • ISSN 1566-5852
  • E-ISSN: 1569-9854
USD
Buy:$35.00 + Taxes

Abstract

Abstract

This study investigates the viability of using large language models (s) to conduct pragmatic annotations of historical texts. The investigation employs a small corpus of witness depositions and compares Claude 3.5 Sonnet — an that excels in reasoning over text — with two human annotators over their performance in the pragmatic annotation of Early Modern English (od) texts. The study also compares the model’s annotations on modernised and original versions of the corpus to explore if od spelling variations affect its performance. The results revealed that although the model’s annotations were less satisfactory than human annotators’, it achieved moderate inter-coder agreement and balanced precision and recall, which is desirable in this particular task by maximising identification without sacrificing accuracy. Furthermore, the prevalent spelling variations did not significantly impair the model’s ability to recognise epistemic stance in the original od texts. Therefore, we propose a human– collaboration approach for historical pragmatic annotation.

© John Benjamins Publishing Company
Loading

Article metrics loading...

/content/journals/10.1075/jhp.25011.hua
2025-12-04
2026-01-13

  • From This Site

    /content/journals/10.1075/jhp.25011.hua
    dcterms_title,dcterms_subject,pub_keyword
    -contentType:Journal -contentType:Contributor -contentType:Concept -contentType:Institution
    10
    5
Loading full text...

Full text loading...

References

  1. Aikhenvald, Alexandra Y.
    2004 Evidentiality. Oxford: Oxford University Press. 10.1093/oso/9780199263882.001.0001
     https://doi.org/10.1093/oso/9780199263882.001.0001 [Google Scholar]
  2. Anthropic
    Anthropic 2024 “Introducing Claude 3.5 Sonnet”. Anthropic. Published21 June 2024. Accessed23 May 2025at: https://www.anthropic.com/news/claude-3-5-sonnet
     [Google Scholar]
  3. Biber, Douglas
    2004 “Historical Patterns for the Grammatical Marking of Stance: A Cross-Register Comparison”. Journal of Historical Pragmatics5 (1): 107–136. 10.1075/jhp.5.1.06bib
     https://doi.org/10.1075/jhp.5.1.06bib [Google Scholar]
  4. Boggel, Sandra
    2009 Metadiscourse in Middle English and Early Modern English Religious Texts: A Corpus-based Study. Frankfurt am Main: Peter Lang.
     [Google Scholar]
  5. Bromhead, Helen
    2009 The Reign of Truth and Faith: Epistemic Expressions in 16th and 17th Century English. Berlin and New York: Mouton de Gruyter. 10.1515/9783110216028
     https://doi.org/10.1515/9783110216028 [Google Scholar]
  6. Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever and Dario Amodei
    2020 “Language Models Are Few-shot Learners”. [v4] Wednesday 22 July 2020. arXiv. Accessed20 January 2025. 10.48550/arXiv.2005.14165
     https://doi.org/10.48550/arXiv.2005.14165 [Google Scholar]
  7. Campesato, Oswald
    2024 Large Language Models: An Introduction. Boston: Mercury Learning and Information. 10.1515/9781501520587
     https://doi.org/10.1515/9781501520587 [Google Scholar]
  8. Chafe, Wallace L. and Johanna Nichols
    (eds) 1986 Evidentiality: The Linguistic Coding of Epistemology. Norwood, New Jersey: Ablex.
     [Google Scholar]
  9. Chockalingam, Annamalai, Ankur Patel, Shashank Verma and Tiffany Yeung
    2023 A Beginner’s Guide to Large Language Models: Part 1. NVIDIA. See: https://resources.nvidia.com/en-us-large-language-model-ebooks/llm-ebook-part1
     [Google Scholar]
  10. Devlin, Jacob, Ming-Wei Chang, Kenton Lee and Kristina Toutanova
    2019 “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume11 (Long and Short Papers), 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics.
     [Google Scholar]
  11. Dong, Qingxiu, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li and Zhifang Sui
    2024 “A Survey on In-Context Learning”. InYaser Al-Onaizan, Mohit Bansal and Yun-Nung Chen (eds), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 1107–1128. Miami, Florida, USA. 12–16 November 2024. Kerrville: Association for Computational Linguistics. 10.18653/v1/2024.emnlp‑main.64
     https://doi.org/10.18653/v1/2024.emnlp-main.64 [Google Scholar]
  12. Faller, Martina T.
    2002 “Semantics and Pragmatics of Evidentials in Cuzco Quechua”. (PhD thesis.) Stanford, California: Stanford University. See: personalpages.manchester.ac.uk/staff/martina.t.faller/documents/Thesis-A4.pdf
     [Google Scholar]
  13. Fonteyn, Lauren
    2020 “What about Grammar? Using BERT Embeddings to Explore Functional-Semantic Shifts of Semi-lexical and Grammatical Constructions”. InProceedings of the Workshop on Computational Humanities Research (CHR 2020), volume27231of CEUR Workshop Proceedings, 257–268. See: ceur-ws.org/Vol-2723/short15.pdf
     [Google Scholar]
  14. Gao, Yunfan, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang and Haofen Wang
    2024 “Retrieval-Augmented Generation for Large Language Models: A Survey”. [v5] Wednesday 27 March 2024. arXiv. Accessed7 June 2025. 10.48550/arXiv.2312.10997
     https://doi.org/10.48550/arXiv.2312.10997 [Google Scholar]
  15. Garside, Roger, Geoffrey Leech and Tony McEnery
    (eds) 1997 Corpus Annotation: Linguistic Information from Computer Text Corpora. London and New York: Routledge. 10.4324/9781315841366
     https://doi.org/10.4324/9781315841366 [Google Scholar]
  16. Gisev, Natasa, J. Simon Bell and Timothy F. Chen
    2013 “Interrater Agreement and Interrater Reliability Key Concepts, Approaches, and Applications”. Research in Social and Administrative Pharmacy9 (3): 330–338. 10.1016/j.sapharm.2012.04.004
     https://doi.org/10.1016/j.sapharm.2012.04.004 [Google Scholar]
  17. Giulianelli, Mario, Marco Del Tredici and Raquel Fernández
    2020 “Analysing Lexical Semantic Change with Contextualised Word Representations”. InDan Jurafsky, Joyce Chai, Natalie Schluter and Joel Tetreault (eds), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 3960–3973. Online. 5–10 July 2020. Kerrville: Association for Computational Linguistics. See: https://www.aclweb.org/anthology/2020.acl-main.365. 10.18653/v1/2020.acl‑main.365
     https://doi.org/10.18653/v1/2020.acl-main.365 [Google Scholar]
  18. Gray, Bethany, Douglas Biber and Turo Hiltunen
    2011 “The Expression of Stance in Early (1665–1712) Publications of the Philosophical Transactions and Other Contemporary Medical Prose: Innovations in a Pioneering Discourse”. InIrma Taavitsainen and Päivi Pahta (eds), Medical Writing in Early Modern English, 221–257. Cambridge: Cambridge University Press. 10.1017/CBO9780511921193.013
     https://doi.org/10.1017/CBO9780511921193.013 [Google Scholar]
  19. Grund, Peter J.
    2012 “The Nature of Knowledge: Evidence and Evidentiality in the Witness Depositions from the Salem Witch Trials”. American Speech87 (1): 7–38. 10.1215/00031283‑1599941
     https://doi.org/10.1215/00031283-1599941 [Google Scholar]
  20. 2017 “Description, Evaluation and Stance: Exploring the Forms and Functions of Speech Descriptors in Early Modern English”. Nordic Journal of English Studies16 (1): 41–73. 10.35360/njes.394
     https://doi.org/10.35360/njes.394 [Google Scholar]
  21. Harju, Anika and Rob van der Goot
    2025 “How to Age BERT Well: Continuous Training for Historical Language Adaptation”. InHansi Hettiarachchi, Tharindu Ranasinghe, Paul Rayson, Ruslan Mitkov, Mohamed Gaber, Damith Premasiri, Fiona Anting Tan and Lasitha Uyangodage (eds), Proceedings of the First Workshop on Language Models for Low-Resource Languages, 258–267. Abu Dhabi, UAE. 20 January 2025. Kerrville: Association for Computational Linguistics.
     [Google Scholar]
  22. Hiltunen, Turo and Jukka Tyrkkö
    2011 “Verbs of Knowing: Discursive Practices in Early Modern Vernacular Medicine”. InIrma Taavitsainen and Päivi Pahta (eds), Medical Writing in Early Modern English, 44–73. Cambridge: Cambridge University Press. 10.1017/CBO9780511921193.005
     https://doi.org/10.1017/CBO9780511921193.005 [Google Scholar]
  23. Huang, Ding
    2023 “Formulaic Sequences in Early Modern English: A Corpus-Assisted Historical Pragmatic Study”. (PhD thesis.) Heidelberg, Germany: Heidelberg University. 10.11588/heidok.00033750
     https://doi.org/10.11588/heidok.00033750 [Google Scholar]
  24. Huang, Ding and Jiajin Xu
    2025 “Supplementary Materials”. ResearchGate. 10.13140/RG.2.2.32357.10726
     https://doi.org/10.13140/RG.2.2.32357.10726 [Google Scholar]
  25. Jurafsky, Daniel and James H. Martin
    2025 Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. (Third edition.) Online manuscript released12 January 2025. See: https://web.stanford.edu/~jurafsky/slp3
     [Google Scholar]
  26. Kamath, Uday, Kevin Keenan, Garrett Somers and Sarah Sorenson
    2024 Large Language Models: A Deep Dive. Bridging Theory and Practice. Cham, Switzerland: Springer. 10.1007/978‑3‑031‑65647‑7
     https://doi.org/10.1007/978-3-031-65647-7 [Google Scholar]
  27. Kärkkäinen, Elise
    2003 Epistemic Stance in English Conversation: A Description of Its Interactional Functions, with A Focus on ‘I think’. Amsterdam: John Benjamins. 10.1075/pbns.115
     https://doi.org/10.1075/pbns.115 [Google Scholar]
  28. Kytö, Merja and Terry Walker
    2006 Guide to A Corpus of English Dialogues 1560–1760. Uppsala: Acta Universitatis Upsaliensis.
     [Google Scholar]
  29. Landert, Daniela
    2024 Methods in Historical Corpus Pragmatics: Epistemic Stance in Early Modern English. Cambridge and New York: Cambridge University Press. 10.1017/9781009237369
     https://doi.org/10.1017/9781009237369 [Google Scholar]
  30. Landis, J. Richard and Gary G. Koch
    1977 “The Measurement of Observer Agreement for Categorical Data”. Biometrics33 (1): 159–174. 10.2307/2529310
     https://doi.org/10.2307/2529310 [Google Scholar]
  31. Liu, Zhiwei, Kailai Yang, Tianlin Zhang, Qianqian Xie and Sophia Ananiadou
    2024 “Emollms: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysis”. [v2] Tuesday 18 June 2024. arXiv. Accessed20 January 2025. 10.1145/3637528.3671552
     https://doi.org/10.1145/3637528.3671552 [Google Scholar]
  32. McEnery, Tony and Andrew Hardie
    2012 Corpus Linguistics. Cambridge: Cambridge University Press.
     [Google Scholar]
  33. Manjavacas, Enrique and Lauren Fonteyn
    2021 “MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450–1950)”. InMika Hämäläinen, Khalid Alnajjar, Niko Partanen and Jack Rueter (eds), Proceedings of the Workshop on Natural Language Processing for Digital Humanities (NLP4DH 2021), 23–36. Online. 19 December 2021. NIT Silchar, India: the Natural Language Processing Association of India (NLPAI). See: https://aclanthology.org/2021.nlp4dh-1.4/
     [Google Scholar]
  34. 2022 “Adapting vs. Pre-training Language Models for Historical Languages”. Journal of Data Mining & Digital HumanitiesNLP4DH1: 1–19. 10.46298/jdmdh.9152
     https://doi.org/10.46298/jdmdh.9152 [Google Scholar]
  35. Meta AI
    Meta AI 2024 “Introducing Meta Llama 3: The Most Capable Openly Available llm to Date”. Meta AI. Published18 April 2024. Accessed23 May 2025at: https://ai.meta.com/blog/meta-llama-3/
     [Google Scholar]
  36. Naveed, Humza, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes and Ajmal Mian
    2024 “A Comprehensive Overview of Large Language Models”. [v10] Thursday 17 October 2024. arXiv. Accessed20 January 2025. 10.48550/arXiv.2307.06435
     https://doi.org/10.48550/arXiv.2307.06435 [Google Scholar]
  37. Nuyts, Jan
    2000 Epistemic Modality, Language and Conceptualization: A Cognitive-Pragmatic Perspective. Amsterdam: John Benjamins.10.1075/hcp.5
     https://doi.org/10.1075/hcp.5 [Google Scholar]
  38. OpenAI
    OpenAI 2024 “Hello GPT-4o”. OpenAI. PublishedMay 13, 2024. Accessed23 May 2025at: https://openai.com/index/hello-gpt-4o/
     [Google Scholar]
  39. OpenAI
    OpenAI. n.d. “Retrieval Augmented Generation (RAG) and Semantic Search for GPTs”. OpenAI Help Center. Accessed7 June 2025at: https://help.openai.com/en/articles/8868588-retrieval-augmented-generation-rag-and-semantic-search-for-gpts
     [Google Scholar]
  40. Qiu, Xipeng, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai and Xuanjing Huang
    2020 “Pre-Trained Models for Natural Language Processing: A Survey”. Science China Technological Sciences631: 1872–1897. 10.1007/s11431‑020‑1647‑3
     https://doi.org/10.1007/s11431-020-1647-3 [Google Scholar]
  41. Qwen Team
    Qwen Team 2024 “Hello Qwen2”. Qwen. Published15 July 2024. Accessed23 May 2025at: https://qwenlm.github.io/blog/qwen2/
     [Google Scholar]
  42. Simon-Vandenbergen, Anne-Marie and Karin Aijmer
    2007 The Semantic Field of Modal Certainty: A Corpus-based Study of English Adverbs. Berlin and New York: Mouton de Gruyter. 10.1515/9783110198928
     https://doi.org/10.1515/9783110198928 [Google Scholar]
  43. Skansi, Sandro
    2018 Introduction to Deep Learning: From Logical Calculus to Artificial Intelligence. Cham, Switzerland: Springer. 10.1007/978‑3‑319‑73004‑2
     https://doi.org/10.1007/978-3-319-73004-2 [Google Scholar]
  44. Squartini, Mario
    2016 “Interactions between Modality and Other Semantic Categories”. InJan Nuyts and Johan van der Auwera (eds), The Oxford Handbook of Modality and Mood, 50–67. Oxford: Oxford University Press.
     [Google Scholar]
  45. Taavitsainen, Irma
    2018 “Historical Corpus Pragmatics”. InAndreas H. Jucker, Klaus P. Schneider and Wolfram Bublitz (eds), Methods in Pragmatics, 527–553. Berlin and Boston: De Gruyter Mouton. 10.1515/9783110424928‑021
     https://doi.org/10.1515/9783110424928-021 [Google Scholar]
  46. Taavitsainen, Irma and Andreas H. Jucker
    2010 “Trends and Developments in Historical Pragmatics”. InAndreas H. Jucker and Irma Taavitsainen (eds), Historical Pragmatics, 3–30. Berlin and New York: De Gruyter Mouton. 10.1515/9783110214284.1.3
     https://doi.org/10.1515/9783110214284.1.3 [Google Scholar]
  47. Tharwat, Alaa
    2021 “Classification Assessment Methods”. Applied Computing and Informatics17 (1): 168–192. 10.1016/j.aci.2018.08.003
     https://doi.org/10.1016/j.aci.2018.08.003 [Google Scholar]
  48. Varnum, Michael E. W., Nicolas Baumard, Mohammad Atari and Kurt Gray
    2024 “Large Language Models Based on Historical Text Could Offer Informative Tools for Behavioral Science”. PNAS121 (42): e2407639121. 10.1073/pnas.2407639121
     https://doi.org/10.1073/pnas.2407639121 [Google Scholar]
  49. Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le and Denny Zhou
    2022 “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”. On the 36th Conference on Neural Information Processing Systems (NeurIPS 2022). New Orleans, USAand online. [v6]10 January 2023. arXiv. Accessed18 January 2025. 10.48550/arXiv.2201.11903
     https://doi.org/10.48550/arXiv.2201.11903 [Google Scholar]
  50. Whitt, Richard J.
    2023 “Epistemic Space and Key Concepts in Early and Late Modern Medical Discourse: An Exploration of Two Genres”. English Language and Linguistics27 (2): 241–269. 10.1017/S136067432200034X
     https://doi.org/10.1017/S136067432200034X [Google Scholar]
  51. Yao, Ben, Yazhou Zhang, Qiuchi Li and Jing Qin
    2024a “Is Sarcasm Detection a Step-by-Step Reasoning Process in Large Language Models?” [v2] 24 August 2024. arXiv. Accessed17 January 2025. 10.48550/arXiv.2407.12725
     https://doi.org/10.48550/arXiv.2407.12725 [Google Scholar]
  52. Yao, Shunyu, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao and Karthik Narasimhan
    2024b “Tree of Thoughts: Deliberate Problem Solving with Large Language Models”. [v2] Sunday 3 December 2023. arXiv. Accessed17 January 2025. 10.48550/arXiv.2305.10601
     https://doi.org/10.48550/arXiv.2305.10601 [Google Scholar]
  53. Yu, Danni, Luyang Li, Hang Su and Matteo Fuoli
    2024 “Assessing the Potential of llm-assisted Annotation for Corpus-based Pragmatics and Discourse Analysis: The Case of Apology”. International Journal of Corpus Linguistics29 (4): 534–561. 10.1075/ijcl.23087.yu
     https://doi.org/10.1075/ijcl.23087.yu [Google Scholar]
  54. Zhao, Wayne Xin, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie and Ji-Rong Wen
    2024 “A Survey of Large Language Models”. [v15] Sunday 13 October 2024. arXiv. Accessed20 January 2025. 10.48550/arXiv.2303.18223
     https://doi.org/10.48550/arXiv.2303.18223 [Google Scholar]
/content/journals/10.1075/jhp.25011.hua
Loading
Assessing the potential of using large language models for pragmatic annotation of historical texts
Journal of Historical Pragmatics 26, 406 (2025); https://doi.org/10.1075/jhp.25011.hua
/content/journals/10.1075/jhp.25011.hua
/content/journals/10.1075/jhp.25011.hua
Loading

Data & Media loading...

  • Article Type: Research Article
Keyword(s): Early Modern English; epistemic stance; historical pragmatics; large language models; pragmatic annotation

Most Cited

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error