1887
  1. Home
  2. e-Journals
  3. International Journal of Corpus Linguistics
  4. Volume 30, Issue 2
  5. Article
Volume 30, Issue 2
  • ISSN 1384-6655
  • E-ISSN: 1569-9811
EPUB
PDF

Abstract

Our study focuses on replicability, which entails researchers’ ability to achieve similar results to a prior study using identical methods but a different yet comparable dataset. We address the challenge of stance detection (determining whether a document is “favorable,” “against,” or “neutral” toward a target), building on prior research underscoring the value of linguistic markers as complementary features for sentiment detection that enable more accurate stance classification. We utilize the Stance in Replies and Quotes (SRQ) dataset, which contains annotated discussion-based responses. Employing a rule-based methodology that emphasizes linguistic features, we examine whether the classification accuracy remains within a similar error margin as observed in a previous study of another dataset. Consistency is a necessary condition for robustness and generalizability, ultimately enhancing trust in the methodology. The replication of the model and its adaptability to the new data context demonstrate that it is competitive compared to existing machine learning studies.

© John Benjamins Publishing Company
Available under the CC BY 4.0 license.
Loading

Article metrics loading...

/content/journals/10.1075/ijcl.24132.rev
2025-09-12
2026-08-11

  • From This Site

    /content/journals/10.1075/ijcl.24132.rev
    dcterms_title,dcterms_subject,pub_keyword
    -contentType:Journal -contentType:Contributor -contentType:Concept -contentType:Institution
    10
    5
Loading full text...

Full text loading...

/deliver/fulltext/ijcl.24132.rev.html?itemId=/content/journals/10.1075/ijcl.24132.rev&mimeType=html&fmt=ahah

References

  1. Arnold, T., & Tilton, L.
    (2022) Wrappers around Stanford CoreNLP tools (version 0.4–2) [R package]. https://cran.r-project.org/web/packages/coreNLP/coreNLP.pdf
     [Google Scholar]
  2. Baker, M.
    (2016) 1,500 scientists lift the lid on reproducibility. Nature, 5331, 452–454. 10.1038/533452a
     https://doi.org/10.1038/533452a [Google Scholar]
  3. Benoit, K.
    (2018) LIWCalike [R package]. https://github.com/kbenoit/LIWCalike
     [Google Scholar]
  4. Bettis, R. A., Helfat, C. E., & Shaver, J. M.
    (2016) The necessity, logic, and forms of replication. Strategic Management Journal, 37(11), 2193–2203. 10.1002/smj.2580
     https://doi.org/10.1002/smj.2580 [Google Scholar]
  5. Blitzer, J., Dredze, M., & Pereira, F.
    (2007) Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. InA. Zaenen & A. van den Bosch (Eds.) Proceedings of the 45th annual meeting of the Association of Computational Linguistics (pp.440–447). Association for Computational Linguistics. https://aclanthology.org/P07-1056.pdf
     [Google Scholar]
  6. Buntain, C., & Golbeck, J.
    (2017) Automatically identifying fake news in popular twitter threads. In2017 IEEE international conference on SmartcCloud (pp.208–215). Institute of Electrical and Electronic Engineers. 10.1109/SmartCloud.2017.40
     https://doi.org/10.1109/SmartCloud.2017.40 [Google Scholar]
  7. Coutellec, L.
    (2019) Ethics and scientific integrity in biomedical research. debates on trust, robustness, and relevance. Handbook of research ethics and scientific integrity, 1–14. HAL Open Science. 10.1007/978‑3‑319‑76040‑7_36‑1
     https://doi.org/10.1007/978-3-319-76040-7_36-1 [Google Scholar]
  8. Crible, L.
    (2022) Studying discourse from corpus and experimental data: Bridging the methodological gap. Discours, 301. journals.openedition.org/discours/12024
     [Google Scholar]
  9. Dipper, S.
    (2008) Theory-driven and corpus-driven computational linguistics, and the use of corpora. InA. Lüdeling & M. Kytö (Eds.), Corpus linguistics. An international handbook (pp.68–96). Mouton de Gruyter.
     [Google Scholar]
  10. Ehret, K., & Taboada, M.
    (2020) The interplay of complexity and subjectivity in opinionated discourse. Discourse Studies, 23(2), 141–165. 10.1177/1461445620966923
     https://doi.org/10.1177/1461445620966923 [Google Scholar]
  11. Finkel, J. R., Grenager, T., & Manning, C.
    (2005) Incorporating non-local information into information extraction systems by Gibbs sampling. InK. Knight, H. T. Ng, & K. Oflazer (Eds.) Proceedings of the 43rd annual meeting of the Association for Computational Linguistics (ACL’05). Association for Computational Linguistics. 10.3115/1219840.1219885
     https://doi.org/10.3115/1219840.1219885 [Google Scholar]
  12. Funer, F.
    (2022) The Deception of certainty: How non-interpretable machine learning outcomes challenge the epistemic authority of physicians. A deliberative-relational approach. Medicine, Health Care and Philosophy, 251, 167–178. 10.1007/s11019‑022‑10076‑1
     https://doi.org/10.1007/s11019-022-10076-1 [Google Scholar]
  13. Ghosh, S., Singhania, P., Singh, S., Rudra, K., & Ghosh, S.
    (2019) Stance detection in web and social media: a comparative study. InF. Crestani, M. Braschler, J. Savoy, A. Rauber, H. Müller, D. E. Losada, G. H. Bürki, L. Cappellato, & N. Ferro (Eds.) Experimental IR meets multilinguality, multimodality, and interaction: 10th international conference of the Cross-Language Evaluation Forum (CLEF) Association. (pp.75–87). Springer. 10.1007/978‑3‑030‑28577‑7_4
     https://doi.org/10.1007/978-3-030-28577-7_4 [Google Scholar]
  14. Gillings, M., Mautner, G., & Baker, P.
    (2023) Corpus-assisted discourse studies. Cambridge University Press. 10.1017/9781009168144
     https://doi.org/10.1017/9781009168144 [Google Scholar]
  15. Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G.
    (2016) Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 311, 337–350. 10.1007/s10654‑016‑0149‑3
     https://doi.org/10.1007/s10654-016-0149-3 [Google Scholar]
  16. Grieve, J.
    (2021) Observation, experimentation, and replication in linguistics. Linguistics, 59(5), 1343–1356. 10.1515/ling‑2021‑0094
     https://doi.org/10.1515/ling-2021-0094 [Google Scholar]
  17. Grimmer, J., Roberts, M. E., & Stewart, B. M.
    (2022) Text as data: A new framework for machine learning and the social sciences. Princeton University Press.
     [Google Scholar]
  18. Hartmann, J., Heitmann, M., Siebert, C., & Schamp, C.
    (2023) More than a feeling: Accuracy and application of sentiment analysis. International Journal of Research in Marketing, 40(1), 75–87. 10.1016/j.ijresmar.2022.05.005
     https://doi.org/10.1016/j.ijresmar.2022.05.005 [Google Scholar]
  19. Joseph, K., Shugars, S., Gallagher, R., Green, J., Mathé, A. Q., An, Z., & Lazer, D.
    (2021) (Mis)Alignment between stance expressed in social media data and public opinion surveys. InM.-F. Moens, X. Huang, L. Specia, & S. W.-T. Yih (Eds.), Proceedings of the 2021 conference on empirical methods in Natural Language Processing (pp.312–324). Association for Computational Linguistics. 10.18653/v1/2021.emnlp‑main.27
     https://doi.org/10.18653/v1/2021.emnlp-main.27 [Google Scholar]
  20. Kilgarriff, A.
    (2005) Language is never, ever, ever, random. Corpus Linguistics and Linguistic Theory, 1(2), 263–276. 10.1515/cllt.2005.1.2.263
     https://doi.org/10.1515/cllt.2005.1.2.263 [Google Scholar]
  21. Küçük, D., & Can, F.
    (2020) Stance detection: A survey. ACM Computing Surveys (CSUR), 53(1), 1–37. 10.1145/3369026
     https://doi.org/10.1145/3369026 [Google Scholar]
  22. Lamprecht, A.-L., Garcia, L., Kuzak, M., Martinez, C., Arcila, R., Martin Del Pico, E., Dominguez Del Angel, V., van de Sandt, S., Ison, J., Martinez, P. A., McQuilton, P., Valencia, A., Harrow, J., Psomopoulos, F., Gelpi, J. L., Chue Hong, N., Goble, C., & Capella-Gutierrez, S.
    (2020) Towards FAIR principles for research software. Data Science, 3(1), 37–59. 10.3233/DS‑190026
     https://doi.org/10.3233/DS-190026 [Google Scholar]
  23. Levshina, N.
    (2015) How to do linguistics with R: Data exploration and statistical analysis. John Benjamins. 10.1075/z.195
     https://doi.org/10.1075/z.195 [Google Scholar]
  24. Lewandowsky, S., & Oberauer, K.
    (2020) Low replicability can support robust and efficient science. Nature Communication, 111, 358. 10.1038/s41467‑019‑14203‑0
     https://doi.org/10.1038/s41467-019-14203-0 [Google Scholar]
  25. Li, S., & Zong, C.
    (2008) Multi-domain adaptation for sentiment classification: Using multiple classifier combining methods. InProceedings of the 2008 international conference on Natural Language Processing and knowledge engineering (pp.1–8). Institute of Electrical and Electronic Engineers. 10.1109/NLPKE.2008.4906772
     https://doi.org/10.1109/NLPKE.2008.4906772 [Google Scholar]
  26. Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S., & McClosky, D.
    (2014) The Stanford CoreNLP natural language processing toolkit. InK. Bontcheva & J. Zhu (Eds.), Proceedings of the 52nd annual meeting of the association for computational linguistics: System demonstrations (pp.55–60). 10.3115/v1/P14‑5010
     https://doi.org/10.3115/v1/P14-5010 [Google Scholar]
  27. Mohammad, S. M., & Turney, P. D.
    (2013) NRC word-emotion association lexicon (version 0.92). National Research Council, Canada. www.saifmohammad.com/WebDocs/NRCemotionlexicon.pdf
     [Google Scholar]
  28. Munafò, M. R., & Smith, G. D.
    (2018) Robust research needs many lines of evidence. Nature, 5531, 399–401. 10.1038/d41586‑018‑01023‑3
     https://doi.org/10.1038/d41586-018-01023-3 [Google Scholar]
  29. Ng, L. H. X., & Carley, K. M.
    (2022) Is my stance the same as your stance? A cross validation study of stance detection datasets. Information Processing & Management, 59(6), 103070. 10.1016/j.ipm.2022.103070
     https://doi.org/10.1016/j.ipm.2022.103070 [Google Scholar]
  30. Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Struhl, M. K., Nuijten, M. B., Rohrer, J. M., Romero, F., Scheel, A. M., Scherer, L. D., Schönbrodt, F. D., & Vazire, S.
    (2022) Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology, 731, 719–748. 10.1146/annurev‑psych‑020821‑114157
     https://doi.org/10.1146/annurev-psych-020821-114157 [Google Scholar]
  31. Pamungkas, E. W., Basile, V., & Patti, V.
    (2019) Stance classification for rumour analysis in twitter: Exploiting affective information and conversation structure. arXiv preprint, 1901.01911. 10.48550/arXiv.1901.01911
     https://doi.org/10.48550/arXiv.1901.01911 [Google Scholar]
  32. Peels, R., & Bouter, L.
    (2023) Replication and trustworthiness. Accountability in Research, 30(2), 77–87. 10.1080/08989621.2021.1963708
     https://doi.org/10.1080/08989621.2021.1963708 [Google Scholar]
  33. Pennebaker, J. W., Booth, R. J., & Francis, M. E.
    (2007) Linguistic Inquiry and Word Count: LIWC [Computer software]. https://www.liwc.app/
     [Google Scholar]
  34. Popper, K.
    (1959) The Logic of scientific discovery. Hutchison.
     [Google Scholar]
  35. Reveilhac, M., & Schneider, G.
    (2023) Replicable semi-supervised approaches to state-of-the-art stance detection of tweets. Information Processing & Management, 60(2), 103199. 10.1016/j.ipm.2022.103199
     https://doi.org/10.1016/j.ipm.2022.103199 [Google Scholar]
  36. Schiller, B., Daxenberger, J., & Gurevych, I.
    (2021) Stance detection benchmark: How robust is your stance detection?KI-Künstliche Intelligenz, 35(3), 329–341. 10.1007/s13218‑021‑00714‑w
     https://doi.org/10.1007/s13218-021-00714-w [Google Scholar]
  37. Schneider, G., Hundt, M., & Oppliger, R.
    (2016) Part-of-speech in historical corpora: Tagger evaluation and ensemble systems on ARCHER. InS. Dipper, F. Neubarth, & H. Zinsmeister (Eds.), Proceedings of the 13th conference on Natural Language Processing, KONVENS 2016 (pp.256–264). Bochumer Linguistische Arbeitsberichte. 10.5167/uzh‑135065
     https://doi.org/10.5167/uzh-135065 [Google Scholar]
  38. Schneider, G., & Lauber, M.
    (2019) Statistics for linguists: A patient, slow-paced introduction to statistics and to the programming language R. University of Zurich. 10.5167/uzh‑183632
     https://doi.org/10.5167/uzh-183632 [Google Scholar]
  39. Schreiber-Gregory, D.
    (2018) Regulation techniques for multicollinearity: Lasso, ridge, and elastic nets. Proceedings of Western users of SAS software conferences 2018. https://www.lexjansen.com/wuss/2018/131_Final_Paper_PDF.pdf
     [Google Scholar]
  40. Schwab, S., Janiaud, P., Dayan, M., Amrhein, V., Panczak, R., Palagi, P. M., Hemkens, L. G., Ramon, M., Rothen, N., Senn, S., Furrer, E., & Held, L.
    (2022) Ten simple rules for good research practice. PLOS Computational Biology, 18(6), e1010139. 10.1371/journal.pcbi.1010139
     https://doi.org/10.1371/journal.pcbi.1010139 [Google Scholar]
  41. Sönning, L. and Werner, V.
    (2021) The replication crisis, scientific revolutions, and linguistics. Linguistics, 59(5), 1179–1206. 10.1515/ling‑2019‑0045
     https://doi.org/10.1515/ling-2019-0045 [Google Scholar]
  42. Taboada, M.
    (2016) Sentiment analysis: An overview from linguistics. Annual Review of Linguistics, 21, 325–347. 10.1146/annurev‑linguistics‑011415‑040518
     https://doi.org/10.1146/annurev-linguistics-011415-040518 [Google Scholar]
  43. Tausczik, Y. R., & Pennebaker, J. W.
    (2010) The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24–54. 10.1177/0261927X09351676
     https://doi.org/10.1177/0261927X09351676 [Google Scholar]
  44. Tognini-Bonelli, E.
    (2001) Corpus linguistics at work. John Benjamins. 10.1075/scl.6
     https://doi.org/10.1075/scl.6 [Google Scholar]
  45. Vajjala, S., & Balasubramaniam, R.
    (2022) What do we really know about state of the art NER?. InN. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, & S. Piperidis (Eds.), Proceedings of the thirteenth Language Resources and Evaluation Conference (LREC) (pp.5983–5993). https://aclanthology.org/2022.lrec-1.643/
     [Google Scholar]
  46. Villa-Cox, R., Kumar, S., Babcock, M., & Carley, K. M.
    (2020) Stance in Replies and Quotes (SRQ): A new dataset for learning stance in twitter conversations. arXiv preprint, 2006.00691. 10.48550/arXiv.2006.00691
     https://doi.org/10.48550/arXiv.2006.00691 [Google Scholar]
  47. Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., Gonzalez-Beltran, A., Gray, A. J. G., Groth, P., Goble, C., Grethe, J. S., Heringa, J., ’t Hoen, P. A. C., Hooft, R., Kuhn, T., Kok, R., Kok, J., Lusher, S. J., Martone, M. E., Mons, A., Packer, A. L., Persson, B., Rocca-Serra, P., Roos, M., van Schaik, R., Sansone, S.-A., Schultes, E., Sengstag, T., Slater, T., Strawn, G., Swertz, M. A., Thompson, M., van der Lei, J., van Mulligen, E., Velterop, J., Waagmeester, A., Wittenburg, P., Wolstencroft, K., Zhao, J., & Mons, B.
    (2016) The FAIR guiding principles for scientific data management and stewardship. Scientific Data, 31, 160018. 10.1038/sdata.2016.18
     https://doi.org/10.1038/sdata.2016.18 [Google Scholar]
  48. Wilson, M.
    (2009) Quality matters: Correctness, robustness and reliability. Overload, 17(93). https://accu.org/journals/overload/17/93/wilson_1585/
     [Google Scholar]
  49. Winter, B.
    (2019) Statistics for linguists: An introduction using R. Routledge. 10.4324/9781315165547
     https://doi.org/10.4324/9781315165547 [Google Scholar]
  50. Winter, B., & Grice, M.
    (2021) Independence and generalizability in linguistics. Linguistics, 59(5), 1251–1277. 10.1515/ling‑2019‑0049
     https://doi.org/10.1515/ling-2019-0049 [Google Scholar]
  51. Young, L., & Soroka, S.
    (2012) Affective news: The automated coding of sentiment in political texts. Political Communication, 29(2), 205–231. 10.1080/10584609.2012.671234
     https://doi.org/10.1080/10584609.2012.671234 [Google Scholar]
  52. Zojaji, Z., & Tork Ladani, B.
    (2022) Adaptive cost-sensitive stance classification model for rumor detection in social networks. Social Networking Analysis and Mining, 121, 134. 10.1007/s13278‑022‑00952‑2
     https://doi.org/10.1007/s13278-022-00952-2 [Google Scholar]
/content/journals/10.1075/ijcl.24132.rev
Loading
Evaluating a transparent and interpretable approach to stance detection using linguistic markers in social media data
International Journal of Corpus Linguistics 30, 195 (2025); https://doi.org/10.1075/ijcl.24132.rev
/content/journals/10.1075/ijcl.24132.rev
/content/journals/10.1075/ijcl.24132.rev
Loading

Data & Media loading...

  • Article Type: Research Article
Keyword(s): replicability; robustness; social media; stance detection; transferability

Most Cited

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error