Volume 25, Issue 2
  • ISSN 0929-998X
  • E-ISSN: 1569-9765
Buy:$35.00 + Taxes



Despite a growing awareness of methodological issues, the literature on has not so far provided adequate answers to some of the key challenges involved in reliably identifying and classifying evaluative language expressions. This article presents a stepwise method for the manual annotation of in text that is designed to optimize reliability, replicability and transparency. The procedure consists of seven steps, from the creation of a context-specific annotation manual to the statistical analysis of the quantitative data derived from the manually-performed annotations. By presenting this method, the article pursues the twofold purpose of (i) providing a practical tool that can facilitate more reliable, replicable and transparent analyses, and (ii) fostering a discussion of the best practices that should be observed when manually annotating .


Article metrics loading...

Loading full text...

Full text loading...


  1. Artstein, Ron & Massimo Poesio
    2008 Inter-coder agreement for computational linguistics. Computational Linguistics34(4). 555–596.10.1162/coli.07‑034‑R2
    https://doi.org/10.1162/coli.07-034-R2 [Google Scholar]
  2. Baayen, R. Harald
    2008Analyzing linguistic data: A practical introduction to Statistics using R. Cambridge: CUP.10.1017/CBO9780511801686
    https://doi.org/10.1017/CBO9780511801686 [Google Scholar]
  3. Bartalesi Lenzi, Valentina , Giovanni Moretti & Rachele Sprugnoli
    2012 CAT: the CELCT Annotation Tool. In Nicoletta Calzolari , Khalid Choukri , Thierry Declerck , Mehmet Uğur Doğan , Bente Maegaard , Joseph Mariani , Asuncion Moreno , Jan Odijk & Stelios Piperidis (eds.), Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC12), Istanbul: European Language Resources Association (ELRA). Available online at: www.lrec-conf.org/proceedings/lrec2012/pdf/216_Paper.pdf [Accessed22 September 2015]
    [Google Scholar]
  4. Bednarek, Monika
    2006Evaluation in media discourse: Analysis of a newspaper corpus. London: Continuum.
    [Google Scholar]
  5. 2008Emotion talk across corpora. Basingstoke: Palgrave.10.1057/9780230285712
    https://doi.org/10.1057/9780230285712 [Google Scholar]
  6. 2009 Language patterns and ATTITUDE. Functions of Language16(2). 165–192.10.1075/fol.16.2.01bed
    https://doi.org/10.1075/fol.16.2.01bed [Google Scholar]
  7. 2014 An astonishing season of destiny! Evaluation in blurbs used for advertising TV series. In Geoff Thompson & Laura Alba-Juez (eds.), 197–220.
    [Google Scholar]
  8. Ben-Aaron, Diana
    2005 Given and news: Evaluation in newspaper stories about national anniversaries. Text & Talk25(5). 691–718.10.1515/text.2005.25.5.691
    https://doi.org/10.1515/text.2005.25.5.691 [Google Scholar]
  9. Carretero, Marta & Maite Taboada
    2014 Graduation within the scope of Attitude in English and Spanish consumer reviews of books and movies. In Geoff Thompson & Laura Alba-Juez (eds.), 221–239.
    [Google Scholar]
  10. Cohen, Jacob
    1960 A coefficient of agreement for nominal scales. Educational and Psychological Measurement20(1). 37–46.10.1177/001316446002000104
    https://doi.org/10.1177/001316446002000104 [Google Scholar]
  11. Cozby, Paul & Scott Bates
    2011Methods in behavioral research, 11th edn.New York, NY: McGraw-Hill.
    [Google Scholar]
  12. Dice, Lee R.
    1945 Measures of the amount of ecologic association between species. Ecology26(3). 297–302.10.2307/1932409
    https://doi.org/10.2307/1932409 [Google Scholar]
  13. Divjak, Dagmar
    2006 Ways of intending: Delineating and structuring near synonyms. In Stefan Th. Gries & Anatol Stefanowitsch (eds.), Corpora in cognitive linguistics: Corpus-based approaches to syntax and lexis, 19–56. Berlin: Mouton.
    [Google Scholar]
  14. Don, Alexanne
    2007 An approach to the analysis of textual identity through profiles of evaluative disposition. InProceedings of the Australian Systemic Functional Linguistics Association 2007 Conference. Available at: www.asfla.org.au/category/asfla2007/. [Accessed22 September 2015]
    [Google Scholar]
  15. Fuoli, Matteo
    2012 Assessing social responsibility: A quantitative analysis of Appraisal in BP’s and IKEA’s social reports. Discourse & Communication6(1). 55–81.10.1177/1750481311427788
    https://doi.org/10.1177/1750481311427788 [Google Scholar]
  16. Fuoli, Matteo & Dylan Glynn
    2013 Computer-assisted manual annotation of evaluative language expressions: Bridging discourse and corpus approaches. Paper presented atthe Evaluative Language and Corpus Linguistics Workshop, Lancaster University, 22 July 2013.
    [Google Scholar]
  17. Fuoli, Matteo & Charlotte Hommerberg
    2015 Optimizing transparency, reliability and replicability: Annotation principles and inter-coder agreement in the quantification of evaluative expressions. Corpora10(3). 315–349.10.3366/cor.2015.0080
    https://doi.org/10.3366/cor.2015.0080 [Google Scholar]
  18. Fuoli, Matteo & Carita Paradis
    2014 A model of trust-repair discourse. Journal of Pragmatics74. 52–69.10.1016/j.pragma.2014.09.001
    https://doi.org/10.1016/j.pragma.2014.09.001 [Google Scholar]
  19. Geeraerts, Dirk , Stefan Grondelaers & Peter Bakema
    1994The structure of lexical variation. Berlin: Mouton.10.1515/9783110873061
    https://doi.org/10.1515/9783110873061 [Google Scholar]
  20. Glynn, Dylan
    2009 Polysemy, syntax, and variation: A usage-based method for Cognitive Semantics. In Vyvyan Evans & Stéphanie Pourcel (eds.), New Directions in Cognitive Linguistics, 77–104. Amsterdam: Benjamins.10.1075/hcp.24.08gly
    https://doi.org/10.1075/hcp.24.08gly [Google Scholar]
  21. Gries, Stefan Th.
    1999 Particle movement: A cognitive and functional approach. Cognitive Linguistics10(2). 105–146.10.1515/cogl.1999.005
    https://doi.org/10.1515/cogl.1999.005 [Google Scholar]
  22. 2013Statistics for linguistics with R: A practical introduction. Berlin: Mouton.10.1515/9783110307474
    https://doi.org/10.1515/9783110307474 [Google Scholar]
  23. Hommerberg, Charlotte & Alexanne Don
    2015 Appraisal and the language of wine appreciation: A critical discussion of the potential of the Appraisal framework as a tool to analyse specialised genres. Functions of Language22(2). 161–191.10.1075/fol.22.2.01hom
    https://doi.org/10.1075/fol.22.2.01hom [Google Scholar]
  24. Hood, Susan
    2004Appraising research: Taking a stance in academic writing. Sydney: University of Technology PhD thesis.
    [Google Scholar]
  25. Hunston, Susan
    2004 Counting the uncountable: Problems of identifying evaluation in a text and in a corpus. In Alan Partington , John Morley & Louann Haarman (eds.), Corpora and Discourse, 157–188. Bern: Peter Lang.
    [Google Scholar]
  26. 2011Corpus approaches to evaluation: Phraseology and evaluative language. New York, NY: Taylor & Francis.
    [Google Scholar]
  27. Kirk, Jerome & Marc L. Miller
    1986Reliability and validity in qualitative research. Beverly Hills, CA: Sage.10.4135/9781412985659
    https://doi.org/10.4135/9781412985659 [Google Scholar]
  28. Krippendorf, Klaus
    2004Content Analysis: An introduction to its methodology. Thousand Oaks, CA: Sage.
    [Google Scholar]
  29. Lipovsky, Caroline
    2008 Constructing affiliation and solidarity in job interviews. Discourse & Communication2(4). 411–432.10.1177/1750481308095938
    https://doi.org/10.1177/1750481308095938 [Google Scholar]
  30. 2011It’s really a great presentation!: Appraising candidates in job interviews. Linguistics and the Human Sciences4(2). 161–185.10.1558/lhs.v4i2.161
    https://doi.org/10.1558/lhs.v4i2.161 [Google Scholar]
  31. 2013 Negotiating ones expertise through appraisal in CVs. Linguistics and the Human Sciences8(3). 307–333.10.1558/lhs.v8i3.307
    https://doi.org/10.1558/lhs.v8i3.307 [Google Scholar]
  32. Mackay, James & Jean Parkinson
    2009My very own mission impossible: An APPRAISAL analysis of student teacher reflections on a design and technology project. Text & Talk29(6). 729–753.10.1515/TEXT.2009.037
    https://doi.org/10.1515/TEXT.2009.037 [Google Scholar]
  33. Macken-Horarik, Mary & Anne Isaac
    2014 Appraising Appraisal. In Geoff Thompson & Laura Alba-Juez (eds.), 67–92.
    [Google Scholar]
  34. Marshall, Christine , Ralph Adendorff & Vivian de Klerk
    2010 The role of APPRAISAL in the NRF Rating System: An analysis of Judgement and Appreciation in peer reviewers’ reports. Southern African Linguistics and Applied Language Studies27(4). 391–412.10.2989/SALALS.2009.
    https://doi.org/10.2989/SALALS.2009. [Google Scholar]
  35. Martin, J. R.
    2000 Beyond exchange: Appraisal systems in English. In Susan Hunston & Geoff Thompson (eds.), Evaluation in text: Authorial stance and the construction of discourse, 142–175. Oxford: OUP.
    [Google Scholar]
  36. Martin, J. R. & David Rose
    2003Working with discourse: Meaning beyond the clause. London: Continuum.
    [Google Scholar]
  37. Martin, J. R. & Peter R. R. White
    2005The language of evaluation: Appraisal in English. London: Palgrave Macmillan.10.1057/9780230511910
    https://doi.org/10.1057/9780230511910 [Google Scholar]
  38. Mauranen, Anna & Marina Bondi
    2003 Evaluative language use in academic discourse. Journal of English for Academic Purposes2(4). 269–271.10.1016/S1475‑1585(03)00045‑6
    https://doi.org/10.1016/S1475-1585(03)00045-6 [Google Scholar]
  39. O’Donnell, Mick
    2008 Demonstration of the UAM CorpusTool for text and image annotation. Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Demo Session. 13–16.10.3115/1564144.1564148
    https://doi.org/10.3115/1564144.1564148 [Google Scholar]
  40. 2012 Appraisal analysis and the computer. Revista Canaria de Estudios Ingleses65. 115–130.
    [Google Scholar]
  41. 2014 Exploring identity through Appraisal Analysis: A corpus annotation methodology. Linguistics and the Human Sciences9(1). 95–116.
    [Google Scholar]
  42. Page, Ruth E.
    2003 An analysis of APPRAISAL in childbirth narratives with special consideration of gender and storytelling style. Text & Talk23(2). 211–237.
    [Google Scholar]
  43. Paradis, Carita
    2001 Adjectives and boundedness. Cognitive Linguistics12(1). 47–64.10.1515/cogl.12.1.47
    https://doi.org/10.1515/cogl.12.1.47 [Google Scholar]
  44. Paradis, Carita , Joost van de Weijer , Caroline Willners & Magnus Lindgren
    2012 Evaluative polarity of antonyms. Lingue e linguaggio11(2). 199–214.
    [Google Scholar]
  45. Pounds, Gabrina
    2010 Attitude and subjectivity in Italian and British hardnews reporting: The construction of a culture-specific ‘reporter’ voice. Discourse Studies12(1). 106–137.10.1177/1461445609346777
    https://doi.org/10.1177/1461445609346777 [Google Scholar]
  46. 2011This property offers much character and charm: Evaluation in the discourse of online property advertising. Text & Talk31(2). 195–220.10.1515/text.2011.009
    https://doi.org/10.1515/text.2011.009 [Google Scholar]
  47. Pustejovsky, James & Amber Stubbs
    2012Natural language annotation for machine learning. Sebastopol, CA: O’Reilly Media.
    [Google Scholar]
  48. Read, Jonathon & John Carroll
    2012 Annotating expressions of Appraisal in English. Language Resources and Evaluation46(3). 421–447.10.1007/s10579‑010‑9135‑7
    https://doi.org/10.1007/s10579-010-9135-7 [Google Scholar]
  49. Ryshina-Pankova, Marianna
    2014 Exploring argumentation in course-related blogs through ENGAGEMENT. In Geoff Thompson & Laura Alba-Juez (eds.), 281–302.
    [Google Scholar]
  50. Santamaría-García, Carmen
    2014 Evaluative discourse and politeness in university students’ communication through social networking sites. In Geoff Thompson & Laura Alba-Juez (eds.), 387–411.
    [Google Scholar]
  51. Taboada, Maite & Marta Carretero
    2012 Contrastive analyses of evaluation in text: Key issues in the design of an annotation system for ATTITUDE applicable to consumer reviews in English and Spanish. Linguistics and the Human Sciences6(1–3). 275–295.
    [Google Scholar]
  52. Taboada, Maite , Marta Carretero & Jennifer Hinnell
    2014 Loving and hating the movies in English, German and Spanish. Languages in Contrast14(1). 127–161.10.1075/lic.14.1.07tab
    https://doi.org/10.1075/lic.14.1.07tab [Google Scholar]
  53. Thompson, Geoff
    2014 AFFECT and emotion, target-value mismatches, and Russian dolls: Refining the APPRAISAL model. In Geoff Thompson & Laura Alba-Juez (eds.), 47–66.
    [Google Scholar]
  54. Thompson, Geoff & Laura Alba-Juez
    (eds.) 2014Evaluation in context. Amsterdam: Benjamins.10.1075/pbns.242
    https://doi.org/10.1075/pbns.242 [Google Scholar]
  55. Thompson, Geoff & Susan Hunston
    2000 Evaluation: An introduction. In Susan Hunston & Geoff Thompson (eds.), Evaluation in text: Authorial stance and the construction of discourse, 1–27. Oxford: OUP.
    [Google Scholar]

Data & Media loading...

  • Article Type: Research Article
This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error