Volume 4, Issue 1
  • ISSN 2542-3851
  • E-ISSN: 2542-386X
Buy:$35.00 + Taxes



In this paper we examine the order of processing of multimodal tweets (text + image). Using an eye tracker, we collected a sample of 36 participants reading 25 humorous tweets. Our conclusions show that the processing of multimodal humorous tweets is in line with the processing of other multimodal texts. The participants were significantly more likely to start from the image, followed by the caption. Other elements, such as the tweet’s “author” (the user name) or elements outside the tweet’s frame, attracted significantly less and later attention. The participants spent significantly more time gazing at the caption, before moving on to another area. The longer the participants spent looking at the tweet, the less predictable their gaze direction became.


Article metrics loading...

Loading full text...

Full text loading...


  1. Añaños, Elena, and Anna Astals Serés
    2013 “¿Imagen o texto? El poder de captar la atención visual de los elementos gráficos analizado con el Eye tracker [Image or text? The power to capture the visual attention of the graphic elements analyzed with the eye tracker].” Grafica1(2): 87–98.
    [Google Scholar]
  2. Attardo, Salvatore
    2020The Linguistics of Humor: An Introduction. Oxford: Oxford University Press. 10.1093/oso/9780198791270.001.0001
    https://doi.org/10.1093/oso/9780198791270.001.0001 [Google Scholar]
  3. Attardo, Salvatore, and Jean-Charles Chabanne
    1992 “Jokes as a text type.” HUMOR: International Journal of Humor Research5(1–2): 165–176. 10.1515/humr.1992.5.1‑2.165
    https://doi.org/10.1515/humr.1992.5.1-2.165 [Google Scholar]
  4. Attardo, Salvatore, Lucy Pickering, Fofo Lomotey, and Shigehito Menjo
    2013 “Multimodality in conversational humor.” Review of Cognitive Linguistics11(2): 402–416. 10.1075/rcl.11.2.12att
    https://doi.org/10.1075/rcl.11.2.12att [Google Scholar]
  5. Attardo, Salvatore, and Victor Raskin
    1991 “Script theory revis(it)ed: Joke similarity and joke representation model.” HUMOR: International Journal of Humor Research4(3–4): 293–347. 10.1515/humr.1991.4.3‑4.293
    https://doi.org/10.1515/humr.1991.4.3-4.293 [Google Scholar]
  6. Boerman, Sophie C., Edith G. Smit, and Lex van Meurs
    2011 “Attention battle; the abilities of brand, visual, and text characteristics of the ad to draw attention versus the diverting power of the direct magazine context.” InAdvances in Advertising Research, vol.2, ed. byShintaro Okazaki, 295–310. Wiesbaden: Gabler Verlag. 10.1007/978‑3‑8349‑6854‑8_19
    https://doi.org/10.1007/978-3-8349-6854-8_19 [Google Scholar]
  7. Borodo, Michał
    2015 “Multimodality, translation and comics.” Perspectives23(1): 22–41. 10.1080/0907676X.2013.876057
    https://doi.org/10.1080/0907676X.2013.876057 [Google Scholar]
  8. Canestrari, Carla
    2018 “The interplay between visual and verbal language in ‘famous last words’ cartoons.” InThe Languages of Humor: Verbal, Visual, and Physical Humor, ed. byArie Sover, 159–166. London: Bloomsbury Academic.
    [Google Scholar]
  9. Chiaro, Delia
    2005 “Verbally expressed humor and translation: An overview of a neglected field.” HUMOR: International Journal of Humor Research18(2): 135–145. 10.1515/humr.2005.18.2.135
    https://doi.org/10.1515/humr.2005.18.2.135 [Google Scholar]
  10. Foulsham, Tom, Dean Wybrow, and Neil Cohn
    2016 “Reading without words: Eye movements in the comprehension of comic strips.” Applied Cognitive Psychology30(4): 566–579. 10.1002/acp.3229
    https://doi.org/10.1002/acp.3229 [Google Scholar]
  11. Gerhardt, Cornelia
    2009 “Multimodal and intertextual humor in the media reception situation.” InHumor in Interaction, ed. byNeal R. Norrick, and Delia Chiaro, 79–100. Amsterdam: John Benjamins. 10.1075/pbns.182.04ger
    https://doi.org/10.1075/pbns.182.04ger [Google Scholar]
  12. Hamrick, Phillip, and Nick B. Pandža
    2020 “Contributions of semantic and contextual diversity to the word frequency effect in L2 lexical access.” Canadian Journal of Experimental Psychology74(1): 25–34. 10.1037/cep0000189
    https://doi.org/10.1037/cep0000189 [Google Scholar]
  13. Hegarty, Mary
    1992 “Mental animation: Inferring motion from static displays of mechanical systems.” Journal of Experimental Psychology: Learning, Memory, and Cognition18(5): 1084–1102.
    [Google Scholar]
  14. Hernández-Méndez, Janet, and Francisco Muñoz-Leiva
    2015 “What type of online advertising is most effective for eTourism 2.0? An eye tracking study based on the characteristics of tourists.” Computers in Human Behavior50: 618–625. 10.1016/j.chb.2015.03.017
    https://doi.org/10.1016/j.chb.2015.03.017 [Google Scholar]
  15. Kaindl, Klaus
    2004 “Multimodality in the translation of humour in comics.” InPerspectives on Multimodality, ed. byElija Ventola, Cassily Charles, and Martin Kaltenbacher, 173–192. Amsterdam: John Benjamins. 10.1075/ddcs.6.12kai
    https://doi.org/10.1075/ddcs.6.12kai [Google Scholar]
  16. Kress, Gunther, and Theo Van Leeuwen
    2001Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.
    [Google Scholar]
  17. Li, Qian, Zhouwei Huang, and Kiel Christianson
    2016 “Visual attention toward tourism photographs with text: An eye-tracking study.” Tourism Management54: 243–258. 10.1016/j.tourman.2015.11.017
    https://doi.org/10.1016/j.tourman.2015.11.017 [Google Scholar]
  18. Liechty, John, Rik Pieters, and Michel Wedel
    2003 “Global and local covert visual attention: Evidence from a Bayesian hidden Markov model.” Psychometrika68(4): 519–541. 10.1007/BF02295608
    https://doi.org/10.1007/BF02295608 [Google Scholar]
  19. Lohse, Gerald L.
    1997 “Consumer eye movement patterns on yellow pages advertising.” Journal of Advertising26(1): 61–73. doi:  10.1080/00913367.1997.10673518
    https://doi.org/10.1080/00913367.1997.10673518 [Google Scholar]
  20. Pieters, Rik, and Michel Wedel
    2004 “Attention capture and transfer in advertising: Brand, pictorial, and text-size effects.” Journal of Marketing68(2): 36–50. 10.1509/jmkg.
    https://doi.org/10.1509/jmkg. [Google Scholar]
  21. 2007 “Goal control of attention to advertising: The Yarbus implication.” Journal of Consumer Research34(2): 224–233. 10.1086/519150
    https://doi.org/10.1086/519150 [Google Scholar]
  22. R: The R Project for Statistical Computing
    R: The R Project for Statistical Computing. “The R Foundation.” https://www.R-project.org/
  23. Radach, Ralph, Stefanie Lemmer, Christian Vorstius, Dieter Heller, and Karina Radach
    2003 “Eye movements in the processing of print advertisements.” InThe Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, ed. byJukka Hyönä, Ralph Radach, and Heiner Deubel, 609–623. Amsterdam: Elsevier. 10.1016/B978‑044451020‑4/50032‑3
    https://doi.org/10.1016/B978-044451020-4/50032-3 [Google Scholar]
  24. Raskin, Victor
    1985Semantics Mechanisms of Humor. Dordrecht: Reidel.
    [Google Scholar]
  25. Rayner, Keith, Caren M. Rotello, Andrew J. Stewart, Jessica Keir, and Susan A. Duffy
    2001 “Integrating text and pictorial information: Eye movements when looking at print advertisements.” Journal of Experimental Psychology: Applied7(3): 219–226.
    [Google Scholar]
  26. Samson, Andrea C., and Oswald Huber
    2007 “The interaction of cartoonist’s gender and formal features of cartoons.” HUMOR: International Journal of Humor Research20(1): 1–25. doi:  10.1515/HUMOR.2007.001
    https://doi.org/10.1515/HUMOR.2007.001 [Google Scholar]
  27. Simarro Vázquez, María
    2016 “Mecanismos de humor verbal en Twitter [Mechanisms of verbal humor on Twitter].” Caracteres: estudios culturales y críticos de la esfera digital5(2): 32–57.
    [Google Scholar]
  28. Tsakona, Villy
    2009 “Language and image interaction in cartoons: Towards a multimodal theory of humor.” Journal of Pragmatics41: 1171–1188. 10.1016/j.pragma.2008.12.003
    https://doi.org/10.1016/j.pragma.2008.12.003 [Google Scholar]
  29. Yus, Francisco
    2016Humour and Relevance. Amsterdam: John Benjamins. 10.1075/thr.4
    https://doi.org/10.1075/thr.4 [Google Scholar]
  30. 2019a “Multimodality in memes: A cyberpragmatic approach.” InAnalyzing Digital Discourse. New Insights and Future Directions, ed. byPatricia Bou-Franch, and Pilar Garcés-Conejos Blitvich, 105–131. Cham: Palgrave Macmillan. 10.1007/978‑3‑319‑92663‑6_4
    https://doi.org/10.1007/978-3-319-92663-6_4 [Google Scholar]
  31. 2019b Incongruity-Resolution Patterns in Meme Communication. Paper presented atthe International Conference on Verbal Humor, University of Alicante, Spain, 23–25 October.
    [Google Scholar]
  32. Zanettin, Federico
    2010 “Humour in translated cartoons and comics.” InTranslation, Humour and the Media, ed. byDelia Chiaro, 34–52. London: Continuum.
    [Google Scholar]

Data & Media loading...

  • Article Type: Research Article
Keyword(s): caption; eye tracking; image; multimodality; processing order; Twitter
This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error