Volume 4, Issue 2
  • ISSN 2542-5277
  • E-ISSN: 2542-5285
Buy:$35.00 + Taxes



The use of captions has grown in recent years in both traditional and new media, particularly in terms of the diversity of style, content, and function. Impact captions have emerged as a popular form of captions for hearing viewers and contain rich multimodal information which is employed to capture viewer attention and enhance engagement, particularly in situations where there is competition for viewer attention. Drawing upon relevance theory, we argue how impact captions could effectively attract and hold visual attention owing to their balance between processing effort and contextual effects. This exploratory study employs a dual-task paradigm and uses authentic materials and viewing situations to further examine the ability of multimodal impact captions to attract and retain overt visual attention amongst a small sample of TV viewers. Our results provide novel insight into the apparent highly individualised efficacy of impact captions, where we identify several variables of interest in participants’ viewing behaviours. We conclude with a discussion of the study’s contributions, limitations, and an outline for future work.


Article metrics loading...

Loading full text...

Full text loading...


  1. Baumgarten, Nicole
    2008 “Yeah, That’s It! Verbal Reference to Visual Information in Film Texts and Film Translations.” Meta: Translators’ Journal53 (1): 6–25.   10.7202/017971ar
    https://doi.org/10.7202/017971ar [Google Scholar]
  2. Bogucki, Łukasz
    2004 “The Constraint of Relevance in Subtitling.” The Journal of Specialised Translation1: 71–88.
    [Google Scholar]
  3. Caffrey, Colm
    2008 “Using Pupilometric, Fixation-Based and Subjective Meastures to Measure of the Processing Effect Experienced when Viewing Subtitled TV Anime with Pop-Up Gloss”. InLooking at Eyes: Eye-tracking Studies of Reading and Translation Processing. Edited bySusanne Göpferich, Arnt Lykke Jakobsen, and Inger Mees, 125–144. Copenhagen: Samfundslitteratur.
    [Google Scholar]
  4. 2009 “Relevant Abuse? Investigating the Effects of an Abusive Subtitling Procedure on the Perception of TV Anime using Eye Tracker and Questionnaire.” PhD thesis. Dublin City University.
    [Google Scholar]
  5. Chung, Marvin and Nicholas Turk-Browne
    2007 “Interactions between Attention and Memory.” Current Opinion in Neurobiology17 (2): 177–184.   10.1016/j.conb.2007.03.005
    https://doi.org/10.1016/j.conb.2007.03.005 [Google Scholar]
  6. Díaz Cintas, Jorge & Remael, Aline
    2021Subtitling: Concepts and Practices. London, UK: Routledge.   10.4324/9781315674278
    https://doi.org/10.4324/9781315674278 [Google Scholar]
  7. Doherty, Stephen
    2016 “The Impact of Translation Technologies on the Process and Product of Translation”. International Journal of Communication (10): 947–969.
    [Google Scholar]
  8. 2020 “Multisensory Integration in Audiovisual Translation”. InMultilingual Mediated Communication and Cognition. Edited byRicardo Muñoz Martín and Sandra Halverson, 155–170. London: Routledge. 10.4324/9780429323867‑7
    https://doi.org/10.4324/9780429323867-7 [Google Scholar]
  9. Ekman, Papul, Wallace V. Friesen, and Joseph C. Hager
    2002The Facial Action Coding System CD-ROM. Salt Lake City: Research Nexus.
    [Google Scholar]
  10. Fritz, Jonathan, Mounya Elhilali, Stephen David, and Shihab Shamma
    2007 “Auditory Attention: Focusing the Searchlight on Sound.” Current Opinion in Neurobiology17 (4): 437–455. 10.1016/j.conb.2007.07.011
    https://doi.org/10.1016/j.conb.2007.07.011 [Google Scholar]
  11. Gambier, Yves
    2018 “Translation Studies, Audiovisual Translation and Reception.” InReception Studies and Audiovisual Trannslation. Edited byElena Di Giovanni and Yves Gambier, 43–68. Ansterdam: John Benjamins.   10.1075/btl.141.04gam
    https://doi.org/10.1075/btl.141.04gam [Google Scholar]
  12. Gibbs, Raymond & Markus Tendahl
    2006 “Cognitive Effort and Effects in Metaphor Comprehension: Relevance Theory and Psycholinguistics”. Mind & Language21 (3): 379–403.   10.1111/j.1468‑0017.2006.00283.x
    https://doi.org/10.1111/j.1468-0017.2006.00283.x [Google Scholar]
  13. Karamitroglou, Fotios
    1998A Proposed Set of Subtitling Standards in Europe. [Online] Available at: www.translationjournal.net/journal/04stndrd.htm
    [Google Scholar]
  14. 2000Towards a Methodology for the Investigation of Norms in Audiovisual Translation. Amsterdam: Rodopi. 10.1163/9789004485891
    https://doi.org/10.1163/9789004485891 [Google Scholar]
  15. Kennedy, Daniel, Brian D’Onofrio, Patrick Quinn, Sven Bölte, Paul Lichtenstein & Terje Falck-Ytter
    2017 “Genetic Influence on Eye Movements to Complex Scenes at Short Timescales”. Current Biology27 (22): 3554–3560.   10.1016/j.cub.2017.10.007
    https://doi.org/10.1016/j.cub.2017.10.007 [Google Scholar]
  16. Knoblauch, Kenneth, Aries Arditi, and Janet Szlyk
    1991 “Effects of Chromatic and Luminance Contrast on Reading”. Journal of the Optical Society of America8 (2): 428–439. 10.1364/JOSAA.8.000428
    https://doi.org/10.1364/JOSAA.8.000428 [Google Scholar]
  17. Koga, Tadamichi
    2013 “Research Report: An Attempt of Quantitative Analysis on Directing Elements in Information and Variety Programs: From an Analytical Survey on Primetime Television Programs.” The NHK Monthly Report on Broadcast Research63 (5): 68–81.
    [Google Scholar]
  18. Kovačič, Irena
    1994Relevance as a Factor in Subtitling Reductions. Amsterdam: John Benjamins. 10.1075/btl.5.35kov
    https://doi.org/10.1075/btl.5.35kov [Google Scholar]
  19. 1995 “Reception of Subtitles. The Non-existent Ideal Viewer”, Translatio (Nouvelles de la FIT/FIT Newsletter)14 (3–4), 376–383. Audiovisual Communication and Language Transfer, International Forum, Strasbourg. 22–24June 1995 Forum, Strasbourg22–24.
    [Google Scholar]
  20. Kress, Gunter, and Theo Van Leeuwen
    2001Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Edward Arnold.
    [Google Scholar]
  21. Kruger, Jan-Louis & Stephen Doherty
    2016 “Measuring Cognitive Load in the Presence of Educational Video: Towards a Multimodal Methodology”. The Australasian Journal of Educational Technology (32): 19–31. 10.14742/ajet.3084
    https://doi.org/10.14742/ajet.3084 [Google Scholar]
  22. Lang, Annie
    1990 “Involuntary Attention and Physiological Arousal Evoked by Structural Features and Emotional Content in TV Commercials”. Communication Research17 (3): 275–299. 10.1177/009365090017003001
    https://doi.org/10.1177/009365090017003001 [Google Scholar]
  23. Legge, Gordon
    2006Psychophysics of Reading. Mahwah, NJ: Lawrence Erlbaum Associates.
    [Google Scholar]
  24. Mangirón Hevia, Carmen
    2013 “Subtitling in Game Localisation: A Descriptive Study.” Perspectives: Studies in Translatology, 21(1): 42–56. 10.1080/0907676X.2012.722653
    https://doi.org/10.1080/0907676X.2012.722653 [Google Scholar]
  25. Orero Clavero, Pilar, Stephen Doherty, Jan-Louis Kruger, Anna Matamala Ripoll, Jan Pedersen, Elisa Perego, Pablo Romero Fresco, Sara Rovira Esteva, Olga Soler Vilageliu & Agnieszka Szarkowska
    2018 “Conducting Experimental Research in Audiovisual Translation (AVT): A Position Paper”. The Journal of Specialised Translation (30): 105–126.
    [Google Scholar]
  26. O’Hagan, Minako
    2010 “Japanese TV Entertainment: Framing Humour with Open Caption Telop”. InTranslation, Humour and the Media. Edited byDalia Chiaro, 70–88. London: Bloomsbury.
    [Google Scholar]
  27. O’Hagan, Minako & Ryoko Sasamoto
    2016 “Crazy Japanese Subtitles? Shedding Light on the Impact of Impact Captions with a Focus on Research Methodology”. InEyetracking and Applied Linguistics. Edited bySilvia Hansen-Schirra and Sambor Grucza, 31–58. Berlin: Language Science Press.
    [Google Scholar]
  28. Paas, Fred, Juhani Tuovinen, Huib Tabbers and Pascal Van Gerven
    2003 Cognitive Load Measurement as a Means to Advance Cognitive Load Theory. Educational Psychologist – EDUC PSYCHOL. 38. 63–71. 10.1207/S15326985EP3801_8
    https://doi.org/10.1207/S15326985EP3801_8 [Google Scholar]
  29. Park, Joseph
    2008 “Ideological Functions of Subtitles in Multimodal Media Texts: Impact Captioning on Korean Television.” Paper presented at the4th International Conference on Multimodality: From Print to Interactive Digital Media: Technology, Multimodal Representation and Knowledge, Singapore Management University, Singapore, July 30–August 1.
    [Google Scholar]
  30. Pashler, Harold
    1994 “Dual-task Interference in Simple Tasks: Data and Theory.” Psychological Bulletin116 (2): 220–244.   10.1037/0033‑2909.116.2.220
    https://doi.org/10.1037/0033-2909.116.2.220 [Google Scholar]
  31. Pérez González, Luis
    2007 “Intervention in New Amateur Subtitling Cultures: A Multimodal Account.” Linguistica Antverpiensia (6): 67–80.
    [Google Scholar]
  32. Quittner, Alexandra, Linda Smith, Mary Osberger, Teresa Mitchell, and Donald Katz
    1994 “The Impact of Audition on the Development of Visual Attention.” Psychological Science5 (6): 347–353. 10.1111/j.1467‑9280.1994.tb00284.x
    https://doi.org/10.1111/j.1467-9280.1994.tb00284.x [Google Scholar]
  33. Sasamoto, Ryoko
    2014 “Impact Caption as a Highlighting Device: Attempts at Viewer Manipulation on TV”. Discourse, Context, and Media6: 1–10.   10.1016/j.dcm.2014.03.003
    https://doi.org/10.1016/j.dcm.2014.03.003 [Google Scholar]
  34. Sasamoto, Ryoko & Stephen Doherty
    2016 “Towards the Optimal Use of Impact Captions on TV Programmes”. InConflict and Communication: A Changing Asia in a Globalising World. Edited byMinako O’Hagan and Qi Zhang, 210–247. Hauppauge, NY: Nova Science Publishers.
    [Google Scholar]
  35. Sasamoto, Ryoko & Minako O’Hagan
    2020 “Relevance, Style and Multimodality: Typographical Features as Stylistic Devices”. InRelevance Theory, Figuration, and Continuity in Pragmatics. Edited byA. Piskorska, 194–228. Amsterdam: John Benjamins. 10.1075/ftl.8.07sas
    https://doi.org/10.1075/ftl.8.07sas [Google Scholar]
  36. Sasamoto, Ryoko, Minako O’Hagan & Stephen Doherty
    2017 “Telop, Affect, and Media Design: A Multimodal Analysis of Japanese TV Programs”. Television & New Media, 18 (5): 427–440. 10.1177/1527476416677099
    https://doi.org/10.1177/1527476416677099 [Google Scholar]
  37. Schultheis, Hoger. and Jameson, Anthony
    2004 “Assessing Cognitive Load in Adaptive Hypermedia Systems: Physiological and Behavioural Methods.” InAdaptive Hypermedia and Adaptive Web-based Systems. Edited byPaul de Bra and Wolfgang Nejdl, 18–24. Eindhoven, Netherlands: Springer Verlag. 10.1007/978‑3‑540‑27780‑4_26
    https://doi.org/10.1007/978-3-540-27780-4_26 [Google Scholar]
  38. Shiota, Eiko
    2003 “関連性理論とテロップ の理解 [Relevance Theory and Understanding of Telop].” The Graduate School Review of the English Language and Literature31: 63–91.
    [Google Scholar]
  39. Shitara, Kaoru
    2011 “The Change of Telops on NHK Variety Shows.” Bulletin of Mukogawa Women’s University, Humanities and Social Science59: 1–9.
    [Google Scholar]
  40. Sperber, Dan and Deirdre Wilson
    1995Relevance: Communication and Cognition. Oxford: Blackwells.
    [Google Scholar]
  41. 1986/1995Relevance: Communication and Cognition, 2nd ed.Oxford: Blackwells.
    [Google Scholar]
  42. Taylor, Christopher
    2003 “Multimodal Transcription in the Analysis, Translation and Subtitling of Italian Films.” The Translator9 (2): 191–205. 10.1080/13556509.2003.10799153
    https://doi.org/10.1080/13556509.2003.10799153 [Google Scholar]
  43. 2004 “Multimodal Text Analysis and Subtitling”. InPerspectives on Multimodality. Edited byEija Ventola, Cassily Charles and Martin Kaltenbacher, 153–172. Amsterdam, Netherlands: John Benjamins. 10.1075/ddcs.6.11tay
    https://doi.org/10.1075/ddcs.6.11tay [Google Scholar]
  44. Telebi Kiroku
    Telebi Kiroku 2013 Honmadekka TV Shichoritsu 2013. Telebi kirokuAvailable at: asobigokoro.info/honmadekka2013/
    [Google Scholar]
  45. Wickens, Christopher
    1991 “Processing Resources and Attention.” InMultiple-task Performance. Edited byD. Damos, 3–34. London: Taylor & Francis.
    [Google Scholar]
  46. Wilson, Deirdre & Dan Sperber
    2002 “Truthfulness and Relevance.” Mind111 (443): 583–632. 10.1093/mind/111.443.583
    https://doi.org/10.1093/mind/111.443.583 [Google Scholar]

Data & Media loading...

  • Article Type: Research Article
Keyword(s): audiovisual translation; impact captions; multimodal; relevance theory; subtitles
This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error