1887
  1. Home
  2. e-Journals
  3. Translation Spaces
  4. Volume 11, Issue 1
  5. Article
Volume 11, Issue 1
  • ISSN 2211-3711
  • E-ISSN: 2211-372X
USD
Buy:$35.00 + Taxes

Abstract

Abstract

Empirical research on the cognitive processing of () products has been thriving over the past decade. While the use of cutting-edge experimental tools such as eye trackers has drawn increasing scholarly attention and accelerated the progress in understanding the complex mental processes involved in the reception of multimodal AVT products, relatively less attention has been devoted to the importance of establishing a theoretical framework or cognitive model that can explain and predict the behaviours observed in empirical experiments. By reviewing numerous theories or cognitive models relevant to AVT research in explaining how different perceptual and cognitive systems operate for understanding multimodal products, this paper calls for engagement with these theoretical frameworks and models to work towards a robust model that can generate testable hypotheses for the integration and interaction of multiple sources of information involved in the processing of AVT or other multimodal products.

© John Benjamins Publishing Company
Loading

Article metrics loading...

/content/journals/10.1075/ts.21024.kru
2022-07-19
2024-09-16

  • From This Site

    /content/journals/10.1075/ts.21024.kru
    dcterms_title,dcterms_subject,pub_keyword
    -contentType:Journal -contentType:Contributor -contentType:Concept -contentType:Institution
    10
    5
Loading full text...

Full text loading...

References

  1. Allen, Richard J., Graham J. Hitch, Judit Mate, and Alan D. Baddeley
    2012 “Feature Binding and Attention in Working Memory: A Resolution of Previous Contradictory Findings.” Quarterly Journal of Experimental Psychology65 (12): 2369–83. 10.1080/17470218.2012.687384
     https://doi.org/10.1080/17470218.2012.687384 [Google Scholar]
  2. Ans, Bernard, Serge Carbonnel, and Sylviane Valdois
    1998 “A Connectionist Multiple-Trace Memory Model for Polysyllabic Word Reading.” Psychological Review105 (4): 678–723. 10.1037/0033‑295X.105.4.678‑723
     https://doi.org/10.1037/0033-295X.105.4.678-723 [Google Scholar]
  3. Atkinson, Richard. C., and Richard M. Shiffrin
    1968 “Human Memory: A Proposed System and Its Control Processes.” InThe Psychology of Learning and Motivation: Advances in Research and Theory, edited bySpence, Kenneth W., and Janet Taylor Spence, 89–195. New York: Academic Press.
     [Google Scholar]
  4. Baddeley, Alan David
    2000 “The Episodic Buffer: A New Component of Working Memory?” Trends in Cognitive Sciences4 (11): 417–23. 10.1016/S1364‑6613(00)01538‑2
     https://doi.org/10.1016/S1364-6613(00)01538-2 [Google Scholar]
  5. 2010 “Working memory.” Current biology20 (4), 136–140. 10.1016/j.cub.2009.12.014
     https://doi.org/10.1016/j.cub.2009.12.014 [Google Scholar]
  6. 2012 “Working Memory: Theories, Models, and Controversies.” Annual Review of Psychology63 (1): 1–29. 10.1146/annurev‑psych‑120710‑100422
     https://doi.org/10.1146/annurev-psych-120710-100422 [Google Scholar]
  7. Baddeley, Alan David, and Graham J. Hitch
    1974 “Working memory.” InThe Psychology of Learning and Motivation: Advances in Research and Theory, edited byBower, Gordon H., 47–89. New York: Academic Press.
     [Google Scholar]
  8. Baddeley, Alan David, Richard J. Allen, and Graham J. Hitch
    2011 “Binding in Visual Working Memory: The Role of the Episodic Buffer.” Neuropsychologia49 (6): 1393–1400. 10.1016/j.neuropsychologia.2010.12.042
     https://doi.org/10.1016/j.neuropsychologia.2010.12.042 [Google Scholar]
  9. Chung, Jing-mei
    1999 “The Effects of Using Video Texts Supported with Advance Organizers and Captions on Chinese College Students’ Listening Comprehension: An Empirical Study.” Foreign Language Annals32 (3): 295–308. 10.1111/j.1944‑9720.1999.tb01342.x
     https://doi.org/10.1111/j.1944-9720.1999.tb01342.x [Google Scholar]
  10. Danan, Martine
    2004 “Captioning and Subtitling: Undervalued Language Learning Strategies.” Meta49 (1): 67–77. 10.7202/009021ar
     https://doi.org/10.7202/009021ar [Google Scholar]
  11. de Pedro Ricoy, Raquel
    2012 “Multimodality in Translation: Steps towards Socially Useful Research.” Journal Multimodal Communication1 (2): 181–203. 10.1515/mc‑2012‑0012
     https://doi.org/10.1515/mc-2012-0012 [Google Scholar]
  12. Díaz Cintas, Jorge, and Agnieszka Szarkowska
    2020 “Introduction: Experimental Research in Audiovisual Translation–Cognition, Reception, Production.” The Journal of Specialised Translation, 331, 3–16.
     [Google Scholar]
  13. Forstmann, Birte U., Eric-Jan Wagenmakers, Tom Eichele, Scott Brown, and John T. Serences
    2011 “Reciprocal Relations between Cognitive Neuroscience and Formal Cognitive Models: Opposites Attract?” Trends in Cognitive Sciences15 (6): 272–79. 10.1016/j.tics.2011.04.002
     https://doi.org/10.1016/j.tics.2011.04.002 [Google Scholar]
  14. Gernsbacher, Morton Ann
    2015 “Video Captions Benefit Everyone.” Edited bySusan T. Fiske. Policy Insights from the Behavioral and Brain Sciences2 (1): 195–202. 10.1177/2372732215602130
     https://doi.org/10.1177/2372732215602130 [Google Scholar]
  15. Hintzman, Douglas L.
    1984 “MINERVA 2: A Simulation Model of Human Memory.” Behavior Research Methods, Instruments, & Computers16 (2): 96–101. 10.3758/BF03202365
     https://doi.org/10.3758/BF03202365 [Google Scholar]
  16. Kaindl, Klaus
    2013 “Multimodality and Translation.” InThe Routledge Handbook of Translation Studies, edited byMillán, Carmen, and Francesca Bartrina, 257–269. London: Routledge.
     [Google Scholar]
  17. Kalyuga, Slava, and John Sweller
    2014 “The Redundancy Principle in Multimedia Learning.” InThe Cambridge Handbook of Multimedia Learning, edited byMayer, Richard E., 247–262. New York: Cambridge University Press. 10.1017/CBO9781139547369.013
     https://doi.org/10.1017/CBO9781139547369.013 [Google Scholar]
  18. Ketola, Anne
    2016 “Towards a Multimodally Oriented Theory of Translation: A Cognitive Framework for the Translation of Illustrated Technical Texts.” Translation Studies9 (1): 67–81. 10.1080/14781700.2015.1086670
     https://doi.org/10.1080/14781700.2015.1086670 [Google Scholar]
  19. Kruger, Jan-Louis
    2016 “Psycholinguistics and Audiovisual Translation.” Target. International Journal of Translation Studies28 (2): 276–87. 10.1075/target.28.2.08kru
     https://doi.org/10.1075/target.28.2.08kru [Google Scholar]
  20. 2019 “Eye tracking in Audiovisual Translation Research.” InThe Routledge Handbook of Audiovisual Translation, edited byPérez-González, Luis, 350–366. London: Routledge, Taylor and Francis Group.
     [Google Scholar]
  21. 2020 “Translation, Multimodality and Cognition.” InThe Routledge Handbook of Translation and Cognition, edited byAlves, Fabio, and Arnt Lykke Jakobsen, 433–444. London, UK: Routledge. 10.4324/9781315178127‑28
     https://doi.org/10.4324/9781315178127-28 [Google Scholar]
  22. Kruger, Jan-Louis, and Faans Steyn
    2014 “Subtitles and Eye Tracking: Reading and Performance.” Reading Research Quarterly49 (1): 105–20. 10.1002/rrq.59
     https://doi.org/10.1002/rrq.59 [Google Scholar]
  23. Kruger, Jan-Louis, Natalia Wisniewska, and Sixin Liao
    2021 “Why Subtitle Speed Matters: Evidence from Word Skipping and Rereading.” Applied Psycholinguistics43 (1): 211–36. 10.1017/S0142716421000503
     https://doi.org/10.1017/S0142716421000503 [Google Scholar]
  24. Liao, Sixin, and Jan-Louis Kruger
    2022 “Cognitive Processing of Subtitles: Charting the Future by Mapping the Past.” InThe Routledge Handbook of Translation, Interpreting, and Bilingualism, edited byFerreira, Aline, and John W. Schwieter.
     [Google Scholar]
  25. Liao, Sixin, Lili Yu, Jan-Louis Kruger, and Erik D. Reichle
    2021a “The impact of Audio on the Reading of Inter- vs. Intra-lingual Subtitles: Evidence from Eye Movements.” Applied Psycholinguistics43 (1): 237–69. 10.1017/S0142716421000527
     https://doi.org/10.1017/S0142716421000527 [Google Scholar]
  26. Liao, Sixin, Lili Yu, Erik D. Reichle, and Jan-Louis Kruger
    2021b “Using Eye Movements to Study the Reading of Subtitles in Video.” Scientific Studies of Reading25(5): 417–435. 10.1080/10888438.2020.1823986
     https://doi.org/10.1080/10888438.2020.1823986 [Google Scholar]
  27. Loschky, Lester C., Adam M. Larson, Tim J. Smith, and Joseph P. Magliano
    2020 “The Scene Perception & Event Comprehension Theory (SPECT) Applied to Visual Narratives.” Topics in Cognitive Science12 (1): 311–51. 10.1111/tops.12455
     https://doi.org/10.1111/tops.12455 [Google Scholar]
  28. Lum, Jarrad A. G., Gina Conti-Ramsden, Debra Page, and Michael T. Ullman
    2012 “Working, Declarative and Procedural Memory in Specific Language Impairment.” Cortex48 (9): 1138–54. 10.1016/j.cortex.2011.06.001
     https://doi.org/10.1016/j.cortex.2011.06.001 [Google Scholar]
  29. Mayer, Richard E.
    2014 The Cambridge Handbook of Multimedia Learning. New York: Cambridge University Press. 10.1017/CBO9781139547369
     https://doi.org/10.1017/CBO9781139547369 [Google Scholar]
  30. Paivio, Allan
    1986 Mental Representations: A Dual Coding Approach. Oxford, England: Oxford University Press.
     [Google Scholar]
  31. 1991 “Dual Coding Theory: Retrospect and Current Status.” Canadian Journal of Psychology45 (3): 255–87. 10.1037/h0084295
     https://doi.org/10.1037/h0084295 [Google Scholar]
  32. 2007 Mind and Its Evolution: A Dual Coding Theoretical Approach. Mahwah, NJ: Erlbaum.
     [Google Scholar]
  33. Perego, Elisa, Fabio Del Missier, Marco Porta, and Mauro Mosconi
    2010 “The Cognitive Effectiveness of Subtitle Processing.” Media Psychology13 (3): 243–72. 10.1080/15213269.2010.502873
     https://doi.org/10.1080/15213269.2010.502873 [Google Scholar]
  34. Pérez-González, Luis
    2014 “Multimodality in Translation and Interpreting Studies: Theoretical and Methodological Perspectives.” InA Companion to Translation Studies, edited byBermann, Sandra, and Catherine Porter, 119–131. Malden, MA/Oxford, England: Wiley-Blackwell. 10.1002/9781118613504.ch9
     https://doi.org/10.1002/9781118613504.ch9 [Google Scholar]
  35. Prezenski, Sabine, André Brechmann, Susann Wolff, and Nele Russwinkel
    2017 “A Cognitive Modeling Approach to Strategy Formation in Dynamic Decision Making.” Frontiers in Psychology81 (August): 1–18. 10.3389/fpsyg.2017.01335
     https://doi.org/10.3389/fpsyg.2017.01335 [Google Scholar]
  36. Reichle, Erik D.
    2021 Computational Models of Reading: A Handbook. Oxford: Oxford University Press. 10.1093/oso/9780195370669.001.0001
     https://doi.org/10.1093/oso/9780195370669.001.0001 [Google Scholar]
  37. Reichle, Erik D., Alexander Pollatsek, and Keith Rayner
    2012 “Using E-Z Reader to Simulate Eye Movements in Nonreading Tasks: A Unified Framework for Understanding the Eye–Mind Link.” Psychological Review119 (1): 155–85. 10.1037/a0026473
     https://doi.org/10.1037/a0026473 [Google Scholar]
  38. Reichle, Erik D., Lili Yu, Sixin Liao, and Jan-Louis Kruger
    2021 “Using Simulations to Understand the Reading of Rapidly Displayed Subtitles.” Proceedings of the 43rd Annual Conference of the Cognitive Science Society.
     [Google Scholar]
  39. Schnotz, Wolfgang
    2005 “Integrated Model of Text and Picture Comprehension.” InThe Cambridge Handbook of Multimedia Learning, edited byMayer, Richard E., 49–70. New York: Cambridge University Press. 10.1017/CBO9780511816819.005
     https://doi.org/10.1017/CBO9780511816819.005 [Google Scholar]
  40. Schnotz, Wolfgang, and Maria Bannert
    2003 “Construction and Interference in Learning from Multiple Representation.” Learning and Instruction13 (2): 141–56. 10.1016/S0959‑4752(02)00017‑8
     https://doi.org/10.1016/S0959-4752(02)00017-8 [Google Scholar]
  41. Schnotz, Wolfgang
    2014 “Integrated model of text and picture comprehension.” InThe Cambridge Handbook of Multimedia Learning (2nd ed), edited byMayer, Richard E., 72–103. 10.1017/CBO9781139547369.006
     https://doi.org/10.1017/CBO9781139547369.006 [Google Scholar]
  42. Schnotz, Wolfgang, and Inga Wagner
    2018 “Construction and Elaboration of Mental Models through Strategic Conjoint Processing of Text and Pictures.” Journal of Educational Psychology110 (6): 850–63. 10.1037/edu0000246
     https://doi.org/10.1037/edu0000246 [Google Scholar]
  43. Seeber, Kilian G.
    2017 Multimodal processing in simultaneous interpreting. InThe Handbook of Translation and Cognition, edited bySchwieter, John and Aline Ferreira, 461–475. Joh Wiley & Sons. 10.1002/9781119241485.ch25
     https://doi.org/10.1002/9781119241485.ch25 [Google Scholar]
  44. Squire, Larry R., and Stuart M. Zola
    1996 “Structure and Function of Declarative and Nondeclarative Memory Systems.” Proceedings of the National Academy of Sciences93 (24): 13515–22. 10.1073/pnas.93.24.13515
     https://doi.org/10.1073/pnas.93.24.13515 [Google Scholar]
  45. Szarkowska, Agnieszka, and Piotr Wasylczyk
    2018 “Five Things You Wanted to Know about Audiovisual Translation Research, but Were Afraid to Ask.” Journal of Audiovisual Translation1 (1): 8–25. 10.47476/jat.v1i1.41
     https://doi.org/10.47476/jat.v1i1.41 [Google Scholar]
  46. Szarkowska, Agnieszka, Breno Silva, and David Orrego-Carmona
    2021 “Effects of Subtitle Speed on Proportional Reading Time.” Translation, Cognition & Behavior. 10.1075/tcb.00057.sza
     https://doi.org/10.1075/tcb.00057.sza [Google Scholar]
  47. Taylor, Christopher
    2016 “The Multimodal Approach in Audiovisual Translation.” Target. International Journal of Translation Studies28 (2): 222–36. 10.1075/target.28.2.04tay
     https://doi.org/10.1075/target.28.2.04tay [Google Scholar]
  48. Tuominen, Tiina, Catalina Jiménez Hurtado, and Anne Ketola
    2018 “Why Methods Matter: Approaching Multimodality in Translation Research.” Linguistica Antverpiensia, New Series: Themes in Translation Studies, 171, 1–21.
     [Google Scholar]
  49. Ullman, Michael T.
    2001 “A Neurocognitive Perspective on Language: The Declarative/Procedural Model.” Nature Reviews Neuroscience2 (10): 717–26. 10.1038/35094573
     https://doi.org/10.1038/35094573 [Google Scholar]
  50. 2004 “Contributions of Memory Circuits to Language: The Declarative/Procedural Model.” Cognition92 (1–2): 231–70. 10.1016/j.cognition.2003.10.008
     https://doi.org/10.1016/j.cognition.2003.10.008 [Google Scholar]
  51. Zhong, Ling, Brianne P. Noud, Harriet Pruitt, Steven C. Marcrum, and Erin M. Picou
    2021 “Effects of Text Supplementation on Speech Intelligibility for Listeners with Normal and Impaired Hearing: A Systematic Review with Implications for Telecommunication.” International Journal of Audiology, June, 1–11. 10.1080/14992027.2021.1937346
     https://doi.org/10.1080/14992027.2021.1937346 [Google Scholar]
/content/journals/10.1075/ts.21024.kru
Loading
Establishing a theoretical framework for AVT research
Translation Spaces 11, 12 (2022); https://doi.org/10.1075/ts.21024.kru
/content/journals/10.1075/ts.21024.kru
/content/journals/10.1075/ts.21024.kru
Loading

Data & Media loading...

  • Article Type: Research Article
Keyword(s): audiovisual translation (AVT); cognitive models; multimodal integrated language framework (MILF); subtitles

Most Cited

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error