Multimodal digital storytelling: Integrating information, emotion and social cognition

Digital stories are a very recent multimedia practice by which ordinary people construct short narratives on personal affairs combining voice, images and sometimes music. This paper contributes to the description of this new emergent genre from both a multimodal and a cognitive point of view, by exploring how diverse semiotic channels in digital storytelling provide different kinds of information (factual, emotional, cultural, etc.) which are finally integrated to construct the global meaning of the narrative. For this purpose, we combine Kress and Van Leeuwen’s (1996) scholarly work related to multimodal representation, with the use of some notions of the Mental Spaces and Conceptual Integration theory (Dancygier, 2008; Fauconnier & Turner, 2002). The results of this study are of interest to those concerned with the representational and communicational modes of semiotic resources in storytelling.