Volume 22, Issue 1
  • ISSN 1384-6655
  • E-ISSN: 1569-9811
Buy:$35.00 + Taxes


Based on the Bielefeld Speech and Gesture Alignment Corpus ( Lücking et al. 2013 ), this paper presents a systematic comparison of the linguistic characteristics of unimodal (speech only) and multimodal (gesture-accompanied) forms of language use. The results suggest that each of these two modes of expression is characterized by statistical preferences for certain types of words and grammatical categories. The words that are most frequently accompanied by a manual gesture, when controlled for their total frequency, include unspecific spatial lexemes, various deictic words, and particles that express difficulty in word retrieval or formulation. Other linguistic items, including pronouns and verbs of cognition, show a strong dispreference for being gesture-accompanied. The second part of the paper shows that gestures do not occur within a fixed time window relative to the word(s) they relate to, but the preferred temporal distance varies with the type of functional relation that exists between the verbal and gestural channel.


Article metrics loading...

Loading full text...

Full text loading...


  1. Adolphs, S. , & Carter, R.
    (2013) Spoken corpus linguistics: From monomodal to multimodal. New York, NY: Routledge.
    [Google Scholar]
  2. Alahverdzhieva, K.
    (2013) Alignment of speech and co-speech gesture in a constraint-based grammar. (Unpublished doctoral dissertation), University of Edinburgh, Edinburgh.
    [Google Scholar]
  3. Altman, D. G. , & Bland, J. M.
    (2011) How to obtain the P value from a confidence interval. British Medical Journal, 343(2304). Retrieved fromwww.bmj.com/content/343/bmj.d2304 (last accessedFebruary 2017).
    [Google Scholar]
  4. Bergmann, K. , Aksu, V. , & Kopp, S.
    (2011) The relation of speech and gestures: Temporal synchrony follows semantic synchrony. Paper presented at the2nd Workshop on Gesture and Speech in Interaction, Bielefeld, Germany.
    [Google Scholar]
  5. Bergmann, K. , & Kopp, S.
    (2009) GNetIc – Using Bayesian decision networks for iconic gesture generation. In Z. Ruttkay , M. Kipp , A. Nijholt & H. H. Vilhjálmsson (Eds.), Proceedings of the 9th International Conference on Virtual Agents (pp.76–89). Amsterdam: Springer.
    [Google Scholar]
  6. Bergmann, K. , Kopp, S. , & Eyssel, F.
    (2010) Individualized gesturing outperforms average gesturing – evaluating gesture production in virtual humans. In J. Allbeck , N. Badler , T. Bickmore , C. Pelachaud & A. Safonova (Eds.), Proceedings of the 10th Conference on Intelligent Virtual Agents (pp.104–117). Philadelphia, PA: Springer. doi: 10.1007/978‑3‑642‑15892‑6_11
    https://doi.org/10.1007/978-3-642-15892-6_11 [Google Scholar]
  7. Cienki, A.
    (2012) Usage events of spoken language and the symbolic units we (may) abstract from them. In K. Kosecki & J. Badio (Eds.), Cognitive Processes in Language (pp.149–158). Frankfurt am Main: Peter Lang.
    [Google Scholar]
  8. Damerau, F. J.
    (1993) Generating and evaluating domain-oriented multi-word terms from texts. Information Processing & Management, 29(4), 433–447. doi: 10.1016/0306‑4573(93)90039‑G
    https://doi.org/10.1016/0306-4573(93)90039-G [Google Scholar]
  9. Diemer, S. , Brunner, M. L. , & Schmidt, S.
    (2016) Compiling computer-mediated spoken language corpora. International Journal of Corpus Linguistics, 21(3), 348–371. doi: 10.1075/ijcl.21.3.03die
    https://doi.org/10.1075/ijcl.21.3.03die [Google Scholar]
  10. Enfield, N. J.
    (2004) On linear segmentation and combinatorics in co-speech gesture: A symmetry-dominance construction in Lao fish trap descriptions. Semiotica, 149(1–4), 57–124.
    [Google Scholar]
  11. Fricke, E.
    (2007) Origo, Geste und Raum: Lokaldeixis im Deutschen. Berlin: Walter de Gruyter. doi: 10.1515/9783110897746
    https://doi.org/10.1515/9783110897746 [Google Scholar]
  12. (2009) Multimodal attribution: How gestures are syntactically integrated into spoken language. Paper presented at thefirst Gesture and Speech in Interaction conference (GeSpIn), Poznań, Poland.
    [Google Scholar]
  13. (2012) Grammatik multimodal: Wie Wörter und Gesten zusammenwirken. Berlin: Walter de Gruyter. doi: 10.1515/9783110218893
    https://doi.org/10.1515/9783110218893 [Google Scholar]
  14. Hadar, U. , & Krauss, R. K.
    (1999) Iconic gestures: The grammatical categories of lexical affiliates. Journal of Neurolinguistics, 12(1), 1–12. doi: 10.1016/S0911‑6044(99)00001‑9
    https://doi.org/10.1016/S0911-6044(99)00001-9 [Google Scholar]
  15. Harrison, S.
    (2008) The expression of negation through grammar and gesture. In J. Zlatev , M. Andrén , M. J. Falck & C. Lundmark (Eds.), Studies in Language and Cognition (pp.405–419). Cambridge: Cambridge Scholars Press.
    [Google Scholar]
  16. (2009) Grammar, gesture, and cognition: The case of negation in English (Unpublished doctoral dissertation). University of Bordeaux, Bordeaux, France.
    [Google Scholar]
  17. Hinrichs, E. , Hinrichs, M. , & Zastrow, T.
    (2010) WebLicht: Web-based LRT services for German. In S. Kübler (Ed.), Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics 2010 System Demonstrations (pp.25–29). Uppsala, Sweden.
    [Google Scholar]
  18. Hole, D. , & Klumpp, G.
    (2000) Definite type and indefinite token: The article son in colloquial German. Linguistische Berichte, 182(1), 231–244.
    [Google Scholar]
  19. Kendon, A.
    (1995) Gestures as illocutionary and discourse structure markers in Southern Italian conversation. Journal of Pragmatics, 23(3), 247–279. doi: 10.1016/0378‑2166(94)00037‑F
    https://doi.org/10.1016/0378-2166(94)00037-F [Google Scholar]
  20. (2004) Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. doi: 10.1017/CBO9780511807572
    https://doi.org/10.1017/CBO9780511807572 [Google Scholar]
  21. Knight, D.
    (2011) Multimodality and Active Listenership: A Corpus Approach. London: Continuum Books.
    [Google Scholar]
  22. Knight, D. , Evans, D. , Carter, R. , & Adolphs, S.
    (2009) HeadTalk, HandTalk and the corpus: Towards a framework for multi-modal, multi-media corpus development. Corpora, 4(1), 1–32. doi: 10.3366/E1749503209000203
    https://doi.org/10.3366/E1749503209000203 [Google Scholar]
  23. Kisler, T. , Schiel, F. , & Sloetjes, H.
    (2012) Signal processing via web services: The use case WebMAUS. In E. Hinrichs , H. Neuroth & P. Wittenburg (Eds.), Proceedings of the Service-oriented Architectures (SOAs) workshop at the Digital Humanities Conference 2012 (pp.30–34). Hamburg, Germany.
    [Google Scholar]
  24. Kok, K. I.
    (2016) The grammatical potential of co-speech gesture: A Functional Discourse Grammar perspective. Functions of Language, 23(2), 149–178. doi: 10.1075/fol.23.2.01kok
    https://doi.org/10.1075/fol.23.2.01kok [Google Scholar]
  25. Kok, K. I. , Bergmann, K. , Cienki, A. , & Kopp, S.
    (2016) Mapping out the multifunctionality of speakers’ gestures. Gesture, 15(1), 37–59. doi: 10.1075/gest.15.1.02kok
    https://doi.org/10.1075/gest.15.1.02kok [Google Scholar]
  26. Kok, K. I. , & Cienki, A.
    (2016) Cognitive Grammar and gesture: Points of convergence, advances and challenges. Cognitive Linguistics, 27(1), 67–100. doi: 10.1515/cog‑2015‑0087
    https://doi.org/10.1515/cog-2015-0087 [Google Scholar]
  27. Kopp, S. , Bergmann, K. , & Wachsmuth, I.
    (2008) Multimodal communication from multimodal thinking – towards an integrated model of speech and gesture production. International Journal of Semantic Computing, 2(1), 115–136. doi: 10.1142/S1793351X08000361
    https://doi.org/10.1142/S1793351X08000361 [Google Scholar]
  28. Krauss, R. M. , Chen, Y. , & Gottesman, R. F.
    (2000) Lexical gestures and lexical access: A process model. In D. McNeill (Ed.), Language and Gesture (pp.261–283). Cambridge: Cambridge University Press. doi: 10.1017/CBO9780511620850.017
    https://doi.org/10.1017/CBO9780511620850.017 [Google Scholar]
  29. Ladewig, S. H.
    (2012) Syntactic and semantic integration of gestures into speech: Structural, cognitive, and conceptual aspects (Unpublished doctoral dissertation). European University Viadrina, Frankfurt (Oder).
    [Google Scholar]
  30. Langacker, R. W.
    (1987) Foundations of Cognitive Grammar, Volume I: Theoretical Prerequisites. Stanford: Stanford University Press.
    [Google Scholar]
  31. Leonard, T. , & Cummins, F.
    (2011) The temporal relation between beat gestures and speech. Language and Cognitive Processes, 26(10), 1457–1471. doi: 10.1080/01690965.2010.500218
    https://doi.org/10.1080/01690965.2010.500218 [Google Scholar]
  32. Levy, E. T. , & McNeill, D.
    (1992) Speech, gesture, and discourse. Discourse Processes, 15(3), 277–301. doi: 10.1080/01638539209544813
    https://doi.org/10.1080/01638539209544813 [Google Scholar]
  33. Loehr, D. P.
    (2004) Gesture and intonation (Unpublished doctoral dissertation). Georgetown University, Washington D.C.
    [Google Scholar]
  34. Lücking, A. , Bergmann, K. , Hahn, F. , Kopp, S. , & Rieser, H.
    (2010) The Bielefeld speech and gesture alignment corpus (SaGA). In M. Kipp , J. C. Martin , P. Paggio & D. Heylen (Eds.), Proceedings of the 7th International Conference for Language Resources and Evaluation (pp.92–98). Valetta, Malta.
    [Google Scholar]
  35. (2013) Data-based analysis of speech and gesture: The Bielefeld Speech and Gesture Alignment Corpus (SaGA) and its applications. Journal on Multimodal User Interfaces, 7(1–2), 5–18. doi: 10.1007/s12193‑012‑0106‑8
    https://doi.org/10.1007/s12193-012-0106-8 [Google Scholar]
  36. McCarthy, M. , & Carter, R.
    (1996) Ten criteria for a spoken grammar. In E. Hinkel & S. Fotos (Eds.), New Perspectives in Grammar Teaching in Second Language Classrooms (pp.51–75). Mahwah, NJ: Lawrence Erlbaum.
    [Google Scholar]
  37. McNeill, D.
    (1992) Hand and Mind: What Gestures Reveal about Thought. Chicago, IL: University of Chicago Press.
    [Google Scholar]
  38. (2000) Catchments and contexts: Non-modular factors in speech and gesture production. In D. McNeill (Ed.), Language and Gesture (pp.312–328). Cambridge: Cambridge University Press. doi: 10.1017/CBO9780511620850.019
    https://doi.org/10.1017/CBO9780511620850.019 [Google Scholar]
  39. Morrel-Samuels, P. , & Krauss, R. M.
    (1992) Word familiarity predicts temporal asynchrony of hand gestures and speech. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(3), 615–622.
    [Google Scholar]
  40. Müller, C. , Ladewig, S. H. , & Bressem, J.
    (2013) Gestures and speech from a linguistic perspective: A new field and its history. In C. Müller , A. Cienki , E. Fricke , S. Ladewig , D. McNeill & J. Bressem (Eds.), Body-Language-Communication: An International Handbook on Multimodality in Human Interaction (Vol.1, pp.55–81). Berlin/Boston: De Gruyter Mouton.
    [Google Scholar]
  41. Schiller, A. , Teufel, S. , & Thielen, C.
    (1995) Guidelines für das Tagging deutscher Textcorpora mit STTS. Unpublished report. University of Stuttgart.
    [Google Scholar]
  42. Schoonjans, S.
    (2014a) Is gesture subject to grammaticalization?Papers of the Linguistic Society of Belgium, 8. Retrieved fromuahost.uantwerpen.be/linguist/SBKL/Vol8.htm (last accessedMay 2016).
    [Google Scholar]
  43. (2014b) Modalpartikeln als multimodale Konstruktionen. Eine korpusbasierte Kookkurrenzanalyse von Modalpartikeln und Gestik im Deutschen (Unpublished doctoral dissertation). University of Leuven, Leuven, Belgium.
    [Google Scholar]
  44. Streeck, J.
    (1993) Gesture as communication: Its coordination with gaze and speech. Communications Monographs, 60(4), 275–299. doi: 10.1080/03637759309376314
    https://doi.org/10.1080/03637759309376314 [Google Scholar]
  45. (2002) Grammars, words, and embodied meanings: On the uses and evolution of so and like. Journal of Communication, 52(3), 581–596. doi: 10.1111/j.1460‑2466.2002.tb02563.x
    https://doi.org/10.1111/j.1460-2466.2002.tb02563.x [Google Scholar]
  46. Thurmair, M.
    (1989) Modalpartikeln und ihre Kombinationen. Tübingen: Niemeyer. doi: 10.1515/9783111354569
    https://doi.org/10.1515/9783111354569 [Google Scholar]
  47. Turner, M. , & Steen, F.
    (2012) Multimodal Construction Grammar. In M. Borkent , B. Dancygier & J. A. J. Hinnell (Eds.), Language and the Creative Mind (pp.255–274). Stanford, CA: CSLI Publications.
    [Google Scholar]
  48. van Son, R. , Wesseling, W. , Sanders, E. , & van den Heuvel, H.
    (2008) The IFADV corpus: A free dialog video corpus. Proceedings of the sixth international conference on Language Resources and Evaluation (LREC) (pp.501–508). Marrakech: European Language Resources Association (ELRA).
    [Google Scholar]
  49. Zima, E.
    (2014) English multimodal motion constructions. A construction grammar perspective. Papers of the Linguistic Society of Belgium, 8. Retrieved fromuahost.uantwerpen.be/linguist/SBKL/sbkl2013/Zim2013.pdf (last accessedMay 2016).
    [Google Scholar]

Data & Media loading...

  • Article Type: Research Article
Keyword(s): distributional analysis; gesture; multimodal corpus; relative frequency ratio
This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error