1887
  1. Home
  2. e-Journals
  3. International Journal of Corpus Linguistics
  4. Volume 22, Issue 1
  5. Article
Volume 22, Issue 1
  • ISSN 1384-6655
  • E-ISSN: 1569-9811
GBP
Buy:£15.00 + Taxes

Abstract

Based on the Bielefeld Speech and Gesture Alignment Corpus ( Lücking et al. 2013 ), this paper presents a systematic comparison of the linguistic characteristics of unimodal (speech only) and multimodal (gesture-accompanied) forms of language use. The results suggest that each of these two modes of expression is characterized by statistical preferences for certain types of words and grammatical categories. The words that are most frequently accompanied by a manual gesture, when controlled for their total frequency, include unspecific spatial lexemes, various deictic words, and particles that express difficulty in word retrieval or formulation. Other linguistic items, including pronouns and verbs of cognition, show a strong dispreference for being gesture-accompanied. The second part of the paper shows that gestures do not occur within a fixed time window relative to the word(s) they relate to, but the preferred temporal distance varies with the type of functional relation that exists between the verbal and gestural channel.

© John Benjamins Publishing Company
Loading

Article metrics loading...

/content/journals/10.1075/ijcl.22.1.01kok
2017-07-21
2024-04-19

  • From This Site

    /content/journals/10.1075/ijcl.22.1.01kok
    dcterms_title,dcterms_subject,pub_keyword
    -contentType:Journal -contentType:Contributor -contentType:Concept -contentType:Institution
    10
    5
Loading full text...

Full text loading...

References

  1. Adolphs, S. , & Carter, R.
    (2013) Spoken corpus linguistics: From monomodal to multimodal. New York, NY: Routledge.
     [Google Scholar]
  2. Alahverdzhieva, K.
    (2013) Alignment of speech and co-speech gesture in a constraint-based grammar. (Unpublished doctoral dissertation), University of Edinburgh, Edinburgh.
     [Google Scholar]
  3. Altman, D. G. , & Bland, J. M.
    (2011) How to obtain the P value from a confidence interval. British Medical Journal, 343(2304). Retrieved fromwww.bmj.com/content/343/bmj.d2304 (last accessedFebruary 2017).
     [Google Scholar]
  4. Bergmann, K. , Aksu, V. , & Kopp, S.
    (2011) The relation of speech and gestures: Temporal synchrony follows semantic synchrony. Paper presented at the2nd Workshop on Gesture and Speech in Interaction, Bielefeld, Germany.
     [Google Scholar]
  5. Bergmann, K. , & Kopp, S.
    (2009) GNetIc – Using Bayesian decision networks for iconic gesture generation. In Z. Ruttkay , M. Kipp , A. Nijholt & H. H. Vilhjálmsson (Eds.), Proceedings of the 9th International Conference on Virtual Agents (pp.76–89). Amsterdam: Springer.
     [Google Scholar]
  6. Bergmann, K. , Kopp, S. , & Eyssel, F.
    (2010) Individualized gesturing outperforms average gesturing – evaluating gesture production in virtual humans. In J. Allbeck , N. Badler , T. Bickmore , C. Pelachaud & A. Safonova (Eds.), Proceedings of the 10th Conference on Intelligent Virtual Agents (pp.104–117). Philadelphia, PA: Springer. doi: 10.1007/978‑3‑642‑15892‑6_11
     https://doi.org/10.1007/978-3-642-15892-6_11 [Google Scholar]
  7. Cienki, A.
    (2012) Usage events of spoken language and the symbolic units we (may) abstract from them. In K. Kosecki & J. Badio (Eds.), Cognitive Processes in Language (pp.149–158). Frankfurt am Main: Peter Lang.
     [Google Scholar]
  8. Damerau, F. J.
    (1993) Generating and evaluating domain-oriented multi-word terms from texts. Information Processing & Management, 29(4), 433–447. doi: 10.1016/0306‑4573(93)90039‑G
     https://doi.org/10.1016/0306-4573(93)90039-G [Google Scholar]
  9. Diemer, S. , Brunner, M. L. , & Schmidt, S.
    (2016) Compiling computer-mediated spoken language corpora. International Journal of Corpus Linguistics, 21(3), 348–371. doi: 10.1075/ijcl.21.3.03die
     https://doi.org/10.1075/ijcl.21.3.03die [Google Scholar]
  10. Enfield, N. J.
    (2004) On linear segmentation and combinatorics in co-speech gesture: A symmetry-dominance construction in Lao fish trap descriptions. Semiotica, 149(1–4), 57–124.
     [Google Scholar]
  11. Fricke, E.
    (2007) Origo, Geste und Raum: Lokaldeixis im Deutschen. Berlin: Walter de Gruyter. doi: 10.1515/9783110897746
     https://doi.org/10.1515/9783110897746 [Google Scholar]
  12. (2009) Multimodal attribution: How gestures are syntactically integrated into spoken language. Paper presented at thefirst Gesture and Speech in Interaction conference (GeSpIn), Poznań, Poland.
     [Google Scholar]
  13. (2012) Grammatik multimodal: Wie Wörter und Gesten zusammenwirken. Berlin: Walter de Gruyter. doi: 10.1515/9783110218893
     https://doi.org/10.1515/9783110218893 [Google Scholar]
  14. Hadar, U. , & Krauss, R. K.
    (1999) Iconic gestures: The grammatical categories of lexical affiliates. Journal of Neurolinguistics, 12(1), 1–12. doi: 10.1016/S0911‑6044(99)00001‑9
     https://doi.org/10.1016/S0911-6044(99)00001-9 [Google Scholar]
  15. Harrison, S.
    (2008) The expression of negation through grammar and gesture. In J. Zlatev , M. Andrén , M. J. Falck & C. Lundmark (Eds.), Studies in Language and Cognition (pp.405–419). Cambridge: Cambridge Scholars Press.
     [Google Scholar]
  16. (2009) Grammar, gesture, and cognition: The case of negation in English (Unpublished doctoral dissertation). University of Bordeaux, Bordeaux, France.
     [Google Scholar]
  17. Hinrichs, E. , Hinrichs, M. , & Zastrow, T.
    (2010) WebLicht: Web-based LRT services for German. In S. Kübler (Ed.), Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics 2010 System Demonstrations (pp.25–29). Uppsala, Sweden.
     [Google Scholar]
  18. Hole, D. , & Klumpp, G.
    (2000) Definite type and indefinite token: The article son in colloquial German. Linguistische Berichte, 182(1), 231–244.
     [Google Scholar]
  19. Kendon, A.
    (1995) Gestures as illocutionary and discourse structure markers in Southern Italian conversation. Journal of Pragmatics, 23(3), 247–279. doi: 10.1016/0378‑2166(94)00037‑F
     https://doi.org/10.1016/0378-2166(94)00037-F [Google Scholar]
  20. (2004) Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. doi: 10.1017/CBO9780511807572
     https://doi.org/10.1017/CBO9780511807572 [Google Scholar]
  21. Knight, D.
    (2011) Multimodality and Active Listenership: A Corpus Approach. London: Continuum Books.
     [Google Scholar]
  22. Knight, D. , Evans, D. , Carter, R. , & Adolphs, S.
    (2009) HeadTalk, HandTalk and the corpus: Towards a framework for multi-modal, multi-media corpus development. Corpora, 4(1), 1–32. doi: 10.3366/E1749503209000203
     https://doi.org/10.3366/E1749503209000203 [Google Scholar]
  23. Kisler, T. , Schiel, F. , & Sloetjes, H.
    (2012) Signal processing via web services: The use case WebMAUS. In E. Hinrichs , H. Neuroth & P. Wittenburg (Eds.), Proceedings of the Service-oriented Architectures (SOAs) workshop at the Digital Humanities Conference 2012 (pp.30–34). Hamburg, Germany.
     [Google Scholar]
  24. Kok, K. I.
    (2016) The grammatical potential of co-speech gesture: A Functional Discourse Grammar perspective. Functions of Language, 23(2), 149–178. doi: 10.1075/fol.23.2.01kok
     https://doi.org/10.1075/fol.23.2.01kok [Google Scholar]
  25. Kok, K. I. , Bergmann, K. , Cienki, A. , & Kopp, S.
    (2016) Mapping out the multifunctionality of speakers’ gestures. Gesture, 15(1), 37–59. doi: 10.1075/gest.15.1.02kok
     https://doi.org/10.1075/gest.15.1.02kok [Google Scholar]
  26. Kok, K. I. , & Cienki, A.
    (2016) Cognitive Grammar and gesture: Points of convergence, advances and challenges. Cognitive Linguistics, 27(1), 67–100. doi: 10.1515/cog‑2015‑0087
     https://doi.org/10.1515/cog-2015-0087 [Google Scholar]
  27. Kopp, S. , Bergmann, K. , & Wachsmuth, I.
    (2008) Multimodal communication from multimodal thinking – towards an integrated model of speech and gesture production. International Journal of Semantic Computing, 2(1), 115–136. doi: 10.1142/S1793351X08000361
     https://doi.org/10.1142/S1793351X08000361 [Google Scholar]
  28. Krauss, R. M. , Chen, Y. , & Gottesman, R. F.
    (2000) Lexical gestures and lexical access: A process model. In D. McNeill (Ed.), Language and Gesture (pp.261–283). Cambridge: Cambridge University Press. doi: 10.1017/CBO9780511620850.017
     https://doi.org/10.1017/CBO9780511620850.017 [Google Scholar]
  29. Ladewig, S. H.
    (2012) Syntactic and semantic integration of gestures into speech: Structural, cognitive, and conceptual aspects (Unpublished doctoral dissertation). European University Viadrina, Frankfurt (Oder).
     [Google Scholar]
  30. Langacker, R. W.
    (1987) Foundations of Cognitive Grammar, Volume I: Theoretical Prerequisites. Stanford: Stanford University Press.
     [Google Scholar]
  31. Leonard, T. , & Cummins, F.
    (2011) The temporal relation between beat gestures and speech. Language and Cognitive Processes, 26(10), 1457–1471. doi: 10.1080/01690965.2010.500218
     https://doi.org/10.1080/01690965.2010.500218 [Google Scholar]
  32. Levy, E. T. , & McNeill, D.
    (1992) Speech, gesture, and discourse. Discourse Processes, 15(3), 277–301. doi: 10.1080/01638539209544813
     https://doi.org/10.1080/01638539209544813 [Google Scholar]
  33. Loehr, D. P.
    (2004) Gesture and intonation (Unpublished doctoral dissertation). Georgetown University, Washington D.C.
     [Google Scholar]
  34. Lücking, A. , Bergmann, K. , Hahn, F. , Kopp, S. , & Rieser, H.
    (2010) The Bielefeld speech and gesture alignment corpus (SaGA). In M. Kipp , J. C. Martin , P. Paggio & D. Heylen (Eds.), Proceedings of the 7th International Conference for Language Resources and Evaluation (pp.92–98). Valetta, Malta.
     [Google Scholar]
  35. (2013) Data-based analysis of speech and gesture: The Bielefeld Speech and Gesture Alignment Corpus (SaGA) and its applications. Journal on Multimodal User Interfaces, 7(1–2), 5–18. doi: 10.1007/s12193‑012‑0106‑8
     https://doi.org/10.1007/s12193-012-0106-8 [Google Scholar]
  36. McCarthy, M. , & Carter, R.
    (1996) Ten criteria for a spoken grammar. In E. Hinkel & S. Fotos (Eds.), New Perspectives in Grammar Teaching in Second Language Classrooms (pp.51–75). Mahwah, NJ: Lawrence Erlbaum.
     [Google Scholar]
  37. McNeill, D.
    (1992) Hand and Mind: What Gestures Reveal about Thought. Chicago, IL: University of Chicago Press.
     [Google Scholar]
  38. (2000) Catchments and contexts: Non-modular factors in speech and gesture production. In D. McNeill (Ed.), Language and Gesture (pp.312–328). Cambridge: Cambridge University Press. doi: 10.1017/CBO9780511620850.019
     https://doi.org/10.1017/CBO9780511620850.019 [Google Scholar]
  39. Morrel-Samuels, P. , & Krauss, R. M.
    (1992) Word familiarity predicts temporal asynchrony of hand gestures and speech. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(3), 615–622.
     [Google Scholar]
  40. Müller, C. , Ladewig, S. H. , & Bressem, J.
    (2013) Gestures and speech from a linguistic perspective: A new field and its history. In C. Müller , A. Cienki , E. Fricke , S. Ladewig , D. McNeill & J. Bressem (Eds.), Body-Language-Communication: An International Handbook on Multimodality in Human Interaction (Vol.1, pp.55–81). Berlin/Boston: De Gruyter Mouton.
     [Google Scholar]
  41. Schiller, A. , Teufel, S. , & Thielen, C.
    (1995) Guidelines für das Tagging deutscher Textcorpora mit STTS. Unpublished report. University of Stuttgart.
     [Google Scholar]
  42. Schoonjans, S.
    (2014a) Is gesture subject to grammaticalization?Papers of the Linguistic Society of Belgium, 8. Retrieved fromuahost.uantwerpen.be/linguist/SBKL/Vol8.htm (last accessedMay 2016).
     [Google Scholar]
  43. (2014b) Modalpartikeln als multimodale Konstruktionen. Eine korpusbasierte Kookkurrenzanalyse von Modalpartikeln und Gestik im Deutschen (Unpublished doctoral dissertation). University of Leuven, Leuven, Belgium.
     [Google Scholar]
  44. Streeck, J.
    (1993) Gesture as communication: Its coordination with gaze and speech. Communications Monographs, 60(4), 275–299. doi: 10.1080/03637759309376314
     https://doi.org/10.1080/03637759309376314 [Google Scholar]
  45. (2002) Grammars, words, and embodied meanings: On the uses and evolution of so and like. Journal of Communication, 52(3), 581–596. doi: 10.1111/j.1460‑2466.2002.tb02563.x
     https://doi.org/10.1111/j.1460-2466.2002.tb02563.x [Google Scholar]
  46. Thurmair, M.
    (1989) Modalpartikeln und ihre Kombinationen. Tübingen: Niemeyer. doi: 10.1515/9783111354569
     https://doi.org/10.1515/9783111354569 [Google Scholar]
  47. Turner, M. , & Steen, F.
    (2012) Multimodal Construction Grammar. In M. Borkent , B. Dancygier & J. A. J. Hinnell (Eds.), Language and the Creative Mind (pp.255–274). Stanford, CA: CSLI Publications.
     [Google Scholar]
  48. van Son, R. , Wesseling, W. , Sanders, E. , & van den Heuvel, H.
    (2008) The IFADV corpus: A free dialog video corpus. Proceedings of the sixth international conference on Language Resources and Evaluation (LREC) (pp.501–508). Marrakech: European Language Resources Association (ELRA).
     [Google Scholar]
  49. Zima, E.
    (2014) English multimodal motion constructions. A construction grammar perspective. Papers of the Linguistic Society of Belgium, 8. Retrieved fromuahost.uantwerpen.be/linguist/SBKL/sbkl2013/Zim2013.pdf (last accessedMay 2016).
     [Google Scholar]
http://instance.metastore.ingenta.com/content/journals/10.1075/ijcl.22.1.01kok
Loading
Functional and temporal relations between spoken and gestured components of language
International Journal of Corpus Linguistics 22, 1 (2017); https://doi.org/10.1075/ijcl.22.1.01kok
/content/journals/10.1075/ijcl.22.1.01kok
/content/journals/10.1075/ijcl.22.1.01kok
Loading

Data & Media loading...

  • Article Type: Research Article
Keyword(s): distributional analysis; gesture; multimodal corpus; relative frequency ratio

Most Cited

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error