1887
  1. Home
  2. Books
  3. From Gesture in Conversation to Visible Action as Utterance
  4. Chapter

Including facial gestures in gesture-speech ensembles

image of Including facial gestures in gesture-speech ensembles

Conversational facial gestures fit Kendon’s (2004) specifications of the functions of hand gestures. We illustrate how facial gestures in dialogue, like hand gestures, convey referential content as well as serving pragmatic, interpersonal and interactive functions. Hand and facial gestures often occur together, creating an integrated visual image in gesture–speech ensembles. A semantic features analysis demonstrates how speakers adjust their use of these visible versus audible expressive resources according to context. Speakers who were interacting face-to-face (compared to speakers who could not see their addressee) were significantly more likely to rely on their hand and facial gestures than on their words when describing key semantic features, and their gestures were more likely to convey information that was not in their words.

  • Affiliations: 1: Department of Psychology, University of Victoria

References

  1. Bavelas, J. , and Chovil, N
    1997 “Faces in dialogue.” In The Psychology of Facial Expression, J.A. Russell and J.M. Fernandez-Dols (eds), 334–346. Cambridge: Cambridge University Press. doi: 10.1017/CBO9780511659911.017
     https://doi.org/10.1017/CBO9780511659911.017 [Google Scholar]
  2. Bavelas, J. , Chovil, N. , Lawrie, D.A. , and Wade, A
    1992 “Interactive gestures.”Discourse Processes15: 469–489. doi: 10.1080/01638539209544823
     https://doi.org/10.1080/01638539209544823 [Google Scholar]
  3. Bavelas, J. , Gerwing, J. , and Healing, S
    2014 “Hand and facial gestures in conversational interaction.” In Handbook of Language and Social Psychology, T. Holtgraves (ed.), 111–130. Oxford: Oxford University Press.
     [Google Scholar]
  4. in press “The effect of dialogue on demonstrations: direct quotations, facial portrayals, hand gestures, and figurative references.” (accepted for publication in Discourse Processes)
  5. Bavelas, J. , Gerwing, J. , Sutton, C. , and Prevost, D
    2008 “Gesturing on the telephone: Independent effects of dialogue and visibility.”Journal of Memory and Language58: 495–520. doi: 10.1016/j.jml.2007.02.004
     https://doi.org/10.1016/j.jml.2007.02.004 [Google Scholar]
  6. Bavelas, J. , and Healing, S
    2013 “Reconciling the effects of mutual visibility on gesturing. A review.”Gesture13: 63–92. doi: 10.1075/gest.13.1.03bav
     https://doi.org/10.1075/gest.13.1.03bav [Google Scholar]
  7. Beattie, G. , and Shovelton, H
    1999 “Do iconic hand gestures really contribute anything to the semantic information conveyed by speech? An experimental investigation.”Semiotica123 (1–2): 1–30. doi: 10.1515/semi.1999.123.1‑2.1
     https://doi.org/10.1515/semi.1999.123.1-2.1 [Google Scholar]
  8. 2002 “An experimental investigation of some properties of individual iconic gestures that mediate their communicative power.”British Journal of Psychology93: 179–192. doi: 10.1348/000712602162526
     https://doi.org/10.1348/000712602162526 [Google Scholar]
  9. Birdwhistell, R.L
    .1970 “Masculinity and femininity as display.”Kinesics and Context: Essays on Body Motion. Philadelphia, PA: University of Pennsylvania.
     [Google Scholar]
  10. Brugman, H. , and Russel, A
    2004 “Annotating multimedia/multimodal resources with ELAN.” In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004) , M.T. Lino , M.F. Xavier , F. Ferreira , R. Costa , and R. Silva (eds), 2065–2068. Paris: European Language Resources Association.
     [Google Scholar]
  11. Brunner, L.J
    1979 “Smiles can be back channels.”Journal of Personality and Social Psychology37: 728–734. doi: 10.1037/0022‑3514.37.5.728
     https://doi.org/10.1037/0022-3514.37.5.728 [Google Scholar]
  12. Chovil, N
    1989 Communicative Functions of Facial Displays in Conversation. Ph.D. dissertation, University of Victoria, BC.
     [Google Scholar]
  13. 1991/1992 “Discourse-oriented facial displays in conversation.”Research on Language and Social Interaction25: 163–192. doi: 10.1080/08351819109389361
     https://doi.org/10.1080/08351819109389361 [Google Scholar]
  14. 1991 Social determinants of facial displays. Journal of Nonverbal Behavior15: 141–153. doi: 10.1007/BF01672216
     https://doi.org/10.1007/BF01672216 [Google Scholar]
  15. 1997 “Facing others. A social communicative perspective on facial displays.” In The Psychology of Facial Expression, J.A. Russell and J.M. Fernandez-Dols (eds), 321–333. Cambridge: Cambridge University Press. doi: 10.1017/CBO9780511659911.016
     https://doi.org/10.1017/CBO9780511659911.016 [Google Scholar]
  16. Coates, L.J
    1991 A Collaborative Theory of Inversion: Irony in Dialogue. M.A. thesis, University of Victoria, BC.
     [Google Scholar]
  17. de Ruiter, J.P. , Bangerter, A. , and Dings, P
    2012 “The interplay between gesture and speech in the production of referring expressions: Investigating the tradeoff hypothesis.”Topics in Cognitive Science4 (2): 232–248. doi: 10.1111/j.1756‑8765.2012.01183.x
     https://doi.org/10.1111/j.1756-8765.2012.01183.x [Google Scholar]
  18. Ekman, P
    1979 “About brows: Emotional and conversational signals.” In Human Ethology, M. von Cranach , K. Foppa , W. Lepenies and D. Ploog (eds), 169–249. Cambridge: ­Cambridge University Press.
     [Google Scholar]
  19. 1985 Telling Lies. New York: Norton.
     [Google Scholar]
  20. 1997 “Should we call it expression or communication?”European Journal of Social Sciences10: 333–359.
     [Google Scholar]
  21. Ekman, P. , and Friesen, W.V
    1969 “The repertoire of nonverbal behavior: Categories, origins, usage, and coding.”SemioticaI: 49–98.
     [Google Scholar]
  22. Enfield, N.J
    2001 “‘Lip-pointing’: A discussion of form and function with reference to data from Laos.”Gesture1: 185–221. doi: 10.1075/gest. 1.2.06enf
     https://doi.org/10.1075/gest. 1.2.06enf [Google Scholar]
  23. Fridlund, A.J. , Ekman, P. , and Oster, H
    1987 “Facial expressions of emotion: Review of literature, 1970–1983.”InNonverbal Communication and Behavior (2nd ed.), A.W. Siegman and S. Feldstein , eds. (143–224). Hillsdale, NJ: Erlbaum.
     [Google Scholar]
  24. Gerwing, J. , and Allison, M
    2009 “The relationship between verbal and gestural contributions in conversation: A comparison of three methods.”Gesture9 (3): 313–336. doi: 10.1075/gest. 9.3.03ger
     https://doi.org/10.1075/gest. 9.3.03ger [Google Scholar]
  25. 2011 “The flexible semantic integration of gestures and words: Comparing face-to-face and telephone dialogues.”Gesture11 (3): 308–329. doi: 10.1075/gest. 11.3.03ger
     https://doi.org/10.1075/gest. 11.3.03ger [Google Scholar]
  26. Holler, J. , and Beattie, G
    2002 “A micro-analytic investigation of how iconic gestures and speech represent core semantic features in talk.”Semiotica142: 31–69.
     [Google Scholar]
  27. 2003 “How iconic gestures and speech interact in the representation of meaning: Are both aspects really integral to the process?”Semiotica146: 81–116.
     [Google Scholar]
  28. 2004 “The interaction of iconic gesture and speech.” In Lecture Notes in Computer Science, 5th International Gesture Workshop, Genova, Italy, 2003: Selected Revised Papers , A. Cammurri and G. Volpe (eds), 63–69. Heidelberg: Springer Verlag.
     [Google Scholar]
  29. Holler, J. , and Stevens, R
    2007 “An experimental investigation into the effect of common ground on how speakers use gesture and speech to represent size information in referential communication.”Journal of Language and Social Psychology26: 4–27. doi: 10.1177/0261927X06296428
     https://doi.org/10.1177/0261927X06296428 [Google Scholar]
  30. Holler, J. , and Wilkin, K
    2009 “Communicating common ground: How mutually shared knowledge influences the representation of semantic information in speech and gesture in a narrative task.”Language and Cognitive Processes24: 267–289. doi: 10.1080/01690960802095545
     https://doi.org/10.1080/01690960802095545 [Google Scholar]
  31. Holler, J. , Tutton, M. , and Wilkin, K
    2011 “Co-speech gestures in the process of meaning coordination.” In Proceedings of the 2nd GESPIN – Gesture in Speech and Interaction Conference , Bielefeld, 5–7, September 2011.
     [Google Scholar]
  32. Kendon, A
    2004 Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press.
     [Google Scholar]
  33. Kraut, R.E. , and Johnston, R.E
    1979 “Social and emotional messages of smiling: An ethological approach.”Journal of Personality and Social Psychology61: 743–745.
     [Google Scholar]
  34. Pine, K.J. , Burney, D.J. , and Fletcher, B
    2010 “The semantic specificity hypothesis: When gestures do not depend upon the presence of a listener.”Journal of Nonverbal Behavior34: 169–178. doi: 10.1007/s10919‑010‑0089‑7
     https://doi.org/10.1007/s10919-010-0089-7 [Google Scholar]
  35. Rimé, B
    1982 “The elimination of visible behavior from social interactions: Effects on verbal, nonverbal and interpersonal variables.”European Journal of Social Psychology12: 113–129. doi: 10.1002/ejsp. 2420120201
     https://doi.org/10.1002/ejsp. 2420120201 [Google Scholar]
  36. Sherzer, J
    1973 Verbal and Nonverbal Deixis: The Pointed Lip Gesture among the San Blas Cuna. University of Texas at Austin: Institute of Latin American Studies.
     [Google Scholar]
  37. Tomasello, M. , Hare, B. , Lehmann, H. , and Call, J
    2007 “Reliance on head versus eyes in the gaze following of great apes and human infants: the cooperative eye hypothesis.”Journal of Human Evolution52 (3): 314–320. doi: 10.1016/j.jhevol. 2006.10.001
     https://doi.org/10.1016/j.jhevol. 2006.10.001 [Google Scholar]
  38. Wittenburg, P. , Brugman, H. , Russel, A. , Klassman, A. , and Sloetjes, H
    2006 “ELAN: A professional framework for multimodality research.” In Proceedings of LREC 2006, Fifth International Conference on Language Resources and Evaluation , Genoa.
     [Google Scholar]

References

  1. Bavelas, J. , and Chovil, N
    1997 “Faces in dialogue.” In The Psychology of Facial Expression, J.A. Russell and J.M. Fernandez-Dols (eds), 334–346. Cambridge: Cambridge University Press. doi: 10.1017/CBO9780511659911.017
     https://doi.org/10.1017/CBO9780511659911.017 [Google Scholar]
  2. Bavelas, J. , Chovil, N. , Lawrie, D.A. , and Wade, A
    1992 “Interactive gestures.”Discourse Processes15: 469–489. doi: 10.1080/01638539209544823
     https://doi.org/10.1080/01638539209544823 [Google Scholar]
  3. Bavelas, J. , Gerwing, J. , and Healing, S
    2014 “Hand and facial gestures in conversational interaction.” In Handbook of Language and Social Psychology, T. Holtgraves (ed.), 111–130. Oxford: Oxford University Press.
     [Google Scholar]
  4. in press “The effect of dialogue on demonstrations: direct quotations, facial portrayals, hand gestures, and figurative references.” (accepted for publication in Discourse Processes)
  5. Bavelas, J. , Gerwing, J. , Sutton, C. , and Prevost, D
    2008 “Gesturing on the telephone: Independent effects of dialogue and visibility.”Journal of Memory and Language58: 495–520. doi: 10.1016/j.jml.2007.02.004
     https://doi.org/10.1016/j.jml.2007.02.004 [Google Scholar]
  6. Bavelas, J. , and Healing, S
    2013 “Reconciling the effects of mutual visibility on gesturing. A review.”Gesture13: 63–92. doi: 10.1075/gest.13.1.03bav
     https://doi.org/10.1075/gest.13.1.03bav [Google Scholar]
  7. Beattie, G. , and Shovelton, H
    1999 “Do iconic hand gestures really contribute anything to the semantic information conveyed by speech? An experimental investigation.”Semiotica123 (1–2): 1–30. doi: 10.1515/semi.1999.123.1‑2.1
     https://doi.org/10.1515/semi.1999.123.1-2.1 [Google Scholar]
  8. 2002 “An experimental investigation of some properties of individual iconic gestures that mediate their communicative power.”British Journal of Psychology93: 179–192. doi: 10.1348/000712602162526
     https://doi.org/10.1348/000712602162526 [Google Scholar]
  9. Birdwhistell, R.L
    .1970 “Masculinity and femininity as display.”Kinesics and Context: Essays on Body Motion. Philadelphia, PA: University of Pennsylvania.
     [Google Scholar]
  10. Brugman, H. , and Russel, A
    2004 “Annotating multimedia/multimodal resources with ELAN.” In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004) , M.T. Lino , M.F. Xavier , F. Ferreira , R. Costa , and R. Silva (eds), 2065–2068. Paris: European Language Resources Association.
     [Google Scholar]
  11. Brunner, L.J
    1979 “Smiles can be back channels.”Journal of Personality and Social Psychology37: 728–734. doi: 10.1037/0022‑3514.37.5.728
     https://doi.org/10.1037/0022-3514.37.5.728 [Google Scholar]
  12. Chovil, N
    1989 Communicative Functions of Facial Displays in Conversation. Ph.D. dissertation, University of Victoria, BC.
     [Google Scholar]
  13. 1991/1992 “Discourse-oriented facial displays in conversation.”Research on Language and Social Interaction25: 163–192. doi: 10.1080/08351819109389361
     https://doi.org/10.1080/08351819109389361 [Google Scholar]
  14. 1991 Social determinants of facial displays. Journal of Nonverbal Behavior15: 141–153. doi: 10.1007/BF01672216
     https://doi.org/10.1007/BF01672216 [Google Scholar]
  15. 1997 “Facing others. A social communicative perspective on facial displays.” In The Psychology of Facial Expression, J.A. Russell and J.M. Fernandez-Dols (eds), 321–333. Cambridge: Cambridge University Press. doi: 10.1017/CBO9780511659911.016
     https://doi.org/10.1017/CBO9780511659911.016 [Google Scholar]
  16. Coates, L.J
    1991 A Collaborative Theory of Inversion: Irony in Dialogue. M.A. thesis, University of Victoria, BC.
     [Google Scholar]
  17. de Ruiter, J.P. , Bangerter, A. , and Dings, P
    2012 “The interplay between gesture and speech in the production of referring expressions: Investigating the tradeoff hypothesis.”Topics in Cognitive Science4 (2): 232–248. doi: 10.1111/j.1756‑8765.2012.01183.x
     https://doi.org/10.1111/j.1756-8765.2012.01183.x [Google Scholar]
  18. Ekman, P
    1979 “About brows: Emotional and conversational signals.” In Human Ethology, M. von Cranach , K. Foppa , W. Lepenies and D. Ploog (eds), 169–249. Cambridge: ­Cambridge University Press.
     [Google Scholar]
  19. 1985 Telling Lies. New York: Norton.
     [Google Scholar]
  20. 1997 “Should we call it expression or communication?”European Journal of Social Sciences10: 333–359.
     [Google Scholar]
  21. Ekman, P. , and Friesen, W.V
    1969 “The repertoire of nonverbal behavior: Categories, origins, usage, and coding.”SemioticaI: 49–98.
     [Google Scholar]
  22. Enfield, N.J
    2001 “‘Lip-pointing’: A discussion of form and function with reference to data from Laos.”Gesture1: 185–221. doi: 10.1075/gest. 1.2.06enf
     https://doi.org/10.1075/gest. 1.2.06enf [Google Scholar]
  23. Fridlund, A.J. , Ekman, P. , and Oster, H
    1987 “Facial expressions of emotion: Review of literature, 1970–1983.”InNonverbal Communication and Behavior (2nd ed.), A.W. Siegman and S. Feldstein , eds. (143–224). Hillsdale, NJ: Erlbaum.
     [Google Scholar]
  24. Gerwing, J. , and Allison, M
    2009 “The relationship between verbal and gestural contributions in conversation: A comparison of three methods.”Gesture9 (3): 313–336. doi: 10.1075/gest. 9.3.03ger
     https://doi.org/10.1075/gest. 9.3.03ger [Google Scholar]
  25. 2011 “The flexible semantic integration of gestures and words: Comparing face-to-face and telephone dialogues.”Gesture11 (3): 308–329. doi: 10.1075/gest. 11.3.03ger
     https://doi.org/10.1075/gest. 11.3.03ger [Google Scholar]
  26. Holler, J. , and Beattie, G
    2002 “A micro-analytic investigation of how iconic gestures and speech represent core semantic features in talk.”Semiotica142: 31–69.
     [Google Scholar]
  27. 2003 “How iconic gestures and speech interact in the representation of meaning: Are both aspects really integral to the process?”Semiotica146: 81–116.
     [Google Scholar]
  28. 2004 “The interaction of iconic gesture and speech.” In Lecture Notes in Computer Science, 5th International Gesture Workshop, Genova, Italy, 2003: Selected Revised Papers , A. Cammurri and G. Volpe (eds), 63–69. Heidelberg: Springer Verlag.
     [Google Scholar]
  29. Holler, J. , and Stevens, R
    2007 “An experimental investigation into the effect of common ground on how speakers use gesture and speech to represent size information in referential communication.”Journal of Language and Social Psychology26: 4–27. doi: 10.1177/0261927X06296428
     https://doi.org/10.1177/0261927X06296428 [Google Scholar]
  30. Holler, J. , and Wilkin, K
    2009 “Communicating common ground: How mutually shared knowledge influences the representation of semantic information in speech and gesture in a narrative task.”Language and Cognitive Processes24: 267–289. doi: 10.1080/01690960802095545
     https://doi.org/10.1080/01690960802095545 [Google Scholar]
  31. Holler, J. , Tutton, M. , and Wilkin, K
    2011 “Co-speech gestures in the process of meaning coordination.” In Proceedings of the 2nd GESPIN – Gesture in Speech and Interaction Conference , Bielefeld, 5–7, September 2011.
     [Google Scholar]
  32. Kendon, A
    2004 Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press.
     [Google Scholar]
  33. Kraut, R.E. , and Johnston, R.E
    1979 “Social and emotional messages of smiling: An ethological approach.”Journal of Personality and Social Psychology61: 743–745.
     [Google Scholar]
  34. Pine, K.J. , Burney, D.J. , and Fletcher, B
    2010 “The semantic specificity hypothesis: When gestures do not depend upon the presence of a listener.”Journal of Nonverbal Behavior34: 169–178. doi: 10.1007/s10919‑010‑0089‑7
     https://doi.org/10.1007/s10919-010-0089-7 [Google Scholar]
  35. Rimé, B
    1982 “The elimination of visible behavior from social interactions: Effects on verbal, nonverbal and interpersonal variables.”European Journal of Social Psychology12: 113–129. doi: 10.1002/ejsp. 2420120201
     https://doi.org/10.1002/ejsp. 2420120201 [Google Scholar]
  36. Sherzer, J
    1973 Verbal and Nonverbal Deixis: The Pointed Lip Gesture among the San Blas Cuna. University of Texas at Austin: Institute of Latin American Studies.
     [Google Scholar]
  37. Tomasello, M. , Hare, B. , Lehmann, H. , and Call, J
    2007 “Reliance on head versus eyes in the gaze following of great apes and human infants: the cooperative eye hypothesis.”Journal of Human Evolution52 (3): 314–320. doi: 10.1016/j.jhevol. 2006.10.001
     https://doi.org/10.1016/j.jhevol. 2006.10.001 [Google Scholar]
  38. Wittenburg, P. , Brugman, H. , Russel, A. , Klassman, A. , and Sloetjes, H
    2006 “ELAN: A professional framework for multimodality research.” In Proceedings of LREC 2006, Fifth International Conference on Language Resources and Evaluation , Genoa.
     [Google Scholar]

  • From This Site

    /content/books/9789027269270-z.188.02bav
    dcterms_subject,pub_keyword
    -contentType:Journal
    10
    5
/content/books/9789027269270-z.188.02bav
dcterms_subject,pub_keyword
-contentType:Journal
10
5
Chapter
content/books/9789027269270
Book
false
Loading
This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error