Volume 21, Issue 2
  • ISSN 1572-0373
  • E-ISSN: 1572-0381
Buy:$35.00 + Taxes



This article presents an applied discussion of the possibility of integrating conversation analysis (CA) methodology into that of machine learning. The aim is to improve the detection of that which resembles disengagement in the interaction between a robot and a human. We offer a novel analytical assemblage at the heart of the two disciplines, and namely on the level of the annotation schemes provided by conversation analysis transcription methods. First, we demonstrate that the need for a stable structure in establishing an interaction scenario and in designing robot behaviours does not prevent the emergence of ordinariness or creativity among the participants engaged in this interaction. Secondly, based on an actual case, we emphasize the possibility of systematicness in CA transcription to support the choice (a) of the categories targeted by prediction methods and defined by the annotation scheme, and (b) of the verbal and non-verbal features used to create prediction models.


Article metrics loading...

Loading full text...

Full text loading...


  1. Bartneck, C., Belpaeme, T., Eyssel, F., Kanda, T., Keijsers, M., & Šabanović, S.
    (2019) Human-robot interaction: An introduction. Cambridge: Cambridge University Press.
    [Google Scholar]
  2. Beach, W. A.
    (1993) Transitional regularities for ‘casual’ “Okay” usages. Journal of Pragmatics, 19, 325–352. 10.1016/0378‑2166(93)90092‑4
    https://doi.org/10.1016/0378-2166(93)90092-4 [Google Scholar]
  3. Ben-Youssef, A., Clavel, C., Essid, S., Bilac, M., Chamoux, M., & Lim, A.
    (2017) UE- HRI: a new dataset for the study of user engagement in spontaneous human- robot interactions. Paper presented at the19th ACM International Conference on Multimodal Interaction, Glasgow, UK.
    [Google Scholar]
  4. Breazeal, C.
    (2003) Toward sociable robots. Robotics and Autonomous Systems, 42, 167–175. 10.1016/S0921‑8890(02)00373‑1
    https://doi.org/10.1016/S0921-8890(02)00373-1 [Google Scholar]
  5. Button, G.
    (1991) Conversation-in-a-series. InD. Boden & D. H. Zimmerman (Eds.), Talk and social structure. Studies in Ethnomethodology and Conversation Analysis (pp.251–277). Cambridge: Polity Press.
    [Google Scholar]
  6. Campano, S., Clavel, C., & Pélachaud, C.
    (2015) “I like this painting too”: when an ECA shares appreciations to engage users. Paper presented at the14th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’15), Istanbul, Turkey.
    [Google Scholar]
  7. Cassell, J., Torres, O., & Prevost, S.
    (1999) Turn taking vs. discourse structure: How best to model multimodal conversation. InY. Wilks (Ed.), Machine Conversations (pp.143–154). The Hague: Kluwer. 10.1007/978‑1‑4757‑5687‑6_12
    https://doi.org/10.1007/978-1-4757-5687-6_12 [Google Scholar]
  8. Chapman, D.
    (1992) Computer rules, conversational rules. Computational Linguistics, 18(4), 531–536.
    [Google Scholar]
  9. Clavel, C., Vasilescu, I., & Devillers, L.
    (2011) Fiction support for realistic portrayals of fear-type emotional manifestations. Computer Speech & Language, 25(1), 63–83. 10.1016/j.csl.2010.03.003
    https://doi.org/10.1016/j.csl.2010.03.003 [Google Scholar]
  10. Clavel, C., Cafaro, A., Campano, S., & Pelachaud, C.
    (2016) Fostering User Engagement in Face-to-Face Human-Agent Interactions: A Survey. InA. Esposito & L. C. Jain (Eds.), Toward Robotic Socially Believable Behaving Systems – Volume II: Modeling Social Signals (pp.93–120). Cham: Springer International Publishing. 10.1007/978‑3‑319‑31053‑4_7
    https://doi.org/10.1007/978-3-319-31053-4_7 [Google Scholar]
  11. Clavel, C., & Callejas, Z.
    (2016) Sentiment analysis: from opinion mining to human-agent interaction. IEEE Transactions on affective computing, 7(1), 74–93. 10.1109/TAFFC.2015.2444846
    https://doi.org/10.1109/TAFFC.2015.2444846 [Google Scholar]
  12. Cowie, R., & Cornelius, R. R.
    (2003) Describing the emotional states that are expressed in speech. Speech communication, 40(1–2), 5–32. 10.1016/S0167‑6393(02)00071‑7
    https://doi.org/10.1016/S0167-6393(02)00071-7 [Google Scholar]
  13. Dautenhahn, K.
    (2007) Socially intelligent robots: dimensions of human-robot interaction. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 362(1480), 679–704. 10.1098/rstb.2006.2004
    https://doi.org/10.1098/rstb.2006.2004 [Google Scholar]
  14. Dickerson, P., Robins, B., & Dautenhahn, K.
    (2013) Where the action is: A conversation analytic perspective on interaction between a humanoid robot, a co-present adult and a child with an ASD. Interaction Studies, 14(2), 297–316. 10.1075/is.14.2.07dic
    https://doi.org/10.1075/is.14.2.07dic [Google Scholar]
  15. Duranti, A.
    (1997) Linguistic anthropology. New York: Cambridge University Press. 10.1017/CBO9780511810190
    https://doi.org/10.1017/CBO9780511810190 [Google Scholar]
  16. Duranti, A., & Goodwin, C.
    (1992) Rethinking context: an introduction. InA. Duranti & C. Goodwin (Eds.), Rethinking Context, language as an interactive phenomenon (pp.1–42). Cambridge University Press.
    [Google Scholar]
  17. Fong, T., Nourbakhsh, I., & Dautenhahn, K.
    (2003) A survey of socially interactive robots. Robotics and Autonomous Systems, 42(3–4), 143–166. 10.1016/S0921‑8890(02)00372‑X
    https://doi.org/10.1016/S0921-8890(02)00372-X [Google Scholar]
  18. Ford, C., & Thompson, S.
    (1996) Interactional units in conversation: syntactic, intonational, and pragmatic resources for the management of turns. InE. Ochs, E. Schegloff, & S. Thompson (Eds.), Interaction and Grammar (134-184): Cambridge University Press. 10.1017/CBO9780511620874.003
    https://doi.org/10.1017/CBO9780511620874.003 [Google Scholar]
  19. Garfinkel, H.
    (1967) Studies in ethnomethodology. Engelwood Cliffs: Prentice-Hall.
    [Google Scholar]
  20. Goffman, E.
    (1973) La mise en scène de la vie quotidienne (Vol.2. Les relations en public). Paris: Les Editions de minuit.
    [Google Scholar]
  21. (1983) The interaction order. American Sociological Review, 48(1), pp: 1–17. 10.2307/2095141
    https://doi.org/10.2307/2095141 [Google Scholar]
  22. Goodwin, C.
    (1981) Conversational organization: interaction between speakers and hearers. New York: Academic Press.
    [Google Scholar]
  23. Heritage, J.
    (1991) L’Ethnométhodologie : une approche procédurale de l’action et de la communication. Réseaux CNET, 50, 89–130. 10.3406/reso.1991.1900
    https://doi.org/10.3406/reso.1991.1900 [Google Scholar]
  24. Jefferson, G.
    (1978) Sequential aspects of storytelling in conversation. InJ. Schenkein (Ed.), Studies in the organization of conversational interaction (pp.219–248). New York: New York Academic Press. 10.1016/B978‑0‑12‑623550‑0.50016‑1
    https://doi.org/10.1016/B978-0-12-623550-0.50016-1 [Google Scholar]
  25. Jones, R. A.
    (2017) What makes a robot “social”?Social Studies of Science, 47(4), 556–579. 10.1177/0306312717704722
    https://doi.org/10.1177/0306312717704722 [Google Scholar]
  26. Langlet, C., & Clavel, C.
    (2014) Modelling user’s attitudinal reactions to the agent utterances: focus on the verbal content. Paper presented at the5th International Workshop on Corpora for Research on Emotion, Sentiment & Social Signals (ES3 2014), Reykjavik, Iceland.
    [Google Scholar]
  27. (2018) Detecting User’s Likes and Dislikes for a Virtual Negotiating Agent. Paper presented at the20th ACM International Conference on Multimodal Interaction, Boulder, USA. 10.1145/3242969.3243024
    https://doi.org/10.1145/3242969.3243024 [Google Scholar]
  28. Levinson, S. C.
    (1983) Pragmatics. Cambridge University Press. 10.1017/CBO9780511813313
    https://doi.org/10.1017/CBO9780511813313 [Google Scholar]
  29. Licoppe, C., & Figeac, J.
    (2014) L’Organisation temporelle des engagements visuels dans des situations de multi-activité équipée en milieu urbain. Activités, 11(1).
    [Google Scholar]
  30. Licoppe, C., & Rollet, N.
    (2020, in press) « Je dois y aller ». Analyses de séquences de clôtures entre humains et robot. Réseaux. La Découverte.
    [Google Scholar]
  31. Mondada, L.
    (2006) Participants’ online analysis and multimodal practices: projecting the end of the turn and the closing of the sequence. Discourse Studies, 8(1), pp.117–129. 10.1177/1461445606059561
    https://doi.org/10.1177/1461445606059561 [Google Scholar]
  32. (2008) Documenter l’articulation des ressources multimodales dans le temps : la transcription d’enregistrements vidéos d’interactions. InM. Bilger (Ed.), Données orales. Les enjeux de la transcription (Vol.37), pp.127–156. Perpignan: Presses Universitaires de Perpignan.
    [Google Scholar]
  33. Mohri, M., Rostamizadeh, A., & Talwalkar, A.
    (2012) Foundations of machine learning. MIT press.
    [Google Scholar]
  34. Ochs, E.
    (1979) Transcription as a theory. InB. B. Schieffelin (Ed.), Developmental Pragmatics (pp.42–72). Academic Press.
    [Google Scholar]
  35. Pelachaud, C., & Glas, N.
    (2015a) Definitions of Engagement in Human-Agent Interaction. Paper presented at theInternational Workshop on Engagement in Human Computer Interaction (ENHANCE), Xi’an, China.
    [Google Scholar]
  36. (2015b) Topic transition strategies for an information-giving agent. Paper presented at the15th European Workshop on natural Language Generation, Brighton, UK.
    [Google Scholar]
  37. Pelikan, H. R. M., & Broth, M.
    (2016) Why That Nao?: How Humans Adapt to a Conventional Humanoid Robot in Taking Turns-at-Talk. Paper presented at the2016 CHI Conference on Human Factors in Computing Systems, San Jose, California, USA. 10.1145/2858036.2858478
    https://doi.org/10.1145/2858036.2858478 [Google Scholar]
  38. Pitsch, K., Kuzuoka, H., Suzuki, Y., Süssenbach, L., Luff, P., & Heath, C.
    (2009) “The first five seconds”: Contingent stepwise entry into an interaction as a means to secure sustained engagement in Human-Robot-Interaction. Paper presented at theIEEE International Symposium on Robot and Human Interactive Communication ROMAN 2009, Toyama, Japan.
    [Google Scholar]
  39. Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S.
    (2018) Voice Interfaces in Everyday Life. Montreal QC, Canada: Association for Computing Machinery. 10.1145/3173574.3174214
    https://doi.org/10.1145/3173574.3174214 [Google Scholar]
  40. Rollet, N.
    (2010) All the things you are. Activité multimodale, frontières et musiques improvisées en répétition InN. Andrieux-Reix (Ed.), Frontières. Du linguistique au sémiotique (pp.279–302). Limoges: Lambert-Lucas.
    [Google Scholar]
  41. (2013) “D’accord”. Approche conversationnelle et multimodale d’une forme située dans les appels au Samu-Centre 15. L’Information grammaticale, 139.
    [Google Scholar]
  42. Rollet, N., Jain, V., Licoppe, C., & Devillers, L.
    (2017) Towards Interactional Symbiosis: Epistemic Balance and Co-presence in a Quantified Self Experiment. InL. Gamberini, A. Spagnolli, G. Jacucci, B. Blankertz, & J. Freeman (Eds.), Symbiotic Interaction: 5th International Workshop, Symbiotic 2016, Padua, Italy, September 29–30, 2016, Revised Selected Papers (pp.143–154). Cham: Springer International Publishing. 10.1007/978‑3‑319‑57753‑1_13
    https://doi.org/10.1007/978-3-319-57753-1_13 [Google Scholar]
  43. Sacks, H.
    (1984) On doing ‘being ordinary’. InJ. M. Atkinson & J. Heritage (Eds.), Structures of Social Action (pp.413–429). Cambridge: CUP.
    [Google Scholar]
  44. (1992) Lectures on conversation (Jefferson, G.ed.). Oxford: Blackwell.
    [Google Scholar]
  45. Sacks, H., & Schegloff, E.
    (1973) Opening up closings. Semiotica, 8, 289–326.
    [Google Scholar]
  46. Sacks, H., Schegloff, E. A., & Jefferson, G.
    (1974) A symplest systematics for the organization of turn-taking for conversation. Language, 50, 696–731. 10.1353/lan.1974.0010
    https://doi.org/10.1353/lan.1974.0010 [Google Scholar]
  47. Sadazuka, K., Kuno, Y., Kawashima, M., & Yamazaki, K.
    (2007) Museum Guide Robot with Effective Head Gestures. Paper presented at theInternational Conference on Control, Automation and Systems, Seoul, Korea. 10.1109/ICCAS.2007.4406510
    https://doi.org/10.1109/ICCAS.2007.4406510 [Google Scholar]
  48. Schegloff, E.
    (1987) Between micro and macro: contexts and other connections. InB. Giesen, J. C. Alexander, R. Münch, & N. J. Smelser (Eds.), The Micro-macro link (pp.207–234). Los Angeles: University of California Press.
    [Google Scholar]
  49. (2002) Accounts of conduct in interaction. Interruption, overlap, and turn-taking. InT. J. H. (Ed.), Handbook of sociological theory (pp.287–321). New York: Kluwer Academic/Plenum publishers.
    [Google Scholar]
  50. (2007) Sequence organization in interaction. A Primer in Conversation Analysis. (Vol.1): Cambridge University Press. 10.1017/CBO9780511791208
    https://doi.org/10.1017/CBO9780511791208 [Google Scholar]
  51. Sidner, C. L., & Dzikovska, M.
    (2002, 16–16Oct. 2002) Human-robot interaction: engagement between humans and robots for hosting activities. Paper presented at the4th IEEE International Conference on Multimodal Interfaces, Pittsburgh, USA.
    [Google Scholar]
  52. Sidner, C. L., Lee, C., Kidd, C. D., & Rich, C.
    (2005) Explorations in engagement for humans and robots. Artificial Intelligence, 166, 140–164. 10.1016/j.artint.2005.03.005
    https://doi.org/10.1016/j.artint.2005.03.005 [Google Scholar]
  53. Stivers, T.
    (2008) Stance, Alignment, and Affiliation During Storytelling: When Nodding Is a Token of Affiliation. Research on Language & Social Interaction, 44(1), 31–57. 10.1080/08351810701691123
    https://doi.org/10.1080/08351810701691123 [Google Scholar]
  54. (2015) Coding Social Interaction: A Heretical Approach in Conversation Analysis?Research on Language and Social Interaction, 48(1), 1–19. 10.1080/08351813.2015.993837
    https://doi.org/10.1080/08351813.2015.993837 [Google Scholar]
  55. Šabanović, S., & Chang, W.-L.
    (2016) Socializing robots: constructing robotic sociality in the design and use of the assistive robot PARO. AI and Society, 31(4), 537–551. 10.1007/s00146‑015‑0636‑1
    https://doi.org/10.1007/s00146-015-0636-1 [Google Scholar]
  56. Straub, I.
    (2016) “It looks like a human!” The interrelation of social presence, interaction and agency ascription: a case study about the effects of an android robot on social agency ascription. AI and Society, 31(4), 553–571. 10.1007/s00146‑015‑0632‑5
    https://doi.org/10.1007/s00146-015-0632-5 [Google Scholar]
  57. Suchman, L.
    (2007) Human-Machine Reconfigurations. Plans and situated actions, 2nd edition. New York: Cambridge University Press.
    [Google Scholar]
  58. Yu, Z., Scherer, S., Devault, D., Gratch, J., Stratou, G., Morency, L., & Cassell, J.
    (2013) Multimodal Prediction of Psychological Disorder: Learning Verbal and Nonverbal Commonality in Adjacency Pairs. Paper presented at the17th Workshop Series on the Semantics and Pragmatics of Dialogue, Amsterdam, Netherland.
    [Google Scholar]
  59. Zimmerman, D.
    (2006) How closing matters in emergency telephone calls. Paper presented at theAnnual meeting of the American Sociology Association, Montréal, Canada.
    [Google Scholar]

Data & Media loading...

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error