Socially Acceptable Robot Behavior
  • ISSN 1572-0373
  • E-ISSN: 1572-0381
Buy:$35.00 + Taxes



In the multi-agent setting, it is relevant to model group dynamics of agents, and logic has proved a good tool to do so. We propose an epistemic logic, that allows one to formalize what are the beliefs formed by a group of agents, where several groups exist and agents can pass from a group to another one. We introduce a new modality which allows an agent to reason about the beliefs of other agents. This allows us to model aspects of the “Theory of Mind”, understood as the set of social-cognitive skills involving the ability to attribute and reason about mental states, desires, beliefs, and knowledge of agents. In this paper, we present the logic and illustrate how it can be used to solve “false-belief tasks”, i.e., tests in which an agent should understand that some other agent may develop, under some circumstances, false beliefs.


Article metrics loading...

Loading full text...

Full text loading...


  1. Adam, C., Herzig, A., & Longin, D.
    (2009) A logical formalization of the OCC theory of emotions. Synth., 168 (2), 201–248. 10.1007/s11229‑009‑9460‑9
    https://doi.org/10.1007/s11229-009-9460-9 [Google Scholar]
  2. Aielli, F., Ancona, D., Caianiello, P., Costantini, S., De Gasperis, G., Di Marco, A., … Mascardi, V.
    (2016) FRIENDLY & KIND with your health: Human-friendly knowledge-intensive dynamic systems for the e-health domain. InJ. Bajo (Eds.), Highlights of practical applications of scalable multi-agent systems. The PAAMS collection – international workshops of PAAMS 2016, proceedings (Vol. 616 of Communications in Computer and Information Science, pp. 15–26). Springer. 10.1007/978‑3‑319‑39387‑2_2
    https://doi.org/10.1007/978-3-319-39387-2_2 [Google Scholar]
  3. Alberti, M., Chesani, F., Gavanelli, M., Lamma, E., Mello, P., & Torroni, P.
    (2008) Verifiable agent interaction in abductive logic programming: The SCIFF framework. ACM Trans. Comput. Log., 9(4), 291:1–29:431. 10.1145/1380572.1380578
    https://doi.org/10.1145/1380572.1380578 [Google Scholar]
  4. Amir, E., Andreson, M. L., & Chaudri, V. K.
    (2007) Report on DARPA workshop on self aware computer systems. Tech. Rep., SRI International Menlo Park, USA. (Full text: www.dtic.mil/dtic/tr/fulltext/u2/1002393.pdf)
    [Google Scholar]
  5. Balaji, P. G., & Srinivasan, D.
    (2010) An introduction to multi-agent systems. InD. Srinivasan & L. C. Jain (Eds.), Innovations in MASs and applications (Vol. 310 of Studies in Computational Intelligence). Springer. 10.1007/978‑3‑642‑14435‑6_1
    https://doi.org/10.1007/978-3-642-14435-6_1 [Google Scholar]
  6. Balbiani, P., Fernández-Duque, D., & Lorini, E.
    (2016) A logical theory of belief dynamics for resource-bounded agents. InProceedings of AAMAS 2016 (pp.644–652). ACM.
    [Google Scholar]
  7. Balbiani, P., Fernandez-Duque, D., & Lorini, E.
    (2019) The dynamics of epistemic attitudes in resource-bounded agents. Studia Logica, 107(3), 457–488. 10.1007/s11225‑018‑9798‑4
    https://doi.org/10.1007/s11225-018-9798-4 [Google Scholar]
  8. Baron-Cohen, S., Leslie, A. M., & Frith, U.
    (1985) Does the autistic child have a “Theory of Mind” ?Cognition, 21 (1), 37–46. 10.1016/0010‑0277(85)90022‑8
    https://doi.org/10.1016/0010-0277(85)90022-8 [Google Scholar]
  9. Bolander, T., Dissing, L., & Herrmann, N.
    (2021) DEL-based epistemic planning for human-robot collaboration: Theory and implementation. InM. Bienvenu, G. Lakemeyer, & E. Erdem (Eds.), Proceedings of the 18th international conference on principles of knowledge representation and reasoning, KR 2021, online event, november 3–12, 2021 (pp.120–129). 10.24963/kr.2021/12
    https://doi.org/10.24963/kr.2021/12 [Google Scholar]
  10. Bordini, R. H., Braubach, L., Dastani, M., El Fallah-Seghrouchni, A., Gómez-Sanz, J. J., Leite, J., … Ricci, A.
    (2006) A survey of programming languages and platforms for multi-agent systems. Informatica (Slovenia), 30 (1), 33–44.
    [Google Scholar]
  11. Bordini, R. H., Fisher, M., Visser, W., & Wooldridge, M.
    (2006) Verifying multi-agent programs by model checking. Autonomous Agents and Multi-Agent Systems, 12(2), 239–256. 10.1007/s10458‑006‑5955‑7
    https://doi.org/10.1007/s10458-006-5955-7 [Google Scholar]
  12. Bordini, R. H., & Hübner, J. F.
    (2005) BDI agent programming in AgentSpeak using Jason (tutorial paper). InF. Toni & P. Torroni (Eds.), Computational logic in multi-agent systems, 6th international workshop, CLIMA VI, revised selected and invited papers (Vol. 3900 of Lecture Notes in Computer Science, pp. 143–164). Springer. 10.1007/11750734_9
    https://doi.org/10.1007/11750734_9 [Google Scholar]
  13. Bosse, T., Memon, Z. A., & Treur, J.
    (2011) A recursive BDI agent model for theory of mind and its applications. Applied Artificial Intelligence, 25(1), 1–44. 10.1080/08839514.2010.529259
    https://doi.org/10.1080/08839514.2010.529259 [Google Scholar]
  14. Bösser, T.
    (2001) Autonomous agents. InN. J. Smelser & P. B. Baltes (Eds.), International encyclopedia of the social & behavioral sciences. Science Direct, Elsevier. 10.1016/B0‑08‑043076‑7/00534‑9
    https://doi.org/10.1016/B0-08-043076-7/00534-9 [Google Scholar]
  15. Calegari, R., Ciatto, G., Mascardi, V., & Omicini, A.
    (2021) Logic-based technologies for multi-agent systems: a systematic literature review. Auton. Agents Multi Agent Syst., 35(1), 1. 10.1007/s10458‑020‑09478‑3
    https://doi.org/10.1007/s10458-020-09478-3 [Google Scholar]
  16. Carlsson, M., & Mildner, P.
    (2012) SICStus Prolog – the first 25 years. Theory Pract. Log. Program., 12(1–2), 35–66. 10.1017/S1471068411000482
    https://doi.org/10.1017/S1471068411000482 [Google Scholar]
  17. Clark, K., & Robinson, P.
    (2021) Qulog/TeleoR home page. (staff.itee.uq.edu.au/pjr/HomePages/QulogHome.html)
    [Google Scholar]
  18. Costantini, S.
    (2011) Defining and maintaining agent’s experience in logical agents. InProceedings of LANMR 2011, seventh Latin-American workshop on non-monotonic reasoning (Vol. 804 of CEUR Workshop Proceedings, p. 151–165). CEUR-WS.org
    [Google Scholar]
  19. (2015) ACE: a flexible environment for complex event processing in logical agents. InM. Baldoni, L. Baresi, & M. Dastani (Eds.), Engineering multi-agent systems, third intl. works., EMAS 2015, revised selected papers (Vol. 9318 of Lecture Notes in Computer Science). Springer. 10.1007/978‑3‑319‑26184‑3_5
    https://doi.org/10.1007/978-3-319-26184-3_5 [Google Scholar]
  20. (2022) Ensuring trustworthy and ethical behaviour in intelligent logical agents. J. Log. Comput., 32(2), 443–478. 10.1093/logcom/exab091
    https://doi.org/10.1093/logcom/exab091 [Google Scholar]
  21. Costantini, S., & De Gasperis, G.
    (2018) Flexible goal-directed agents’ behavior via DALI MASs and ASP modules. In2018 AAAI spring symposia. AAAI Press.
    [Google Scholar]
  22. Costantini, S., De Gasperis, G., Dyoub, A., & Pitoni, V.
    (2018) Trustworthiness and safety for intelligent ethical logical agents via interval temporal logic and runtime self-checking. InAAAI spring symposia, Stanford university. AAAI Press.
    [Google Scholar]
  23. Costantini, S., De Gasperis, G., & Migliarini, P.
    (2019) Multi-agent system engineering for emphatic human-robot interaction. In2019 IEEE second international conference on artificial intelligence and knowledge engineering (AIKE) (pp.36–42). 10.1109/AIKE.2019.00015
    https://doi.org/10.1109/AIKE.2019.00015 [Google Scholar]
  24. Costantini, S., De Gasperis, G., & Nazzicone, G.
    (2017) DALI for cognitive robotics: Principles and prototype implementation. InY. Lierler & W. Taha (Eds.), Practical aspects of declarative languages – 19th international symposium, proceedings (Vol. 10137 of Lecture Notes in Computer Science, pp. 152–162). Springer. 10.1007/978‑3‑319‑51676‑9_10
    https://doi.org/10.1007/978-3-319-51676-9_10 [Google Scholar]
  25. Costantini, S., De Gasperis, G., Pitoni, V., & Salutari, A.
    (2017) DALI: A multi agent system framework for the web, cognitive robotic and complex event processing. InD. Monica, A. Murano, S. Rubin, & L. Sauro (Eds.), Proceedings of ICTCS’17 and CILC’17 (Vol. 1949 of CEUR Workshop Proceedings, pp. 286–300). CEUR-WS.org
    [Google Scholar]
  26. Costantini, S., De Lauretis, L., Ferri, C., Giancola, J., & Persia, F.
    (2021) A smart health assistant via DALI logical agents. InS. Monica & F. Bergenti (Eds.), Proceedings of the 36th Italian conference on computational logic, 2021 (Vol. 3002 of CEUR Workshop Proceedings, pp. 173–187). CEUR-WS.org
    [Google Scholar]
  27. Costantini, S., De Lauretis, L., & Persia, F.
    (2022a) Intelligent agents and complex event processing to enhance patient monitoring. InM. Alviano & A. Pieris (Eds.), Proceedings Datalog 2.0 2022, co-located with LPNMR 2022. CEUR-WS.org
    [Google Scholar]
  28. (2022b) An intelligent ecosystem to improve patient monitoring using wearables and artificial intelligence. InR. Calegari & G. Ciatto (Eds.), Proceedings of CILC 2022, 37th Italian conference on computational logic. CEUR-WS.org
    [Google Scholar]
  29. Costantini, S., Dell’Acqua, P., & Pereira, L. M.
    (2011) Conditional learning of rules and plans by knowledge exchange in logical agents. InN. Bassiliades, G. Governatori, & A. Paschke (Eds.), Rule-based reasoning, programming, and applications – proceedings of 5th international symposium, RuleML 2011 (Vol. 6826 of Lecture Notes in Computer Science, pp. 250–265). Springer. 10.1007/978‑3‑642‑22546‑8_20
    https://doi.org/10.1007/978-3-642-22546-8_20 [Google Scholar]
  30. Costantini, S., Dyoub, A., & Pitoni, V.
    (2018) Towards humanized ethical intelligent agents: the role of reflection and introspection. InP. Felli & M. Montali (Eds.), Proceedings of the 33rd Italian conference on computational logic (Vol. 2214 of CEUR Workshop Proceedings, pp. 82–96). CEUR-WS.org. Retrieved fromceur-ws.org/Vol-2214/paper10.pdf
    [Google Scholar]
  31. Costantini, S., Formisano, A., & Pitoni, V.
    (2021a) An epistemic logic for modular development of multi-agent systems. InN. Alechina, M. Baldoni, & B. Logan (Eds.), Engineering multi-agent systems – 9th international workshop, EMAS 2021, revised selected papers (Vol. 13190 of Lecture Notes in Computer Science, pp. 72–91). Springer. 10.1007/978‑3‑030‑75775‑5_8
    https://doi.org/10.1007/978-3-030-75775-5_8 [Google Scholar]
  32. (2021b) An epistemic logic for multi-agent systems with budget and costs. InW. Faber, G. Friedrich, M. Gebser, & M. Morak (Eds.), Logics in artificial intelligence – 17th European conference, JELIA 2021, proceedings (Vol. 12678 of Lecture Notes in Computer Science, pp. 101–115). Springer. 10.1007/978‑3‑030‑75775‑5_8
    https://doi.org/10.1007/978-3-030-75775-5_8 [Google Scholar]
  33. (2021c) A logic of inferable in multi-agent systems with budget and costs. InF. Dignum, A. Lomuscio, U. Endriss, & A. Nowé (Eds.), AAMAS ’21: 20th international conference on autonomous agents and multiagent systems, virtual event, UK, 2021 (pp.1483–1485). ACM. 10.5555/3463952.3464133
    https://doi.org/10.5555/3463952.3464133 [Google Scholar]
  34. (2022a) Cooperation among groups of agents in the epistemic logic L-DINF. InG. Governatori & A.-Y. Turhan (Eds.), Rules and reasoning – 6th international joint conference on rules and reasoning, Rule ML+RR 2022, Berlin, Germany, 2022, proceedings (Vol. 13752 of Lecture Notes in Computer Science). Springer. 10.1007/978‑3‑031‑21541‑4_18
    https://doi.org/10.1007/978-3-031-21541-4_18 [Google Scholar]
  35. (2022b) Modelling agents roles in the epistemic logic L-DINF. InO. Arieli, G. Casini, & L. Giordano (Eds.), Proceedings of the 20th international workshop on non-monotonic reasoning, NMR 2022, part of the federated logic conference FLoC 2022, Haifa, Israel, 2022 (Vol. 3197 of CEUR Workshop Proceedings, pp. 70–79). CEUR-WS.org
    [Google Scholar]
  36. (2022c) Temporalizing epistemic logic L-DINF. InR. Calegari, G. Ciatto, & A. Omicini (Eds.), Proceedings of the 37th Italian conference on computational logic, Bologna, Italy, 2022 (Vol. 3204 of CEUR Workshop Proceedings, pp. 119–133). CEUR-WS.org
    [Google Scholar]
  37. Costantini, S., & Pitoni, V.
    (2020) Towards a logic of “inferable” for self-aware transparent logical agents. InC. Musto, D. Magazzeni, S. Ruggieri, & G. Semeraro (Eds.), Proceedings of XAI.it@AIxIA 2020 (Vol. 2742 of CEUR Workshop Proceedings, pp. 68–79). CEUR-WS.org. Retrieved fromceur-ws.org/Vol-2742/paper6.pdf
    [Google Scholar]
  38. Costantini, S., & Tocchio, A.
    (2002) A logic programming language for multi-agent systems. InS. Flesca, S. Greco, N. Leone, & G. Ianni (Eds.), Logics in artificial intelligence, European conference, JELIA 2002, proceedings (Vol. 2424 of Lecture Notes in Computer Science, pp. 1–13). Springer. 10.1007/3‑540‑45757‑7_1
    https://doi.org/10.1007/3-540-45757-7_1 [Google Scholar]
  39. (2004) The DALI logic programming agent-oriented language. InJ. J. Alferes & J. A. Leite (Eds.), Logics in artificial intelligence, 9th European conference, JELIA 2004, proceedings (Vol. 3229 of Lecture Notes in Computer Science, pp. 685–688). Springer. 10.1007/978‑3‑540‑30227‑8_57
    https://doi.org/10.1007/978-3-540-30227-8_57 [Google Scholar]
  40. (2005) About declarative semantics of logic-based agent languages. InM. Baldoni, U. Endriss, A. Omicini, & P. Torroni (Eds.), Declarative agent languages and technologies III, third international workshop, DALT 2005, selected and revised papers (Vol. 3904 of Lecture Notes in Computer Science, pp. 106–123). Springer. 10.1007/11691792_7
    https://doi.org/10.1007/11691792_7 [Google Scholar]
  41. Costantini, S., Tocchio, A., & Verticchio, A.
    (2005) Communication and trust in the DALI logic programming agent-oriented language. Intelligenza Artificiale, 2 (1), 39–46. (Journal of the Italian Association AI*IA)
    [Google Scholar]
  42. Dastani, M., van Riemsdijk, M. B., & Meyer, J. C.
    (2005) Programming multi-agent systems in 3APL. InMulti-agent programming (Vol. 15 of Multiagent Systems Artificial Societies and Simulated Organizations, pp. 39–67). Springer. 10.1007/0‑387‑26350‑0_2
    https://doi.org/10.1007/0-387-26350-0_2 [Google Scholar]
  43. De Gasperis, G., Costantini, S., & Nazzicone, G.
    (2014, July). DALI multi agent systems framework. DALI GitHub Software Repository. (DALI: github.com/AAAI-DISIM-UnivAQ/DALI) 10.5281/zenodo.11042
    https://doi.org/10.5281/zenodo.11042 [Google Scholar]
  44. de Weerd, H., Verbrugge, R., & Verheij, B.
    (2022) Higher-order theory of mind is especially useful in unpredictable negotiations. Auton. Agents Multi Agent Syst., 36(1), 30. 10.1007/s10458‑022‑09558‑6
    https://doi.org/10.1007/s10458-022-09558-6 [Google Scholar]
  45. Dennis, L. A.
    (2018) The MCAPL framework including the agent infrastructure layer an agent Java pathfinder. J. of Open Source Software, 3 (24), 617. 10.21105/joss.00617
    https://doi.org/10.21105/joss.00617 [Google Scholar]
  46. Dennis, L. A., Bentzen, M. M., Lindner, F., & Fisher, M.
    (2021) Verifiable machine ethics in changing contexts. InThirty-fifth AAAI conference on artificial intelligence, AAAI 2021, thirty-third conference on innovative applications of artificial intelligence, IAAI 2021, the eleventh symposium on educational advances in artificial intelligence, EAAI 2021 (pp.11470–11478). AAAI Press. 10.1609/aaai.v35i13.17366
    https://doi.org/10.1609/aaai.v35i13.17366 [Google Scholar]
  47. Dennis, L. A., Fisher, M., Lincoln, N., Lisitsa, A., & Veres, S. M.
    (2016) Practical verification of decision-making in agent-based autonomous systems. Autom. Softw. Eng., 23(3), 305–359. 10.1007/s10515‑014‑0168‑9
    https://doi.org/10.1007/s10515-014-0168-9 [Google Scholar]
  48. Dissing, L., & Bolander, T.
    (2020) Implementing theory of mind on a robot using dynamic epistemic logic. InC. Bessiere (Ed.), Proceedings of IJCAI 2020 (pp.1615–1621). ijcai.org. 10.24963/ijcai.2020/224
    https://doi.org/10.24963/ijcai.2020/224 [Google Scholar]
  49. Dorri, A., Kanhere, S. S., & Jurdak, R.
    (2018) Multi-agent systems: A survey. IEEE Access, 61, 28573–28593. 10.1109/ACCESS.2018.2831228
    https://doi.org/10.1109/ACCESS.2018.2831228 [Google Scholar]
  50. Duží, M., & Menšík, M.
    (2017) Logic of inferable knowledge. InFrontiers in Artificial Intelligence and Applications, Information Modelling and Knowledge Bases XXVIII, Volume 292. IOS Press.
    [Google Scholar]
  51. Dyoub, A., Costantini, S., Letteri, I., & Lisi, F. A.
    (2021) A logic-based multi-agent system for ethical monitoring and evaluation of dialogues. InA. Formisano (Eds.), Proceedings of the 37th international conference on logic programming (technical communications) (Vol. 345 of Electronic Proceedings in Theoretical Computer Science, pp. 182–188). 10.4204/EPTCS.345.32
    https://doi.org/10.4204/EPTCS.345.32 [Google Scholar]
  52. Dyoub, A., Costantini, S., & Lisi, F. A.
    (2019) Towards ethical machines via logic programming. InB. Bogaerts (Eds.), Proceedings of the 35th international conference on logic programming (technical communications) (Vol. 306 of Electronic Proceedings in Theoretical Computer Science, pp. 333–339). 10.4204/EPTCS.306.39
    https://doi.org/10.4204/EPTCS.306.39 [Google Scholar]
  53. Ferrando, A., Dennis, L. A., Ancona, D., Fisher, M., & Mascardi, V.
    (2018) Verifying and validating autonomous systems: Towards an integrated approach. InC. Colombo & M. Leucker (Eds.), Runtime verification, 18th international conference, RV 2018, proceedings (Vol. 11237 of Lecture Notes in Computer Science, pp. 263–281). Springer. 10.1007/978‑3‑030‑03769‑7_15
    https://doi.org/10.1007/978-3-030-03769-7_15 [Google Scholar]
  54. Ferrando, A., Winikoff, M., Cranefield, S., Dignum, F., & Mascardi, V.
    (2019) On enactability of agent interaction protocols: Towards a unified approach. InL. A. Dennis, R. H. Bordini, & Y. Lespérance (Eds.), Engineering multi-agent systems – 7th international workshop, EMAS 2019, revised selected papers (Vol. 12058 of Lecture Notes in Computer Science, pp. 43–64). Springer. 10.1007/978‑3‑030‑51417‑4_3
    https://doi.org/10.1007/978-3-030-51417-4_3 [Google Scholar]
  55. Ferrario, A., & Loi, M.
    (2022) How explainability contributes to trust in AI. InFacct ’22: 2022 ACM conference on fairness, accountability, and transparency (pp.1457–1466). ACM. 10.1145/3531146.3533202
    https://doi.org/10.1145/3531146.3533202 [Google Scholar]
  56. Fisher, M., Bordini, R. H., Hirsch, B., & Torroni, P.
    (2007) Computational logics and agents: a road map of current technologies and future trends. Computational Intelligence Journal, 23(1), 61–91. 10.1111/j.1467‑8640.2007.00295.x
    https://doi.org/10.1111/j.1467-8640.2007.00295.x [Google Scholar]
  57. Garro, A., Mühlhäuser, M., Tundis, A., Baldoni, M., Baroglio, C., Bergenti, F., & Torroni, P.
    (2019) Intelligent agents: Multi-agent systems. InS. Ranganathan, M. Gribskov, K. Nakai, & C. Schonbach (Eds.), Encyclopedia of bioin formatics and computational biology – Vol. 1 (pp.315–320). Elsevier. 10.1016/B978‑0‑12‑809633‑8.20328‑2
    https://doi.org/10.1016/B978-0-12-809633-8.20328-2 [Google Scholar]
  58. Goldman, A. I.
    (2012) Theory of mind. InE. Margolis, R. Samuels, & S. P. Stich (Eds.), The Oxford handbook of philosophy of cognitive science (Vol. 11). Oxford University Press. 10.1093/oxfordhb/9780195309799.003.0017
    https://doi.org/10.1093/oxfordhb/9780195309799.003.0017 [Google Scholar]
  59. Hindriks, K. V., van der Hoek, W., & Meyer, J. C.
    (2012) GOAL agents instantiate intention logic. InLogic programs, norms and action (Vol. 7360 of Lecture Notes in Computer Science, pp. 196–219). Springer. 10.1007/978‑3‑642‑29414‑3_11
    https://doi.org/10.1007/978-3-642-29414-3_11 [Google Scholar]
  60. Holzmann, G. J.
    (1991) Design and validation of computer protocols. Prentice Hall Intl.: Hemel Hempstead, England.
    [Google Scholar]
  61. Jones, A. V., & Lomuscio, A.
    (2010) Distributed BDD-based BMC for the verification of multi-agent systems. InW. van der Hoek, G. A. Kaminka, Y. Lespérance, M. Luck, & S. Sen (Eds.), 9th international conference on autonomous agents and multiagent systems (AAMAS 2010) (pp.675–682). IFAAMAS.
    [Google Scholar]
  62. Kacprzak, M., Lomuscio, A., & Penczek, W.
    (2004) Verification of multiagent systems via unbounded model checking. InProceedings of the third int. joint conf. on autonomous agents and multiagent systems, AAMAS 04 (p.638–645). ACM Press.
    [Google Scholar]
  63. Kong, J., & Lomuscio, A.
    (2017) Symbolic model checking multi-agent systems against CTL*K specifications. InK. Larson, M. Winikoff, S. Das, & E. H. Durfee (Eds.), Proceedings of the 16th conference on autonomous agents and multiagent systems, AAMAS 2017 (pp.114–122). ACM.
    [Google Scholar]
  64. Lloyd, J. W.
    (1987) Foundations of logic programming, second edition. Berlin: Springer. 10.1007/978‑3‑642‑83189‑8
    https://doi.org/10.1007/978-3-642-83189-8 [Google Scholar]
  65. Lomuscio, A., Lasica, T., & Penczek, W.
    (2002) Bounded model checking for interpreted systems: Preliminary experimental results. InM. G. Hinchey, J. L. Rash, W. Truszkowski, C. A. Rouff, & D. F. Gordon-Spears (Eds.), Formal approaches to agent-based systems, 2nd international workshop, FAABS 2002, revised papers (Vol. 2699 of Lecture Notes in Computer Science, pp. 115–125). Springer. 10.1007/978‑3‑540‑45133‑4_10
    https://doi.org/10.1007/978-3-540-45133-4_10 [Google Scholar]
  66. Lomuscio, A., Qu, H., & Raimondi, F.
    (2017) MCMAS: an open-source model checker for the verification of multi-agent systems. Int. J. Softw. Tools Technol. Transf., 19(1), 9–30. 10.1007/s10009‑015‑0378‑x
    https://doi.org/10.1007/s10009-015-0378-x [Google Scholar]
  67. Martinich, A. P.
    (2009) The philosophy of language. Oxford University Press. (International Fifth Edition)
    [Google Scholar]
  68. McMillan, K. L.
    (1993) Symbolic model checking. Kluwer Academic Publishers. 10.1007/978‑1‑4615‑3190‑6
    https://doi.org/10.1007/978-1-4615-3190-6 [Google Scholar]
  69. Panisson, A. R., Sarkadi, S., McBurney, P., Parsons, S., & Bordini, R. H.
    (2018) On the formal semantics of theory of mind in agent communication. InM. Lujak (Ed.), Agreement technologies – 6th international conference, AT 2018, revised selected papers (Vol. 11327 of Lecture Notes in Computer Science, pp. 18–32). Springer.
    [Google Scholar]
  70. Rao, A. S., & Georgeff, M. P.
    (1991) Modeling rational agents within a BDI-architecture. InJ. F. Allen, R. Fikes, & E. Sandewall (Eds.), Proceedings of the 2nd international conference on principles of knowledge representation and reasoning (KR 91) (pp.473–484). Morgan Kaufmann.
    [Google Scholar]
  71. Reisenzein, R., Hudlicka, E., Dastani, M., Gratch, J., Hindriks, K. V., Lorini, E., & Meyer, J. C.
    (2013) Computational modeling of emotion: Toward improving the inter- and intradisciplinary exchange. IEEE Trans. Affect. Comput., 4 (3), 246–266. 10.1109/T‑AFFC.2013.14
    https://doi.org/10.1109/T-AFFC.2013.14 [Google Scholar]
  72. Rozier, K. Y.
    (2011) Linear temporal logic symbolic model checking. Comput. Sci. Rev., 5 (2), 163–203. 10.1016/j.cosrev.2010.06.002
    https://doi.org/10.1016/j.cosrev.2010.06.002 [Google Scholar]
  73. (2016) Specification: The biggest bottleneck in formal methods and autonomy. InS. Blazy & M. Chechik (Eds.), Verified software. theories, tools, and experiments – 8th international conference, VSTTE 2016, revised selected papers (Vol. 9971 of Lecture Notes in Computer Science, pp. 8–26). 10.1007/978‑3‑319‑48869‑1_2
    https://doi.org/10.1007/978-3-319-48869-1_2 [Google Scholar]
  74. Rozier, K. Y., & Vardi, M. Y.
    (2010) LTL satisfiability checking. Int. J. Softw. Tools Technol. Transf., 12(2), 123–137. 10.1007/s10009‑010‑0140‑3
    https://doi.org/10.1007/s10009-010-0140-3 [Google Scholar]
  75. Sarkadi, S., Panisson, A. R., Bordini, R. H., McBurney, P., & Parsons, S.
    (2018) Towards an approach for modelling uncertain theory of mind in multi-agent systems. InM. Lujak (Ed.), Agreement technologies – 6th international conference, AT 2018, revised selected papers (Vol. 11327 of Lecture Notes in Computer Science, pp. 3–17). Springer. 10.1007/978‑3‑030‑17294‑7_1
    https://doi.org/10.1007/978-3-030-17294-7_1 [Google Scholar]
  76. Stone, P., & Veloso, M. M.
    (2000) Multiagent systems: A survey from a machine learning perspective. Auton. Robots, 8(3), 345–383. 10.1023/A:1008942012299
    https://doi.org/10.1023/A:1008942012299 [Google Scholar]
  77. Tørresen, J., Plessl, C., & Yao, X.
    (2015) Self-aware and self-expressive systems. IEEE Computer, 48(7), 18–20. 10.1109/MC.2015.205
    https://doi.org/10.1109/MC.2015.205 [Google Scholar]
  78. van Ditmarsch, H., Halpern, J. Y., van der Hoek, W., & Kooi, B.
    (2015) Handbook of epistemic logic. College Publications.
    [Google Scholar]
  79. Vardi, M. Y.
    (2001) Branching vs. linear time: Final showdown. InProceedings of the 2001 conf. on tools and algorithms for the construction and analysis of systems, TACAS 2001 (p.1–22). Springer-Verlag. 10.1007/3‑540‑45319‑9_1
    https://doi.org/10.1007/3-540-45319-9_1 [Google Scholar]
  80. Veltman, K., de Weerd, H., & Verbrugge, R.
    (2019) Training the use of theory of mind using artificial agents. J. Multimodal User Interfaces, 13(1), 3–18. 10.1007/s12193‑018‑0287‑x
    https://doi.org/10.1007/s12193-018-0287-x [Google Scholar]
  81. Weyhrauch, R. W.
    (1980) Prolegomena to a theory of mechanized formal reasoning. Artif. Intell., 13(1–2), 133–170. 10.1016/0004‑3702(80)90015‑6
    https://doi.org/10.1016/0004-3702(80)90015-6 [Google Scholar]
  82. Yakouda, M., Abbel, W., Corneille, K. V., & Sinclair, N. D.
    (2020) Multi-agent system: A two-level BDI model integrating theory of mind. International Journal of Engineering Research & Technology (IJERT), 9 (7). Retrieved fromwww.ijert.orgISSN:2278-0181
    [Google Scholar]
  83. Zhao, S., Wang, S., Soleymani, M., Joshi, D., & Ji, Q.
    (2019) Affective computing for large-scale heterogeneous multimedia data: A survey. ACM Transactions on Multimedia Computing, Communications, and Applications, 15(3s), 1–32. 10.1145/3363560
    https://doi.org/10.1145/3363560 [Google Scholar]
  • Article Type: Research Article
Keyword(s): agents and multi-agent systems; epistemic logic; theory of mind

Most Cited

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error