Interaction Studies - Volume 8, Issue 1, 2007

The traditional view of symbol grounding seeks to connect an a priori internal representation or ‘form’ to its external referent. But such a ‘form’ is usually itself systematically composed out of more primitive parts (i.e., it is ‘symbolic’), so this view ignores its grounding in the physics of the world. Some previous work simulating multiple talking/listening agents has effectively taken this stance, and shown how a shared discrete speech code (i.e., vowel system) can emerge. Taking the earlier work of Oudeyer, we have extended his model to include a dispersive force intended to account broadly for a speaker’s motivation to increase auditory distinctiveness. New simulations show that vowel systems result that are more representative of the range seen in human languages. These simulations make many profound abstractions and assumptions. Relaxing these by including more physically and physiologically realistic mechanisms for talking and listening is seen as the key to replicating more complex and dynamic aspects of speech, such as consonant-vowel patterning.

This paper illustrates how external (or social) symbol grounding can be studied in simulations with large populations. We discuss how we can simulate language evolution in a relatively complex environment which has been developed in the context of the New Ties project. This project has the objective of evolving a cultural society and, in doing so, the agents have to evolve a communication system that is grounded in their interactions with their virtual environment and with other individuals. A preliminary experiment is presented in which we investigate the effect of a number of learning mechanisms. The results show that the social symbol grounding problem is a particularly hard one; however, we provide an ideal platform to study this problem.

This paper addresses word learning for human–robot interaction. The focus is on making a robotic agent aware of its surroundings, by having it learn the names of the objects it can find. The human user, acting as instructor, can help the robotic agent ground the words used to refer to those objects. A lifelong learning system, based on one-class learning, was developed (OCLL). This system is incremental and evolves with the presentation of any new word, which acts as a class to the robot, relying on instructor feedback. A novel experimental evaluation methodology, that takes into account the open-ended nature of word learning, is proposed and applied. This methodology is based on the realization that a robot’s vocabulary will be limited by its discriminatory capacity which, in turn, depends on its sensors and perceptual capabilities. The results indicate that the robot’s representations are capable of incrementally evolving by correcting class descriptions, based on instructor feedback to classification results. In successive experiments, it was possible for the robot to learn between 6 and 12 names of real-world office objects. Although these results are comparable to those obtained by other authors, there is a need to scale-up. The limitations of the method are discussed and potential directions for improvement are pointed out.

Taking a distributed view of language, this paper naturalizes symbol grounding. Learning to talk is traced to — not categorizing speech sounds — but events that shape the rise of human-style autonomy. On the extended symbol hypothesis, this happens as babies integrate micro-activity with slow and deliberate adult action. As they discover social norms, intrinsic motive formation enables them to reshape co-action. Because infants link affect to contingencies, dyads develop norm-referenced routines. Over time, infant doings become analysis amenable. The caregiver of a nine-month-old may, for example, prompt the baby to fetch objects. Once she concludes that the baby uses ‘words’ to understand what she says, the infant can use this belief in orienting to more abstract contingencies. New cognitive powers will develop as the baby learns to act in ways that are consistent with a caregiver’s false belief that her baby uses ‘words.’

This paper compares the nascent theory of the ‘semiotic symbol’ in cognitive science with its computational relative. It finds that the semiotic symbol as it is understood in recent practical and theoretical work does not have the resources to explain the role of symbols in cognition. In light of this argument, an alternative model of symbol internalisation, based on Vygotsky, is put forward which goes further in showing how symbols can go from playing intersubjective communicative roles to intrasubjective cognitive ones. Such a formalisation restores the symbol’s cognitive and communicative dimensions to their proper roles.

I present the symbol grounding problem in the larger context of a materialist theory of content and then present two problems for causal, teleo-functional accounts of content. This leads to a distinction between two kinds of mental representations: presentations and symbols; only the latter are cognitive. Based on Milner and Goodale’s dual route model of vision, I posit the existence of precise interfaces between cognitive systems that are activated during object recognition. Interfaces are constructed as a child learns, and is taught, how to interact with its environment; hence, interface structure has a social determinant essential for symbol grounding. Symbols are encoded in the brain to exploit these interfaces, by having projections to the interfaces that are activated by what they symbolise. I conclude by situating my proposal in the context of Harnad’s (1990) solution to the symbol grounding problem and responding to three standard objections.

After reviewing the papers in this special issue, I must conclude that brains are not syntactic engines, but control systems that orient to biological, interindividual, and cultural norms. By themselves, syntactic constraints both underdetermine and overdetermine cognitive operations. So, rather than serving as the basis for general cognition, they are just another kind of empirically acquired constraint. In humans, symbols emerge from a particular sensorimotor activity through a process of contextual broadening that depends on the coordination of conscious and nonconscious processing. This process provides the representational freedom and stability that constitute the human brain’s solution to the frame problem and symbol grounding problem. Symbol formation and grounding is an ongoing process of generalising constraints from particular contexts, selectively enlisting their use, and re-automating them. This process is central to the self-creation of a language-using person with beliefs, agency, and identity.

It has long been asserted that the evolutionary path to spoken language was paved by manual–gestural behaviors, a claim that has been revitalized in response to recent research on mirror neurons. Renewed interest in the relationship between manual and vocal behavior draws attention to its development. Here, the pointing and vocalization of 16.5-month-old infants are reported as a function of the context in which they occurred. When infants operated in a referential mode, the frequency of simultaneous vocalization and pointing exceeded the frequency of vocalization-only and pointing-only responses by a wide margin. In a non-communicative context, combinatorial effects persisted, but in weaker form. Manual–vocal signals thus appear to express the operation of an integrated system, arguably adaptive in the young from evolutionary times to the present. It was speculated, based on reported evidence, that manual behavior increases the frequency and complexity of vocal behaviors in modern infants. There may be merit in the claim that manual behavior facilitated the evolution of language because it helped make available, early in development, behaviors that under selection pressures in later ontogenetic stages elaborated into speech.