Learning Semantic Components from Subsymbolic Multimodal Perception - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2013

Learning Semantic Components from Subsymbolic Multimodal Perception

Olivier Mangin
  • Function : Author
  • PersonId : 884109
Pierre-Yves Oudeyer


Perceptual systems often include sensors from several modalities. However, existing robots do not yet sufficiently discover patterns that are spread over the flow of multimodal data they receive. In this paper we present a framework that learns a dictionary of words from full spoken utterances, together with a set of gestures from human demonstrations and the semantic connection between words and gestures. We explain how to use a nonnegative matrix factorization algorithm to learn a dictionary of components that represent meaningful elements present in the multimodal perception, without providing the system with a symbolic representation of the semantics. We illustrate this framework by showing how a learner discovers word-like components from observation of gestures made by a human together with spoken descriptions of the gestures, and how it captures the semantic association between the two.
Fichier principal
Vignette du fichier
mangin.2013.icdl.pdf (433.77 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-00842453 , version 1 (08-07-2013)


  • HAL Id : hal-00842453 , version 1


Olivier Mangin, Pierre-Yves Oudeyer. Learning Semantic Components from Subsymbolic Multimodal Perception. Joint IEEE International Conference on Development and Learning an on Epigenetic Robotics (ICDL-EpiRob), Aug 2013, Osaka, Japan. ⟨hal-00842453⟩
156 View
429 Download


Gmail Mastodon Facebook X LinkedIn More