Multimodal space representation driven by self-evaluation of predictability - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2014

Multimodal space representation driven by self-evaluation of predictability


PROPRE is a generic and modular neural learning paradigm that autonomously extracts meaningful concepts of multimodal data flows driven by predictability across modalities in an unsupervised, incremental and online way. For that purpose, PROPRE consists of the combination of projection and prediction. Firstly, each data flow is topologically projected with a self-organizing map, largely inspired from the Kohonen model. Secondly, each projection is predicted by each other map activities, by mean of linear regressions. The main originality of PROPRE is the use of a simple and generic predictability measure that compares predicted and real activities for each modal stream. This measure drives the corresponding projection learning to favor the mapping of predictable stimuli across modalities at the system level (i.e. that their predictability measure overcomes some threshold). This predictability measure acts as a self-evaluation module that tends to bias the representations extracted by the system so that to improve their correlations across modalities. We already showed that this modulation mechanism is able to bootstrap representation extraction from previously learned representations with artificial multimodal data related to basic robotic behaviors and improves performance of the system for classification of visual data within a supervised learning context. In this article, we improve the self-evaluation module of PROPRE, by introducing a sliding threshold, and apply it to the unsupervised classification of gestures caught from two time- of-flight (ToF) cameras. In this context, we illustrate that the modulation mechanism is still useful although less efficient than purely supervised learning.
Fichier principal
Vignette du fichier
finalarticle.pdf (1.2 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-01061668 , version 1 (12-09-2014)


  • HAL Id : hal-01061668 , version 1


Mathieu Lefort, Thomas Kopinski, Alexander Gepperth. Multimodal space representation driven by self-evaluation of predictability. Joint IEEE International Conference Developmental Learning and Epigenetic Robotics (ICDL-EPIROB), Oct 2014, Gênes, Italy. ⟨hal-01061668⟩
238 View
196 Download


Gmail Facebook X LinkedIn More