Indexing and Searching 100M Images with Map-Reduce - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2013

Indexing and Searching 100M Images with Map-Reduce

Denis Shestakov
  • Function : Author
  • PersonId : 937626
Gylfi Thór Gudmundsson
  • Function : Author
  • PersonId : 890913
Laurent Amsaleg


Most researchers working on high-dimensional indexing agree on the following three trends: (i) the size of the multimedia collections to index are now reaching millions if not billions of items, (ii) the computers we use every day now come with multiple cores and (iii) hardware becomes more available, thanks to easier access to Grids and/or Clouds. This paper shows how the Map-Reduce paradigm can be applied to indexing algorithms and demonstrates that great scalability can be achieved using Hadoop, a popular Map-Reduce-based framework. Dramatic performance improvements are not however guaranteed a priori: such frameworks are rigid, they severely constrain the possible access patterns to data and scares resource RAM has to be shared. Furthermore, algorithms require major redesign, and may have to settle for sub-optimal behavior. The benefits, however, are many: simplicity for programmers, automatic distribution, fault tolerance, failure detection and automatic re-runs and, last but not least, scalability. We share our experience of adapting a clustering-based high-dimensional indexing algorithm to the Map-Reduce model, and of testing it at large scale with Hadoop as we index 30 billion SIFT descriptors. We foresee that lessons drawn from our work could minimize time, effort and energy invested by other researchers and practitioners working in similar directions.
Fichier principal
Vignette du fichier
icmr115-moise.pdf (155.31 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-00796475 , version 1 (04-03-2013)


  • HAL Id : hal-00796475 , version 1


Diana Moise, Denis Shestakov, Gylfi Thór Gudmundsson, Laurent Amsaleg. Indexing and Searching 100M Images with Map-Reduce. ACM International Conference on Multimedia Retrieval, Apr 2013, Dallas, United States. ⟨hal-00796475⟩
465 View
5100 Download


Gmail Facebook X LinkedIn More