Frugal Gaussian clustering of huge imbalanced datasets through a bin-marginal approach - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Preprints, Working Papers, ... Year : 2021

Frugal Gaussian clustering of huge imbalanced datasets through a bin-marginal approach

Filippo Antonazzo
  • Function : Author
  • PersonId : 1086416
Christophe Biernacki
  • Function : Author
  • PersonId : 923939

Abstract

Clustering conceptually reveals all its interest when the dataset size considerably increases since there is the opportunity to discover tiny but possibly high value clusters which were out of reach with more modest sample sizes. However, clustering is practically faced to computer limits with such high data volume, since possibly requiring extremely high memory and computation resources. In addition, the classical subsampling strategy, often adopted to overcome these limitations, is expected to heavily failed for discovering clusters in the highly imbalanced cluster case. Our proposal first consists in drastically compressing the data volume by just preserving its bin-marginal values, thus discarding the bin-cross ones. Despite this extreme information loss, we then prove identifiability property for the diagonal mixture model and also introduce a specific EM-like algorithm associated to a composite likelihood approach. This latter is extremely more frugal than a regular but unfeasible EM algorithm expected to be used on our bin-marginal data, while preserving all consistency properties. Finally, numerical experiments highlight that this proposed method outperforms subsampling both in controlled simulations and in various real applications where imbalanced clusters may typically appear, such as image segmentation, hazardous asteroids recognition and fraud detection.
Fichier principal
Vignette du fichier
FrugalGaussianClusteringABK.pdf (4.18 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03485364 , version 1 (17-12-2021)

Identifiers

  • HAL Id : hal-03485364 , version 1

Cite

Filippo Antonazzo, Christophe Biernacki, Christine Keribin. Frugal Gaussian clustering of huge imbalanced datasets through a bin-marginal approach. 2021. ⟨hal-03485364⟩
65 View
66 Download

Share

Gmail Mastodon Facebook X LinkedIn More