Clustering-Based Inter-group Correlation Estimation - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2022

Clustering-Based Inter-group Correlation Estimation

Abstract

In this work, we propose a novel non-parametric estimator of the correlation between groups of variables with arbitrary intra-group dependence and in the presence of noise. The challenge resides in the fact that both noise and intra-group correlation can lead to inconsistent correlation estimation using classical approaches. Previous works handle either one or the other but fail to tackle both at the same time. To address this problem, we propose to fully utilize the dependency structures of the data. To that end, we transform the data to a space where the Euclidean distance can be used as a proxy for the sample correlation. We then leverage hierarchical clustering to simultaneously offset the effects of noise and intra-group correlation. We prove our estimator is both consistent and unbiased for an appropriate cut-off height of the dendogram. We also empirically show our approach surpasses state-of-the-art estimators in terms of quality and provide illustrations on real-world datasets.
No file

Dates and versions

hal-03867463 , version 1 (23-11-2022)

Identifiers

  • HAL Id : hal-03867463 , version 1

Cite

Hanâ Lbath, Alexander Petersen, Wendy Meiring, Sophie Achard. Clustering-Based Inter-group Correlation Estimation. ICSDS 2022- IMS International Conference on Statistics and Data Science, Dec 2022, Florence, Italy. ⟨hal-03867463⟩
104 View
0 Download

Share

Gmail Mastodon Facebook X LinkedIn More