Clustering-Based Inter-group Correlation Estimation
Abstract
In this work, we propose a novel non-parametric estimator of the correlation between groups of variables with arbitrary intra-group dependence and in the presence of noise. The challenge resides in the fact that both noise and intra-group correlation can lead to inconsistent correlation estimation using classical approaches. Previous works handle either one or the other but fail to tackle both at the same time. To address this problem, we propose to fully utilize the dependency structures of the data. To that end, we transform the data to a space where the Euclidean distance can be used as a proxy for the sample correlation. We then leverage hierarchical clustering to simultaneously offset the effects of noise and intra-group correlation. We prove our estimator is both consistent and unbiased for an appropriate cut-off height of the dendogram. We also empirically show our approach surpasses state-of-the-art estimators in terms of quality and provide illustrations on real-world datasets.