Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts

Abstract

Originally introduced as a neural network for ensemble learning, mixture of experts (MoE) has recently become a fundamental building block of highly successful modern deep neural networks for heterogeneous data analysis in several applications, including those in machine learning, statistics, bioinformatics, economics, and medicine. Despite its popularity in practice, a satisfactory level of understanding of the convergence behavior of Gaussian-gated MoE parameter estimation is far from complete. The underlying reason for this challenge is the inclusion of covariates in the Gaussian gating and expert networks, which leads to their intrinsically complex interactions via partial differential equations with respect to their parameters. We address these issues by designing novel Voronoi loss functions to accurately capture heterogeneity in the maximum likelihood estimator (MLE) for resolving parameter estimation in these models. Our results reveal distinct behaviors of the MLE under two settings: the first setting is when all the location parameters in the Gaussian gating are non-zeros while the second setting is when there exists at least one zero-valued location parameter. Notably, these behaviors can be characterized by the solvability of two different systems of polynomial equations. Finally, we conduct a simulation study to verify our theoretical results.

Dates and versions

hal-04125056 , version 1 (11-06-2023)

Licence

Identifiers

Cite

Huy Nguyen, Trungtin Nguyen, Khai Nguyen, Nhat Ho. Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts. AISTATS 2024 - 27th International Conference on Artificial Intelligence and Statistics, May 2024, Valencia, Spain. pp.1-31. ⟨hal-04125056⟩
52 View
0 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More