High-Dimensional Private Empirical Risk Minimization by Greedy Coordinate Descent - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2022

High-Dimensional Private Empirical Risk Minimization by Greedy Coordinate Descent

Résumé

In this paper, we study differentially private empirical risk minimization (DP-ERM). It has been shown that the (worst-case) utility of DP-ERM reduces as the dimension increases. This is a major obstacle to privately learning large machine learning models. In high dimension, it is common for some model's parameters to carry more information than others. To exploit this, we propose a differentially private greedy coordinate descent (DP-GCD) algorithm. At each iteration, DP-GCD privately performs a coordinate-wise gradient step along the gradients' (approximately) greatest entry. We show theoretically that DP-GCD can improve utility by exploiting structural properties of the problem's solution (such as sparsity or quasi-sparsity), with very fast progress in early iterations. We then illustrate this numerically, both on synthetic and real datasets. Finally, we describe promising directions for future work.
Fichier principal
Vignette du fichier
greedy-coordinate-descent.pdf (541.23 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03714465 , version 1 (05-07-2022)
hal-03714465 , version 2 (21-10-2022)
hal-03714465 , version 3 (09-04-2023)

Identifiants

  • HAL Id : hal-03714465 , version 1

Citer

Paul Mangold, Aurélien Bellet, Joseph Salmon, Marc Tommasi. High-Dimensional Private Empirical Risk Minimization by Greedy Coordinate Descent. 2022. ⟨hal-03714465v1⟩
143 Consultations
72 Téléchargements

Partager

Gmail Facebook X LinkedIn More