Adapting Language Models When Training on Privacy-Transformed Data - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2021

Adapting Language Models When Training on Privacy-Transformed Data


In recent years, voice-controlled personal assistants have revolutionized the interaction with smart devices and mobile applications. These dialogue tools are then used by system providers to improve and retrain the language models (LMs). Each spoken message reveals personal information, hence, it is necessary to remove the private data from the input utterances. However, this may harm the LM training because privacy-transformed data is unlikely to match the test distribution. This paper aims to fill the gap by focusing on the adaptation of LM initially trained on privacy-transformed utterances. Our data sanitization process relies on named-entity recognition. We propose an LM adaptation strategy over the private data with minimum losses. Class-based modeling is an effective approach to overcome data sparsity in the context of n-gram model training. On the other hand, neural LMs can handle longer contexts which can yield better predictions. Our methodology combines the predictive power of class-based models and the generalization capability of neural models together. With privacy transformation, we have a relative 11% word error rate (WER) increase compared to an LM trained on the clean data. Despite the privacy-preserving, we can still achieve comparable accuracy. Empirical evaluations attain a relative WER improvement of 8% over the initial model.
Fichier principal
Vignette du fichier
Paper_1854.pdf (121.75 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03189354 , version 1 (03-04-2021)
hal-03189354 , version 2 (08-05-2022)


  • HAL Id : hal-03189354 , version 1


Mehmet Ali Tugtekin Turan, Dietrich Klakow, Emmanuel Vincent, Denis Jouvet. Adapting Language Models When Training on Privacy-Transformed Data. LREC 2022 - 13th Language Resources and Evaluation Conference, Aug 2021, Brno, Czech Republic. ⟨hal-03189354v1⟩
224 View
446 Download


Gmail Facebook X LinkedIn More