Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2022

Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles

Abstract

Linking neural representations to linguistic factors is crucial in order to build and analyze NLP models interpretable by humans. Among these factors, syntactic roles (e.g. subjects, direct objects,.. .) and their realizations are essential markers since they can be understood as a decomposition of predicative structures and thus the meaning of sentences. Starting from a deep probabilistic generative model with attention, we measure the interaction between latent variables and realizations of syntactic roles, and show that it is possible to obtain, without supervision, representations of sentences where different syntactic roles correspond to clearly identified different latent variables. The probabilistic model we propose is an Attention-Driven Variational Autoencoder (ADVAE). Drawing inspiration from Transformer-based machine translation models, ADVAEs enable the analysis of the interactions between latent variables and input tokens through attention. We also develop an evaluation protocol to measure disentanglement with regard to the realizations of syntactic roles. This protocol is based on attention maxima for the encoder and on disturbing individual latent variables for the decoder. Our experiments on raw English text from the SNLI dataset show that i) disentanglement of syntactic roles can be induced without supervision, ii) ADVAE separates more syntactic roles than classical sequence VAEs, iii) realizations of syntactic roles can be separately modified in sentences by mere intervention on the associated latent variables. Our work constitutes a first step towards unsupervised controllable content generation. The code for our work is publicly available 1 .
Fichier principal
Vignette du fichier
lqVcfT-CTRLGEN (7).pdf (545.74 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03540084 , version 1 (22-01-2022)

Identifiers

  • HAL Id : hal-03540084 , version 1

Cite

Ghazi Felhi, Joseph Le Roux, Djamé Seddah. Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles. CtrlGen: Controllable Generative Modeling in Language and Vision, Jan 2022, virtual, France. ⟨hal-03540084⟩
97 View
125 Download

Share

Gmail Facebook X LinkedIn More