Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2018

Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction

Résumé

Current state-of-the-art machine translation systems are based on encoder-decoder architectures, that first encode the input sequence, and then generate an output sequence based on the input encoding. Both are interfaced with an attention mechanism that recombines a fixed encoding of the source tokens based on the decoder state. We propose an alternative approach which instead relies on a single 2D convolutional neural network across both sequences. Each layer of our network re-codes source tokens on the basis of the output sequence produced so far. Attention-like properties are therefore pervasive throughout the network. Our model yields excellent results, outperforming state-of-the-art encoder-decoder systems, while being conceptually simpler and having fewer parameters.
Fichier principal
Vignette du fichier
paper.pdf (1.66 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01851612 , version 1 (10-08-2018)
hal-01851612 , version 2 (03-09-2018)
hal-01851612 , version 3 (04-09-2018)

Identifiants

  • HAL Id : hal-01851612 , version 3

Citer

Maha Elbayad, Laurent Besacier, Jakob Verbeek. Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction. CoNLL 2018 - Conference on Computational Natural Language Learning, Oct 2018, Brussels, Belgium. pp.97-107. ⟨hal-01851612v3⟩
1012 Consultations
868 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More