Multitask Prompted Training Enables Zero-Shot Task Generalization - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2022

Multitask Prompted Training Enables Zero-Shot Task Generalization

Victor Sanh
  • Function : Author
Colin Raffel
  • Function : Author
Lintang Sutawika
  • Function : Author
Arnaud Stiegler
  • Function : Author
Teven Le Scao
  • Function : Author
Manan Dey
  • Function : Author
Urmish Thakker
  • Function : Author
Shanya Sharma
  • Function : Author
Eliza Szczechla
  • Function : Author
Gunjan Chhablani
  • Function : Author
Jonathan Chang
  • Function : Author
Mike Tian-Jian Jiang
  • Function : Author
Zheng-Xin Yong
  • Function : Author
Harshit Pandey
  • Function : Author
Michael Mckenna
  • Function : Author
Trishala Neeraj
  • Function : Author
Thibault Fevry
  • Function : Author
Tali Bers
  • Function : Author
Leo Gao
  • Function : Author
Thomas Wolf
  • Function : Author
Alexander M. Rush
  • Function : Author

Abstract

Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learning in language models’ pretraining (Radford et al., 2019). Can zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we develop a system for easily mapping any natural language tasks into a human-readable prompted form. We convert a large set of supervised datasets, each with multiple prompts with diverse wording. These prompted datasets allow for benchmarking the ability of a model to perform completely held-out tasks. We fine-tune a pre-trained encoder-decoder model (Raffel et al., 2020; Lester et al., 2021) on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models up to 16x its size. Further, our approach attains strong performance on a subset of tasks from the BIG-bench benchmark, outperforming models up to 6x its size. All trained models are available at https://github.com/bigscience-workshop/t-zero, and all prompts are available at https://github.com/bigscience-workshop/promptsource.
Fichier principal
Vignette du fichier
2110.08207.pdf (1.99 Mo) Télécharger le fichier
2110.08207 (1).pdf (1.99 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03540072 , version 1 (10-01-2023)

Identifiers

  • HAL Id : hal-03540072 , version 1

Cite

Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, et al.. Multitask Prompted Training Enables Zero-Shot Task Generalization. ICLR 2022 - Tenth International Conference on Learning Representations, Apr 2022, Online, Unknown Region. ⟨hal-03540072⟩
530 View
171 Download

Share

Gmail Facebook X LinkedIn More