Towards a Robust Detection of Language Model-Generated Text: Is ChatGPT that easy to detect? - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

Towards a Robust Detection of Language Model-Generated Text: Is ChatGPT that easy to detect?

Abstract

Recent advances in natural language processing (NLP) have led to the development of large language models (LLMs) such as ChatGPT. This paper proposes a methodology for developing and evaluating ChatGPT detectors for French text, with a focus on investigating their robustness on out-of-domain data and against common attack schemes. The proposed method involves translating an English dataset into French and training a classifier on the translated data. Results show that the detectors can effectively detect ChatGPT-generated text, with a degree of robustness against basic attack techniques in in-domain settings. However, vulnerabilities are evident in out-of-domain contexts, highlighting the challenge of detecting adversarial text. The study emphasizes caution when applying in-domain testing results to a wider variety of content. We provide our translated datasets and models as open-source resources.
Fichier principal
Vignette du fichier
461938.pdf (228.73 Ko) Télécharger le fichier
Origin : Publisher files allowed on an open archive

Dates and versions

hal-04130146 , version 1 (20-06-2023)

Identifiers

  • HAL Id : hal-04130146 , version 1

Cite

Wissam Antoun, Virginie Mouilleron, Benoît Sagot, Djamé Seddah. Towards a Robust Detection of Language Model-Generated Text: Is ChatGPT that easy to detect?. 18e Conférence en Recherche d'Information et Applications -- 16e Rencontres Jeunes Chercheurs en RI -- 30e Conférence sur le Traitement Automatique des Langues Naturelles -- 25e Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues, Jun 2023, Paris, France. pp.14-27. ⟨hal-04130146⟩
66 View
32 Download

Share

Gmail Facebook X LinkedIn More