Communication Dans Un Congrès Année : 2025

bench-MIA: Towards automatic multi-lever benchmark construction for MIA evaluation

Résumé

The lack of transparency in large language models (LLMs) training data has created concerns regarding unauthorized data use. Membership Inference Attacks (MIA) offer a method for detecting potential copyright violations and privacy breaches by determining whether specific data points were included during training. However, existing MIA methods face reliability and generalization challenges due to biases in validation datasets. Since various LLMs are trained on different, often proprietary datasets, creating a single universal dataset and evaluation framework is not feasible. In this work-in-progress paper, we introduce bench-MIA, an approach that automatically constructs tailored benchmarks for MIA evaluation. Bench-MIA leverages the concept of lever to systematically generate evaluation datasets addressing key sources of bias. By providing appropriate evaluation baselines for each dataset, bench-MIA facilitates more accurate and fair MIA assessments. Preliminary experimental results demonstrate the impact of these levers on MIA evaluation and motivate further validation.

Fichier non déposé

Dates et versions

hal-05287192 , version 1 (28-09-2025)

Identifiants

  • HAL Id : hal-05287192 , version 1

Citer

Yanming Li, Cédric Eichler, Nicolas Anciaux, Héber H. Arcolezi, José Maria de Fuentes. bench-MIA: Towards automatic multi-lever benchmark construction for MIA evaluation. APVP 2025 - 15ème Atelier sur la protection de la vie privée, Jun 2025, Chasseneuil-du-Poitou, France. ⟨hal-05287192⟩
188 Consultations
0 Téléchargements

Partager

  • More