bench-MIA: Towards automatic multi-lever benchmark construction for MIA evaluation
Résumé
The lack of transparency in large language models (LLMs) training data has created concerns regarding unauthorized data use. Membership Inference Attacks (MIA) offer a method for detecting potential copyright violations and privacy breaches by determining whether specific data points were included during training. However, existing MIA methods face reliability and generalization challenges due to biases in validation datasets. Since various LLMs are trained on different, often proprietary datasets, creating a single universal dataset and evaluation framework is not feasible. In this work-in-progress paper, we introduce bench-MIA, an approach that automatically constructs tailored benchmarks for MIA evaluation. Bench-MIA leverages the concept of lever to systematically generate evaluation datasets addressing key sources of bias. By providing appropriate evaluation baselines for each dataset, bench-MIA facilitates more accurate and fair MIA assessments. Preliminary experimental results demonstrate the impact of these levers on MIA evaluation and motivate further validation.