What does data leakage in model outputs mean and how can it occur in practice?

• A. It means training data or sensitive attributes are unintentionally revealed through predictions or model behavior, e.g., memorization, overfitting, or side channels.
• B. It means data leakage is prevented by differential privacy.
• C. It only refers to leaking model hyperparameters to competitors.
• D. It is when the model outputs random noise to confuse attackers.

Answer: (B)

Data leakage through model outputs happens when the model’s predictions reveal details about the training data or sensitive attributes that should stay private. This can occur if the model memorizes training examples because it is overfit, so exact or near-exact data from the training set show up in responses to inputs that resemble those examples. It can also arise from target leakage during data preparation, where features inadvertently contain information that should not be available or that would only be known after the fact, causing the model to leak that information through its outputs. In practice, attackers may perform membership inference to guess whether a data record was in the training set by examining predictions or confidence scores, or exploit side channels such as exact probabilities, response timing, or explanations that reveal details about the data the model learned. To reduce leakage, techniques like differential privacy add controlled noise to training or outputs to bound what can be inferred, while other defenses include regularization to curb memorization, careful data curation to remove sensitive attributes, data minimization, and auditing or restricting the granularity of model outputs.