What Is Information Leakage in Machine Learning?

What is information leakage?

Information (or data) leakage is undesired behavior in machine learning during which information that should not be in the training data inflates the model’s ability to learn, causing poor performance at prediction time or in production. Models subject to information leakage do not generalize well to unseen data.

There are multiple types of data leakage, including:

Explicit leakage, in which some features represent transformation of or a proxy for the target variable. This can happen during the pre-processing or feature engineering phase when some transformations are blindly applied to both training and testing data.
Independent and identically distributed, or i.i.d., samples. i.i.d (). For time-series problems, this can happen when data is randomly split and some training examples occur after the testing observations, also known as implicit leakage. Under these conditions, data available to a model could potentially include some future information that the model would not have access to in a real-world scenario.

Diagram comparing two train/test data-splitting methods along a time axis. The top bar, "Temporal Split," is one continuous blue Training Data region followed by a single orange Testing Data region, so all test data comes after all training data. The bottom bar, "Random Split," alternates many small blue and orange segments across the timeline, with curved arrows pointing from training segments into nearby testing segments to indicate information leakage between them.

Why is information leakage important?

Avoiding or detecting information leakage early is important to prevent models from learning the wrong signals and overestimating their value before they go into production. In addition to following data science best practices, model interpretability is a great tool to identify and fight information leakage.

How C3 AI helps organizations detect and avoid information leakage

At C3 AI, data scientists are well-versed in information leakage problems and how to detect them. C3 AI carefully splits the data into separate groups – training, validation, and test sets – and keeps the test set intact to report the final performance after the model has been optimized on the validation set. For time-series data, C3 AI applications always apply a cut-off timestamp or time-series cross-validation.

Diagram titled "Walk-Forward Optimization for Time-Series" showing four stacked, progressively right-shifted bars. Each bar has a long blue "Training" segment followed by a short orange "Test" segment (folds #1 through #4), so each fold trains on earlier data and tests on the next period. Below runs a left-and-right time axis with tick marks; vertical arrows drop from each fold's test segment down into a single orange "Projected Performance" bar, indicating that the out-of-sample test results combine into the projected performance.

C3 AI applications meticulously track the data lineage from raw data all the way to engineered variables to ensure no detrimental information is leaking. In the C3 AI Reliability application, for example, all the features used to predict a failure are derived from data observed and logged before the failure event log time.

A horizontal timeline bar split in two: a green "usable" section on the left and a pink "not usable" section on the right, sitting on a rightward "time" axis arrow. A downward black triangle labeled "moment of prediction" marks the boundary — data before the moment of prediction is usable, data after it is not.

The C3 AI Platform leverages several interpretability frameworks, such as SHAP (SHapley Additive exPlanation) and ELI5 (Explain Like I’m 5), during development of AI systems to detect information leakage by examining feature contributions.