Using Predictive Power Score to Pinpoint Non-linear Correlations
As an artificial intelligence and machine learning expert, I‘m always on the lookout for powerful tools to help make sense of complex, messy real-world data. One surprisingly underutilized technique I‘ve come to appreciate is predictive power score (PPS), a way to detect predictive relationships between variables that can handily surface non-linear correlations that other methods miss.
Developed by data scientists at 8080 Labs, PPS is designed as an alternative to standard correlation coefficients like Pearson‘s r or Spearman‘s rho. While these tried-and-true correlations are great for quantifying the strength of linear relationships, they can completely whiff on non-linear patterns that are incredibly common in real data.
PPS works by fitting a machine learning model to predict one variable from another, then comparing that model‘s performance to a naive baseline. The score ranges from 0 to 1, with 0 meaning the model does no better than the baseline (indicating no predictive power) and 1 meaning the model achieves perfect prediction. By default, PPS uses a decision tree regressor or classifier under the hood, but this can be swapped out for other models as needed.
Mathematically, PPS is defined as:
$$
PPS(X, Y) = 1 – \frac{score(model_{X \rightarrow Y})}{score(baseline_Y)}
$$
where $score$ is a model performance metric like mean squared error or F1 score, $model_{X \rightarrow Y}$ is a model trained to predict Y from X, and $baseline_Y$ is a naive model that always predicts the median of Y (for regression) or the most frequent class of Y (for classification).
To make PPS scores more comparable across different problems, the library also supports normalized PPS, which rescales the score by the performance range of a perfect model vs a naive model:
$$
PPS_{norm}(X, Y) = \frac{score(baselineY) – score(model{X \rightarrow Y})}{score(baseline_Y) – score(perfect_Y)}
$$
One of the standout features of PPS is that it is asymmetric – the predictive power of X for Y can be quite different than the predictive power of Y for X. This is in stark contrast to standard correlation coefficients which are always symmetric. The asymmetry of PPS makes it more reflective of the directional, imbalanced relationships we often care about in the real world.
For example, consider a simple nonlinear relationship:
import numpy as np
import pandas as pd
# Generate example data
df = pd.DataFrame()
df["x"] = np.random.uniform(-2, 2, 10000)
df["error"] = np.random.uniform(-0.5, 0.5, 10000)
df["y"] = df["x"]**2 + df["error"]Here the variable x ranges from -2 to 2 while y is calculated as x^2 plus random noise. If we calculate the Pearson correlation between x and y, we get a value very close to 0:
df["x"].corr(df["y"]) 0.0014This implies that there is basically no linear relationship at all between x and y, which we know is misleading! In contrast, the normalized PPS of x predicting y is:
from ppscore import score
score(df, "x", "y"){‘ppscore‘: 0.6518, ‘case‘: ‘regression‘, ‘is_valid_score‘: True,
‘metric‘: ‘mean absolute error‘, ‘baseline_score‘: 1.0012,
‘model_score‘: 0.3479, ‘model‘: DecisionTreeRegressor()}With a PPS of 0.65, the predictive power of x for y is quite high, correctly identifying the strong quadratic relationship between them. However, if we calculate the PPS in the other direction:
score(df, "y", "x") {‘ppscore‘: 0.0, ‘case‘: ‘regression‘, ‘is_valid_score‘: True,
‘metric‘: ‘mean absolute error‘, ‘baseline_score‘: 1.1551,
‘model_score‘: 1.1557, ‘model‘: DecisionTreeRegressor()}The PPS plummets to 0! This reflects the fact that while x can predict y very well through the quadratic equation, y cannot uniquely predict x since each y value maps to two possible x values thanks to the symmetry of x^2. The asymmetry of PPS mirrors this real-world asymmetry in a way correlation simply cannot.
Another major advantage of PPS is that it seamlessly handles both numeric and categorical variables, while correlation is only defined for numeric variables. Behind the scenes, PPS uses different model performance metrics depending on the datatypes – mean squared error or mean absolute error for numeric targets and F1 score or similar for categorical targets.
To see PPS in action on a real dataset, let‘s calculate the PPS matrix for the classic Titanic survival data:
from ppscore import matrix
titanic_df = pd.read_csv("titanic.csv")
matrix(titanic_df)This generates a PPS matrix indicating the predictive power of each variable for every other variable:
| x | y | ppscore |
|---|---|---|
| Age | Fare | 0.4113 |
| Age | Parch | 0.1607 |
| Age | Pclass | 0.3353 |
| … | … | … |
| Pclass | Survived | 0.2195 |
| Sex | Survived | 0.5358 |
| SibSp | Survived | 0.0154 |
Visualizing this matrix as a heatmap makes it easy to quickly identify strong predictive relationships:
import seaborn as sns
pps_matrix = matrix(titanic_df)[["x", "y", "ppscore"]]
pps_matrix = pps_matrix.pivot(index="x", columns="y", values="ppscore")
sns.heatmap(pps_matrix, vmin=0, vmax=1, cmap="Blues", linewidths=0.5, annot=True)
Right away, some interesting patterns pop out. We see that the Sex variable has by far the highest predictive power for Survived with a PPS over 0.53. This reflects the well-known fact that women were more likely to survive the Titanic disaster than men.
More surprisingly, we notice that the Fare column, representing ticket price, has very high predictive power for many other variables like Pclass (passenger class), Embarked (port of embarkation) and even Survived. With a PPS of 0.42 for Survived, Fare is almost as predictive as Sex! This suggests that ticket price encoded a lot of information about a passenger‘s wealth and social status which in turn influenced their survival outcomes.
Digging deeper, we see that Fare also has moderately high mutual predictive power with less obvious variables like Parch (number of parents/children aboard). It seems unlikely that ticket price would directly affect family structure, so there may be some latent variables like wealth or nationality that explain both. Extracting PPS subscores for individual classes could help identify these confounding relationships.
Another variable with intriguingly high predictive power in the Titanic dataset is the passenger‘s ticket number. Ticket has a PPS over 0.5 for predicting Fare, Cabin, Embarked and Parch. What could ticket number possibly tell us about a passenger?
To investigate, we can group the dataframe by ticket number and spot check some of the high PPS relationships:
titanic_df.groupby("ticket").agg(
avg_fare=("fare", "mean"),
num_family=("parch", "mean"),
class_mode=("pclass", lambda x: x.mode())
)| ticket | avg_fare | num_family | class_mode |
|---|---|---|---|
| 110152 | 86.50 | 0.00 | 3 |
| 110413 | 108.90 | 2.00 | 1 |
| 110465 | 52.00 | 1.00 | 1 |
| 110564 | 263.00 | 3.00 | 1 |
| … | … | … | … |
Inspecting individual tickets, a pattern emerges – the same ticket number often refers to groups of passengers from the same family who booked together, especially in first and second class. So the ticket number actually encodes useful information about family structure and group membership that would be lost looking at each passenger individually!
For a final example, let‘s look at the asymmetric predictive relationship between Age and Pclass (passenger class). The PPS matrix shows that Pclass predicts Age with a score of 0.33, while Age predicts Pclass with only 0.16. This means that knowing a passenger‘s class gives you more information about their likely age than vice versa.
We can validate this asymmetry by plotting the distributions directly:
sns.histplot(data=titanic_df, x="age", hue="pclass", multiple="stack")
Looking at the distribution of age within each passenger class, we see that 1st class skews significantly older while 3rd class skews younger. So if we know a passenger was in 1st class, we could reasonably infer they were probably middle aged or older, while a 3rd class ticket suggests a younger passenger. But if we only knew a passenger‘s age, it would be much harder to guess their class, especially for passengers in their 20s or 30s who were spread across all three classes.
This is a great example of how an asymmetric predictive relationship can arise from an underlying asymmetric data generating process. It also demonstrates how visualizing the actual distributions can help confirm and explain the results we see in the PPS matrix.
So as we‘ve seen, predictive power score is a powerful and versatile technique for uncovering both linear and non-linear relationships between variables in a dataset. By computing a PPS matrix and combining it with other exploratory visualizations, we can quickly build up a detailed understanding of the predictive structure of our data.
This understanding can then directly inform our next analysis steps. We might use PPS to guide feature selection by identifying and combining predictive variable sets. We could improve model performance by dropping low PPS variables that only add noise. Asymmetric PPS pairs could reveal potential target leakage we need to investigate. And high PPS cliques could suggest latent structures or concepts we may want to extract.
It‘s important to note that while PPS is a very useful tool, it‘s not a complete replacement for other techniques. Linear correlations are still important and often more easily interpretable than PPS. And certain complex non-linear relationships may require more powerful techniques like mutual information, maximal information coefficient, or Kraskov estimation to reveal.
Nonetheless, I believe predictive power score deserves a place in every data scientist‘s toolbox. Especially in the early stages of exploring a new dataset, it can quickly surface unexpected patterns and relationships that could otherwise remain hidden.
Some current limitations of PPS include its relatively slow computation on large datasets (since it requires fitting many models), and its opaque scores which can sometimes obscure the actual shape of the discovered relationships. There is also interesting research to be done on extracting PPS subscores for individual classes or clusters to find predictive relationships unique to certain subgroups.
As the field of data science and machine learning continues to evolve, I‘m excited to see how techniques like predictive power score can help us scale human intuition and pattern recognition to larger and more complex datasets. By taking full advantage of the power of both linear and non-linear relationships, we can build richer understanding to solve harder real-world problems. The future is bright at the intersection of statistics and machine learning!
