Understanding Scikit-Learn Estimators: fit, transform, fit_transform and predict

ByJordan Brown

Scikit-learn is the most widely used Python library for machine learning, thanks to its consistent API, versatile set of tools, and helpful documentation. When working with scikit-learn, understanding how its estimator objects work is key to using the library effectively. Most estimators implement four key methods: fit(), transform(), fit_transform(), and predict(). In this article, we‘ll dive into what each of these methods do, when to use them, and how they work together in a typical machine learning workflow. We‘ll focus particularly on fit_transform() and how it can simplify the process of fitting and applying preprocessing transformations.

The fit() Method

The fit() method is used for learning parameters from training data. Many scikit-learn estimators accept data in their constructor, but don‘t actually do anything with that data until fit() is called. The fit() method takes training data, learns parameters from that data, and returns the estimator object with the learned parameters:

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(penalty=‘l2‘, C=1.0) 
model.fit(X_train, y_train)

After calling fit(), the model object will have learned its parameters (coefficients and intercept) from the training features X_train and labels y_train. Hyperparameters like the regularization type and strength are specified when the estimator is constructed.

Internally, fit() does different things depending on the type of estimator:

  • For supervised models like LogisticRegression, it will optimize the model parameters to minimize a loss function on the training data
  • For unsupervised models like KMeans, it will learn the cluster centroids from the training data
  • For preprocessing transformers like StandardScaler, it will compute statistics like the mean and standard deviation of the training data

But in each case, the purpose is to learn parameters from the provided training data. The fit() method is only ever called on training data, never on test data.

The transform() Method

The transform() method is used for applying a transformation to data using previously learned parameters. Scikit-learn transformers (like scalers, normalizers, encoders) implement transform() to modify input data:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
scaler.fit(X_train)

X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

Here, fit() is first called on the training data X_train to compute the mean and standard deviation of each feature. Then transform() is called on both X_train and X_test to standardize the data using the learned mean and standard deviation.

The key point is that transform() does not learn anything from the data it is called on – it only applies a transformation using previously learned parameters. The learning is done by calling fit() first. transform() is used on both training and test data.

Internally, most transformers work on NumPy arrays or SciPy sparse matrices. Sparse input will typically be preserved as sparse output. For example, here is how StandardScaler handles dense NumPy input vs sparse CSR matrix input:

from sklearn.preprocessing import StandardScaler
import numpy as np
from scipy.sparse import csr_matrix

dense_data = np.array([[0, 1], 
                       [2, 3]])

sparse_data = csr_matrix(dense_data)

scaler = StandardScaler()

# Standardize dense data
dense_scaled = scaler.fit_transform(dense_data)
print(dense_scaled.toarray())
# Output:
# [[-1.  -1. ]
#  [ 1.   1. ]]

# Standardize sparse data  
sparse_scaled = scaler.fit_transform(sparse_data)
print(type(sparse_scaled))
# Output: 
# <class ‘scipy.sparse.csr.csr_matrix‘>

The dense NumPy array is transformed to another dense NumPy array, while the sparse CSR matrix is transformed to another sparse CSR matrix. Preserving sparsity can save a lot of memory when working with large, high-dimensional sparse datasets, which are common in applications like natural language processing with bag-of-words or TF-IDF features.

The fit_transform() Method

Many scikit-learn transformers provide a fit_transform() method which is equivalent to calling fit() and transform() in succession on the same input data:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()

X_train_scaled = scaler.fit_transform(X_train)

Using fit_transform() is a convenience to avoid redundantly specifying the input data. But it‘s important to only call fit_transform() on training data, never test data. If we used fit_transform() on the test data, the transformer would learn the mean and standard deviation of the test data, leading to a form of data leakage:

# Don‘t do this!
X_test_scaled = scaler.fit_transform(X_test)

The test data should always be transformed using parameters learned from the training data, never its own parameters. In a typical machine learning workflow using a transformer, we fit and transform the training data using fit_transform(), then transform the test data separately using transform() with the already fit transformer:

from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# Scale data
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) 
X_test_scaled = scaler.transform(X_test)

# Train model 
model = LogisticRegression()
model.fit(X_train_scaled, y_train)  

# Evaluate model
model.score(X_test_scaled, y_test)

The transformer learns parameters from the training data, which are then used to transform both the training and test data identically. The model is trained on the transformed training data and evaluated on the transformed test data. Keeping the train/test split consistent throughout fitting and evaluating the model is crucial.

fit_transform() is also useful for transformers that change the dimensionality or type of the input data, like TF-IDF vectorization for text. TF-IDF represents documents as sparse vectors where each element corresponds to the importance of a word:

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    ‘This is the first document.‘,
    ‘This document is the second document.‘,
    ‘And this is the third one.‘,
    ‘Is this the first document?‘,
]

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)

print(X.shape)
# Output: (4, 9)

The vectorizer is fit to the training corpus and transforms the text to a sparse matrix of TF-IDF features in a single step by calling fit_transform(). The same vectorizer can then be used to transform new documents at test time or in production using transform().

When using a transformer inside a scikit-learn Pipeline (discussed below), fit_transform() will automatically be called on the training data for each transformer step before fitting the final estimator. For an example of why this is important, consider a model with a text vectorizer followed by a classifier:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

model = make_pipeline(CountVectorizer(), LogisticRegression())

model.fit(X_train, y_train)

When fit() is called on the pipeline, it will fit_transform() the CountVectorizer on the training text X_train, then fit() the LogisticRegression on the vectorized training text. The test text will only be vectorized at prediction time. This ensures the model never sees the test data during training, even just to build the vocabulary.

The predict() Method

Once an estimator is fit, we use the predict() method to make predictions on new data. For supervised learning models, predict() takes input features and returns the predicted labels or values:

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

The model is first fit on the training data X_train and y_train, then used to predict labels for the test features X_test. Calling predict() on the training data X_train would give an overly optimistic evaluation, since the model has already seen that data.

For transformers and other estimators that don‘t make predictions, predict() is usually not implemented. These estimators typically use transform() to modify input data at test time instead of making predictions.

Scikit-Learn Pipelines

A pipeline chains together multiple estimators (usually transformers and a final predictor) into a single estimator object. This makes it easy to apply a series of transformations followed by a model fit in one step. Here‘s an example pipeline that standardizes numeric features, one-hot encodes categorical features, and fits a random forest:

from sklearn.compose import make_column_transformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline

preprocessor = make_column_transformer(
    (StandardScaler(), [‘age‘, ‘fare‘]),
    (OneHotEncoder(handle_unknown=‘ignore‘), [‘embarked‘, ‘sex‘]),
)

model = make_pipeline(preprocessor, RandomForestClassifier())

model.fit(X_train, y_train)
y_pred = model.predict(X_test)

When fit() is called on the pipeline, it will:

  1. Call fit_transform() on each transformer in the ColumnTransformer to learn their parameters and transform the training data
  2. Call fit() on the RandomForestClassifier to train it on the transformed training data

At prediction time, calling predict() on the pipeline will:

  1. Call transform() on each transformer to transform the test data
  2. Call predict() on the RandomForestClassifier to make predictions on the transformed test data

Using a pipeline ensures that fit_transform() is only ever called on the training data, while transform() is called on the test data, avoiding any data leakage issues. It also makes the code more readable by abstracting away the details of the transformations.

Cross-Validation

When doing cross-validation, it‘s important that any transformations are done inside the CV loop to avoid data leakage between folds. With scikit-learn, this means using pipelines and fitting them repeatedly on each training fold, instead of fitting transformers and models separately:

from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

model = make_pipeline(StandardScaler(), LogisticRegression())

cv_scores = cross_val_score(model, X, y, cv=5)

Inside each fold of 5-fold CV, the pipeline will be refit from scratch on the training folds using fit_transform() to scale the training portion, then evaluated on the held out validation fold scaled with transform(). This gives a valid estimate of model performance, while fitting the scaler separately on all of X before doing CV would let the model see distributional information about the validation folds during training, leading to overfitting.

Real-World Usage and Benefits

Scikit-learn‘s fit_transform() method and pipelines may seem like just conveniences, but they are actually crucial abstractions for avoiding data leakage and building robust machine learning models. Without fit_transform(), it would be easy to accidentally fit a transformer on the test data, allowing the model to cheat by learning about the distribution of the test set.

And without pipelines, even if we were careful to fit the transformers only on the training data, we would have to keep track of each step, making sure to call fit_transform() on the training data and transform() on the test data for each transformer before fitting the final model. This gets unwieldy fast, especially if we have a more complex series of transformations.

In practice, essentially any nontrivial machine learning model will require some form of preprocessing like scaling, encoding categorical variables, feature extraction, or dimensionality reduction. So understanding fit_transform() and pipelines is key to being effective with scikit-learn.

Some real-world use cases of pipelines and fit_transform():

  • Kaggle competitions: Almost all winning solutions use extensive, carefully engineered pipelines of transformers fit with fit_transform() to extract useful features and avoid leakage
  • Anomaly detection: Density-based anomaly detection models like Local Outlier Factor need to be fit on standardized data, which can be done safely in a pipeline with fit_transform()
  • Customer churn prediction: Typically involves transforming raw customer attributes and interactions into useful predictors using fit_transform(), then training a binary classifier
  • Image classification: Convolutional neural networks expect standardized pixel values, which can be scaled properly with fit_transform() in a preprocessing pipeline before flowing into the network

Overall, fit_transform() and pipelines make it easy to apply a series of transformations to raw data, giving us a clean, informative representation to train powerful models on, all while preventing data leakage. Along with scikit-learn‘s wide variety of estimators, they form the backbone of the library.

Conclusion

Scikit-learn‘s estimator API, based on fit(), transform(), fit_transform() and predict(), is one of the library‘s greatest strengths. Understanding how these methods work together allows you to leverage scikit-learn‘s huge variety of models and transformers safely and effectively. Some key takeaways:

  • fit() learns model parameters from training data
  • transform() applies a transformation to data using learned parameters
  • fit_transform() fits and transforms in one step, but should only be used on training data to avoid leakage
  • predict() makes predictions on new data using a fit model
  • Pipelines automate the process of fitting and transforming through a series of steps

With these tools, you can build robust, powerful machine learning models for a variety of real-world applications. Scikit-learn‘s careful API design makes it easy to do the right thing, but it‘s still important to be aware of the distinctions between fitting, transforming, and predicting to use the library to its full potential.

References

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

You May Like to Read,

Similar Posts