Fit KDE with tuned bandwidth

As a machine learning practitioner, one of the most common and frustrating challenges you‘ll face is insufficient training data. We‘ve all been there – you‘re excited to build a cutting-edge predictive model, but your dataset is small, incomplete, or lacks diversity. What should you do?

Having too little data to effectively train a model is problematic because machine learning algorithms rely on learning patterns and relationships from many example observations. Without enough representative samples, models tend to memorize noise instead of uncovering true underlying patterns. This often manifests as:

Overfitting – When a model performs great on its training data but poorly on new data, indicating it failed to generalize
Underfitting – When a model is too simple to capture the complexity of the data, leading to high bias and poor predictive power

Insufficient data is especially common when dealing with rare events, underrepresented classes/groups, or when data collection is difficult and expensive (e.g. medical images). But don‘t despair – there are proven strategies to squeeze the most out of a limited dataset! Let‘s walk through four powerful techniques and how to implement them.

1. Simplify Your Model Architecture

With big data, we have the luxury of building large, complex models with millions of parameters to capture intricate patterns. But with small datasets, simpler is often better. Complex models are prone to overfitting on insufficient data by essentially "memorizing" training samples instead of learning generalizable concepts.

The solution is to reduce model complexity by:

• Decreasing the number of parameters, layers, neurons, etc.
• Using linear models, naïve Bayes, or decision trees/forests instead of deep neural nets
• Applying strong regularization to constrain learned representations

Here‘s an example of training a simple logistic regression model in Python using scikit-learn:

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
logreg = LogisticRegression()

logreg.fit(X_train, y_train)
y_pred = logreg.predict(X_test)

We can further improve performance by combining multiple weaker models into a more robust ensemble. Techniques like voting, bagging, and boosting allow us to train diverse models and aggregate their predictions. Even if each individual learner is limited, the "wisdom of the crowd" can yield impressive results.

from sklearn.ensemble import VotingClassifier
estimators = [
(‘logistic‘, LogisticRegression()),
(‘decision_tree‘, DecisionTreeClassifier()), 
(‘svm‘, SVC())
]
ensemble = VotingClassifier(estimators)
ensemble.fit(X_train, y_train)  

2. Leverage Transfer Learning

Another powerful approach, especially for computer vision and NLP tasks, is to leverage the hard work already done by others. Transfer learning allows us to take a pre-trained model (usually on a large dataset) and adapt it to our own domain and task.

The intuition is that many datasets share common low-level features or concepts. For example, a conv net originally trained on ImageNet to classify objects like cats and cars has already learned to detect useful visual patterns like edges, shapes, and textures. We can extract those feature representations and fine-tune the model on our small dataset, giving us a huge head start compared to training from scratch.

Some popular sources of pre-trained models include:

• Keras and PyTorch model zoos
• Hugging Face Transformers library for NLP
• Model repositories like TensorFlow Hub or PyTorch Hub

To perform transfer learning, we typically remove the original output layer of the pre-trained model and replace it with a new one matched to our dataset‘s classes. We then freeze most of the model‘s weights and fine-tune only the new output layer (and optionally some of the top layers).

import tensorflow as tf
import tensorflow_hub as hub
model = tf.keras.Sequential([
hub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/feature_vector/4", 
input_shape=(224,224,3),
trainable=False),

tf.keras.layers.Dense(num_classes, activation=‘softmax‘)
])
model.compile(...)
history = model.fit(...)

Transfer learning enables us to achieve great results with orders of magnitude less data than the original model required. However, it does assume there is some overlap between the source and target domains. Performance may suffer if our data is radically different from what the model was originally trained on.

3. Augment Your Training Data

Data augmentation is a sneaky trick to artificially expand a small dataset by programmatically generating new samples from existing ones. The goal is to increase the quantity and diversity of training data without actually collecting new samples.

Some common augmentation techniques for images include:

• Flipping (horizontal/vertical)
• Rotating
• Scaling
• Cropping
• Translating
• Adjusting brightness/contrast
• Adding noise

As long as the transformations preserve the core content of the image, the model can learn to be more robust by seeing many variations of each sample. Just be careful not to go overboard and create data that is unrealistic or unlikely to occur in the real world.

Augmenting images in Keras is simple using the ImageDataGenerator class:

from tensorflow.keras.preprocessing.image import ImageDataGenerator
datagen = ImageDataGenerator(
rotation_range=20,
width_shift_range=0.1,
height_shift_range=0.1,
brightness_range=[0.8,1.2], 
horizontal_flip=True,
zoom_range=0.1
)
datagen.fit(X_train)
history = model.fit(
datagen.flow(X_train, y_train, batch_size=32),
validation_data=(X_test, y_test),
epochs=50,
verbose=2
)

For tabular data and time series, augmentation is less visual but still valuable. Simple techniques include:

• Adding small amounts of noise
• Interpolating between samples
• Decomposing periodic series into seasonal components

Data augmentation provides an easy way to get more mileage out of limited data. While it‘s no substitute for real samples, it can significantly improve generalization and combat overfitting when data is scarce. However, always be cognizant of maintaining realistic data distributions with your augmentations.

4. Generate Synthetic Data

Our final tactic may feel like cheating, but it‘s a proven approach for situations where real-world data is limited or difficult to obtain. Synthetic data generation creates artificial samples from scratch, allowing us to expand datasets while preserving their format and statistical properties.

The concept is to fit a statistical model on a small dataset, then use that model to generate new samples that resemble the original data distribution. Some methods include:

• Multivariate Gaussian distributions for continuous features
• Markov chains for categorical/sequential data
• Generative adversarial networks (GANs) for media like images, audio, and video

Synthetic data enable us to create many realistic samples to augment or even replace real data. It provides an alternative in domains with high costs or privacy concerns around data collection and sharing. However, the quality of synthetic data will depend heavily on the realism of the generating model – unrealistic samples can potentially make downstream models perform worse.

Here‘s an example of fitting a kernel density estimate (KDE) on the Olivetti faces dataset, then sampling synthetic faces from it:

from sklearn.datasets import fetch_olivetti_faces
from sklearn.neighbors import KernelDensity
from sklearn.model_selection import GridSearchCV
faces = fetch_olivetti_faces()

params = {‘bandwidth‘: np.arange(0.01, 1, 0.05)}
grid_search = GridSearchCV(KernelDensity(), params)
grid_search.fit(faces.data)
kde = grid_search.bestestimator

synthetic_faces = kde.sample(8)

Synthetic data generation is still an active area of research, especially for complex, high-dimensional domains. While not perfect, it can be a lifesaver when real data is scarce. Always evaluate the quality of synthetic samples before training downstream models.

Putting It All Together

We covered four powerful techniques for dealing with insufficient data in machine learning – simplifying models, transfer learning, data augmentation, and synthetic data generation. While I presented them independently, they can absolutely be combined for even greater effect!

For example, you could simplify a model architecture while also transfer learning from a pre-trained checkpoint. Or augment real samples with synthetic ones. The optimal approach will always depend on your unique situation and constraints.

Some additional tips:

• Always question if you have exhausted options for collecting more real data before turning to techniques like synthetic generation
• Experiment with different approaches and combinations to see what works best for your data and model
• Look to the literature and ML community for cutting-edge techniques in data-constrained domains like medicine, autonomous driving, and robotics

At the end of the day, the goal is to extract maximum performance from minimum data. Insufficient data is a common challenge but it doesn‘t have to be a showstopper. By leveraging the strategies we covered, you can build robust models that generalize well to the real world.

I hope this deep dive has been informative and gotten your gears turning! What other techniques have you used for small datasets? Share your war stories in the comments below. Until next time, happy modeling!