6 Challenging Data Science Projects to Become a Better Data Scientist in 2025

Why Data Science Projects Are Essential to Honing Your Skills

As a data scientist, it‘s easy to get caught up in the day-to-day responsibilities of your job and neglect your own learning and development. But in a field that is evolving as rapidly as data science, falling behind on the latest techniques and tools is a sure-fire path to obsolescence.

One of the best ways to keep your skills sharp is by taking on data science projects outside of your regular work. Participating in the open source community and building your own projects is an invaluable way to challenge yourself, explore new areas, and demonstrate your abilities to potential employers or clients.

Open source projects in particular have several advantages:

• The code is freely available to study and adapt for your own use cases
• There is often an active community to learn from and contribute to
• They encompass the end-to-end data science lifecycle from data engineering to deployment
• They leverage cutting-edge tools and architectures
• They tend to be extremely well-documented

In this post, I‘ve curated 6 compelling open source data science projects from 2022-2024 that will help you grow your skills in areas like deep learning, NLP, computer vision, data engineering, graphs, and AI.

While these projects are certainly challenging, I‘ve included links to relevant tutorials and courses for each one to help you bridge the gap. Remember, the goal is to stretch outside your comfort zone – that‘s where real growth happens. Let‘s dive in!

Project 1: Seg-Any – Image Segmentation on Any Category

Image segmentation is a hot area of computer vision research that aims to partition an image into multiple segments or objects. It has applications in everything from autonomous vehicles to medical image diagnosis.

Seg-Any is a state-of-the-art image segmentation model developed by Google Research that can generate accurate segmentations for any visual category, even categories not seen during training. It builds upon prior work in large language models, few-shot learning, and zero-shot transfer to achieve remarkable flexibility and generalization.

Some key features of Seg-Any:

• Based on Transformers, can process both image and text inputs
• Trained on a huge dataset of images and corresponding text captions
• Achieves strong zero-shot performance on unseen categories
• Can perform competitively with just a few examples per category

Another innovation of Seg-Any is its ability to output segmentations in a vector format like SVG that is more compact and interpretable than a raster mask. This allows for interesting applications like example-based image editing.

While the full Seg-Any model is quite large, the GitHub repo includes code and pre-trained weights to run a smaller version. To get the most out of this project, I‘d recommend being familiar with PyTorch, Transformers, and general computer vision and deep learning techniques.

Relevant tutorials and courses:

• A Comprehensive Guide to Convolutional Neural Networks
• Getting Started with PyTorch for Deep Learning
• Stanford CS231n: Convolutional Neural Networks for Visual Recognition

Project 2: DistAug – Data Augmentation Strategies for Text

Data augmentation, or artificially increasing the size of the training dataset, is a widely used technique to improve the performance of deep learning models, especially when labeled data is scarce. While data augmentation has been well-studied for images, it is still an active area of research for text data.

DistAug is a Python library that implements various data augmentation strategies for text classification tasks. It leverages large pre-trained language models like BERT to generate diverse augmented examples that preserve the label of the original example.

Some notable augmentation strategies in DistAug:

• Random word swaps and deletions
• Paraphrasing via back-translation
• Generating plausible examples with a finetuned GPT model
• Mixing and interpolating between examples
• Augmenting the latent space directly

In experiments on several benchmark text classification datasets, the authors show that DistAug can achieve significant improvements, especially in low-resource settings. It also outperforms strong baselines like EDA and UDA.

To make the most of DistAug, you should have a solid foundation in NLP concepts like language models, text embeddings, and data augmentation. Experience with popular NLP libraries like Transformers and Spacy will also be helpful.

Relevant tutorials and courses:

• Natural Language Processing with Deep Learning
• Data Augmentation for NLP: Best Practices from Language Model Pre-training
• Intro to Fine-tuning Pretrained Language Models

Project 3: Merlion – An End-to-End Machine Learning Platform for Time Series

Time series data is ubiquitous across domains like finance, healthcare, and IoT. Despite its importance, many data scientists lack dedicated tools and expertise in time series analysis and forecasting.

Merlion is an open source machine learning platform built specifically for time series developed by researchers at Salesforce. It aims to provide a one-stop shop for common time series tasks such as:

• Anomaly detection
• Forecasting
• Data imputation
• Change point detection

Under the hood, Merlion implements many classical time series analysis techniques as well as state-of-the-art deep learning models like Transformers and Neural ODEs. It also includes utilities for time series feature engineering, model evaluation, and explaining model predictions.

One nice aspect of Merlion is that it has a unified and easy-to-use Python API for all tasks. This allows you to seamlessly mix and match different techniques in your time series pipelines. It also integrates with other popular data science libraries like Pandas, scikit-learn, and PyTorch.

To get started with Merlion, it‘s good to have some background in time series analysis concepts like seasonality, autocorrelation, and evaluation metrics. Experience with Python data manipulation and machine learning libraries will also help you get up to speed quickly.

Relevant tutorials and courses:

• Introduction to Time Series Analysis in Python
• ARIMA Modeling with R
• Forecasting with Prophet by Facebook

Project 4: EvoPrompting – Evolving Better Prompts for Large Language Models

2022 saw an explosion of interest in large language models (LLMs) and their potential for few-shot and zero-shot learning. LLMs like GPT-3 and PaLM have shown remarkable abilities to generalize to new tasks given only a short textual prompt describing the task.

However, designing effective prompts is still largely a manual and error-prone process. The quality of prompts can vary widely, leading to inconsistent model performance.

EvoPrompting is a project that aims to automate prompt engineering by using evolutionary algorithms to search for high-performing prompts. It starts with a set of seed prompts and iteratively improves them through selection, mutation, and crossover operations.

The fitness function used to evaluate prompt quality is based on expected model performance on a held-out validation set. This allows EvoPrompting to optimize for prompts that generalize well rather than just fitting the idiosyncrasies of the training data.

In experiments across a variety of language tasks, EvoPrompting is able to find prompts that outperform hand-crafted and baseline prompts. The evolved prompts are also typically shorter and more concise.

To run and extend EvoPrompting, you should be comfortable with Python and have a good grasp of concepts like language models, prompt engineering, and evolutionary algorithms. Access to compute resources like GPUs will also be important for training and evaluating prompts.

Relevant tutorials and courses:

• Prompt Engineering 101
• Evolutionary Algorithms for Optimization
• Stanford CS224N: Natural Language Processing with Deep Learning

Project 5: CausalNex – Causal Inference with Bayesian Networks

Causal inference, or the problem of determining cause-and-effect relationships from data, is a core challenge in many domains like healthcare, economics, and online experimentation. While traditional machine learning excels at capturing associations, it struggles to disentangle correlation from causation.

CausalNex is a Python library that helps data scientists apply causal inference techniques using Bayesian networks. Bayesian networks are graphical models that encode the probabilistic relationships between variables and can be used for both causal reasoning and prediction.

Some key features of CausalNex:

• Structure learning algorithms to infer the causal graph from data
• Support for incorporating domain knowledge as priors
• Built-in methods for causal inference tasks like intervention and counterfactuals
• Integration with other libraries like NumPy, Pandas, and scikit-learn

CausalNex also includes detailed tutorials and case studies showing how to apply causal inference to real-world problems like customer churn prediction and uplift modeling. The documentation does a nice job explaining key concepts and walking through the API.

To use CausalNex effectively, it‘s important to have a solid foundation in probability, statistics, and causal inference principles. Familiarity with Bayesian methods and graphical models is also very helpful, though not strictly required.

Relevant tutorials and courses:

• Introduction to Causal Inference
• Probabilistic Graphical Models Specialization
• Causal Inference Bootcamp

Project 6: RL4RS – Reinforcement Learning for Recommender Systems

Recommender systems are a core component of many digital products and services. Traditional approaches model recommendation as a supervised learning problem and train on historical user interaction data.

However, this offline approach fails to capture the inherently interactive nature of user behavior and can lead to suboptimal performance in production. Users‘ preferences may change over time and they respond dynamically to the recommendations they are shown.

Reinforcement learning (RL) provides a promising alternative paradigm that frames recommendation as a sequential decision making problem. By learning a policy that maps states (e.g. user context, item features) to actions (recommendations), RL agents can adapt to user feedback in real-time and optimize for long-term engagement metrics.

RL4RS is a project that provides implementations of several state-of-the-art RL-based recommender systems. It includes both value-based methods like Q-learning and policy gradient methods like REINFORCE.

Some notable features of RL4RS:

• Supports both collaborative filtering and content-based approaches
• Includes utilities for simulating user environments and evaluation
• Scales up to large datasets by leveraging GPU acceleration and distributed training
• Integrates with other RecSys libraries like LightFM and Surprise

To dive into RL4RS, you should be familiar with the fundamentals of recommender systems as well as reinforcement learning concepts like MDPs, value functions, and policy optimization. Experience with deep learning frameworks like PyTorch or TensorFlow is also valuable.

Relevant tutorials and courses:

• Introduction to Recommender Systems
• Foundations of Deep RL
• Building Recommender Systems with Machine Learning and AI

Stretch Your Skills with Open Source Data Science Projects

I hope this curated list of open source data science projects has inspired you to roll up your sleeves and start building. Remember, the goal is not to understand everything perfectly from the outset, but rather to challenge yourself incrementally and learn by doing.

If a particular project seems daunting, start by running the code and replicating the results. Then try to extend it in small ways, whether that‘s exploring a different dataset, tweaking the model architecture, or adding a new feature.

As you make progress, share your work with the community and don‘t hesitate to ask for feedback. You‘ll be surprised at how willing people are to help and collaborate. Feel free to share your experiences in the comments below!