30 Challenging Open Source Data Science Projects to Master in 2025

Introduction

As we enter 2024, the fields of data science, machine learning and AI continue to rapidlyevolve. Exciting innovations emerge constantly, expanding the realm of what‘s possible. At the same time, the open source community has become an indispensable resource, democratizing access to cutting-edge tools and knowledge.

For data science practitioners, hands-on projects provide an ideal way to acquire new skills and gain practical experience. Working on challenging projects pushes you out of your comfort zone, enabling faster growth. Contributing to open source also allows you to learn from and collaborate with leading experts from around the world.

To support your learning journey, we‘ve curated a list of 30 stimulating data science projects to tackle in 2024. We‘ve included projects spanning multiple domains such as machine learning, deep learning, NLP, computer vision, and big data. There are projects suitable for different experience levels, from beginner-friendly to highly advanced.

So without further ado, here are 30 open source data science projects to help you sharpen your skills in 2024:

Machine Learning Projects

1. Time Series Forecasting with Prophet and GluonTS

Time series forecasting is a crucial skill with many business applications. Facebook‘s Prophet and Amazon‘s GluonTS are two popular open source libraries for building robust forecasting models.

This project involves using Prophet and GluonTS to forecast metrics like sales, web traffic, or demand. You‘ll get experience with time series data preparation, model building, evaluation and visualization. Consider experimenting with different algorithms like ARIMA or DeepAR.

Helpful resources:

• Prophet quick start guide: https://facebook.github.io/prophet/docs/quick_start.html
• GluonTS tutorials: https://ts.gluon.ai/tutorials/forecasting/quick_start_tutorial.html

2. Anomaly Detection on Time Series Data

Anomaly detection is the task of identifying rare events or observations which differ from the norm. It has useful applications in fraud detection, system health monitoring, and more.

For this project, use unsupervised learning algorithms like Isolation Forest, Local Outlier Factor or Autoencoders to detect anomalies in time series data. Good datasets to try include Yahoo‘s benchmark dataset or NAB (Numenta Anomaly Benchmark). Focus on designing robust features and choosing appropriate model thresholds.

Helpful resources:

• Overview of anomaly detection techniques: https://www.anodot.com/blog/what-is-anomaly-detection/
• PyOD library for anomaly detection: https://pyod.readthedocs.io/

3. Customer Churn Prediction with Interpretable Models

Customer churn, or attrition, is a significant problem faced by many companies. Predicting which customers are likely to churn can enable targeted retention efforts.

This project focuses on building interpretable ML models to predict churn. Use techniques like logistic regression, decision trees or RuleFit to construct models that are both accurate and human-understandable. Practice explaining your models using libraries like LIME, SHAP, or Skater.

Helpful resources:

• A short tutorial on interpretable ML: https://towardsdatascience.com/interpretable-machine-learning-with-xgboost-9ec80d148d27
• Open source interpretability libraries: https://github.com/slundberg/shap, https://github.com/marcotcr/lime, https://github.com/oracle/Skater

Deep Learning Projects

4. Image Super-Resolution with GANs

Image super-resolution is the task of recovering a high-resolution image from a low-resolution input. Recent advances in GANs have enabled remarkable progress on this challenging problem.

For this project, you‘ll build a GAN-based model for single image super-resolution (SISR). Experiment with different architectures like SRGAN, ESRGAN or SFTGAN. Train your model on standard benchmarks like Set5, Set14 or BSD100. Focus on faithful texture synthesis and realistic perceptual quality.

Helpful resources:

• SISR with Deep Learning overview: https://cv.snu.ac.kr/research/EDSR/singleimage_SR_using_DL.html
• Collection of SISR links and resources: https://github.com/ChaofWang/Awesome-Super-Resolution

5. Fine-Tuning CLIP for Zero-Shot Visual Classification

OpenAI‘s CLIP model has exhibited impressive zero-shot classification capabilities. It can recognize objects without any task-specific fine-tuning, by leveraging the information in its text encoder.

In this project, fine-tune CLIP on a custom dataset to perform zero-shot visual classification. Explore techniques like prompt engineering, classifier calibration and transductive fine-tuning to boost accuracy. Analyze what visual concepts CLIP has learned and where it still struggles.

Helpful resources:

• Intro to CLIP and zero-shot learning: https://www.pinecone.io/learn/openai-clip/?utm_source=google&utm_medium=cpc&utm_campaign=Retargeting&gclid=CjwKCAjw7IeUBhBbEiwADhiEMQVw89gUyv3BuSlU2oSyHGhM1gIcEEn39H-Y3T6RD73KZNkBJQqA1RoCOFMQAvD_BwE
• Fine-tuned CLIP models and code: https://github.com/haltakov/natural-language-image-search

6. Efficient Transformers for Mobile and Edge Devices

Transformer models like BERT have become ubiquitous in NLP. However, their large size and compute requirements make them challenging to deploy on resource-constrained devices.

This project aims to build efficient Transformer variants that can run on mobile or edge devices. Explore techniques like distillation, quantization, pruning, and architecture search to shrink model size while preserving performance. Benchmark your models on metrics like accuracy, latency, and memory footprint.

Helpful resources:

• Transformer model compression survey: https://arxiv.org/abs/2009.06732
• Mobile deep learning overview: https://machinethink.net/blog/mobile-deep-learning/

Computer Vision Projects

7. Improved Road Lane Detection with Transformers

Accurate lane detection is a critical perception task for self-driving vehicles. Transformer attention mechanisms have recently shown promise in boosting road scene understanding.

For this project, build a lane detection model that incorporates Transformers to capture global context. Use a dataset like CULane or TuSimple. Aim to design an architecture that improves robustness to challenging scenarios like poor lighting, occlusion, or faded lane markings.

Helpful resources:

• SegFormer: A Transformer model for semantic segmentation: https://arxiv.org/abs/2105.15203
• State-of-the-art lane detection methods: https://github.com/cardwing/Codes-for-Lane-Detection

8. Image-to-Image Translation for Virtual Try-On

Virtual try-on allows visualizing how clothing items would look when worn by a person. It has applications in fashion, e-commerce, and gaming.

This project explores unpaired image-to-image translation techniques for virtual try-on. Given a photo of a clothing item and a person, train a model to "dress" the person with that item. Experiment with architectures like CycleGAN, UNIT, or MUNIT. Focus on preserving the person‘s pose and identity while realistically transferring the garment‘s appearance.

Helpful resources:

• Intro to image-to-image translation: https://machinelearningmastery.com/what-is-image-to-image-translation/
• VITON: An Image-based Virtual Try-on Network: https://arxiv.org/abs/1711.08447

9. Improved Deepfake Detection

Deepfakes, or synthetic media generated by AI, have rapidly grown more convincing. Detecting deepfakes is important for preventing misuse.

For this project, build an ensemble model to discriminate between real and deepfake videos. Use techniques like attention, multi-scale feature fusion, and temporal reasoning to spot artifacts. Train and evaluate your model on large-scale datasets like FaceForensics++, DeeperForensics-1.0 or Celeb-DF.

Helpful resources:

• Overview of deepfake detection methods: https://www.mdpi.com/2076-3417/11/7/3413
• FaceForensics benchmark and datasets: https://github.com/ondyari/FaceForensics

NLP Projects

10. Fine-Tuned LLMs for Dialogue Agents

Large language models (LLMs) like GPT-3 have shown remarkable performance on many NLP tasks. Fine-tuning LLMs is a promising approach for building engaging dialogue agents.

This project involves fine-tuning a pre-trained LLM for open-ended dialogue. Use frameworks like Hugging Face or Microsoft‘s DialoGPT. Collect a diverse dataset of human conversations. Experiment with techniques like retrieval augmentation, safety filtering, and persona conditioning. Evaluate your model with both automated and human metrics.

Helpful resources:

• Fine-tuning language models guide: https://huggingface.co/blog/how-to-generate
• DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation: https://arxiv.org/abs/1911.00536

11. Abstractive Text Summarization with Longformer Attention

Abstractive summarization, or rewriting text in a more concise form, is a challenging NLP task. Longformer attention mechanisms can help capture long-range dependencies.

For this project, build an abstractive summarizer using a Longformer-based encoder-decoder architecture. Train and evaluate your model on datasets like CNN/DailyMail, XSum, or arXiv. Analyze how varying the attention window size impacts fluency, faithfulness and length of the generated summaries.

Helpful resources:

• Intro to text summarization approaches: https://www.machinelearningplus.com/nlp/text-summarization-approaches-nlp-example/
• Longformer paper: https://arxiv.org/abs/2004.05150

12. Multilingual Named Entity Recognition with Cross-Lingual Transfer Learning

Named entity recognition (NER) is the task of detecting mentions of entities like people, organizations, or locations in text. Building accurate NER models for low-resource languages remains challenging.

This project focuses on improving multilingual NER using cross-lingual transfer learning. Start with a pre-trained multilingual model like XLM-R or mBERT. Then fine-tune it on labeled NER data in a high-resource language like English. Finally, adapt the model to a low-resource target language using techniques like zero-shot transfer, word-level adversarial training, or meta-learning.

Helpful resources:

• Awesome cross-lingual NLP resources: https://github.com/sebastianruder/awesome-cross-lingual-nlp
• Cross-lingual Transfer Learning for NER tutorial: https://towardsdatascience.com/cross-lingual-transfer-learning-for-ner-d9a5baf6e7df

Big Data Projects

13. Distributed Inference with Ray and PyTorch

Scaling inference for deep learning models to handle huge volumes of requests is a critical challenge. Distributed frameworks like Ray can help parallelize inference across a cluster.

This project demonstrates how to build a scalable inference system using Ray and PyTorch. Design an architecture that can serve a PyTorch model across multiple nodes. Use techniques like model sharding, micro-batching, and input pipelining to optimize throughput and latency. Load test your system to evaluate performance under different traffic patterns.

Helpful resources:

• Scalable PyTorch model inference with Ray: https://docs.ray.io/en/latest/serve/tutorials/pytorch.html
• Efficient model serving at scale: https://technology.condenast.com/story/efficient-model-serving-at-scale

14. Anomaly Detection on Streaming Data with Apache Flink

Detecting anomalies in real-time data streams is essential for monitoring critical systems and catching issues early. Apache Flink is a powerful framework for stream processing.

This project implements an anomaly detection pipeline using Flink. Ingest a stream of time series data from a source like Kafka. Use Flink‘s DataStream API to compute streaming aggregates and run an anomaly detection algorithm like HBOS, xStream or RRCF. Output the detected anomalies to a sink for alerting or further analysis.

Helpful resources:

• Flink streaming concepts: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/concepts/flink-architecture/
• River: Online machine learning in Python: https://riverml.xyz/latest/

15. Large-Scale Product Recommendation with PySpark

Building recommender systems that can handle millions of users and items is a big data challenge. PySpark provides distributed implementations of collaborative filtering and other recommendation algorithms.

For this project, develop a product recommender using PySpark‘s MLlib library. Start with a large dataset of user-item interactions like the Amazon Reviews dataset. Experiment with different algorithms like ALS, logistic regression, or neural collaborative filtering. Optimize your model‘s hyperparameters through cross-validation. Deploy your trained model and evaluate its performance on live traffic.

Helpful resources:

• Building a recommender system with PySpark tutorial: https://towardsdatascience.com/building-a-recommendation-system-using-pyspark-e28702042e68
• Intro to recommender systems: https://www.kdnuggets.com/2019/09/introduction-recommender-systems.html

Closing Thoughts

We‘ve covered a broad range of data science projects spanning different domains and levels of complexity. Whether you‘re a beginner looking for a place to start, or an experienced practitioner eager to expand your skillset, there‘s something here for you.

Remember that becoming a proficient data scientist requires consistent practice. Don‘t hesitate to dive into a project even if it seems daunting at first. Break the problem down, research unfamiliar concepts, and reach out for help when you get stuck. With determination and focused effort, you‘ll make steady progress.

Beyond the specific technical skills you gain from each project, you‘ll also grow your problem-solving abilities, creativity, and resilience. You‘ll learn to ask insightful questions, validate assumptions, and communicate results effectively. These core competencies will serve you well throughout your data science career.

Finally, consider sharing your completed projects publicly on platforms like GitHub, Kaggle or your personal blog. Showcasing your best work demonstrates your abilities and helps you connect with like-minded professionals. You never know what opportunities might arise from putting your work out there!

Best of luck with your learning journey! Remember to embrace challenges, stay curious, and never stop honing your craft. Here‘s to a productive and rewarding 2024 filled with data science growth.