# The Ultimate Guide to Setting Up a Python Virtual Environment for Machine Learning and Deep Learning on macOS

- Canonical: https://33rdsquare.com/a-quick-guide-to-setting-up-a-virtual-environment-for-machine-learning-and-deep-learning-on-macos/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

## Introduction

Machine learning (ML) and artificial intelligence (AI) are transforming industries and solving previously intractable problems across science and engineering. At the heart of this revolution are the open-source libraries, frameworks and tools of the Python data science ecosystem. Libraries like NumPy, Pandas, Scikit-learn, TensorFlow, and PyTorch have become essential building blocks powering everything from breakthrough research to production applications.

However, the rapid pace of development in ML and AI means that new versions of these core libraries are released frequently, often with significant API changes or performance improvements. For example, TensorFlow 2.0 integrated eager execution by default, PyTorch 1.0 introduced a new JIT compiler, and scikit-learn 0.22 added support for compositional ML pipelines. Upgrading an existing project to use the latest versions can lead to frustrating dependency conflicts or even break the code entirely.

This is where Python virtual environments come in. Virtual environments allow you to create isolated "sandboxes" for Python projects that contain all the necessary libraries without interfering with other projects or the system installation. They make it easy to manage multiple projects with different dependencies on the same machine.

According to the 2020 Kaggle Machine Learning & Data Science survey, 89% of data scientists and ML engineers use virtual environments for their projects. 68% use the conda tool and 20% use pip with tools like virtualenv.

In this guide, we‘ll walk through best practices for setting up a Python virtual environment for ML and AI development on macOS. We‘ll cover installing Python and key ML libraries, creating reproducible environments, and expert tips to supercharge your workflow. Let‘s dive in!

## Step 1: Install Python 3

macOS comes with Python 2.7 pre-installed, but Python 2 reached end-of-life on January 1, 2020. Python 3 is the present and future of the language, with many new features and performance improvements. All major scientific Python libraries now only support Python 3.

The official installer from python.org is the easiest way to get started, but I recommend using the Homebrew package manager to install Python on macOS. Homebrew makes it easy to install and manage multiple Python versions and switch between them for different projects. It also plays nicely with virtualenv, as we‘ll see shortly.

To install Homebrew, open a Terminal and run:

```
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```

Once Homebrew is installed, you can install the latest Python 3.x by running:

```
brew install python
```

This installs Python along with pip, the standard Python package installer. You can verify the installation by running:

```
$ python3 --version
Python 3.9.1

$ pip3 --version
pip 21.0.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)
```

## Step 2: Install virtualenv

virtualenv is a tool to create isolated Python environments. It creates a folder that contains all the executables and libraries needed for a specific Python version and set of packages. To install virtualenv using pip, run:

```
pip3 install virtualenv
```

There are alternative tools for creating Python virtual environments, like the built-in venv module or the conda tool from Anaconda. I prefer virtualenv for its simplicity, wide adoption, and handy features like the ability to use a Python interpreter from any path.

## Step 3: Create a virtual environment

To create a new virtual environment for a project, navigate to the project directory and run:

```
python3 -m virtualenv myproject_env
```

This creates a new folder called `myproject_env` containing a self-contained Python installation. You can name the environment folder anything you want, but I like to use the `_env` suffix to make it clear that it‘s a virtual environment.

## Step 4: Activate the virtual environment

Before you can use the new virtual environment, you need to activate it using the `source` command:

```
source myproject_env/bin/activate
```

You‘ll notice that the prompt changes to show the name of the active environment:

```
(myproject_env) $
```

Now any Python packages you install using pip will be installed in this environment, isolated from other projects and the system Python.

## Step 5: Install ML and data science packages

With the virtual environment activated, you can install the packages needed for ML development. According to the Python Developers Survey 2020, the most popular data science packages are:

- NumPy (used by 64% of respondents): The fundamental package for scientific computing in Python, providing fast array operations and linear algebra routines.
- Pandas (55%): A powerful data manipulation and analysis library, providing data structures like DataFrames and tools for cleaning, transforming, and merging datasets.
- Matplotlib (51%): The most widely used 2D plotting library in Python, allowing you to create publication-quality figures and interactive visualizations.
- Scikit-learn (39%): The go-to library for machine learning in Python, with a wide range of algorithms for classification, regression, clustering, dimensionality reduction, and model evaluation.
- TensorFlow (18%) and PyTorch (13%): The two most popular deep learning frameworks in Python, used for building and training neural networks on GPUs and TPUs.

You can install all of these core packages using pip:

```
pip install numpy pandas matplotlib scikit-learn tensorflow torch
```

I also recommend installing Jupyter Notebook for interactive development and exploration:

```
pip install jupyter
```

To verify the installation, launch a Jupyter Notebook and import the packages:

```
jupyter notebook
```

```
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
import tensorflow as tf
import torch
```

If the notebook runs without any `ImportError`s, your virtual environment is set up and ready for ML development!

## Managing virtual environments

Here are some tips and best practices I‘ve learned for managing Python virtual environments:

- Use a separate environment for each project to avoid conflicts and ensure reproducibility.
- Specify precise package versions in a `requirements.txt` file to enable recreating the exact environment on a different machine or at a later time. Create one using `pip freeze > requirements.txt` and install from it using `pip install -r requirements.txt`.
- Use meaningful and descriptive names for your environments, like `project_name_env`. Avoid generic names like `env` or `venv`.
- Delete unused or outdated environments to conserve disk space and reduce clutter. You can use the `deactivate` command to exit the current environment.
- Consider using a tool like virtualenvwrapper to make it easier to manage multiple environments and streamline common tasks.

One of the key benefits of virtual environments is enabling reproducible research and deployment. By capturing the exact dependencies used for an experiment or application in code, you can ensure that others can reproduce your work and that it will continue to run correctly in the future, even as libraries evolve.

For data science work, I‘m a fan of the "virtual environment per notebook" approach, creating a new environment for each exploratory analysis or experiment. This makes it easy to iterate quickly without worrying about breaking previous work.

## Conda: An alternative for ML

While virtualenv is a great general-purpose tool for Python development, many data scientists and ML researchers use the conda package manager and environment system from Anaconda. Conda is tailored for data science and provides pre-built binaries for hard-to-compile packages like NumPy, SciPy, and XGBoost.

The main advantages of conda over pip + virtualenv are:

- Unified package and environment management through a single tool
- Pre-built packages optimized for performance on Intel and AMD CPUs
- Support for non-Python dependencies like CUDA and HDF5
- Easy installation of specific Python versions in environments

To get started with conda, download and install Anaconda from [https://www.anaconda.com/products/individual](https://www.anaconda.com/products/individual). Then use the `conda create` command to create a new environment:

```
conda create --name myproject_env python=3.9 numpy pandas scikit-learn tensorflow
```

Activate the environment using:

```
conda activate myproject_env
```

According to the Anaconda State of Data Science 2020 report, 74% of data scientists use conda for package and environment management, compared to 50% using pip + virtualenv.

The choice between conda and pip + virtualenv depends on your specific needs and preferences. For general Python development, virtualenv is lightweight and flexible. For ML and data science, conda can simplify the setup process and optimize performance. Many data scientists use both in their workflows.

## Conclusion

In this guide, we‘ve covered the essentials of setting up a Python virtual environment for machine learning and AI development on macOS. We walked through:

- Installing Python 3 using Homebrew
- Creating and activating a virtual environment with virtualenv
- Installing key ML packages like NumPy, Pandas, Scikit-learn, TensorFlow, and PyTorch
- Best practices for managing virtual environments and ensuring reproducibility
- Using conda as an alternative environment and package manager

By leveraging virtual environments in your data science and ML projects, you can create reproducible code, avoid dependency conflicts, and accelerate your iterations. You can experiment fearlessly with new libraries and tools, knowing that your other projects are safely isolated.

As the field of AI continues to advance at a breakneck pace and Python libraries evolve to keep up, virtual environments will become even more indispensable for productive development. They allow you to harness the latest algorithmic and computational innovations from the community while keeping your workflows stable and organized.

Building ML and AI systems that solve real-world problems is immensely challenging and rewarding work. Having a clean, reproducible, and optimized development environment is one key to delivering those solutions faster and with fewer headaches. I hope this guide has equipped you with the knowledge and tools to do just that.

Now, go forth and create amazing things! Push the boundaries of what‘s possible with ML and AI. Contribute your own tools and insights to the open-source ecosystem. And above all, keep learning and experimenting. The next big breakthrough in AI could be just a `virtualenv` away!

---

Source: [The Ultimate Guide to Setting Up a Python Virtual Environment for Machine Learning and Deep Learning on macOS](https://33rdsquare.com/a-quick-guide-to-setting-up-a-virtual-environment-for-machine-learning-and-deep-learning-on-macos/)