# Supercharging Mathematical Operations in Python with NumPy: An AI/ML Perspective - Canonical: https://33rdsquare.com/mathematical-operations-in-python-with-numpy/ - Published: 2024-09-03 - Author: Jordan Brown - Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/) --- Python has emerged as the preeminent language for artificial intelligence and machine learning, thanks in no small part to its powerful ecosystem of numerical computing libraries. Chief among these is NumPy, which provides the core data structures and functions that enable high-performance mathematical operations in Python. For anyone working in AI/ML, NumPy is an absolutely essential tool to master. Its multi-dimensional arrays and linear algebra routines form the computational backbone of virtually all ML workflows in Python. In this article, we‘ll dive deep into NumPy from an AI/ML perspective, exploring how it can supercharge your models and applications. We‘ll cover advanced usage patterns, performance optimization, and integration with higher-level ML frameworks. By the end, you‘ll appreciate NumPy‘s pivotal role in the Python data science stack and understand how to leverage it for machine learning success. ## Why NumPy for AI/ML? At the heart of machine learning are mathematical operations on large, multi-dimensional datasets. These include matrix multiplication, eigenvalue decomposition, statistical summarization, and more. Performing these operations efficiently is critical, as ML datasets often contain millions or billions of data points and models may have millions of parameters. While Python has built-in data structures like lists and the `math` module for scalar math, these are inadequate for the demands of modern ML workflows. This is where NumPy comes in. NumPy is built on a powerful N-dimensional array object that allows homogeneous data (i.e. all of the same type) to be stored contiguously in memory. Compared to standard Python lists which contain pointers to data scattered throughout memory, NumPy‘s arrays are cache-friendly and can be operated on with vectorized functions that are implemented in optimized C code. The result is orders-of-magnitude better performance. To illustrate, let‘s compare the execution time of squaring a million numbers with Python math vs. NumPy: ``` import math import numpy as np n = 1000000 # Python loop start = time.time() result = [] for i in range(n): result.append(math.pow(i, 2)) end = time.time() print(f"Python loop: {end - start:.3f} seconds") # NumPy start = time.time() a = np.arange(n) result = np.power(a, 2) end = time.time() print(f"NumPy: {end - start:.3f} seconds") ``` ``` Python loop: 2.075 seconds NumPy: 0.012 seconds ``` NumPy is a staggering ~175x faster! This is a toy example, but the performance gap is similar for real-world ML computations on much larger matrices and tensors. Besides raw speed, NumPy also simplifies ML code with an expressive syntax for array indexing, reshaping, broadcasting, and more. This allows data to be concisely transformed and fed into ML models. Essential building blocks like distance metrics, convolutions, and activation functions can be written succinctly with NumPy primitives. ## NumPy in the ML Ecosystem NumPy‘s multi-dimensional arrays are the standard representation for data and model parameters in Python ML libraries. Pandas, the most popular library for data manipulation, is built on NumPy arrays. Scikit-learn, the leading ML library, uses NumPy arrays as inputs and outputs for all of its algorithms. Deep learning frameworks like TensorFlow and PyTorch use NumPy-like tensor objects and can interface directly with NumPy data. In fact, converting data to NumPy arrays is often the first step in an ML pipeline after loading a dataset. This sets the stage for further data preprocessing, feature engineering, and training models. Here‘s an example of training a basic feedforward neural network on the classic iris dataset using NumPy and scikit-learn: ``` from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.neural_network import MLPClassifier # Load data iris = load_iris() X, y = iris.data, iris.target # Convert to NumPy arrays X, y = np.array(X), np.array(y) # Split into train/test sets X_train, X_test, y_train, y_test = train_test_split(X, y) # Fit a neural network model = MLPClassifier(hidden_layer_sizes=(10,10,10), max_iter=1000) model.fit(X_train, y_train) # Evaluate accuracy print(f"Accuracy: {model.score(X_test, y_test):.3f}") ``` NumPy arrays are used to hold the input features `X` and target labels `y`. The raw data is converted from Python lists to NumPy arrays before being split into train and test subsets. The MLPClassifier is a basic neural network model that accepts NumPy arrays to fit on the training data. Model accuracy is evaluated by calling `score()` with the test set NumPy arrays. This simple example demonstrates NumPy‘s central role in the ML workflow. Data is passed between preprocessing steps, models, and evaluation metrics as NumPy arrays. This allows the different components to interoperate seamlessly. ## Advanced NumPy for ML Beyond the basics, NumPy has many advanced features that are valuable for ML applications. Here are a few key ones: ### Broadcasting Broadcasting allows arrays with different shapes to be used in arithmetic operations without creating large intermediate arrays. This is useful for e.g. subtracting a mean vector from a matrix, or multiplying a matrix with a vector. NumPy‘s broadcasting rules automatically align the dimensions of the arrays based on their shapes. ``` A = np.random.rand(3, 3) b = np.random.rand(3) C = A - b[:,None] ``` Here `b` has shape `(3,)` while `A` has shape `(3, 3)`. Normally they could not be subtracted, but broadcasting allows `b` to be subtracted from each column of `A` to produce `C` with shape `(3, 3)`. Broadcasting is more memory-efficient than tiling `b` to match `A`‘s shape. ### Masked Arrays Masked arrays are NumPy arrays that have a boolean mask indicating invalid or missing values. Mathematical operations on masked arrays automatically ignore the masked values. This is convenient for handling incomplete data, a common scenario in ML. ``` X = np.array([[1, 2, 3], [4, 5, np.nan], [7, 8, 9]]) mask = np.isnan(X) mX = np.ma.masked_array(X, mask) print(mX.mean(axis=0)) # [4. 5. 6.] ``` The `mean` calculation ignores the NaN value thanks to the mask. Masked arrays are helpful for keeping data and masks aligned through different array operations. ### Structured Arrays Structured arrays are ndarrays with compound data types, similar to structs in C. Each element can have multiple named fields of different data types. This allows heterogeneous data like ML datasets with both numeric and categorical features to be represented compactly in a single array. ``` dtype = [(‘feature1‘, float), (‘feature2‘, int), (‘label‘, ‘U10‘)] data = [(1.5, 10, ‘apple‘), (2.0, 20, ‘banana‘), (2.5, 30, ‘orange‘)] a = np.array(data, dtype=dtype) print(a[‘feature1‘]) # [1.5 2. 2.5] print(a[‘label‘]) # [‘apple‘ ‘banana‘ ‘orange‘] ``` ### Polynomial Fitting NumPy also provides functions for fitting polynomials to data points, a common task in ML. The `polyfit` function does least squares polynomial fitting with a given degree: ``` x = np.array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0]) y = np.array([0.0, 0.8, 0.9, 0.1, -0.8, -1.0]) coeffs = np.polyfit(x, y, deg=3) print(coeffs) # [-0.16071429 0.50071429 -0.23214286 0.08035714] ``` The coefficients of the best-fit polynomial can then be used to interpolate or extrapolate values with `np.polyval()`. ## Accelerating NumPy Despite NumPy‘s strong performance out of the box, there are ways to accelerate it even further for demanding ML applications: - [Numba](http://numba.pydata.org/) is an LLVM-based JIT compiler for Python code. It can compile NumPy-centric Python functions to native machine code, often with significant speedups. Numba is especially effective at optimizing custom numerical algorithms not already implemented in NumPy‘s C codebase. - [Cython](https://cython.org/) is an optimizing static compiler that extends Python with C-like type declarations. Cython can produce major performance gains for NumPy-heavy code by providing low-level access to NumPy‘s underlying C arrays. - [Dask](https://dask.org/) scales NumPy operations to multiple cores or distributed clusters. It provides a clone of the NumPy API with support for larger-than-memory arrays and parallel execution. - [CuPy](https://cupy.dev/) implements NumPy‘s API with GPU acceleration, enabling fast execution of NumPy code on NVIDIA CUDA GPUs. These tools help scale NumPy to meet the increasing computational demands of modern AI/ML projects. ## Comparison to MATLAB MATLAB has historically been the dominant platform for numerical computing. However, Python with NumPy has emerged as a compelling open-source alternative. While MATLAB has a mature set of mathematical and statistical functions, NumPy covers many of the same bases. Here are a few areas relevant to AI/ML where NumPy is comparable or superior: - **Syntax:** NumPy uses a more mainstream, Pythonic syntax than MATLAB, which can reduce cognitive overhead. For example, NumPy indexes arrays starting from 0 rather than 1. - **Performance:** While MATLAB uses MKL and other proprietary libraries to boost performance, NumPy relies mostly on open-source libraries that have been heavily optimized by the community. NumPy can be faster than MATLAB for many common operations. - **Ecosystem:** Python has a much richer ecosystem for AI/ML, with libraries like scikit-learn, Pandas, Keras, PyTorch, and more. NumPy arrays integrate seamlessly with these tools. - **Cost:** Python and NumPy are free and open source, while MATLAB can cost thousands of dollars per seat. Overall, NumPy provides a capable, high-performance numerical computing foundation for Python ML projects without the cost or vendor lock-in of MATLAB. ## NumPy in Research and Industry NumPy is the numerical engine powering many groundbreaking AI/ML applications. Here are a few examples: - **ImageNet and Computer Vision:** NumPy was a critical component in the original AlexNet architecture that kicked off the deep learning revolution in computer vision. Pre-trained models like VGG and ResNet are distributed as collections of NumPy arrays storing the learned weights. - **NLP and Transformers:** Libraries like spaCy and Hugging Face Transformers use NumPy arrays to represent word embeddings and language model parameters. NumPy‘s linear algebra routines enable efficient attention mechanisms and optimization. - **Reinforcement Learning:** OpenAI Gym, the most popular platform for developing and comparing RL algorithms, returns observations as NumPy arrays. RL libraries like Stable Baselines use NumPy to implement algorithms like DQN and PPO. - **Robotics with ROS:** Robot Operating System (ROS), the standard platform for robotics applications, uses NumPy arrays to represent sensor data, point clouds, and other signals. NumPy enables efficient processing of robotics data for perception and control. As you can see, NumPy plays a foundational role across diverse domains. It lays the groundwork for high-level AI/ML libraries in Python. ## Conclusion NumPy is the bedrock of AI/ML in Python. Its multi-dimensional arrays are the default data structure for representing datasets, model parameters, and inputs/outputs across the ML ecosystem. NumPy‘s vectorized operations, broadcasting, and linear algebra routines enable efficient and concise implementation of ML algorithms. In this article, we‘ve explored NumPy from an AI/ML perspective. We‘ve seen how its performance and API design are uniquely suited to ML workflows. We‘ve looked at advanced features and extensions that make NumPy even more powerful for data science. If you‘re working in ML with Python, NumPy is an essential part of your toolkit. Take the time to master its core features, and don‘t hesitate to dive into its advanced capabilities. NumPy mastery will pay dividends in your AI/ML career. --- Source: [Supercharging Mathematical Operations in Python with NumPy: An AI/ML Perspective](https://33rdsquare.com/mathematical-operations-in-python-with-numpy/)