12 Matrix Operations You Must Master to Excel at Deep Learning
Deep learning has taken the world by storm in recent years, powering breakthrough advances in areas like computer vision, natural language processing, and reinforcement learning. At the core of deep learning algorithms lies the fundamental language of linear algebra and matrix operations.
Whether you‘re a beginner getting started with deep learning, or an experienced practitioner looking to level up your skills, having a solid grasp of essential matrix operations is critical. In this article, we‘ll dive into 12 key matrix operations that every deep learning engineer should have in their toolbox.
But first, let‘s take a step back – why are matrix operations so important in deep learning? The key reason is that the most basic building block of deep learning models, the artificial neuron, relies on matrix multiplication to take its inputs, apply weights, and generate an output. When you stack neurons together in a multilayer network, you‘re essentially performing a series of matrix multiplications down the layers.
So by understanding matrix operations at a deep level, you‘ll gain valuable intuition about what‘s happening under the hood of deep learning models. You‘ll be able to reason about model architectures, debug issues, and implement new ideas more effectively.
With that context in mind, let‘s jump into our key matrix operations! We‘ll provide code snippets using the ubiquitous NumPy library in Python to illustrate each one.
1. Creating and Manipulating Matrices
Before we can perform operations on matrices, we need to be able to create and work with them in code. The NumPy library provides a convenient way to create and manipulate multi-dimensional arrays (including 1-D vectors and 2-D matrices).
import numpy as np
# Create a matrix
A = np.array([[1, 2, 3],
[4, 5, 6]])
print(A)Output:
[[1 2 3]
[4 5 6]]We can inspect properties of the matrix like its shape (number of rows and columns), and the total number of elements:
print(A.shape) # (2, 3)
print(A.size) # 6 Accessing and modifying individual elements is straightforward using indexing:
print(A[0,0]) # 1
A[1,1] = 10
print(A)[[1 2 3]
[4 10 6]]2. Matrix Addition and Subtraction
Adding and subtracting matrices is an element-wise operation – corresponding elements are added/subtracted to produce the result matrix, which has the same shape as the input matrices. In order for addition/subtraction to be possible, the input matrices must have identical shapes.
A = np.array([[1, 2],
[3, 4]])
B = np.array([[5, 6],
[7, 8]])
C = A + B
print(C) [[ 6 8]
[10 12]] Subtraction works similarly:
D = B - A
print(D)[[4 4]
[4 4]]3. Matrix Transpose
The transpose of a matrix is an operation that flips the matrix over its diagonal, switching the row and column indices of the matrix. So an m x n matrix becomes an n x m matrix after transposition.
A = np.array([[1, 2],
[3, 4],
[5, 6]])
print(A.T)[[1 3 5]
[2 4 6]]Taking the transpose of a matrix is a common operation in many deep learning computations, such as in the backpropagation algorithm.
4. Converting Between Dense and Sparse Matrices
In some cases, we may be dealing with sparse matrices – matrices where most of the elements are zero. Using a dense representation (storing all elements) for sparse matrices is inefficient. There are various sparse representations that only store the non-zero elements.
Converting between dense and sparse representations is useful in optimizing computations and saving memory. Let‘s see an example:
import numpy as np
from scipy.sparse import csr_matrix
A = np.array([[0, 0, 0],
[0, 1, 0],
[3, 0, 3]])
# Convert to sparse (CSR format)
S = csr_matrix(A)
print(S) (1, 1) 1
(2, 0) 3
(2, 2) 3We can convert back to a dense representation too:
B = S.todense()
print(B)[[0 0 0]
[0 1 0]
[3 0 3]]5. Matrix Multiplication
One of the most critical operations in deep learning is matrix multiplication. Unlike element-wise multiplication, matrix multiplication is a special operation defined only for certain compatible shapes.
For two matrices A (m x n) and B (n x p) to be multiplied, the number of columns in A must equal the number of rows in B. The result is a matrix C of shape (m x p).
A = np.array([[1, 2],
[3, 4]])
B = np.array([[5, 6],
[7, 8]])
C = np.dot(A,B)
print(C) [[19 22]
[43 50]]Matrix multiplication shows up all over the place in deep learning algorithms, from forward propagation of inputs, to calculation of gradients via backpropagation.
6. Element-wise Operations
Sometimes we want to perform operations on each element of a matrix. This can be done by simply defining Python functions and using NumPy‘s vectorize() function.
def add_100(i):
return i + 100
vectorized_add_100 = np.vectorize(add_100)
A = np.array([[1, 2],
[3, 4]])
B = vectorized_add_100(A)
print(B) [[101 102]
[103 104]]We can use this technique to apply all kinds of custom functions to matrices, element-wise.
7. Aggregation Operations
Aggregation operations reduce a matrix to a single scalar value. Examples include summing all elements, finding the minimum or maximum value, or calculating the mean or standard deviation.
A = np.array([[1, 2],
[3, 4]])
print(np.sum(A)) # 10
print(np.min(A)) # 1
print(np.max(A)) # 4
print(np.mean(A)) # 2.5
print(np.std(A)) # 1.118033988758. Matrix Norms
A matrix norm is a function that assigns a positive value to a matrix that quantifies its "size". There are various types of matrix norms, each with different properties.
The Frobenius norm, for example, is the square root of the sum of the absolute squares of the matrix‘s elements. It treats the matrix as a long vector.
from numpy.linalg import norm
A = np.array([[1, 2],
[3, 4]])
print(norm(A)) # 5.47722557505
print(norm(A, ‘fro‘)) # 5.47722557505Other common matrix norms include the L1 norm (maximum absolute column sum) and the L2 norm (maximum singular value).
9. Matrix Inverse
The inverse of a square matrix A is a matrix A^-1 such that A A^-1 = A^-1 A = I, where I is the identity matrix. Not all square matrices have an inverse. If an inverse exists, the matrix is said to be invertible or non-singular.
from numpy.linalg import inv
A = np.array([[1., 2.], [3., 4.]])
A_inv = inv(A)
print(A_inv)[[-2. 1. ]
[ 1.5 -0.5]]We can verify it‘s indeed the inverse:
print(np.dot(A, A_inv))[[1.00000000e+00 1.11022302e-16]
[0.00000000e+00 1.00000000e+00]]The result is the identity matrix, up to some small numerical error.
10. Eigenvalues and Eigenvectors
Eigenvalues and eigenvectors are important properties of square matrices. For a square matrix A, a scalar λ and a non-zero vector v are an eigenvalue and eigenvector of A if:
Av = λv
Eigenvalues and eigenvectors have many applications, including in principal component analysis (PCA) for dimensionality reduction.
from numpy.linalg import eig
A = np.array([[1, 2],
[3, 4]])
eigenvalues, eigenvectors = eig(A)
print(eigenvalues) # [-0.37228132 5.37228132]
print(eigenvectors) [[-0.82456484 -0.41597356]
[ 0.56576746 -0.90937671]]11. Singular Value Decomposition (SVD)
The Singular Value Decomposition (SVD) is a matrix decomposition method for reducing a matrix to its constituent parts in order to make certain subsequent matrix calculations simpler.
For a matrix A, the SVD is:
A = UΣV^T
where U and V are orthogonal matrices (their columns are orthonormal) and Σ is a diagonal matrix of positive numbers called singular values.
from numpy.linalg import svd
A = np.array([[1, 2],
[3, 4],
[5, 6]])
U, S, VT = svd(A)
print(U)
print(S)
print(VT)[[-0.3863177 -0.89954315 0.2012068 ]
[-0.57480185 0.41170442 -0.70727752]
[-0.7633059 0.14886469 0.67634824]]
[9.508032 0.77286964]
[[-0.61962948 -0.78489445]
[-0.78489445 0.61962948]]SVD has applications in compression, dimensionality reduction, and in the calculation of the pseudoinverse.
12. Matrix Pseudoinverse
The pseudoinverse of a matrix A, denoted as A^+, is a generalization of the matrix inverse that can be computed for any matrix, even if it is not square or not of full rank.
It has many applications, including computing least squares solutions to linear equations. The pseudoinverse can be computed using the SVD:
A^+ = VΣ^+U^T
where Σ^+ is formed by taking the reciprocal of each non-zero element on the diagonal of Σ, leaving the zeros in place, and then taking the transpose of the resulting matrix.
from numpy.linalg import pinv
A = np.array([[1, 2],
[3, 4],
[5, 6]])
A_pinv = pinv(A)
print(A_pinv)[[-0.94444444 0.44444444 0.05555556]
[ 0.72222222 -0.22222222 -0.05555556]]Conclusion
In this post, we‘ve covered 12 essential matrix operations that every deep learning practitioner should know. From basic manipulation of matrices to more advanced concepts like eigendecomposition and singular value decomposition, these operations form the foundation upon which deep learning algorithms are built.
By taking the time to understand these matrix operations at a deep level, you‘ll be well-equipped to understand, implement, and debug deep learning models. You‘ll develop intuition about what‘s happening mathematically "under the hood" when you define and train neural networks.
But the learning doesn‘t stop here. To truly master these concepts, it‘s important to practice implementing these operations in code (NumPy is a great tool for this), and to explore how they‘re used in various deep learning algorithms and architectures.
Additionally, seeking out visual explanations and intuitive analogies can be very helpful in cementing your understanding. For example, you might think of matrix multiplication as a way of applying a linear transformation to a vector, or singular value decomposition as a way of finding the "axes of variation" in a matrix.
As you continue on your deep learning journey, always keep these foundational matrix operations in mind. They‘ll serve as a North Star, guiding your understanding and enabling you to navigate the complex landscape of deep learning with confidence and clarity.
Happy learning!
