Implementing the Perceptron Algorithm from Scratch in Python

Introduction

The perceptron algorithm, developed by Frank Rosenblatt in 1957, is one of the earliest and simplest types of artificial neural networks. While today‘s deep learning models are much more complex, the perceptron was a groundbreaking development that established many of the key concepts neural networks still rely on. This article will explain what the perceptron is, the theory behind how it works, and show you how to implement it from scratch in Python.

A perceptron models how a single neuron in the brain works. It takes in several binary inputs and produces a single binary output. During training, it learns what weights to assign each input to predict the correct output. While very simple compared to modern neural networks with many layers and units, the perceptron was revolutionary as a first model of how the brain could learn.

Perceptron Theory

A perceptron has the following key components:

• A vector of input features X=(x1, x2, …, xn)
• A vector of weights W=(w1, w2, …, wn)
• An activation function f

To make a prediction, the perceptron computes the dot product between the input features and weights vectors. This weighted sum z is then passed to the activation function f to produce the final output ŷ:

z = w1x1 + w2x2 + … + wn*xn
ŷ = f(z)

The activation function is typically the unit step function, which outputs 1 if z is greater than some threshold (usually 0) and 0 otherwise. This makes the perceptron a binary classifier.

During training, the perceptron algorithm optimizes the weights to correctly predict the true output y. The weights are initialized randomly. For each training example, the perceptron makes a prediction ŷ. It then computes the error e between the true and predicted output:

e = y – ŷ

If the prediction was correct (e=0), no changes are made. If the prediction was incorrect, the weights are updated in the direction of the input vector X:

If e=1: W = W + X (weights too low)
If e=-1: W = W – X (weights too high)

This weight update rule makes sense intuitively. If the perceptron predicted 0 but should have predicted 1 (e=1), it adds the input vector to the weights to increase the weighted sum so the activation function is more likely to output 1 next time. If it predicted 1 but should have predicted 0 (e=-1), it subtracts the input vector from the weights to decrease the weighted sum.

The full training procedure loops through all examples multiple times, updating the weights each time, until the perceptron correctly classifies all examples (or reaches some maximum number of iterations). This is a simple optimization process called the perceptron learning rule.

Implementing the Perceptron in Python

Now that you understand how the perceptron works in theory, let‘s implement it in Python! We‘ll define a Perceptron class that encapsulates the weights and methods for training and prediction:

import numpy as np

class Perceptron:
    def __init__(self, n_inputs, max_epochs=100, learning_rate=0.1):
        self.weights = np.zeros(n_inputs + 1) # +1 for bias
        self.max_epochs = max_epochs
        self.learning_rate = learning_rate

    def predict(self, x):
        z = np.dot(x, self.weights[1:]) + self.weights[0] # Compute weighted sum
        return 1 if z > 0 else 0 # Apply activation function

    def train(self, X, y):
        for _ in range(self.max_epochs):
            for i in range(len(X)):
                x = np.insert(X[i], 0, 1) # Insert bias
                y_hat = self.predict(x)
                self.weights += self.learning_rate * (y[i] - y_hat) * x # Update weights

Let‘s break this down:

• The constructor init initializes the weights to a zero vector with length n_inputs + 1. The extra weight is the bias term, which shifts the decision boundary.
• The predict method computes the dot product between the input x and weights, adds the bias term, and applies the step function.
• The train method loops through the training examples for max_epochs iterations. For each example, it makes a prediction, computes the error, and updates the weights according to the perceptron learning rule.

We can use this class to train a perceptron to predict the AND logic gate:

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 0, 0, 1])

p = Perceptron(n_inputs=2)
p.train(X, y)

print(p.predict([0, 0])) # 0
print(p.predict([0, 1])) # 0 
print(p.predict([1, 0])) # 0
print(p.predict([1, 1])) # 1

The perceptron learns weights that correctly predict the output of the AND gate for all four possible inputs. We could similarly train it to learn OR, NAND, and other linearly separable functions.

Limitations of Perceptrons

While groundbreaking, perceptrons have significant limitations. The key drawback is they can only learn to solve linearly separable problems. If the classes cannot be separated by a single straight line, the perceptron learning rule will never converge.

The canonical example of an inseparable problem perceptrons cannot solve is the XOR logic gate. XOR outputs 1 only when the two inputs are different. No linear decision boundary can separate the two classes:

x1  x2  XOR
0   0   0
0   1   1
1   0   1 
1   1   0

To overcome this limitation, we must stack multiple perceptrons together in layers to construct a multilayer perceptron (MLP) network. The hidden layers learn to represent nonlinear combinations of the inputs. Given enough units, MLPs can approximate any continuous function and solve inseparable problems like XOR.

All modern neural network architectures, from MLPs to convolutional and recurrent networks, are built on this basic idea of stacking many simple units together in layers to model complex nonlinear patterns. In this sense, the perceptron was the critical building block for today‘s deep learning revolution.

Conclusion

The perceptron algorithm was a major milestone in artificial intelligence that established the key principles of neural networks. While much simpler than modern architectures, the perceptron introduced the ideas of weighted inputs, activation functions, and iterative learning rules. Implementing a perceptron from scratch in Python is surprisingly straightforward and clearly illustrates these core concepts.

However, understanding the perceptron‘s limitations is crucial. Perceptrons can only solve linearly separable problems and fail on tasks like XOR. Stacking perceptrons together into multilayer networks overcomes this constraint and allows neural networks to model complex nonlinear patterns.

I hope this article gave you a solid intuition for what perceptrons are and how they work! The perceptron may be an outdated model today but it‘s still a great avenue for building your understanding of neural networks from the ground up. Let me know in the comments if you have any other questions!