Implementing Artificial Neural Network Classification in Python from Scratch

ByJordan Brown

Artificial neural networks (ANNs) are powerful machine learning algorithms inspired by the structure and function of biological neural networks in the human brain. ANNs have achieved remarkable performance on a variety of tasks, especially classification problems where the goal is to predict a category or class label for a given input.

Some common applications of ANNs for classification include:

  • Image classification: Labeling an input image as belonging to one of several predefined categories
  • Sentiment analysis: Classifying a piece of text as expressing positive, negative or neutral sentiment
  • Spam email detection: Determining if an email message is spam or not based on its content
  • Medical diagnosis: Classifying a patient as having a certain disease or not based on their symptoms and test results

While popular deep learning libraries like TensorFlow and PyTorch provide high-level APIs for easily building ANNs, it‘s very instructive to understand how ANNs actually work under the hood by implementing one from scratch. Not only will this deepen your understanding of ANNs, it will also enable you to customize your neural network for your specific use case.

In this blog post, we‘ll walk through how to build and train an ANN for classification in Python from scratch, using only Numpy for matrix math operations. We‘ll apply our ANN to a real-world dataset and evaluate its performance. Let‘s get started!

Steps to Implement an ANN in Python

Here is an overview of the steps we‘ll follow to build our neural network classifier:

  1. Import required libraries
  2. Load and explore the dataset
  3. Preprocess the data
  4. Initialize the neural network parameters
  5. Implement forward propagation
  6. Implement backward propagation
  7. Update parameters with gradient descent
  8. Train the model on the training data
  9. Evaluate performance on the test set
  10. Make predictions with the trained model

We‘ll now go through each of these steps in detail, with code examples.

Step 1: Import Libraries

First, we need to import the Python libraries we‘ll be using. We‘ll use Numpy for matrix operations and Pandas for data loading and manipulation. We‘ll also use matplotlib for visualizing our results.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

Step 2: Load and Explore Data

Next, we‘ll load our dataset into a Pandas DataFrame. For this example, we‘ll use the famous Iris flower dataset which consists of 50 samples from each of three species of Iris (setosa, versicolor, and virginica). Four features were measured from each sample: the length and the width of the sepals and petals, in centimeters. 

dataset = pd.read_csv(‘iris.csv‘)
print(dataset.head())
print(dataset.describe())

This will print out the first few rows and summary statistics of our dataset, which help us get a sense of the data we‘re working with. The output will look something like:

   sepal_length  sepal_width  petal_length  petal_width      species
0           5.1          3.5           1.4          0.2  Iris-setosa
1           4.9          3.0           1.4          0.2  Iris-setosa
2           4.7          3.2           1.3          0.2  Iris-setosa
3           4.6          3.1           1.5          0.2  Iris-setosa
4           5.0          3.6           1.4          0.2  Iris-setosa

       sepal_length  sepal_width  petal_length  petal_width
count    150.000000   150.000000    150.000000   150.000000
mean       5.843333     3.054000      3.758667     1.198667
std        0.828066     0.433594      1.764420     0.763161
min        4.300000     2.000000      1.000000     0.100000
25%        5.100000     2.800000      1.600000     0.300000
50%        5.800000     3.000000      4.350000     1.300000
75%        6.400000     3.300000      5.100000     1.800000
max        7.900000     4.400000      6.900000     2.500000

We see the dataset has 150 total examples across the 3 classes, with 4 numeric features and 1 categorical label (the species).

Step 3: Preprocess Data

Real-world data is often messy and needs cleaning and preprocessing before we can feed it to our models. Some common preprocessing steps include:

  • Handling missing data: Identifying and filling in (imputing) missing values
  • Encoding categorical variables: Converting text labels to numeric values
  • Feature scaling: Normalizing feature values to a consistent range
  • Splitting data into train/validation/test sets: To evaluate model performance on unseen data

Since the Iris dataset is already quite clean, we won‘t need to do much preprocessing. We will split our data into features (X) and labels (y), then split into train and test sets:

X = dataset.iloc[:, :-1].values  
y = dataset.iloc[:, -1].values

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

We set aside 20% of the data as a test set for evaluating our model‘s performance.

Next, let‘s convert our text class labels to integers using scikit-learn‘s LabelEncoder:

from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder()
y_train = encoder.fit_transform(y_train)
y_test = encoder.transform(y_test)

The fit_transform() method finds all unique labels and generates a mapping, which transform() uses to encode any label data.

Finally, we‘ll scale our feature data to have zero mean and unit variance using sklearn‘s StandardScaler:

from sklearn.preprocessing import StandardScaler  
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)  
X_test = scaler.transform(X_test)

Standardizing inputs tends to speed up learning and leads to faster convergence.

Step 4: Initialize Neural Network Parameters

Now we‘re ready to start building our neural net! We‘ll use a simple network architecture with one hidden layer.

First we need to choose the number of neurons for each layer. The input layer size is determined by the number of features in the data (4). The output layer will have one neuron for each class (3 for iris species). For the hidden layer size, we‘ll pick a number between the input and output sizes, let‘s say 10.

input_size = 4 
hidden_size = 10
output_size = 3

Next we initialize our weight matrices W1 and W2 to small random numbers and biases b1, b2 to zeros:

W1 = np.random.randn(input_size, hidden_size) * 0.01
b1 = np.zeros((1, hidden_size)) 
W2 = np.random.randn(hidden_size, output_size) * 0.01  
b2 = np.zeros((1, output_size))

The dimensions of the weight matrices are set based on the sizes of the layers they connect.

Step 5: Implement Forward Propagation

In each forward pass through the network, we take our input data X and compute the predicted output ŷ. Information flows from the input layer, through the hidden layer, to the output layer. At each layer, we compute a weighted sum of the inputs from the previous layer and apply an activation function.

Here‘s the code for forward propagation:

def forward_prop(X, W1, b1, W2, b2):
  Z1 = np.dot(X, W1) + b1
  A1 = np.maximum(0, Z1)  # ReLU activation
  Z2 = np.dot(A1, W2) + b2 
  A2 = 1 / (1 + np.exp(-Z2))  # Sigmoid activation
  return A1, A2

We use the ReLU (rectified linear unit) activation in the hidden layer and sigmoid activation in the output layer since we‘re doing binary classification.

Step 6: Implement Backpropagation

Backpropagation is the key step that allows neural networks to learn. We compute the model‘s prediction error based on the current parameters, then propagate that error signal back through the network layers to calculate gradients. We can then use these gradients to update the weights in the direction that reduces the error.

Here‘s the implementation of backpropagation:

def backward_prop(X, y, A1, A2, W2): 
  m = len(y)
  dZ2 = A2 - y.reshape(-1, 1)
  dW2 = 1/m * np.dot(A1.T, dZ2)
  db2 = 1/m * np.sum(dZ2, axis=0)

  dA1 = np.dot(dZ2, W2.T)
  dZ1 = dA1.copy()
  dZ1[A1 <= 0] = 0
  dW1 = 1/m * np.dot(X.T, dZ1) 
  db1 = 1/m * np.sum(dZ1, axis=0)

  return dW1, db1, dW2, db2

We compute the gradients by taking partial derivatives of the loss with respect to each parameter using the chain rule of calculus. The ReLU gradient is 0 for negative inputs and 1 otherwise.

Step 7: Update Parameters

Finally, we update the model parameters W and b by taking a small step in the direction of the negative gradient, controlled by the learning rate α:

def update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha):
  W1 -= alpha * dW1
  b1 -= alpha * db1
  W2 -= alpha * dW2
  b2 -= alpha * db2
  return W1, b1, W2, b2

By repeating this process of forward prop, backprop and parameter updates over many iterations, the model will learn weights that map the inputs to the correct outputs.

Step 8: Train the Model

Now we‘ll put it all together to train the model on our data:

def train(X, y, epochs, alpha):
  W1 = np.random.randn(input_size, hidden_size) * 0.01
  b1 = np.zeros((1, hidden_size))
  W2 = np.random.randn(hidden_size, output_size) * 0.01
  b2 = np.zeros((1, output_size))

  for i in range(epochs):
    A1, A2 = forward_prop(X, W1, b1, W2, b2)
    dW1, db1, dW2, db2 = backward_prop(X, y, A1, A2, W2)
    W1, b1, W2, b2 = update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha)

    if i % 100 == 0:
      loss = get_loss(A2, y)
      print(f"Epoch {i}, loss: {loss:.3f}")

  return W1, b1, W2, b2

The get_loss function computes the cross-entropy loss between the predicted and true labels.

Let‘s train our model for 1000 epochs using a learning rate of 0.1:

W1, b1, W2, b2 = train(X_train, y_train, epochs=1000, alpha=0.1)

We print out the loss every 100 epochs to monitor convergence.

Step 9: Evaluate Performance

To see how well our model performs, let‘s use it to make predictions on the test set and compare to the true labels:

def predict(X, W1, b1, W2, b2):
  A1, A2 = forward_prop(X, W1, b1, W2, b2)
  y_pred = (A2 > 0.5).astype(int)
  return y_pred

y_pred = predict(X_test, W1, b1, W2, b2) 
print(classification_report(y_test, y_pred))

This will print out metrics like precision, recall, F1-score for each class as well as overall accuracy.

Step 10: Make New Predictions

We can now use the trained model to predict the class for new Iris flower examples:

sample = np.array([[5.1, 3.5, 1.4, 0.2]])
sample_scaled = scaler.transform(sample)
yhat = predict(sample_scaled, W1, b1, W2, b2)
print(f"Predicted class: {encoder.inverse_transform(yhat)[0]}")

The inverse_transform method maps the prediction back to the original label.

And there you have it – a fully functional neural network classifier built from scratch! Of course, there are many ways to improve our basic implementation, such as:

  • Adding more hidden layers
  • Using different activation functions like tanh, leaky ReLU
  • Applying regularization to prevent overfitting
  • Using a validation set to tune hyperparameters
  • Minibatch gradient descent instead of full batch

I encourage you to experiment with the code and try out your own ideas. I hope this hands-on introduction gave you a deeper understanding of the core concepts behind artificial neural networks. While libraries like TensorFlow abstract away a lot of the implementation details, peeking under the hood will make you a better deep learning practitioner.

What interesting applications of ANNs will you tackle next? The possibilities are endless!

Additional Resources:

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

You May Like to Read,

Similar Posts