Neural Networks from Scratch in Python and R: A Step-by-Step Guide

Neural networks have revolutionized the field of machine learning and artificial intelligence in recent years. From computer vision to natural language processing to speech recognition, neural networks power many of the intelligent applications we use every day. But how exactly do neural networks work under the hood?

In this in-depth tutorial, we‘ll peel back the layers and build a neural network completely from scratch using Python and R. By coding a neural network from the ground up, you‘ll gain a thorough understanding of the core mechanics of this powerful machine learning model. Let‘s dive in!

Neural Networks 101

At a high level, a neural network is a machine learning model loosely inspired by the structure of the human brain. It consists of interconnected nodes organized into layers:

• An input layer which takes in your data
• One or more hidden layers where the learning occurs
• An output layer that makes predictions

Information flows through the neural network from the input layer to the output layer. Each node has a weight and the network learns by adjusting these weights.

Here‘s a simple neural network architecture:

[Neural network diagram]

A neural network makes predictions through a two-step process:

1. Forward propagation: The input data is fed through the network to generate an output
2. Backward propagation: The network‘s weights are adjusted to minimize the prediction error

This two-step training process is repeated many times on many training examples until the network‘s predictions are optimized. With that conceptual foundation, let‘s now implement a neural network from scratch in Python!

Coding a Neural Network from Scratch in Python

We‘ll build a simple feedforward neural network with one hidden layer to perform binary classification on a toy dataset. Our implementation will use numpy, a popular Python library for numerical computing.

First, let‘s generate a dummy dataset:

import numpy as np

# Generate dummy data
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])

Next, we‘ll define our neural network architecture. We‘ll use a 2-2-1 architecture: 2 input nodes, 2 hidden layer nodes, and 1 output node.

# Define network architecture 
num_input_nodes = 2
num_hidden_nodes = 2 
num_output_nodes = 1

Now let‘s randomly initialize our weight matrices. We‘ll have a weights matrix between the input and hidden layer (W1) and a weights matrix between the hidden and output layer (W2).

# Initialize weights
W1 = np.random.randn(num_input_nodes, num_hidden_nodes)
W2 = np.random.randn(num_hidden_nodes, num_output_nodes)

Next we need an activation function to introduce non-linearity into our network. We‘ll use the sigmoid function:

# Sigmoid activation function
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

Now we‘re ready to implement the forward propagation step. This will take an input, pass it through the network, and generate an output prediction.

# Forward propagation
def forward(x, W1, W2):
    # Hidden layer
    z1 = np.dot(x, W1)
    a1 = sigmoid(z1)

    # Output layer
    z2 = np.dot(a1, W2)
    a2 = sigmoid(z2)

    return a2

Let‘s break this down. First we compute the weighted sum of the inputs to the hidden layer (z1) and pass it through the activation function to get the hidden layer output (a1). We then compute the weighted sum of the hidden layer outputs (z2) and pass it through the activation function to get the final prediction (a2).

To train the network, we need a way to quantify how wrong its predictions are. We‘ll use mean squared error as our loss function:

# Loss function 
def loss(prediction, y):
    return np.mean((prediction - y)**2)

To improve its predictions, the network needs to adjust its weights via backpropagation. Here‘s how we‘ll implement that:

# Backpropagation
def backprop(x, y, prediction, W1, W2):
    # Compute gradients
    dL_dz2 = 2 * (prediction - y) * sigmoid(prediction) * (1 - sigmoid(prediction))
    dL_dW2 = np.dot(a1.T, dL_dz2)

    dL_da1 = np.dot(dL_dz2, W2.T)
    dL_dz1 = dL_da1 * sigmoid(a1) * (1 - sigmoid(a1))
    dL_dW1 = np.dot(x.T, dL_dz1)

    # Update weights
    W1 -= learning_rate * dL_dW1  
    W2 -= learning_rate * dL_dW2

    return W1, W2

This looks complex, but it‘s just applying the chain rule to compute gradients. We first calculate how much the loss changes with respect to the output layer weights (dL_dW2). Then we calculate how much the loss changes with respect to the hidden layer weights (dL_dW1). Finally, we update the weights by taking a small step in the negative gradient direction.

We‘re ready to train! Let‘s put it all together:

# Train 
learning_rate = 0.1
losses = []

for epoch in range(1000):
    for x, y in zip(X, y):
        # Forward pass
        prediction = forward(x, W1, W2)

        # Compute loss  
        L = loss(prediction, y)
        losses.append(L)

        # Backpropagation
        W1, W2 = backprop(x, y, prediction, W1, W2)

print(f"Final Loss: {losses[-1]:.4f}")        

We train for 1,000 epochs, performing forward propagation, computing loss, and backpropagation for each training example in each epoch. After training, our final loss is quite low, indicating our network has learned to fit the training data well.

Let‘s visualize the training process:

# Plot loss over time  
import matplotlib.pyplot as plt

plt.figure(figsize=(12,5))
plt.plot(losses)
plt.xlabel("Iteration")
plt.ylabel("Loss")
plt.show()

[Plot of loss decreasing over iterations]

We can see the loss steadily decreasing as the network learns.

Finally, let‘s test our trained network on new data:

# Test
test_data = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
predictions = [forward(x, W1, W2) for x in test_data]

print(f"Predictions: {[f‘{x[0]:.4f}‘ for x in predictions]}")

Our network successfully learned the XOR pattern!

Implementing a Neural Network from Scratch in R

Now let‘s implement the same neural network in R. The code will look quite similar.

First, we need to load required packages and generate the dummy data:

library(dplyr)

# Generate dummy data
X <- matrix(c(0, 0, 0, 1, 1, 0, 1, 1), nrow=4, byrow=TRUE) 
y <- matrix(c(0, 1, 1, 0), nrow=4)

Next, define the network architecture and randomly initialize weights:

# Define network architecture
num_input_nodes <- ncol(X) 
num_hidden_nodes <- 2
num_output_nodes <- 1

# Initialize weights  
W1 <- matrix(rnorm(num_input_nodes * num_hidden_nodes), nrow=num_input_nodes)
W2 <- matrix(rnorm(num_hidden_nodes * num_output_nodes), nrow=num_hidden_nodes)

Define the sigmoid activation function, loss function, and forward propagation:

# Sigmoid activation 
sigmoid <- function(x){
  1 / (1 + exp(-x))
}

# Loss function
loss <- function(prediction, y){
  mean((prediction - y)^2)
}

# Forward propagation  
forward <- function(x, W1, W2){
  # Hidden layer
  z1 <- x %*% W1
  a1 <- sigmoid(z1)

  # Output layer
  z2 <- a1 %*% W2
  a2 <- sigmoid(z2)

  return(a2)
}

Implement backpropagation:

backprop <- function(x, y, prediction, a1, W1, W2){
  # Gradients
  dL_dz2 <- 2 * (prediction - y) * sigmoid(prediction) * (1 - sigmoid(prediction)) 
  dL_dW2 <- t(a1) %*% dL_dz2

  dL_da1 <- dL_dz2 %*% t(W2)
  dL_dz1 <- dL_da1 * sigmoid(a1) * (1 - sigmoid(a1))
  dL_dW1 <- t(x) %*% dL_dz1

  # Update weights
  W1 <- W1 - learning_rate * dL_dW1
  W2 <- W2 - learning_rate * dL_dW2

  return(list(W1, W2))
}

Train the network:

# Train
learning_rate <- 0.1
num_epochs <- 1000
losses <- c()

for(epoch in 1:num_epochs){

  for(i in 1:nrow(X)){
    x <- X[i,]

    # Forward propagation 
    prediction <- forward(x, W1, W2)

    # Compute loss
    L <- loss(prediction, y[i])
    losses <- c(losses, L)

    # Backpropagation
    a1 <- sigmoid(x %*% W1) 
    weights <- backprop(x, y[i], prediction, a1, W1, W2)
    W1 <- weights[[1]] 
    W2 <- weights[[2]]
  }

}

cat(sprintf("Final Loss: %.4f\n", losses[length(losses)]))

Visualize the training process:

# Plot loss 
plot(losses, type="l", xlab="Iteration", ylab="Loss", main="Training Progress")

And finally test on new data:

# Test  
test_data <- X
predictions <- apply(test_data, 1, function(x) forward(x, W1, W2))

cat(sprintf("Predictions: %s\n", paste(sprintf("%.4f", predictions), collapse=", ")))

The R implementation gives the same results as the Python version. The core concepts are the same in any programming language.

Neural Network Applications and Advances

We‘ve walked through how to code up a simple neural network from scratch. In practice, you‘d use higher level libraries like TensorFlow or PyTorch that provide abstractions and speed optimizations. But knowing what‘s happening under the hood conceptually is invaluable for debugging more complex models.

So what are neural networks used for? Some major applications include:

• Computer Vision: Convolutional Neural Networks (CNNs) can classify images, detect objects, segment images and more with human-like or even superhuman accuracy. CNNs power vision systems in self-driving cars, facial recognition, medical image analysis and many other domains.

• Natural Language Processing: Recurrent Neural Networks (RNNs) and Transformers can understand, analyze and generate human language. They are the workhorse behind language translation, text summarization, sentiment analysis, chatbots and more. Large language models like GPT-3 can engage in open-ended dialogue and even write code.

• Speech Recognition: Deep neural networks can transcribe speech to text in real time, powering voice assistants like Siri and Alexa as well as automatic closed captioning systems.

• Recommender Systems: Neural networks can learn user preferences from behavior data and generate highly personalized recommendations, fueling recommendation engines at Netflix, Spotify, Amazon and elsewhere.

• Robotics: Deep reinforcement learning, often using neural networks, allows robots to learn complex behaviors through trial and error, such as grasping objects or walking over uneven terrain.

Since 2012 and the deep learning revolution, neural networks have grown much larger and more sophisticated. Here are some of the latest advancements as of 2023:

• Transformers have become the dominant architecture for sequential data, replacing older recurrent neural networks. Transformers power large language models, but are also increasingly used in other domains like vision and robotics.

• Neural Architecture Search (NAS) automates the process of designing neural network architectures. NAS systems can discover novel architectures that outperform hand-designed ones.

• Diffusion models are an alternative to GANs for image and audio generation that have produced some of the most realistic synthetic data to date.

• Foundational models are large pretrained models that can be adapted or "fine-tuned" for a wide variety of downstream tasks with little labeled data. Large language models like GPT-3 are the most famous examples.

• Efficient Transformers like the Reformer, Longformer, and Linformer aim to reduce the memory and compute requirements of standard Transformers, allowing them to scale to even longer sequences.

• Graph Neural Networks can directly operate on graph-structured data like molecules, social networks, and knowledge graphs. They have shown promising results on tasks like drug discovery and fraud detection.

It‘s an exciting time for neural networks and deep learning. We‘ve made immense progress in the last decade, but there is still much to discover. Novel neural network architectures and training methods are an active area of research that may lead to more powerful, efficient, and interpretable models in the future.

Wrapping Up

In this post, we started with the basics of how neural networks work, then implemented a simple neural network from scratch in both Python and R. By coding it ourselves, we gained an intimate understanding of the core forward and backward propagation algorithms under the hood.

We also discussed some of the current major application areas for neural networks and recapped the most exciting recent advancements in neural network architectures and algorithms as of 2023.

I hope this tutorial gave you a solid foundation for understanding neural networks. Of course, we only scratched the surface – there are many more architectures and training techniques we didn‘t cover. But now you have the conceptual building blocks to go out and learn about Convolutional Neural Networks, Recurrent Neural Networks, Reinforcement Learning and other more advanced topics.

Neural networks are a powerful and versatile tool that will likely only grow more prevalent and impactful in the coming years. I believe it‘s important for anyone working in software, and really any technical field, to have at least a basic working knowledge of neural networks given their growing ubiquity. I hope this post helped demystify them and bring you up to speed on the fundamentals.

Thanks for reading! Let me know if you have any questions or insights to share.