A Hands-on Introduction to Reinforcement Learning with Python

Reinforcement learning (RL) is an exciting area of machine learning that has seen tremendous progress and real-world impact in recent years. From achieving superhuman performance in complex games like Go and Starcraft to enabling robots to learn advanced locomotion and manipulation skills, RL has proven to be a powerful framework for artificial intelligence.

In this post, we‘ll dive into the key concepts of reinforcement learning, understand how it differs from other machine learning paradigms, and walk through hands-on Python code examples to get a practical feel for RL. By the end, you‘ll have a solid foundation to start applying RL to your own projects!

What is Reinforcement Learning?

At a high level, reinforcement learning is a computational approach to learning from interaction. An RL agent learns by directly interacting with an environment, receiving feedback in the form of positive or negative rewards, and adapting its behavior to maximize its cumulative reward over time.

This is quite different from other types of machine learning:

• In supervised learning, an algorithm learns from labeled example data to map inputs to known output labels. It is a form of learning from instruction.

• In unsupervised learning, an algorithm identifies patterns and structure in unlabeled data. It is a form of learning from patterns.

• In reinforcement learning, an agent learns through trial-and-error interactions with a dynamic environment. It is a form of learning from interaction.

RL is inspired by how humans and animals learn – we don‘t always have clearly labeled data or an instructor telling us the right action to take. More often, we learn through experience, taking actions, observing outcomes, and adapting our future behavior to achieve our goals.

Key Concepts in Reinforcement Learning

To formalize how an RL agent learns, let‘s walk through the key terminology:

• Agent: The algorithm or entity that is learning to make decisions and take actions. This could be a physical robot, a character in a game, or a program trading stocks.

• Environment: The world that the agent interacts with and learns from. The environment determines the state the agent is in, takes the agent‘s actions as input, and returns the next state and reward feedback. Environments can range from simulated games and robot simulators to the real physical world.

• State: A description of the agent‘s current situation in the environment. A chess board position, a robot‘s sensor readings, or a user‘s browsing history could all be states.

• Action: A decision the agent can take to interact with the environment. Moving a chess piece, a robot joint motor command, or showing an ad to a user are all examples of actions.

• Reward: A scalar feedback signal from the environment indicating how well the agent is doing. Rewards can be positive or negative. Winning a game, reaching a goal position, or getting a click on an ad could be positive rewards. Losing a game, running out of battery, or showing an irrelevant ad could be negative rewards.

• Policy: The agent‘s strategy or behavior function that maps states to actions. This is what we are trying to learn – an optimal policy that maximizes long-term cumulative reward.

• Value Function: A prediction of the expected long-term cumulative reward from a given state or state-action pair. This helps the agent evaluate the goodness of states and therefore take actions that will lead to better states.

The goal in reinforcement learning is to learn an optimal policy – a behavioral strategy that maximizes the long-term cumulative reward the agent receives from the environment.

The Reinforcement Learning Loop

The agent learns this optimal policy through repeated interaction with the environment in a feedback loop:

1. The agent observes the current state s of the environment
2. Based on the current policy, the agent takes an action a
3. The environment transitions to a new state s‘ and emits a reward r
4. The agent updates its policy and/or value function based on this experience (s, a, r, s‘)
5. The process repeats from step 1 with the new state s‘

Over many cycles of this interaction loop, the agent incrementally improves its policy, learning to take actions that lead to better long-term outcomes as measured by the reward signal.

Under the hood, most RL algorithms can be understood as trying to approximate solutions to Markov Decision Processes (MDPs). An MDP is a mathematical framework for modeling sequential decision making under uncertainty, which perfectly captures the setting of an agent interacting with an environment. The key aspects of an MDP are:

• A set of states S the environment can be in
• A set of actions A the agent can take
• A transition function T(s,a,s‘) specifying the probability of transitioning from state s to state s‘ when taking action a
• A reward function R(s,a,s‘) specifying the reward received when transitioning from s to s‘ with action a
• A discount factor γ between 0 and 1 that weights the importance of immediate vs future rewards

Given an MDP, the goal is to find a policy π(s) (a mapping from states to actions) that maximizes the expected cumulative discounted reward:

This expression sums up the core idea – find a policy that maximizes the expected cumulative reward over trajectories of experience in the environment.

With this theoretical foundation in mind, let‘s now turn to practical RL algorithms and code examples in Python.

Hands-on RL in Python

There are several excellent open source libraries that make it easy to implement and experiment with reinforcement learning in Python. Some of the most popular ones are:

• OpenAI Gym: A toolkit for developing and comparing RL algorithms, providing a collection of benchmark environments from classic control problems to Atari games.

• Stable Baselines: A set of high-quality implementations of RL algorithms in PyTorch, with unified interfaces and well-tuned hyperparameters.

• RLlib: A scalable library for RL that provides both high-level APIs for training popular algorithms and low-level building blocks for customization.

Let‘s walk through a simple example using OpenAI Gym to solve the classic CartPole balancing problem with the Q-learning RL algorithm. In this problem, a pole is attached by an unactuated joint to a cart which moves along a track. The system is controlled by applying a force of +1 or -1 to the cart. The pendulum starts upright, and the goal is to prevent it from falling over by increasing and reducing the cart‘s velocity.

First, install the gym library:

pip install gym

Next, let‘s create the CartPole environment:

import gym

env = gym.make(‘CartPole-v1‘)

We can inspect the state and action spaces:

print(env.observation_space) 
# Box([-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38], [4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38], (4,), float32)
print(env.action_space)  
# Discrete(2)

The state is a 4D vector indicating the cart position, cart velocity, pole angle, and pole angular velocity. The actions are discrete – either push left (0) or push right (1).

Now let‘s define our Q-learning agent. Q-learning maintains a table of Q-values Q(s,a) estimating the expected return of taking action a in state s. The Q-values are updated based on experience using the Bellman equation:

We can discretize the continuous state space into bins to make the Q-table tractable:

import numpy as np

# Discretize the continuous state space for each of the 4 dimensions
NUM_BUCKETS = (1, 1, 6, 3)  # (x, x‘, theta, theta‘)
NUM_ACTIONS = env.action_space.n  # (left, right)

# Initialize Q table
q_table = np.zeros(NUM_BUCKETS + (NUM_ACTIONS,))

Next we define a function to discretize the state:

def discretize(obs):
    upper_bounds = [env.observation_space.high[0], 0.5, env.observation_space.high[2], math.radians(50)]
    lower_bounds = [env.observation_space.low[0], -0.5, env.observation_space.low[2], -math.radians(50)]
    ratios = [(obs[i] - lower_bounds[i]) / (upper_bounds[i] - lower_bounds[i]) for i in range(len(obs))]
    new_obs = [int(round((NUM_BUCKETS[i] - 1) * ratios[i])) for i in range(len(obs))]
    new_obs = [min(NUM_BUCKETS[i] - 1, max(0, new_obs[i])) for i in range(len(obs))]
    return tuple(new_obs)

Finally, we are ready to train the agent using Q-learning:

import math

# Hyperparameters
NUM_EPISODES = 1000
MAX_T = 250
ALPHA = 0.1
GAMMA = 1.0
EPSILON_START = 1.0
EPSILON_END = 0.01
EPSILON_DECAY = 500

epsilon = EPSILON_START

# Run Q learning algorithm
for episode in range(NUM_EPISODES):
    # Initialize environment for this episode
    obs = env.reset()
    state = discretize(obs)

    done = False
    truncated = False

    while not done and not truncated:
        # Get action from Q table
        if np.random.uniform(0, 1) < epsilon:
            action = env.action_space.sample()
        else:
            action = np.argmax(q_table[state])

        # Take action and get new measurements
        next_obs, reward, done, truncated, _ = env.step(action)
        next_state = discretize(next_obs)

        # Update Q value 
        q_table[state][action] += ALPHA * (reward + GAMMA * np.max(q_table[next_state]) - q_table[state][action])

        state = next_state

        if done or truncated:
            break

    # Decay epsilon
    epsilon = max(EPSILON_END, EPSILON_START - episode / EPSILON_DECAY)

We run for some number of episodes, following an epsilon-greedy policy. With probability epsilon we take a random action, otherwise we take the action with the highest Q-value for the current state. We update Q-values based on the experience, gradually decaying epsilon over time to more exploitation vs exploration.

Let‘s see how our trained agent performs:

# Evaluate agent
obs, _ = env.reset()
state = discretize(obs)

done = False
truncated = False

while not done and not truncated:
    action = np.argmax(q_table[state]) 
    next_obs, reward, done, truncated, _ = env.step(action)
    env.render()
    state = discretize(next_obs)

env.close()   

The trained agent is able to balance the pole well! This is a simple example, but the same principles of Q-learning can be scaled to more complex problems with function approximation instead of a discrete table. More advanced RL algorithms build on the ideas of Q-learning to achieve even better performance.

Applications and Frontiers

Beyond toy examples, reinforcement learning has seen exciting success in many real-world domains:

In robotics, RL has been used to learn dexterous manipulation skills, locomotion for legged robots, and vision-based robotic grasping policies. For example, researchers at OpenAI trained a five-fingered robot hand to solve a Rubik‘s cube using RL.

In game-playing, RL systems have achieved superhuman performance in checkers, chess, Go, Atari games, StarCraft, and Dota. DeepMind‘s AlphaGo used RL to become the first computer program to defeat a world champion at the complex game of Go.

RL is also being applied to optimize systems in energy, transportation, healthcare, finance, and more. For instance, RL can be used to learn optimal control policies for HVAC systems to minimize energy usage while meeting temperature constraints.

Recommender systems and online advertising can also be formulated as RL problems, learning policies to show relevant content to users to maximize long-term engagement.

Some key challenges and frontiers in RL research include:

Sample efficiency – RL often requires a large amount of interaction data, which can be costly or infeasible to obtain. There is ongoing research into off-policy, model-based, and transfer learning approaches to improve sample efficiency.

Safety – Ensuring RL agents behave safely and reliably, avoiding negative side effects and respecting constraints while exploring and optimizing.

Generalization – Making RL agents robust to changes in the environment and able to handle novel situations not seen during training.

Multi-agent RL – Extending single-agent RL to settings with multiple interacting agents, such as in autonomous driving or decentralized control systems.

Explainability – Understanding the strategies learned by RL agents and being able to explain their behavior, which is important for building trust and accountability.

Conclusion

Reinforcement learning is a powerful framework for learning sequential decision making policies through interaction with an environment. In this post, we covered the key concepts in RL, walked through a hands-on Python example, and discussed real-world applications and current frontiers.

RL has seen impressive achievements but there are still many exciting challenges to tackle. To learn more, check out these great resources:

• Reinforcement Learning: An Introduction (2nd Edition) by Richard S. Sutton and Andrew G. Barto – A comprehensive textbook introduction to RL

• Spinning Up in Deep RL by OpenAI – An educational resource to learn about deep RL

• CS285 Deep Reinforcement Learning at UC Berkeley – Lecture videos and assignments for a graduate deep RL course

• RLlib Documentation – Examples and tutorials for using RLlib to solve RL problems

I hope this post gave you a practical foundation to start exploring RL! The field is evolving rapidly and there are many opportunities to apply RL to impactful problems. Experiment with the code examples, extend them to harder problems, and aim to make an impact with RL.