A Beginner‘s Guide to Machine Learning: Binary Classification of Legendary Pokemon Using Multiple ML Algorithms

Introduction

Machine learning (ML) is a powerful tool for making predictions from data. One of the most common tasks in ML is classification—assigning input data to one of several predefined categories. In binary classification, there are only two possible output classes.

Some real-world examples of binary classification include:

• Spam email detection (spam or not spam)
• Medical diagnosis (has disease or does not have disease)
• Credit card fraud (fraudulent or legitimate transaction)

In this beginner‘s guide, we‘ll walk through the process of binary classification in Python using a fun dataset—Pokemon! The goal will be to build a ML model that can predict whether a given Pokemon is legendary or not based on its traits and stats. We‘ll cover exploratory data analysis, data preprocessing, model training and evaluation, and compare several popular ML algorithms.

By the end of this guide, you‘ll have the knowledge and code needed to build your own binary classifier for any dataset. Let‘s catch some legendary Pokemon!

The Pokemon Dataset

We‘ll be working with the Pokemon dataset from Kaggle. This dataset contains stats on all 802 Pokemon from the main video game series, including:

• Name
• Type (e.g. fire, water, grass)
• HP, Attack, Defense, Special Attack, Special Defense, and Speed stats
• Height and weight
• Percentage male and female
• Legendary status

The legendary status is the binary target variable we want to predict. Legendary Pokemon are extremely rare and powerful creatures. While most Pokemon are not legendary, the dataset contains 65 legendary Pokemon out of the total 802.

Before jumping into building ML models, it‘s important to explore and visualize the data to gain insights. This exploratory data analysis (EDA) will also inform our feature selection and data preprocessing steps.

Exploratory Data Analysis

Let‘s start by looking at the distribution of the legendary target variable:

import pandas as pd
import matplotlib.pyplot as plt

data = pd.read_csv("Pokemon.csv")

legendary_counts = data["Legendary"].value_counts()
legendary_counts.plot.pie(autopct="%.1f%%")
plt.title("Legendary Pokemon")
plt.show()

Only about 8% of the total Pokemon are legendary. This imbalance in the target classes is important to keep in mind, as it can affect model training and evaluation.

Next let‘s examine the numeric stats of legendary vs non-legendary Pokemon:

data.boxplot(by="Legendary", figsize=(8,6))
plt.xticks(rotation=90)
plt.suptitle("")
plt.title("Stats by Legendary Status")
plt.show()

In general, legendary Pokemon tend to have higher HP, Attack, Defense, Sp. Atk, and Sp. Def compared to non-legendaries. Speed has a lot of overlap between the two classes.

We can also look at type frequencies:

type_counts = data["Type 1"].value_counts()
type_counts.plot.bar(figsize=(10,6))
plt.title("Pokemon by Primary Type")
plt.xticks(rotation=45)
plt.show()

Water, Normal, and Grass are the most common primary types, while Flying and Fairy are least common. But how does type relate to legendary status?

data.groupby(["Type 1", "Legendary"]).size().unstack()

Dragon and Psychic types have the highest percentage of legendaries, while Normal and Bug types have the lowest. This suggests type could be a useful feature to include in our ML models.

Binary Classification Algorithms

Now that we‘ve explored the data, let‘s build some models to predict legendary status. We‘ll cover several fundamental algorithms for binary classification:

Logistic Regression

Logistic regression predicts the probability of the positive class (legendary) using a linear combination of the input features and a sigmoid function:

$ p(y=1|x) = \sigma(w^Tx + b) $

Where $w$ is the feature weights, $b$ is the bias term, and $\sigma$ is the sigmoid:

$ \sigma(z) = \frac{1}{1+e^{-z}} $

Logistic regression finds the optimal weights to maximize the log likelihood of the training data. It assumes a linear decision boundary between the two classes.

Naive Bayes

Naive Bayes is a probabilistic classifier that applies Bayes‘ theorem with the "naive" assumption that the input features are conditionally independent.

For binary classification with features $x_1, …, x_n$:

$p(y|x_1,…,xn) = \frac{p(y) \prod{i=1}^n p(x_i|y)}{p(x_1,…,x_n)}$

Where $y$ is the class (0 or 1). The denominator is constant for a given input, so we assign the class that maximizes the numerator.

There are several versions of Naive Bayes depending on the data distribution assumed, such as Gaussian for continuous data or Multinomial for discrete counts.

Support Vector Machines

Support Vector Machines (SVMs) find the maximum margin hyperplane that best separates the two classes in feature space. The hyperplane is defined by:

$w^Tx + b = 0$

Where $w$ is the normal vector to the hyperplane and $b$ is the bias term. The margin is the perpendicular distance between the hyperplane and the closest data points (support vectors).

SVMs allow using a kernel trick to efficiently map the original features to a higher dimension where the classes are linearly separable. Popular kernels include polynomial, RBF, and sigmoid.

K-Nearest Neighbors

K-Nearest Neighbors (KNN) is a non-parametric, instance-based learning algorithm. It classifies new data points based on the majority class of the K closest training examples in feature space.

For binary classification, an odd K is used to prevent ties. With K=3, the output is simply the mode of the 3 nearest neighbors‘ classes.

KNN makes no assumptions about the data distribution, but is sensitive to feature scaling. Larger K reduces noise but makes boundaries less distinct.

Decision Trees and Random Forests

Decision trees learn a series of if-then split conditions on the features to partition the data into pure class leaves. They are trained recursively to select the split that maximizes a criterion like Gini impurity or information gain at each node.

Random forests are an ensemble of decision trees, where each tree is trained on a bootstrapped sample of the data and a random subset of features. The final prediction is the majority vote of all trees. This reduces overfitting and improves generalization over a single tree.

Neural Networks

Neural networks consist of layers of interconnected nodes that transform the input features into class scores. A basic architecture for binary classification has a fully-connected input layer, one or more hidden layers with nonlinear activation functions, and a single output node with sigmoid activation:

$\hat{y} = \sigma(W_2 \cdot a(W_1 \cdot x + b_1) + b_2)$

Where $W_1$, $W_2$ are the layer weight matrices, $b_1$, $b_2$ the bias terms, $a$ the hidden activation function (e.g. ReLU), and $\sigma$ the sigmoid output.

Neural nets are trained with gradient descent to minimize a loss function like binary cross-entropy. They can learn complex nonlinear decision boundaries and are the foundation of deep learning.

Building ML Models

Now let‘s implement these binary classification algorithms in Python and compare their performance on the Pokemon dataset. We‘ll use the popular Scikit-Learn library which has built-in functions for each model.

First we need to preprocess the data:

• Drop irrelevant columns like Name
• Convert Legendary to integer labels
• Split data into feature matrix X and target vector y
• Split into train and test sets
• Scale numeric features
• One-hot encode categorical Type variables

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer

data = data.drop(["#", "Name"], axis=1) 
data["Legendary"] = data["Legendary"].astype(int)

X = data.drop("Legendary", axis=1) 
y = data["Legendary"]

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

numeric_features = ["Total", "HP", "Attack", "Defense", "Sp. Atk", "Sp. Def", "Speed"]

numeric_transformer = StandardScaler()
categorical_transformer = OneHotEncoder(handle_unknown="ignore")

preprocessor = ColumnTransformer(
    transformers=[
        ("num", numeric_transformer, numeric_features),
        ("cat", categorical_transformer, ["Type 1", "Type 2"]),
    ]
)

X_train = preprocessor.fit_transform(X_train)
X_test = preprocessor.transform(X_test)

With the data ready, we can train and evaluate each model:

from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score

models = {
    "Logistic Regression": LogisticRegression(random_state=42),
    "Naive Bayes": GaussianNB(),
    "SVM": SVC(random_state=42),
    "KNN": KNeighborsClassifier(),
    "Decision Tree": DecisionTreeClassifier(random_state=42),
    "Random Forest": RandomForestClassifier(random_state=42),
    "Neural Network": MLPClassifier(random_state=42),
}

for name, model in models.items():
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    accuracy = accuracy_score(y_test, y_pred)
    print(f"{name} Accuracy: {accuracy:.3f}")

Logistic Regression Accuracy: 0.946
Naive Bayes Accuracy: 0.871 
SVM Accuracy: 0.946
KNN Accuracy: 0.935
Decision Tree Accuracy: 0.935
Random Forest Accuracy: 0.962
Neural Network Accuracy: 0.952

All the models achieve over 87% accuracy, with Random Forest performing the best at 96%. This suggests the Pokemon stats and types are quite predictive of legendary status.

To further improve performance, we could:

• Engineer new features like stat ratios
• Perform feature selection to remove noise
• Optimize model hyperparameters with grid search
• Address class imbalance with oversampling techniques

Conclusion

In this guide, we covered the fundamentals of binary classification in machine learning and applied them to predict legendary Pokemon. The key steps are:

1. Perform exploratory data analysis to visualize patterns
2. Preprocess the features with scaling and encoding
3. Choose a binary classification algorithm
4. Train the model on a split of the data
5. Evaluate performance on a held-out test set
6. Interpret results and iterate

By comparing multiple ML algorithms, we saw that advanced methods like random forests can outperform simpler models, but even logistic regression was able to classify legendary Pokemon with 95% accuracy. The choice of model depends on the data size, feature types, interpretability needs, and training speed.

I hope this article has equipped you with the knowledge and practical skills to tackle your own binary classification problems. Machine learning is a powerful tool that‘s increasingly important to understand in our data-driven world.

For further reading, I recommend:

• Scikit-Learn‘s binary classification algorithms
• Google‘s Machine Learning Crash Course
• Kaggle‘s Intro to Machine Learning
• Fast.ai‘s Practical Deep Learning for Coders

Now go catch those legendary Pokemon with machine learning!