A Beginner‘s Guide to Named Entity Recognition with spaCy

Have you ever wondered how Siri or Alexa are able to understand and extract key information from the sentences you speak? One of the core natural language processing (NLP) techniques that powers this capability is named entity recognition, or NER for short.

In this in-depth, hands-on tutorial, we‘ll walk through what NER is, how it works, and most importantly – how you can implement it yourself in Python using the powerful open-source library spaCy. By the end, you‘ll be able to extract entities like people, places, organizations, and more from raw text. Let‘s dive in!

What is Named Entity Recognition?

Named entity recognition is the task of identifying and categorizing key information (entities) in text. An entity can be any word or series of words that consistently refers to the same thing. Every detected entity is classified into a pre-defined category, such as a person, place, organization, time, or events.

For example, let‘s say we have the following sentence:

"Microsoft was founded by Bill Gates and Paul Allen on April 4, 1975, in Albuquerque, New Mexico."

An NER model would detect and classify the following entities:

• Microsoft – Organization
• Bill Gates – Person
• Paul Allen – Person
• April 4, 1975 – Time
• Albuquerque – Place
• New Mexico – Place

NER is a form of information extraction that has many useful applications, such as:

• Extracting key details from articles, documents, or web pages
• Organizing and categorizing content by topics
• Identifying mentions of companies and people for analysis
• Automating data entry by extracting structured data from unstructured text
• Improving search algorithms and chatbots by understanding questions better

NER is a key building block of NLP that enables language understanding beyond simple keywords.

Introduction to spaCy

To perform NER in Python, we‘ll be using the spaCy library. spaCy is a free, open-source library for advanced NLP. It‘s designed to be fast, efficient, and production-ready.

Some key features of spaCy include:

• Pre-trained models for many languages
• Support for training custom models
• Easy-to-use API for common NLP tasks
• Fast processing that‘s scalable to large volumes
• Interoperability with deep learning libraries like TensorFlow and PyTorch

spaCy is well-suited for NER because it comes with pre-trained models that can detect a wide range of named entities out-of-the-box. It also makes it straightforward to train your own custom NER model, as we‘ll see later on.

Setting Up Your Environment

Before we can get started with the hands-on part of this tutorial, you‘ll need to have spaCy installed. You can install spaCy with pip:

pip install spacy

spaCy comes with different pre-trained models for each language. For this tutorial, we‘ll be using the English model, which you can download using the following command:

python -m spacy download en_core_web_sm

This will download the small English model, which is sufficient for our examples. There are also larger models available that are more accurate but take longer to load.

spaCy NER Tutorial

Now let‘s get started implementing NER with spaCy! We‘ll walk through the steps with explanations and code samples.

Step 1: Import libraries and load data

First, let‘s import the libraries we need and load our sample data. We‘ll use some example sentences from Wikipedia articles.

import spacy

text = "Microsoft was founded by Bill Gates and Paul Allen on April 4, 1975, in Albuquerque, New Mexico. It is one of the largest software companies headquartered in Washington state."

Step 2: Load the spaCy model

Next, we need to load the pre-trained spaCy model for English that we downloaded earlier:

nlp = spacy.load("en_core_web_sm")

This will load the small English model into a nlp object that we can use to process text.

Step 3: Process the text

Now we can pass our text to the nlp object to process it and perform NER:

doc = nlp(text)

This runs the text through spaCy‘s processing pipeline, which tokenizes the text, performs part-of-speech tagging, parses dependencies, and most importantly for us – detects named entities.

Step 4: Extract detected entities

To get the detected entities, we can iterate through the doc.ents property:

for ent in doc.ents:
    print(ent.text, ent.label_)

This will print out each entity text along with its predicted label. The output looks like:

Microsoft ORG
Bill Gates PERSON
Paul Allen PERSON
April 4, 1975 DATE
Albuquerque GPE
New Mexico GPE
Washington GPE

We can see that spaCy has successfully detected and classified the entities in our text! The labels tell us that Microsoft is an organization, Bill Gates and Paul Allen are people, April 4, 1975 is a date, and Albuquerque, New Mexico, and Washington are geopolitical entities (places).

Step 5: Visualize the entities

spaCy also provides a convenient way to visualize the detected entities using the displacy module. We can render the doc with the entities highlighted:

from spacy import displacy

displacy.render(doc, style="ent")

This will display the text with the entities color-coded by their label. Here‘s what the visualization looks like:

[Visualization of entities using displacy]

The visualization really helps bring the entities to life and makes it easy to see the relationships between them at a glance.

Evaluating NER Performance

So far we‘ve seen that spaCy can detect entities in our sample text with reasonable accuracy. But how well does it actually perform in general?

To evaluate an NER model, we typically use a labeled dataset that has the "true" entities marked. We can then compare the model‘s predicted entities to the ground truth and calculate metrics like precision, recall and F1 score.

Precision measures the percentage of entities detected by the model that are actually correct. Recall measures the percentage of actual entities that were detected by the model. F1 score is the harmonic mean of precision and recall.

spaCy makes it easy to evaluate a model‘s performance on a labeled dataset using the evaluate method. For example:

scores = nlp.evaluate("data/labeled_data.json")
print(scores["ents_p"], scores["ents_r"], scores["ents_f"])

This evaluates the loaded nlp model on a labeled dataset loaded from a JSON file and prints out the precision, recall and F1 scores for the entities.

Training a Custom NER Model

While spaCy‘s default pre-trained models are great for many use cases, sometimes you may need to detect custom entities that aren‘t included in the default. For example, maybe you want to detect product names, book titles, or other domain-specific entities.

The good news is that spaCy makes it relatively straightforward to train your own custom NER model! The basic steps are:

1. Create a blank nlp model
2. Add a new entity recognizer to the pipeline
3. Prepare training data in the format spaCy expects
4. Call nlp.begin_training() and loop for a number of iterations:
5. Shuffle the training data
6. Loop through each text, nlp.update() with the text and annotations
7. Save the trained model to disk

Here‘s a simplified code snippet showing these steps:

import spacy
import random

model = spacy.blank("en")
ner = nlp.create_pipe("ner")
nlp.add_pipe(ner)
ner.add_label("PRODUCT")

train_data = [
    ("buy iPad Pro", {"entities": [(4, 11, "PRODUCT")]}),
    ("order iPhone 12", {"entities": [(6, 15, "PRODUCT")]}),
    ("purchase MacBook Air", {"entities": [(9, 20, "PRODUCT")]}),
]

nlp.begin_training()
for itn in range(5):
    random.shuffle(train_data) 
    for text, annotations in train_data:
        nlp.update([text], [annotations])
nlp.to_disk("product_ner_model")

This trains a blank English model to detect a custom PRODUCT entity based on a few examples. The key steps are creating the new entity type, providing example training data in the expected format of tuples of (text, {"entities": [(start, end, label)]}), and calling nlp.update() to show the model the text and labels.

After training, we can save out the model so we can load it later for inference on new data. Training a custom model does require having labeled example data, but it allows you to detect any entity types relevant to your use case.

Improving NER with Rules

Another way to customize and improve NER results is by adding rules on top of the statistical model. spaCy supports rule-based matching to find and label entities based on patterns you define.

For example, let‘s say we want to detect versions of Microsoft Windows operating system in text:

import spacy
from spacy.matcher import PhraseMatcher

nlp = spacy.load("en_core_web_sm")
matcher = PhraseMatcher(nlp.vocab)

patterns = [nlp.make_doc(name) for name in ["Windows 10", "Windows 8.1", "Windows 7"]]
matcher.add("WINDOWS", None, *patterns)

text = "Sales of PCs with Windows 10 pre-installed have surpassed those with Windows 8.1 and Windows 7 combined."
doc = nlp(text)

matches = matcher(doc)
for match_id, start, end in matches:
    entity = doc[start:end]
    print(entity.text, entity.label_)

This creates a PhraseMatcher with patterns for the Windows version names we want to detect. It then finds matches of those patterns in the input doc. Finally, it prints out the matched text and assigns a label of "WINDOWS".

The output is:

Windows 10 WINDOWS
Windows 8.1 WINDOWS
Windows 7 WINDOWS

This is a simple example, but it demonstrates how you can use rules to match specific entity patterns that may not be captured by the statistical model. Rules can help improve the precision of your NER results for entities with predictable patterns.

Case Study: Analyzing News Articles

To show a realistic application of using spaCy for NER, let‘s consider an example of analyzing a collection of news articles to extract key entities mentioned.

Imagine you have a dataset of 1000 news articles covering various topics like politics, business, sports and entertainment. Your goal is to get an overview of the most frequently mentioned people, places and organizations.

You could use spaCy‘s NER to process each article and extract entities, something like:

import spacy
import pandas as pd
from collections import Counter

nlp = spacy.load("en_core_web_sm")

news = pd.read_csv("news_dataset.csv")

entities = []
for article in news["article_text"]:
    doc = nlp(article) 
    for ent in doc.ents:
        entities.append((ent.text, ent.label_))

entity_counts = Counter(entities)
top_entities = entity_counts.most_common(10)

print(top_entities)

This would load in the news dataset from a CSV file, process each article‘s text with spaCy NER, collect all detected entities in a list of tuples (entity_text, entity_label), count the frequencies of each entity using a Counter, and finally print out the top 10 most frequent entities.

The result might look something like:

[(‘Donald Trump‘, ‘PERSON‘), 235,
 (‘United States‘, ‘GPE‘), 212,
 (‘Google‘, ‘ORG‘), 187,
 (‘White House‘, ‘FAC‘), 156,
 ...
]

This gives you a quick snapshot of the most frequent and important entities discussed across the articles, which could help summarize the topics covered. Of course, you can further slice and dice the entities to analyze things like the top people or places mentioned, or the entities associated with articles on certain topics.

The key point is that by leveraging NER to extract entities from raw text, you unlock the ability to analyze and derive insights from large collections of unstructured text data.

Next Steps

In this guide, we‘ve covered the basics of how to use the spaCy library to perform named entity recognition on text data in Python. We‘ve walked through the key steps of loading a pre-trained NER model, extracting entities from text, evaluating NER performance, training a custom model, adding rule-based improvements, and analyzing a real-world news dataset.

To continue learning about NER and NLP with spaCy, here some great resources to check out next:

• The official spaCy documentation for API references and usage guides
• spaCy course for in-depth tutorials and exercises
• Real-world spaCy projects for examples and inspiration

Additional techniques to explore include using word vectors and deep learning to boost NER performance, building end-to-end pipelines that combine multiple NLP tasks, and deploying your trained models in production applications.

Thanks for reading, and happy entity extraction with spaCy!