A Comprehensive Guide to Named Entity Recognition Using spaCy in Python

Named entity recognition, or NER for short, is a fundamental task in natural language processing (NLP) that involves identifying and categorizing key information in unstructured text. Also known as entity chunking or extraction, NER is the process of parsing through text to locate named entities – the important nouns like people, places, organizations, dates, quantities, monetary values, percentages, and more – and classifying them into predefined categories.

NER is a form of information extraction that enables you to automatically structure and categorize the unstructured named entities scattered throughout your text corpus. By distilling the key named entities, NER allows you to transform a block of text into actionable structured data that can be used for analysis, visualization, knowledge graphs, and machine learning applications.

Some common use cases and applications of NER include:

• Identifying people, companies, and locations in news articles for knowledge discovery
• Extracting product names, prices, and specifications from e-commerce listings
• Recognizing the names of genes, proteins, diseases, and drugs from biomedical research
• Categorizing the names of ingredients, recipes, and food in restaurant reviews
• Detecting key events, dates, times, and participants from meeting notes to automate calendaring
• Tagging and indexing important entities in documents for efficient information retrieval

As you can see, NER has wide-ranging applications across domains like business, finance, healthcare, and more. By automating the identification and structuring of key named entities in text data, NER can greatly improve the efficiency and scalability of information extraction compared to manual annotation.

In this article, we‘ll take a deep dive into NER using the popular open-source Python library spaCy. With its exceptionally fast performance, concise API, and wide range of features, spaCy has become one of the go-to libraries for production-grade NLP in Python. We‘ll walk through how to use spaCy to perform NER on your own text data in Python, as well as cover some advanced topics and best practices to help you get the most out of spaCy for your NER projects.

Setting Up spaCy for NER in Python

Before we can get started with NER in spaCy, we need to make sure we have spaCy and its English language models installed. You can install spaCy with a simple pip command:

pip install spacy

Next, we‘ll download the English language model we want to use. For this tutorial, we‘ll use the small English model, but you can also use the medium or large models for improved accuracy at the expense of speed and memory.

python -m spacy download en_core_web_sm

With spaCy and the language model downloaded, we‘re ready to start using it for NER in Python.

Performing NER with spaCy

To demonstrate NER with spaCy, let‘s try it out on a sample news headline:

import spacy

# Load the language model
nlp = spacy.load("en_core_web_sm")

# Sample text
text = "Tesla CEO Elon Musk moves Tesla headquarters from Palo Alto, California to Austin, Texas"

# Run NER
doc = nlp(text)

# Print detected entities
for ent in doc.ents:
    print(ent.text, ent.start_char, ent.end_char, ent.label_)

Output:
Tesla 0 5 ORG
Elon Musk 9 18 PERSON
Tesla 28 33 ORG
Palo Alto 48 57 GPE
California 59 69 GPE
Austin 73 79 GPE
Texas 81 86 GPE

Here we can see that spaCy has identified several named entities in the text, including organizations (Tesla), people (Elon Musk), and geopolitical entities like cities and states (Palo Alto, California, Austin, Texas).

The spaCy NER model predicts both the text spans and categories for each entity. It returns the entity text, start and end character offsets, and the entity label.

Some of the default entity categories recognized by spaCy include:

• PERSON: People, including fictional characters
• ORG: Companies, agencies, institutions, etc.
• GPE: Geopolitical entities, i.e. countries, cities, states
• LOC: Non-GPE locations, mountain ranges, bodies of water
• PRODUCT: Objects, vehicles, foods, etc. (not services)
• DATE: Absolute or relative dates or periods
• CARDINAL: Numerals that do not fall under another type
• MONEY: Monetary values, including unit
• PERCENT: Percentage expressions

These default categories provide broad coverage, but you can also train spaCy to recognize custom entities relevant to your specific domain.

Visualizing Named Entities with displaCy

In addition to printing out the entities, spaCy also offers a built-in visualization tool called displaCy for rendering detected entities in a readable format.

We can use displaCy to visualize the entities in our example with just a few lines of code:

from spacy import displacy

# Render entities
displacy.render(doc, style="ent", jupyter=True)

This will display the text with the entities highlighted in color:

The displaCy visualizer is highly customizable, allowing you to control things like the entity colors, page template, and more. It‘s useful for debugging your NER models and visualizing results.

Training a Custom Named Entity Recognizer

While the pre-trained spaCy models provide good baseline performance, you‘ll often want to train a custom NER model to recognize entities specific to your domain.

Training a custom NER model in spaCy involves:

1. Annotating your training data with entity labels
2. Defining your custom entity labels
3. Training a new entity recognizer using the annotated data

To demonstrate, let‘s walk through a simplified example of training spaCy to recognize entities in restaurant reviews.

First, we‘ll create some training data by annotating review text with our desired entities:

TRAIN_DATA = [
    ("The pizza at Reggiano‘s is amazing!", {"entities": [(11, 20, "RESTAURANT"), (24, 30, "FOOD")]}),
    ("I loved the tacos from Paco‘s Tacos", {"entities": [(13, 18, "FOOD"), (24, 35, "RESTAURANT")]}),
    ("The sushi at Sushi Dojo blew me away", {"entities": [(4, 9, "FOOD"), (13, 23, "RESTAURANT")]}),
    ("Bento Box has the best teriyaki in town", {"entities": [(0, 9, "RESTAURANT"), (22, 30, "FOOD")]}),
    ("The service at Reggiano‘s is impeccable", {"entities": [(14, 24, "RESTAURANT")] })
]

Next, we‘ll define our custom entity labels:

LABEL = ["RESTAURANT", "FOOD"]

Then, we can train a new entity recognizer by updating an existing spaCy model with our examples:

import random
from spacy.training import Example
from spacy.util import minibatch, compounding

# Load the model and create a new NER pipe
nlp = spacy.load("en_core_web_sm")  
ner = nlp.add_pipe("ner")

# Add labels
for label in LABEL:
    ner.add_label(label)

# Convert data to spaCy format
examples = []
for text, annotations in TRAIN_DATA:
    examples.append(Example.from_dict(nlp.make_doc(text), annotations))

# Disable other pipeline components  
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != "ner"]
with nlp.disable_pipes(*other_pipes):

    # Initial random weights
    nlp.begin_training()

    # Train for 30 iterations
    for itn in range(30):

        # Shuffle examples each iteration        
        random.shuffle(examples)
        losses = {}

        # Batch examples with compound size        
        batches = minibatch(examples, size=compounding(4.0, 32.0, 1.001))

        # Update model with each batch        
        for batch in batches:
            nlp.update(batch, losses=losses, drop=0.5)
            print("Losses", losses)

After training, our custom NER model can now identify restaurants and foods in new, unseen text:

doc = nlp("I had the best burger at Burger Shack yesterday!")

for ent in doc.ents:
    print(ent.text, ent.label_)

Output:
burger FOOD
Burger Shack RESTAURANT

This is just a toy example, but it demonstrates the general workflow for training a custom NER model in spaCy. To build a truly robust model, you‘ll want to use a much larger annotated training corpus.

Advanced NER with spaCy

Beyond the basics, spaCy offers several advanced features for leveling up your NER pipelines:

• Rule-based matching: In addition to statistical NER, spaCy lets you implement rule-based entity matching using token-based rules and regular expressions. This is useful for quickly matching large terminology lists or guaranteed entities that have consistent patterns, like IDs or phone numbers.

• Entity linking: Entity linking involves resolving textual entities to unique identifiers in a knowledge base. SpaCy‘s EntityLinker allows you to link entities to knowledge bases like Wikipedia or custom KBs for disambiguation.

• Injecting word vectors: SpaCy supports initializing NER models with pre-trained word embeddings, which can give a nice boost in accuracy especially if you have a small training dataset. You can plug in static word vectors or contextualized embeddings from language models.

• Model tuning and pretraining: SpaCy provides a config system for precisely controlling your model architecture, hyperparameters, and training settings. You can leverage transfer learning by pretraining the weights of earlier pipeline components or even the NER layer itself. The spaCy project templates make it easy to manage end-to-end NER workflows.

• Visualizing models and errors: In addition to the displaCy visualizer, spaCy offers tools for interactively debugging and visualizing your custom NER models. The spacy.explain method lets you inspect descriptions for model decisions, and you can profile the speed and accuracy of your components.

Comparing spaCy to Other NLP Libraries

SpaCy is a popular choice for NER among Python developers, but there are other notable open-source libraries for NLP in Python. Two other leading libraries are the Natural Language Toolkit (NLTK) and Stanza.

NLTK is a longstanding Python NLP library that provides modules for a wide range of text processing tasks. For NER, NLTK uses a conditional random field (CRF) sequential tagger trained on datasets like CoNLL2002. Compared to spaCy, NLTK has a larger ecosystem of algorithms and datasets, but spaCy is built for better performance and developer productivity.

Stanza (formerly StanfordNLP) is a Python NLP library built on top of PyTorch. Stanza uses a bi-LSTM neural architecture and features a multilingual NER model that currently supports 66 languages. In terms of usage, Stanza has a different API design than spaCy and is more focused on research use cases compared to spaCy‘s emphasis on production.

The Future of NER

NER has been an active research area for decades, but recent advancements in deep learning and language models have dramatically improved the state-of-the-art in NER. Transformers-based language models like BERT have achieved human parity on several NER benchmarks by learning powerful contextualized representations that can better handle ambiguous and complex entities.

Some promising frontiers in NER include:

• Distantly and weakly supervised approaches that can learn from large datasets without explicit annotations
• Zero and few-shot techniques that can recognize entities from limited examples
• Active learning workflows that intelligently suggest informative examples for users to annotate
• Domain-adaptive pretraining and fine-tuning of language models for specialized domains
• Scalable solutions for NER on long-text and multi-document settings
• Multimodal NER that can utilize textual, visual, and acoustic context

As language models and knowledge graphs continue to evolve, so too will our approaches to NER. Automated NER will become ever more proficient at extracting the latent entities that underlie our natural language. The dream of machines that can read and understand text relies on capable and robust NER.

Conclusion

In this article, we explored named entity recognition in Python using the spaCy library. We covered the fundamentals of how to use spaCy to detect and label entities in text, walked through training a custom NER model on restaurant review data, and discussed advanced techniques like rule-based matching and entity linking.

Whether you‘re working on document understanding, knowledge base population, or text mining applications, NER with spaCy is a powerful tool to have in your NLP toolkit. By following the concepts and code recipes from this guide, you‘re well on your way to building production-grade NER solutions.