A Beginner‘s Guide to Named Entity Recognition (NER)

Introduction

Named Entity Recognition, commonly known as NER, is a fundamental task in Natural Language Processing (NLP) that focuses on identifying and categorizing named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc.

NER is a form of information extraction that enables computers to understand key details in text in a structured way. It allows machines to automatically scan entire articles, reports, or webpages and locate & classify important named entities without human intervention. As such, NER is a key component in many downstream NLP applications such as:

Automatically highlighting key people, places and events in news articles
Extracting symptoms, medications, and procedures from medical records
Identifying product and brand name mentions in social media posts
Anonymizing personal data in legal documents
Powering intelligent Q&A, chatbots, and search systems

At a high level, NER takes raw text as input, applies a statistical model to identify potential entities, and outputs those entities classified into categories. For example, given the sentence "Apple CEO Tim Cook spoke at the WorldWide Developer Conference in San Francisco", an NER system should identify:

[Apple]_ORGANIZATION [CEO]_TITLE [Tim Cook]_PERSON spoke at the [WorldWide Developer Conference]_EVENT in [San Francisco]_LOCATION

Modern NER systems can handle huge volumes of text in many languages with human-level quality. This article will give you a practical overview of how Named Entity Recognition works under the hood and survey the different approaches, tools and applications you should know about as an NLP practitioner.

A Brief History of NER

While the term "Named Entity" was first introduced at the 6th Message Understanding Conference in 1996, the task of identifying proper names in text dates back to the 1990s. Early NER systems primarily relied on handcrafted rules and heuristics, such as capitalization patterns, to locate potential named entities.

In the 2000s, machine learning techniques like Hidden Markov Models (HMMs), Maximum Entropy Markov Models (MEMMs), and Conditional Random Fields (CRFs) came into prominence. These approaches leveraged large annotated training corpora to automatically learn patterns and features associated with different named entity types from data.

Starting in the early 2010s, deep learning models using neural networks, such as RNNs, LSTMs and CNNs, achieved state-of-the-art results on many NER tasks. More recently, large pre-trained language models like BERT and its variants, which learn contextual word embeddings from massive unlabeled text corpora, have pushed NER performance to new heights.

Today, NER is considered an essential first step in many NLP pipelines and new model architectures, approaches to leveraging knowledge bases, and techniques like transfer learning continue to be active research areas.

How NER Works

Under the hood, Named Entity Recognition is formulated as a sequence labeling task. The model takes in a sequence of tokens (words) as input and outputs a corresponding sequence of labels indicating the presence and type of named entities.

Defining the NER Task

More formally, given an input sequence of n tokens x = (x₁, x₂, …, x_n), an NER model aims to generate a sequence of tags y = (y₁, y₂, …, y_n), where each y_i comes from a predefined tagset.

A commonly used tagging scheme is IOB (Inside-Outside-Beginning) tagging, where each token is labeled as either beginning an entity (B-type), being inside an entity (I-type), or not belonging to an entity (O). The following example illustrates IOB formatted output:

Tim/B-PER Cook/I-PER is/O the/O CEO/O of/O Apple/B-ORG Inc/I-ORG

This indicates that "Tim Cook" is a person (PER) mention and "Apple Inc" is an organization (ORG). The other tokens are marked with the outside (O) tag.

Feature Extraction

To make accurate entity tagging decisions, NER models need to learn good representations of the input text. This involves extracting various lexical, syntactic, and semantic features about each token and its surrounding context, such as:

Token text
Lowercased token text
Token lemma
Part-of-speech tag
Dependency parse information
Word shape (capitalization, punctuation, digits)
Gazetteer and dictionary features
Word embeddings

Historically, this feature engineering process required significant manual effort and domain expertise. Modern neural NER architectures have alleviated this challenge by automatically learning relevant features in an end-to-end fashion.

Sequence Labeling Models

With extracted word-level features in hand, the next step is to train a sequence labeling model to predict the most likely tag sequence given the input tokens. Different model architectures make different assumptions about the dependencies between tags and how they relate to the input.

Here are a few of the most popular sequence labeling approaches used in NER:

Hidden Markov Models (HMMs) – A generative probabilistic sequence model that assumes each tag only depends on the previous tag (Markov assumption). Relies on tag-token emission probabilities and tag transition probabilities.
Conditional Random Fields (CRFs) – A discriminative undirected graphical model that captures dependencies between neighboring output tags and input features. Models the conditional probability of the entire output sequence given the input. Allows incorporating rich overlapping features.
Recurrent Neural Networks (RNNs) – A neural network that captures long-distance dependencies by maintaining a hidden state vector that acts as a "memory" for previous tokens in the sequence. Vanilla RNNs struggle with very long sequences due to vanishing/exploding gradients.
Long Short-Term Memory Networks (LSTMs) – A type of RNN with specialized gating mechanisms (input, output, forget gates) that enable learning very long-range dependencies. Widely used in state-of-the-art NER models.
Transformers – An attention-based neural network architecture that dispenses with recurrent connections in favor of modeling pairwise relationships between tokens via self-attention. Pre-training transformer language models on huge unsupervised corpora has yielded significant improvements across many NLP tasks, including NER.

Evaluation Metrics

Evaluating the performance of an NER model requires comparing the model‘s predicted entity spans and types to manually annotated ground truth labels. The standard metrics used are:

Precision – What fraction of the named entities predicted by the model are actually correct? Precision = True Positives / (True Positives + False Positives)
Recall – What fraction of the actual named entities in the data are captured by the model? Recall = True Positives / (True Positives + False Negatives)
F1 Score – The harmonic mean of precision and recall that provides a balanced measure of the model‘s accuracy. F1 = 2 (precision recall) / (precision + recall)

Models are typically evaluated in two ways: exact match, which requires predicted entity spans to precisely match the gold labels, and partial match, which awards partial credit for inexact matches based on token-level accuracy.

Tools and Libraries

There are many open-source tools and libraries in various programming languages that provide easy-to-use implementations of state-of-the-art NER models. Here are a few of the most popular:

spaCy – An industrial-strength NLP library in Python with fast, robust NER models built-in. Supports training custom models on your own data.
Stanford CoreNLP – A Java toolkit from Stanford University that provides various NLP tools, including a CRF-based NER model.
NLTK – A Python library for NLP research with interfaces to several NER implementations like the Stanford NER and Illinois NER.
Flair – A powerful PyTorch NLP library that allows you to mix and match different word embeddings and use pre-trained state-of-the-art sequence labeling models for NER.
AllenNLP – An open-source NLP research library in Python built on PyTorch that makes it easy to train custom NER models using modern neural architectures.

Conclusion

Named Entity Recognition is a key component of modern NLP pipelines that enables extracting valuable structured knowledge from unstructured text. We‘ve covered the fundamentals of how NER works, starting from early rule-based approaches to state-of-the-art neural architectures.

While great strides have been made in NER performance, there remain many open challenges. Handling emerging and rare entity types, domain-specific lingo, noisy and informal text are all active areas of research. Recent work in few-shot learning, unsupervised NER, and connecting NER with knowledge bases offer exciting future directions.

To learn more, check out:

We hope this guide has given you a practical foundation in Named Entity Recognition to apply to your own projects. Happy entity hunting!

A Beginner‘s Guide to Named Entity Recognition (NER)

Introduction

A Brief History of NER