Introduction to Natural Language Processing and Tokenization

As humans, we communicate using natural language – the words, phrases, and sentences that make up our everyday speech and writing. But for computers to understand and process human language, it first needs to be converted into a structured format they can work with. This is where the field of Natural Language Processing, or NLP, comes in.

NLP is a branch of artificial intelligence focused on enabling computers to understand, interpret, and generate human language. Some common applications you may have encountered include:

• Language translation (e.g. Google Translate)
• Chatbots and virtual assistants (e.g. Siri, Alexa)
• Text classification (e.g. spam filters, sentiment analysis)
• Information extraction (e.g. named entity recognition, keyword extraction)
• Text summarization
• Question answering
• And many more

Under the hood, NLP systems leverage techniques from computer science, linguistics, and machine learning to model and analyze language. A typical NLP pipeline consists of the following key steps:

1. Tokenization
2. Text Cleaning
3. Part-of-Speech Tagging
4. Named Entity Recognition
5. Syntactic Parsing
6. Semantic Analysis

In this post, we‘ll dive deep into the details of that crucial first step – tokenization. We‘ll explore what tokenization is, survey different tokenization methods and libraries, and walk through hands-on code examples, with a special focus on the Gensim library‘s tokenization capabilities. By the end, you‘ll have a solid foundation to begin working with tokenization in your own NLP projects.

What is Tokenization?

Put simply, tokenization is the process of breaking down a piece of text into smaller units called tokens. Typically, these tokens are individual words, but they could also be phrases, sentences, or other meaningful elements depending on the application.

For example, consider the following text:

"After landing in New York, John took a taxi to his hotel on 5th Avenue."

Tokenizing this sentence would produce the following list of word tokens:

[‘After‘, ‘landing‘, ‘in‘, ‘New‘, ‘York‘, ‘John‘, ‘took‘, ‘a‘, ‘taxi‘, ‘to‘, ‘his‘, ‘hotel‘, ‘on‘, ‘5th‘, ‘Avenue‘]

Tokenization is an essential first step in the NLP pipeline because it converts unstructured text into a standardized list of tokens that can be fed as input into later steps like part-of-speech tagging, parsing, named entity extraction, and so on.

While tokenization may seem straightforward, there are many nuances and design decisions to consider:

• How to handle punctuation? Should punctuation marks be discarded or kept as separate tokens?
• What about contractions like "don‘t" or "we‘ll"? Should these be split into "do" + "n‘t" and "we" + "‘ll" or left as single tokens?
• How to deal with hyphenated words, URLs, email addresses, hashtags, and other special entities?
• Should capitalization be preserved or normalized to lowercase?
• Should very common "stopwords" like "the", "a", "and", etc. be filtered out?

The answers to these questions depend on the specific use case and requirements. Luckily, we don‘t have to write tokenization logic from scratch – there are a number of excellent open source libraries available in Python to help. Let‘s take a tour of some of the most popular ones, starting with Gensim.

Tokenization with Gensim

Gensim is a mature open-source library for topic modeling and document similarity retrieval, with excellent performance on large text collections. It also provides convenient utilities for text preprocessing, including a streamlined API for tokenization.

The main entry point is the gensim.utils.tokenize() function, which takes in a string of text and returns a list of tokens using some sensible defaults. Here‘s a basic example:

from gensim.utils import tokenize

text = "After landing in New York, John took a taxi to his hotel on 5th Avenue."

tokens = list(tokenize(text))

print(tokens)

Output:

[‘after‘, ‘landing‘, ‘in‘, ‘new‘, ‘york‘, ‘john‘, ‘took‘, ‘taxi‘, ‘to‘, ‘his‘, ‘hotel‘, ‘on‘, ‘5th‘, ‘avenue‘]

By default, tokenize() does the following:

• Splits the text on whitespace
• Removes punctuation
• Lowercases all tokens

These are generally reasonable defaults, but we can customize the behavior if needed by passing in additional arguments. For example, to disable lowercasing and keep uppercase tokens intact:

tokens = list(tokenize(text, lowercase=False))

print(tokens) 

Output:

[‘After‘, ‘landing‘, ‘in‘, ‘New‘, ‘York‘, ‘John‘, ‘took‘, ‘taxi‘, ‘to‘, ‘his‘, ‘hotel‘, ‘on‘, ‘5th‘, ‘Avenue‘]

We can also provide our own custom regular expression pattern to tokenize() using the to_lower and token_pattern arguments. This gives us finer-grained control over which characters should be included in tokens. For instance, to preserve contractions:

from gensim.utils import tokenize

text = "I‘m afraid we can‘t make it tonight. Let‘s reschedule for next week."

tokens = list(tokenize(text, token_pattern=r‘(?u)\b\w+\‘\w+\b|\b\w+‘))

print(tokens)

Output:

["i‘m", ‘afraid‘, ‘we‘, "can‘t", ‘make‘, ‘it‘, ‘tonight‘, "let‘s", ‘reschedule‘, ‘for‘, ‘next‘, ‘week‘]

In addition to word tokenization, Gensim also provides a convenient utility for sentence tokenization – that is, splitting a text into a list of sentences:

from gensim.summarization.textcleaner import split_sentences

text = "After landing in New York, John took a taxi to his hotel on 5th Avenue. He was excited to explore the city. First on his list was visiting the Statue of Liberty."

sentences = split_sentences(text)

print(sentences)

Output:

[‘After landing in New York, John took a taxi to his hotel on 5th Avenue.‘,
 ‘He was excited to explore the city.‘,
 ‘First on his list was visiting the Statue of Liberty.‘]

The split_sentences() function uses an internal decision tree to detect sentence boundaries based on punctuation and capitalization patterns. It handles many edge cases like abbreviations, numbers, and parentheses that trip up simplistic approaches.

NLTK and Keras Tokenizers

In addition to Gensim, two other widely used Python libraries for tokenization are NLTK and Keras.

NLTK (Natural Language Toolkit) is a comprehensive platform for building NLP programs in Python. It provides modules for many different text processing tasks, plus a suite of helpful resources like corpus readers and trained models.

For tokenization, the main functions to use are:

• nltk.word_tokenize() – Tokenizes a string into a list of words. Uses a sophisticated regex-based approach under the hood.

• nltk.sent_tokenize() – Tokenizes a string into a list of sentences.

Here‘s a quick example:

import nltk

text = "After landing in New York, John took a taxi to his hotel on 5th Avenue."

word_tokens = nltk.word_tokenize(text)
print(word_tokens)

sent_tokens = nltk.sent_tokenize(text)  
print(sent_tokens)

Output:

[‘After‘, ‘landing‘, ‘in‘, ‘New‘, ‘York‘, ‘,‘, ‘John‘, ‘took‘, ‘a‘, ‘taxi‘, ‘to‘, ‘his‘, ‘hotel‘, ‘on‘, ‘5th‘, ‘Avenue‘, ‘.‘]

[‘After landing in New York, John took a taxi to his hotel on 5th Avenue.‘] 

The word tokenizer preserves punctuation by default, and the sentence tokenizer is able to handle the mid-sentence comma and period appropriately.

While NLTK is a dedicated NLP library, Keras is a high-level deep learning framework that provides its own utilities for tokenization as part of its text preprocessing module.

The main class to use is keras.preprocessing.text.Tokenizer, which can be instantiated to create a tokenizer object with customizable parameters. It also supports advanced features like one-hot encoding and sequence padding for deep learning models. Here‘s a basic usage example:

from keras.preprocessing.text import Tokenizer

texts = [
    "After landing in New York, John took a taxi to his hotel on 5th Avenue.",
    "He was excited to explore the city.",
    "First on his list was visiting the Statue of Liberty.",
]

tokenizer = Tokenizer(num_words=100)
tokenizer.fit_on_texts(texts)

print(tokenizer.word_index)

Output:

{‘to‘: 1, ‘his‘: 2, ‘was‘: 3, ‘he‘: 4, ‘on‘: 5, ‘in‘: 6, ‘new‘: 7, ‘york‘: 8, ‘john‘: 9, ‘after‘: 10, ‘landing‘: 11, ‘took‘: 12, ‘taxi‘: 13, ‘hotel‘: 14, ‘5th‘: 15, ‘avenue‘: 16, ‘excited‘: 17, ‘explore‘: 18, ‘the‘: 19, ‘city‘: 20, ‘first‘: 21, ‘list‘: 22, ‘visiting‘: 23, ‘statue‘: 24, ‘of‘: 25, ‘liberty‘: 26}

The Tokenizer builds a vocabulary index of the most frequently occurring words, which we can use to convert input texts into integer sequences:

sequences = tokenizer.texts_to_sequences(texts)

print(sequences)

Output:

[[10, 11, 6, 7, 8, 9, 12, 13, 1, 2, 14, 5, 15, 16], [4, 3, 17, 1, 18, 19, 20], [21, 5, 2, 22, 3, 23, 19, 24, 25, 26]]

These integer sequences can then be fed into Keras‘ deep learning layers for tasks like text classification, language modeling, and sequence-to-sequence learning.

Tokenization Challenges and Considerations

While the libraries we‘ve covered so far make tokenization easy to implement, there are still some challenges and design tradeoffs to keep in mind.

One issue is handling out-of-vocabulary (OOV) words. Since tokenizers typically build a fixed-size vocabulary from a training corpus, they may encounter new words during inference that they don‘t know how to handle. Strategies to deal with this include:

• Ignoring OOV words entirely
• Mapping OOV words to a special "unknown" token
• Using subword tokenization algorithms like Byte-Pair Encoding (BPE) that can encode novel words from known subword units
• Using character-level tokenization

Another challenge is dealing with languages that don‘t use spaces to delimit words, like Chinese, Japanese, and Thai. For these languages, more sophisticated techniques like dictionary-based or statistical word segmentation are needed.

Tokenization can also become tricky when working with domain-specific texts heavy in jargon, acronyms, or non-standard spellings (think social media). Off-the-shelf tokenizers may need to be customized or augmented with domain-specific knowledge.

It‘s also worth noting that there isn‘t always a single "right answer" for how to tokenize a given text. Different applications may warrant different tokenization schemes. A sentiment analysis model, for instance, might want to preserve emoticons and punctuation for detecting tone. Meanwhile, an information extraction system might be okay with stripping those out. The key is to align the tokenization approach with the end task.

Tokenization Use Cases

We‘ve hinted at some of the use cases for tokenization throughout this post, but let‘s recap a few key ones:

• Text Classification – Tokenization is often the first step in pipelines for classifying articles into topics, detecting the sentiment of reviews, or identifying the language of a document. The tokens are typically fed into machine learning models like Naive Bayes or logistic regression as features.

• Named Entity Recognition – NER systems aim to identify and extract entities like people, places, organizations, and dates from unstructured text. Tokenization is necessary to chunk the text into word-level units that can be classified by the NER model.

• Machine Translation – MT systems need to tokenize both the source and target language text in order to map words and phrases between them. Subword tokenization schemes are often used to handle rare words.

• Text Summarization – Extractive summarization approaches typically operate at the sentence level, so robust sentence tokenization is key. From there, sentences can be ranked and selected to form a compressed version of the original text.

• Chatbots and Virtual Assistants – Chatbots need to tokenize user queries in order to parse intents, extract entities, and formulate relevant responses. Handling misspellings, slang, and contextual ambiguity are important challenges.

• Social Media Analysis – Tokenizing social media posts often requires handling emojis, hashtags, @-mentions, and other unique constructs. Preserving case and punctuation can also be important for tasks like sentiment analysis and author attribution.

Conclusion

In this post, we‘ve taken a deep dive into tokenization, a foundational step in many NLP workflows. We‘ve covered:

• What tokenization is and why it‘s important
• Different tokenization methods and libraries, with a focus on Gensim
• Key challenges and considerations, like handling OOV words and domain-specific text
• Common use cases and applications

To sum up, while tokenization may seem like a straightforward task at first glance, there are a number of subtle decisions and tradeoffs to keep in mind. The best approach will depend on the specific language, domain, and end application. Luckily, open source libraries like Gensim, NLTK, and Keras make it easy to get started and prototype different strategies.

Looking ahead, tokenization is just the first step in a typical NLP pipeline. In future posts, we‘ll explore more advanced topics like part-of-speech tagging, parsing, named entity recognition, and vectorization. We‘ll also dive into how transformer-based models like BERT are changing the game with pre-trained, contextualized representations that can be fine-tuned for a variety of downstream tasks.

In the meantime, you can apply what you‘ve learned here to tokenize your own datasets and start building basic NLP models. Try comparing how different tokenization schemes affect the performance of a text classifier or sentiment analyzer. Or see how far you can push the off-the-shelf tokenizers before you need something custom. The best way to learn is to get your hands dirty and experiment!

I hope this post has given you a solid foundation in tokenization to build on in your NLP journey. Feel free to leave any questions or feedback in the comments below. Happy tokenizing!