The Essential Guide to Feature Engineering in NLP

Feature engineering is one of the most important yet often overlooked aspects of building effective machine learning models for natural language processing (NLP) tasks. While powerful neural network architectures like transformers have achieved state-of-the-art results on many NLP benchmarks, feature engineering still plays a critical role, especially when training data is limited or for productionizing models.

At its core, feature engineering is the process of using domain knowledge to extract informative features from raw data that help machine learning algorithms learn and generalize better. In the context of NLP, this means going beyond just representing text as a sequence of words or characters and deriving more semantically meaningful attributes.

In this guide, we‘ll dive deep into the most important feature engineering techniques used in modern NLP and how to apply them to real-world text datasets. Whether you‘re a beginner looking to get started with NLP or an experienced practitioner seeking to take your models to the next level, understanding and leveraging these techniques can significantly boost your NLP performance.

Bag-of-Words and TF-IDF

The bag-of-words model is perhaps the simplest way to numerically represent text for machine learning. As its name suggests, bag-of-words represents a piece of text, such as a sentence or document, as an unordered collection (or "bag") of its constituent words.

Specifically, bag-of-words creates a vocabulary of all the unique words in a text corpus and then represents each document as a vector where each element corresponds to the count of a vocabulary word in that document. For example:

Document 1: "the quick brown fox jumped over the lazy dog"
Document 2: "the lazy dog slept all day"

Vocabulary: [the, quick, brown, fox, jumped, over, lazy, dog, slept, all, day]

Bag-of-words representation:
Doc 1: [2, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0] Doc 2: [1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1]

The main advantage of bag-of-words is its simplicity – it‘s straightforward to understand and implement. However, it has several limitations. Bag-of-words loses all information about word order, which is often important for meaning. It also suffers from data sparsity, as most words in the vocabulary will have a count of zero for any given document.

To address the latter issue, a common technique is to weight the raw word counts by the term frequency-inverse document frequency (TF-IDF). The TF-IDF value increases proportionally to the number of times a word appears in a document, but is offset by the word‘s frequency across the entire corpus, which helps to reflect how important a word is to a document.

Mathematically, TF-IDF for a word t in a document d from a corpus D is calculated as:

TF-IDF(t, d, D) = TF(t, d) * IDF(t, D)

where TF(t, d) is the number of times term t appears in document d, and IDF(t, D) = log(N / DF(t)) where N is the total number of documents and DF(t) is the number of documents that contain term t.

In our example above, the words "the", "lazy", and "dog" appear in both documents, so they would have a lower TF-IDF score, while the other words unique to each document like "quick" and "slept" would receive a higher score.

While bag-of-words with TF-IDF weighting was commonly used with classic ML algorithms like Naive Bayes and SVMs, it has been largely superseded by more advanced techniques. However, it remains a good baseline and is still useful in certain applications like information retrieval.

N-Grams and Part-of-Speech Tagging

The next level up from bag-of-words is using n-grams, which are contiguous sequences of n words (or characters). N-grams provide a way to incorporate local word order and phrases into the text representation.

The most common type are word n-grams:

• Unigrams: "the", "quick", "brown", "fox"
• Bigrams: "the quick", "quick brown", "brown fox"
• Trigrams: "the quick brown", "quick brown fox"

Character n-grams, i.e. substrings of characters, can also be used:

• 3-grams: "the", "he ", "e q", " qu", "qui", "uic", …
• 4-grams: "the ", "he q", "e qu", " qui", "quic", …

Using n-grams, especially bigrams and trigrams, can help capture common phrases and short-range word dependencies. They are particularly useful for more syntactic tasks like language modeling or POS tagging.

Character n-grams are more robust to spelling variations and capture sub-word patterns that are helpful for tasks like sentiment analysis where word stems are informative (e.g. "love", "loved", "lovely" all indicate positive sentiment).

However, n-grams also have downsides – they greatly increase the feature space and hence memory footprint and training time. They still don‘t capture long-range dependencies or broader semantic relationships. Therefore, they are best used in combination with other techniques.

Another classic NLP technique is part-of-speech (POS) tagging, which labels each word in a text with its grammatical category, such as noun, verb, adjective, etc. POS tags can provide useful information about the syntactic structure of text. Some examples:

"The quick brown fox jumped over the lazy dog"
DET ADJ ADJ NOUN VERB ADP DET ADJ NOUN

Incorporating POS tags as features allows machine learning models to learn patterns based on syntax, not just the words themselves. This can improve performance on tasks like named entity recognition, where capitalized proper nouns are important to detect. POS tag sequences are also used directly in applications like grammar correction.

While POS tagging used to be done using rules or statistical models, modern POS taggers are usually based on deep learning sequence models trained on labeled corpora. As such, using POS tags now generally means running a pre-trained model as a preprocessing step, rather than training on POS features directly.

Semantic Features

While n-grams and POS tags capture local syntax, a key challenge in NLP feature engineering is extracting higher-level semantic information. Some important semantic features include:

Named Entity Recognition: NER identifies and classifies named entities in text, such as people, organizations, locations, etc. This provides valuable information about who and what a piece of text is discussing. For example:

"Elon Musk‘s Tesla Buys $1.5 Billion in Bitcoin"
PERSON ORG MONEY GPE

Common NER features include entity type counts/ratios (e.g. number of person mentions divided by total number of words) as well as particular entity tokens (e.g. one-hot encoding of all company names). Like POS tagging, NER is now typically done via pre-trained deep learning models.

Sentiment Scores: For subjective text like reviews or social media, the sentiment or overall emotional tone is highly informative. Sentiment analysis models predict sentiment labels (positive, negative, neutral) or scores (e.g. star ratings or a continuous sentiment scale).

Sentiment scores can be used as direct features indicating the proportion or intensity of different sentiments in a piece of text. Alternatively, sentiment analysis can first be applied to individual words, sentences, or entities in a text, and then those fine-grained sentiments can be aggregated into sentiment features for the whole text.

Topic Models: Topic models are algorithms that discover the main themes or topics pervading a text corpus. They work by finding groups of semantically related words that tend to occur together. The famous latent Dirichlet allocation (LDA) model learns topic-word distributions and represents each document as a mixture of those topics.

As features, topic models can generate a topic vector for each document, where each dimension represents the probability of that document discussing a particular topic. This helps machine learning models distinguish between documents with different thematic content.

Topic features are useful for categorization and retrieval tasks, such as identifying what news articles are about or recommending similar documents. However, standard topic models do not capture word order and so do not always align with human interpretations. More recent neural topic models aim to improve coherence and incorporate contextual information.

Word and Document Embeddings

In the last decade, the dominant paradigm in NLP feature learning has shifted from the above statistical approaches to neural word embeddings. Word embeddings are dense vector representations of words that capture their syntactic and semantic properties.

The key idea is that words with similar meanings and grammatical functions will have similar embedding vectors. This is based on the distributional hypothesis that words with similar co-occurrence patterns have similar meanings.

Popular word embedding algorithms include:

• word2vec: Predicts a target word given its context (continuous bag-of-words) or predicts the context given a target word (skip-gram). Learns embeddings as weights of shallow neural network trained on these prediction tasks.

• GloVe: Combines global matrix factorization with local context window methods. Factorizes word-word co-occurrence matrix to learn embeddings.

• fastText: Extension of word2vec that learns embeddings for character n-grams, allowing handling of out-of-vocabulary words.

Using pre-trained word embeddings as input features allows NLP models to leverage semantic and syntactic information learned from large unlabeled corpora. This is especially useful when downstream training data is limited.

The same concept of embeddings can be extended to the sentence or document level. Classical document embedding techniques like doc2vec (PV-DBOW and PV-DM) learn a vector for each document in addition to the word vectors, allowing an entire text to be represented as a dense embedding.

More recent transformer language models like BERT learn contextual word embeddings that dynamically change based on surrounding words, better capturing meaning in context. Embeddings from the special [CLS] token or pooled output can also serve as full-text embeddings.

Document embeddings are extremely versatile features that can be used for most NLP tasks, as they aim to capture the overall semantics of a piece of text in a single vector representation. They are particularly well-suited for tasks involving long texts and semantic similarity comparisons.

Choosing the Right Features for the Task

With the plethora of NLP feature engineering techniques available, it can be challenging to know which ones to apply when. The key is to consider the specific needs of your task and data.

For syntactic tasks that depend heavily on word order and phyrasemstructure, like language modeling or POS tagging, n-grams are a good choice. Sequence tagging tasks like NER benefit from incorporating POS and other syntactic features.

When dealing with subjective, opinion-oriented text, sentiment-related features are indispensable. For document classification or retrieval, topic and embedding features help capture the high-level semantics.

In general, it‘s best to start with the simplest techniques like bag-of-words with TF-IDF, and gradually add more advanced features and assess their impact on validation performance. Using pre-trained models (e.g. transformer embeddings) as input features is a good default when available, but domain-specific features may still be helpful.

Ultimately, the goal of feature engineering is to capture as much task-relevant information in the input as possible, while avoiding adding noise. This requires a combination of domain expertise to identify salient attributes, and empirical experimentation to validate their utility.

The Future of NLP Feature Engineering

As of 2024, end-to-end deep learning approaches have come to dominate NLP, often achieving state-of-the-art performance without explicit feature engineering. Transfer learning from huge language models pre-trained on web-scale data has reduced the need for domain-specific feature extraction in many cases.

However, this does not mean feature engineering is obsolete. Cutting-edge language models still struggle to capture some types of domain knowledge, especially for specialized verticals. Injecting manually engineered features into the input or model architecture can improve performance and data efficiency.

Multimodal learning involving both text and other modalities like images is also an increasingly important area where feature engineering is needed to effectively combine different types of information. How to design features that optimally align textual and visual semantics remains an open research question.

In the near future, we can expect a hybrid approach to become common – leveraging the power of pre-trained language models as a strong base, and then fine-tuning with domain-specific features to eke out further gains. Automating the feature engineering process itself, e.g. with learned combinatorial feature spaces or neural architecture search, may also emerge as a way to get the benefits of both customization and scale.

Regardless, the central tenet of feature engineering will always hold true: to build an effective NLP model, you must first understand your data and encode what is important in a machine-understandable way. As long as there are new NLP tasks and domains to tackle, feature engineering will remain an essential tool in any NLP practitioner‘s toolbox.