A Comprehensive Guide to Building Language Models in Python

Language models are a fundamental component of modern natural language processing (NLP). Whether it‘s machine translation, text generation, spelling correction, or any application involving language understanding, chances are a language model is working behind the scenes to enable that capability.

In this guide, we‘ll take a deep dive into language models, focusing on the classic n-gram approach as well as modern neural architectures. We‘ll cover the key concepts, walk through how to implement them in Python, and explore powerful pre-trained models you can leverage in your own projects. By the end, you‘ll have a solid understanding of how language models work and be equipped to apply them to a wide range of NLP tasks.

Language Modeling Basics

At its core, a language model aims to capture the statistical properties of language by learning the likelihood of word sequences from training data. With a well-trained model, you can then predict the probability of new sequences, generate realistic text, or identify unlikely word combinations.

More formally, a language model estimates the probability distribution over sequences of words:

P(w1, w2, …, wn)

That is, given any sequence of n words, the model assigns a probability to that sequence, with more likely sequences having higher probabilities.

N-Gram Language Models

The classic approach to language modeling is using n-grams. An n-gram is simply a contiguous sequence of n words from a text. For example:

• Unigram (n=1): "the", "cat", "sat"
• Bigram (n=2): "the cat", "cat sat", "sat on"
• Trigram (n=3): "the cat sat", "cat sat on", "sat on the"

The intuition behind n-gram models is that instead of computing the probability of an entire word sequence, which is often infeasible, we can approximate it by breaking the sequence into smaller chunks and assuming each word depends only on the previous n-1 words.

For example, instead of computing:
P(the cat sat on the mat)

We compute:
P(the) P(cat|the) P(sat|the cat) P(on|cat sat) P(the|sat on) * P(mat|on the)

This relies on the Markov assumption that the probability of a word depends only on the previous n-1 words, rather than the full context. While not entirely true in language, this turns out to be a surprisingly effective approximation.

Bigram Language Models

Let‘s dive deeper into bigram models, one of the most commonly used n-gram variants. In a bigram model, we assume each word depends only on the immediately preceding word. The probability of a sequence becomes:

P(w1, w2, …, wn) ≈ P(w1) P(w2|w1) P(w3|w2) … P(wn|w(n-1))

To compute the individual bigram probabilities P(wn|w(n-1)), we simply count how often each word occurs after the preceding word in the training data and normalize:

P(wn|w(n-1)) = count(w(n-1), wn) / count(w(n-1))

Where count(w(n-1), wn) is the number of times the bigram (w(n-1), wn) appears in the training corpus, and count(w(n-1)) is the total count of w(n-1).

Here‘s how we can implement a basic bigram model in Python:

from collections import defaultdict

class BigramModel:
    def __init__(self):
        self.bigram_counts = defaultdict(lambda: defaultdict(int))
        self.unigram_counts = defaultdict(int)

    def train(self, corpus):
        for sentence in corpus:
            words = sentence.split()
            for i in range(len(words) - 1):
                w1, w2 = words[i], words[i+1]
                self.bigram_counts[w1][w2] += 1
                self.unigram_counts[w1] += 1

    def probability(self, w1, w2):
        return self.bigram_counts[w1][w2] / self.unigram_counts[w1]

    def generate(self, context, n):
        for i in range(n):
            choices = list(self.bigram_counts[context].items())
            probs = [count / self.unigram_counts[context] for _, count in choices]
            context = np.random.choice([w for w, _ in choices], p=probs)
            yield context

The model maintains counts of bigrams and unigrams seen during training. To generate text, it repeatedly samples the next word based on the conditional bigram probabilities given the preceding word.

While this bigram implementation is straightforward, in practice there are additional considerations like smoothing (to handle unseen bigrams) and backoff (to fall back to unigram probabilities when necessary). The NLTK library provides a more complete implementation you can use out of the box.

Limitations of N-Gram Models

While n-gram models are simple and efficient, they have some notable limitations:

1. Limited context: By relying on a fixed window of previous words, n-gram models struggle to handle long-range dependencies in language. Increasing the context size by using higher-order n-grams (e.g. 4-grams or 5-grams) quickly becomes intractable due to data sparsity.

2. Lack of generalization: N-gram models can only generate word sequences that have been observed in the training data. They cannot generalize to understand the underlying semantics and produce novel sequences.

3. Curse of dimensionality: As the vocabulary size grows, the number of possible n-grams explodes exponentially, leading to data sparsity issues. This requires vast amounts of training data and computation to estimate probabilities reliably.

Neural Language Models

To address the limitations of n-gram models, researchers have turned to neural networks to learn more expressive language representations. Neural language models, powered by architectures like recurrent neural networks (RNNs) and transformers, can capture complex patterns and long-range dependencies in language.

At a high level, neural language models work by:

1. Representing words as dense vectors (embeddings) capturing semantic similarities
2. Processing sequences of word vectors through hidden layers to build up context
3. Outputting a probability distribution over the vocabulary for the next word

Here‘s a simple example of building a character-level RNN language model in Python using Keras:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Embedding

# Prepare the data
text = open(‘input.txt‘).read()
chars = sorted(list(set(text)))
char_indices = {c: i for i, c in enumerate(chars)}
indices_char = {i: c for i, c in enumerate(chars)}

# Create training sequences and targets
maxlen = 40
step = 3
sequences = []
targets = []
for i in range(0, len(text) - maxlen, step):
    sequence = text[i:i+maxlen]
    target = text[i+maxlen]
    sequences.append([char_indices[c] for c in sequence])
    targets.append(char_indices[target])

# Define the model architecture 
model = Sequential([
    Embedding(len(chars), 256, input_length=maxlen),
    LSTM(512),
    Dense(len(chars), activation=‘softmax‘)
])
model.compile(loss=‘sparse_categorical_crossentropy‘, optimizer=‘adam‘)

# Train the model
model.fit(sequences, targets, epochs=30)

# Generate new text
seed_text = ‘The quick brown fox‘
generated = seed_text
for i in range(100):
    x = np.zeros((1, maxlen), dtype=int)
    x[0, :len(seed_text)] = [char_indices[c] for c in seed_text[-maxlen:]]
    probs = model.predict(x).reshape(-1)
    next_char = indices_char[np.random.choice(len(chars), p=probs)]
    generated += next_char
    seed_text = seed_text[1:] + next_char
print(generated)

This model learns to predict the next character based on the previous 40 characters. By sampling from the model‘s output distribution repeatedly, we can generate new text that mimics the style of the training data.

Of course, this is just a toy example – modern neural language models are far more sophisticated, leveraging techniques like attention, transformers, and unsupervised pre-training on massive corpora. But the core principles remain the same.

Leveraging Pre-trained Language Models

Training large-scale language models from scratch requires vast amounts of data and compute resources. Fortunately, there are powerful pre-trained models available that you can leverage for a wide range of NLP tasks. One of the most influential is GPT (Generative Pre-trained Transformer) from OpenAI.

GPT-2, released in 2019, is a large transformer-based language model trained on a dataset of 8 million web pages. It achieved state-of-the-art results in many language tasks and sparked discussions about the potential impacts of increasingly powerful language models.

Using GPT-2 for text generation is straightforward with the transformers library:

from transformers import GPT2LMHeadModel, GPT2Tokenizer

model = GPT2LMHeadModel.from_pretrained(‘gpt2‘)
tokenizer = GPT2Tokenizer.from_pretrained(‘gpt2‘)

prompt = "In a shocking finding, scientists discovered a herd of unicorns living in a remote, " \
         "previously unexplored valley, in the Andes Mountains. Even more surprising to the " \
         "researchers was the fact that the unicorns spoke perfect English."

input_ids = tokenizer.encode(prompt, return_tensors=‘pt‘)

outputs = model.generate(input_ids, max_length=200, num_return_sequences=3)

for i, output in enumerate(outputs):
    print(f"Sample {i+1}: {tokenizer.decode(output, skip_special_tokens=True)}")

This code loads the pre-trained GPT-2 model and generates three continuations of the given prompt. The results are often shockingly coherent and creative, demonstrating the power of large language models.

Since the release of GPT-2, even larger and more capable models have emerged, such as GPT-3, PaLM, and Chinchilla. These models push the boundaries of what‘s possible with language generation and understanding.

Conclusion

Language models are a cornerstone of modern NLP, enabling a wide range of applications from machine translation to conversational AI. We‘ve seen how n-gram models provide a simple yet effective approach to modeling word sequences, while neural language models offer more expressive representations at the cost of increased complexity.

With the advent of large pre-trained models like GPT, it‘s now possible to leverage state-of-the-art language modeling capabilities without the need for massive compute resources. As these models continue to advance, they open up exciting possibilities for building more intelligent, language-aware systems.

Whether you‘re a researcher pushing the boundaries of NLP or a practitioner looking to incorporate language understanding into your applications, a solid grasp of language modeling techniques is essential. Armed with the knowledge from this guide, you‘re well-equipped to start building your own language models and exploring the vast potential of NLP.