Detecting Fake News with Deep Learning and Keras

The proliferation of fake news online has become a major problem in recent years. False stories disguised as legitimate journalism can spread rapidly on social media, misleading people and even influencing important events like elections. As the volume of information on the internet continues to grow, identifying fake content manually becomes increasingly infeasible. However, artificial intelligence and machine learning offer a potential solution.

In this post, we‘ll walk through building a deep learning model using Keras to automatically classify news articles as real or fake. While not a production-ready system, it will demonstrate an approach to this challenge and some key concepts in natural language processing. We‘ll especially focus on the important step of tokenizing text data with the Keras Tokenizer.

The Dataset

To train and evaluate our model, we‘ll be using a dataset of approximately 45,000 articles compiled from various sources. The fake articles come from over 500 unreliable websites identified by fact-checkers. The real articles are from the Reuters news agency. The data is fairly evenly split between the two classes.

While this dataset is a good starting point, it does have some notable limitations. The articles are focused only on US news and the legitimate articles come from a single source. Some have also identified data leakage issues, where features only appear in one class, making classification artificially easy. We‘ll aim to address these problems in the pre-processing steps.

Text Pre-Processing with the Keras Tokenizer

Neural networks require numeric input, so our first challenge is converting the article text to numbers. We‘ll do this using the Tokenizer and pad_sequences utilities built into Keras.

The Tokenizer converts text documents to sequences of integers. It works by:

1. Splitting each text into words (tokens)
2. Generating a dictionary mapping words to unique integers
3. Converting each text into a sequence of those integers

For example, consider the phrases:

"the cat sat on the mat"
"the dog played in the garden"

The Tokenizer would create a dictionary like:

{
‘the‘: 1,
‘cat‘: 2,
‘sat‘: 3,
‘on‘: 4,
‘mat‘: 5,
‘dog‘: 6,
‘played‘: 7,
‘in‘: 8,
‘garden‘: 9
}

And then encode the phrases as:
[[1, 2, 3, 4, 1, 5] [1, 6, 7, 8, 1, 9]]

We can configure the Tokenizer to ignore punctuation, convert to lowercase, filter out rare words, and more. This gives us fine-grained control over how the text is handled.

After tokenizing, we use pad_sequences to ensure all the integer sequences are the same length, either truncating long ones or padding short ones with zeros. This is necessary for feeding batches of data into the neural network.

Here‘s what the code looks like:

from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Define the Tokenizer
tokenizer = Tokenizer(num_words=MAX_VOCAB_SIZE, 
                      filters=‘!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n‘,
                      lower=True)
# Fit it on the texts
tokenizer.fit_on_texts(train_texts)

# Convert texts to sequences 
train_sequences = tokenizer.texts_to_sequences(train_texts)
test_sequences = tokenizer.texts_to_sequences(test_texts)

# Pad the sequences to a fixed length
train_padded = pad_sequences(train_sequences, maxlen=MAX_SEQUENCE_LENGTH, 
                             padding=PADDING_TYPE, truncating=TRUNCATING_TYPE)
test_padded = pad_sequences(test_sequences, maxlen=MAX_SEQUENCE_LENGTH,
                            padding=PADDING_TYPE, truncating=TRUNCATING_TYPE)  

After these steps, our articles will be represented as uniform-length sequences of integers ready for input into an embedding layer.

Word Embeddings

While we could feed the integer token sequences directly into a neural network, that has some big drawbacks. The integers are arbitrary and don‘t capture any semantic relationships between words. The sequences would also be very high-dimensional with mostly zeros (one-hot encoding).

Word embeddings solve this by mapping the large integer space to a lower-dimensional vector space where similar words have similar encodings. We can either learn this embedding as part of the model training, or load a pre-trained embedding.

Keras offers an Embedding layer that can learn a specialized embedding for the specific text in our dataset. All we need to provide is the vocabulary size and desired embedding dimensionality and it handles the rest:

from tensorflow.keras.layers import Embedding

embedding = Embedding(MAX_VOCAB_SIZE, EMBEDDING_SIZE, 
                      input_length=MAX_SEQUENCE_LENGTH)

Alternatively, we can use pre-trained word vectors like those from Word2Vec, GloVe, or fastText. These are learned from huge text corpora and capture general word relationships. Using pre-trained embeddings is a form of transfer learning that can improve model performance, especially when training data is limited.

For this project, we‘ll try spaCy‘s en_core_web_sm vectors. These are 300-dimensional vectors trained on web text, so they should be well-suited to our news articles. Loading them into Keras is straightforward:

import spacy

# Load the spacy model
nlp = spacy.load(‘en_core_web_sm‘)

# Build a mapping of our tokens to spaCy‘s vector indices  
embedding_matrix = np.zeros((MAX_VOCAB_SIZE, EMBEDDING_SIZE))
for word, i in tokenizer.word_index.items():
    embedding_vector = nlp(word)[0].vector
    if i < MAX_VOCAB_SIZE:
        embedding_matrix[i] = embedding_vector

# Load this matrix into an Embedding layer 
embedding = Embedding(MAX_VOCAB_SIZE, EMBEDDING_SIZE, 
                      weights=[embedding_matrix],
                      input_length=MAX_SEQUENCE_LENGTH,
                      trainable=False)

By setting trainable=False, we keep the embedding fixed, letting the model use the pre-trained relationships between words.

Model Architecture

With our text represented numerically and embedded into a semantic vector space, we‘re ready to define a neural network architecture for our binary text classification problem.

Since we‘re working with variable-length sequences of words, a recurrent neural network (RNN) is a natural fit. Specifically, we‘ll use a bidirectional gated recurrent unit (GRU).

GRUs are similar to the popular long short-term memory units (LSTMs) in learning long-range dependencies, but are cheaper to compute. Running the GRU in both directions can further improve performance, as it gives the network access to contextual information on both sides of each word.

Here‘s a basic architecture using the Keras functional API:

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense, Bidirectional, GRU, BatchNormalization

# The input will be sequence of MAX_SEQUENCE_LENGTH integers
sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype=‘int32‘)

# Embed the integer sequences 
embedded_sequences = embedding(sequence_input)

# Pass the embedded sequences through a bidirectional GRU 
x = Bidirectional(GRU(64, return_sequences=True))(embedded_sequences)
x = Bidirectional(GRU(64))(x)

# Apply batch normalization 
x = BatchNormalization()(x)

# Feed into final dense layer for classification
preds = Dense(1, activation=‘sigmoid‘)(x)

# Compile the model
model = Model(sequence_input, preds)
model.compile(loss=‘binary_crossentropy‘,
              optimizer=‘adam‘,
              metrics=[‘accuracy‘])  

A few key points:

• The input shape is determined by our maximum sequence length
• The embedding is the first layer, mapping sequences to vectors
• Two bidirectional GRU layers learn the text representations
• Batch normalization improves training speed and generalization
• A sigmoid output produces a probability between 0-1

After training, the model achieved an impressive 99% accuracy on the test set using the Keras embedding and 95% using spaCy.

Caveats and Next Steps

While these results are encouraging, it‘s important to remember the limitations of our dataset. A model trained only on a narrow slice of US political news may not generalize well to other kinds of articles or cultural contexts. The suspicious degree of separation between the classes also suggests the model may be learning superficial patterns rather than deep semantic understanding.

Improving the training data should be the first priority for a production fake news detector. Gathering a more diverse, representative, and carefully curated dataset is crucial. Augmenting the data with paraphrasing, translation, or text generation techniques could also help.

On the modeling side, attention mechanisms, transformers, and multi-task learning are all promising areas to explore. Attention can help the model focus on the most salient parts of long articles. Transformers are becoming the dominant architecture for NLP. And jointly learning related tasks like stance detection or sentiment analysis could yield more robust representations.

Finally, model interpretability is especially important for a high-stakes application like fake news detection. Understanding what features the model is basing its predictions on is essential for ensuring fairness, reliability, and freedom from manipulation. Visualization techniques like attention heatmaps or layer activations could provide valuable insight here.

Conclusions

Fake news is a challenging problem, but deep learning offers powerful tools for automated detection. In this post, we walked through the key steps of building a Keras model for fake news classification:

1. Preparing a dataset of fake and real articles
2. Pre-processing the text using the Keras Tokenizer and padding
3. Mapping tokens to a dense vector space with word embeddings
4. Training a recurrent neural network to classify articles

While our basic model achieved high accuracy, truly solving this problem will require further iteration on datasets, architectures, and interpretability. Still, this serves as a solid introduction to an important NLP application and the core concepts involved.

With responsible AI development and deployment, deep learning could prove to be a valuable asset in the fight against online misinformation. By augmenting human fact-checkers with intelligent fake news detectors, we can hopefully make the internet a more trustworthy place.