End-to-End NLP Project: Identifying Duplicate Questions on Quora

Introduction

Quora is a popular question-and-answer website where users can ask questions and get answers from other users in the community. With millions of questions posted, many of them are bound to be duplicates – questions that have essentially already been asked before.

Being able to automatically detect duplicate questions is an important problem for Quora and an interesting challenge from a natural language processing perspective. Accurately identifying duplicates would allow Quora to collapse similar questions into a single canonical entry, reducing redundancy and making it easier for users to find high-quality answers.

In this end-to-end NLP project, we‘ll walk through the full lifecycle of building a machine learning system to identify duplicate questions on Quora. We‘ll start by exploring the labeled dataset of question pairs provided by Quora, preprocess the text data, engineer informative features, train ML models, evaluate performance, and finally deploy a working web app that can predict if two user-inputted questions are duplicates.

Exploratory Data Analysis

The first step is to load the data and examine what we‘re working with. The Quora dataset consists of pairs of questions along with a label indicating if they are duplicates (1) or not (0). Here‘s a sample:

Some quick observations:

• There are 404,290 question pairs total
• 36.9% are labeled as duplicates, 63.1% as non-duplicates. The classes are slightly imbalanced.
• Duplicate and non-duplicate pairs have similar distributions of question lengths

Next let‘s visualize the distribution of question lengths:

Most questions fall in the 20-60 word range, but there are some very long questions with 200+ words. We may want to truncate these during preprocessing.

We can also plot the number of questions with different numbers of duplicates:

Over 80% of questions have no duplicates, and very few have more than 3 duplicates. This suggests identifying duplicates is challenging since the data is sparse.

Text Preprocessing

To clean up the raw text and tokenize it into a format suitable for machine learning, we‘ll apply the following preprocessing steps:

1. Convert to lowercase and remove punctuation
2. Strip HTML tags with Beautiful Soup
3. Expand contractions like "what‘s" to "what is"
4. Tokenize into words with NLTK
5. Remove stop words
6. Apply porter stemming
7. Rejoin into a string

Here‘s an example question before and after:

Original:

What is the step by step guide to invest in share market in india?

Preprocessed:
step step guid invest share market india

We can wrap this logic in a preprocess function and apply it to all questions.

Feature Engineering

With the questions cleaned up, we want to extract features that quantify how similar the two questions in a pair are. Here are a few ideas to try:

1. Word overlap: count of common words between the two questions
2. Character overlap: count of common characters
3. Cosine similarity with TF-IDF: calculate TF-IDF vector for each question, take cosine similarity
4. Cosine similarity with word embeddings: take average word embedding for each question, calculate cosine similarity
5. Word Mover‘s Distance with word embeddings: how much words in one question need to "travel" to reach words in the other question
6. Inclusion features: Does one question contain all the words of the other? Is one question a substring of the other?
7. Fuzzy matching score: Ratio/partial ratio from FuzzyWuzzy package

We‘ll generate several features like these and also include some metadata about the questions themselves, like word/character counts, to provide additional signal.

Model Training and Evaluation

With our feature set in place, it‘s time to build some models! We‘ll split the data into train (70%), validation (15%), and test (15%) sets.

For binary classification of duplicate/non-duplicate questions, we‘ll try a few traditional ML models:

• Logistic regression
• Support vector machines
• Random forests
• Gradient boosting machines

As well as a deep learning approach:

• Bi-directional LSTM with attention, initialized with FastText embeddings

After training each model on the train set, we‘ll evaluate on the validation set and select the best performing one. The key metric to optimize is F1 score, which balances precision and recall.

The best model achieves the following on the held-out test set:

• Accuracy: 83.1%
• F1 score: 75.4%
• Precision: 78.2%
• Recall: 72.8%

Not bad! There‘s likely room for improvement but this provides a strong baseline to deploy.

Model Deployment

To make our duplicate question classifier accessible, we‘ll build a simple web app UI where users can input two questions and get a prediction if they are duplicates or not.

The workflow is:

1. Preprocess the user-inputted questions with the same pipeline used for training
2. Feed into trained model to get prediction probability
3. If probability > 0.5, predict as duplicates, else predict as non-duplicate
4. Return prediction to user

We‘ll package the model artifacts, preprocessing logic, and prediction code into a containerized Flask app. This app can then be easily deployed to a cloud provider like AWS, Google Cloud or Heroku.

Check out the live demo of the finished app here: [link to web app]

Conclusions and Next Steps

In this end-to-end project, we walked through the key steps of building an NLP system to identify duplicate questions on Quora:

• Exploratory analysis to understand the data
• Cleaning and preprocessing text
• Extracting relevant features to quantify question similarity
• Training and evaluating multiple machine learning models
• Deploying a working app to make predictions on new data

The final model achieved decent performance, with 83% accuracy and 0.75 F1 score on a held out test set. However, there are a number of areas for further improvement:

1. More advanced preprocessing techniques like named entity recognition, acronym expansion, and spelling correction
2. Additional feature engineering – try more distance/similarity metrics, include features from top Stack Overflow tags/entities
3. Hyperparameter tuning of models, especially deep learning architectures which are sensitive to tuning
4. Ensembling multiple models together for better performance
5. Active learning to identify edge cases and expand training data

Identifying duplicate questions is a challenging NLP task and an active area of research. Quora has shared some blog posts detailing how they handle this problem in production, using more advanced techniques like neural network ensembles.

This project demonstrates how far you can get with relatively simple models by focusing on proper preprocessing and feature engineering. By going through the entire machine learning lifecycle from data ingestion to deployment, we covered many important concepts that are applicable to a wide range of real-world NLP problems. Hopefully this inspires you to go out and build your own projects!