Build a Powerful Chat Assistant for PDFs and Articles Without an OpenAI API Key

Introduction

Imagine having your own personal AI assistant that can instantly answer questions based on the contents of any PDF document or web article, without the need for an expensive API key. In this in-depth guide, we‘ll walk through how to build just that using open-source NLP libraries and models.

The power of large language models has opened up exciting opportunities in natural language processing, but relying on APIs like OpenAI can get costly. Fortunately, high-quality open-source alternatives now exist that allow you to create your own powerful applications at a fraction of the cost.

Our chat assistant will be able to take a question from the user, search through a provided PDF or webpage to find the most relevant passages, and extract an answer – all on your local machine or server. This has huge potential for creating knowledgebases, research tools, customer support chatbots, and more.

Workflow Overview

At a high level, our chat assistant will follow these steps:

1. Extract text content from a supplied PDF document or webpage URL
2. Split the text into smaller chunks
3. Convert each text chunk and the user‘s question into a vector representation called an embedding
4. Use semantic search to find the text chunks most relevant to answering the question
5. Pass the question and relevant text to a question-answering model to generate a final answer

By breaking down the problem into discrete steps, we can build a pipeline using modular open-source components. Let‘s dive into the details of each step and see how to implement this in Python.

Extracting Text from PDFs and Webpages

The first step is to extract the raw text content from a PDF or webpage. For PDFs, we‘ll use the PyPDF2 library:

from PyPDF2 import PdfReader

def extract_text_from_pdf(file):
    reader = PdfReader(file)
    text = ""
    for page in reader.pages:
        text += page.extract_text()
    return text

For webpages, we‘ll use the Requests library to fetch the HTML and BeautifulSoup to parse and extract the relevant content:

import requests
from bs4 import BeautifulSoup

def extract_text_from_url(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, "html.parser")
    text = " ".join([p.get_text() for p in soup.find_all("p")])
    return text

These functions provide a simple way to go from a PDF or URL to a string of text that we can work with in the next steps.

Splitting Text into Chunks

To efficiently process longer documents, we‘ll split the text into smaller chunks. This allows us to perform semantic search at a more granular level to find the most relevant snippets. The langchain library provides a convenient CharacterTextSplitter for this:

from langchain.text_splitter import CharacterTextSplitter

def split_text(text):
    text_splitter = CharacterTextSplitter(
        separator="\n",
        chunk_size=500,
        chunk_overlap=50,
        length_function=len,
    )
    chunks = text_splitter.split_text(text)
    return chunks

Here we configure the splitter to create chunks of around 500 characters each, with a 50 character overlap between chunks. This ensures that relevant information isn‘t lost at the boundaries between chunks.

Generating Embeddings

An embedding is a vector representation that captures the semantic meaning of a piece of text. By generating embeddings for our text chunks and the user‘s question, we can perform semantic search to find the most relevant chunks for answering the question.

While the OpenAI API provides embedding functionality, we‘ll use the open-source all-MiniLM-L6-v2 model via the Sentence-Transformers library. This model was trained on a large dataset to generate high-quality sentence and paragraph embeddings.

from sentence_transformers import SentenceTransformer

def get_embeddings(texts):
    model = SentenceTransformer("all-MiniLM-L6-v2")
    embeddings = model.encode(texts)
    return embeddings

We simply pass a list of texts to the model and it returns a list of embedding vectors. We‘ll generate embeddings for each of our text chunks as well as the user‘s question.

Semantic Search

Now that we have vector representations of our chunks and question, we can perform a semantic search to find the chunks most relevant to the question. This is done by computing the cosine similarity between the question embedding and each chunk embedding.

Cosine similarity measures the angle between two vectors – the smaller the angle, the more similar the vectors are. We can use the util.semantic_search function from Sentence-Transformers to do this efficiently:

from sentence_transformers import util

def semantic_search(query_embedding, chunk_embeddings, top_k=3):
    similarities = util.cos_sim(query_embedding, chunk_embeddings)
    sorted_indices = similarities.argsort(descending=True)
    top_k_indices = sorted_indices[0][:top_k]
    top_k_chunks = [chunks[i] for i in top_k_indices]
    return top_k_chunks

This returns the top_k most similar chunks to the question embedding. We can configure top_k depending on how much context we want to pass to the question-answering model.

Generating the Answer

The final step is to pass the user‘s question and the most relevant text chunks to a question-answering model, which will extract the final answer. While you could use OpenAI‘s question-answering API, there are open-source models that provide similar capabilities.

Some good options are:

• BERT models fine-tuned for QA on datasets like SQuAD
• Distilled versions of GPT-3 models like GPT-Neo
• Purpose-built open-source models like BiDAF (Bi-Directional Attention Flow)

Here‘s an example using a DistilBERT model fine-tuned for QA:

from transformers import pipeline

def get_answer(question, chunks):
    qa_model = pipeline("question-answering", model="distilbert-base-cased-distilled-squad")

    result = qa_model(question=question, context="\n".join(chunks))
    answer = result["answer"]
    return answer

By combining the extracted context with strong question-answering models, we can generate accurate and relevant answers from our documents.

Putting It All Together

We can now combine all of these components into a complete script:

# qa.py

from PyPDF2 import PdfReader
import requests
from bs4 import BeautifulSoup
from langchain.text_splitter import CharacterTextSplitter
from sentence_transformers import SentenceTransformer, util
from transformers import pipeline

def extract_text_from_pdf(file):
    reader = PdfReader(file)
    text = ""
    for page in reader.pages:
        text += page.extract_text()
    return text

def extract_text_from_url(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, "html.parser")
    text = " ".join([p.get_text() for p in soup.find_all("p")])
    return text

def split_text(text):
    text_splitter = CharacterTextSplitter(
        separator="\n",
        chunk_size=500,
        chunk_overlap=50,
        length_function=len,
    )
    chunks = text_splitter.split_text(text)
    return chunks

def get_embeddings(texts):
    model = SentenceTransformer("all-MiniLM-L6-v2")
    embeddings = model.encode(texts)
    return embeddings

def semantic_search(query_embedding, chunk_embeddings, top_k=3):
    similarities = util.cos_sim(query_embedding, chunk_embeddings)
    sorted_indices = similarities.argsort(descending=True)
    top_k_indices = sorted_indices[0][:top_k]
    top_k_chunks = [chunks[i] for i in top_k_indices]
    return top_k_chunks

def get_answer(question, chunks):
    qa_model = pipeline("question-answering", model="distilbert-base-cased-distilled-squad")

    result = qa_model(question=question, context="\n".join(chunks))
    answer = result["answer"]
    return answer

text = extract_text_from_pdf("document.pdf") 
# or extract_text_from_url("https://example.com/article")

chunks = split_text(text)
chunk_embeddings = get_embeddings(chunks)

query = "What is the main idea of the article?"
query_embedding = get_embeddings([query])[0]

relevant_chunks = semantic_search(query_embedding, chunk_embeddings)
answer = get_answer(query, relevant_chunks)

print(answer)

With this, you have a complete, powerful chat assistant that can answer questions based on any provided PDF or web article. The key steps are:

1. Extracting text from the source
2. Splitting it into chunks
3. Generating embeddings for semantic search
4. Using a question-answering model to extract the final answer

By leveraging high-quality open-source models, you can build your own version of ChatGPT‘s API capabilities at a fraction of the cost. This approach is highly extensible – you can swap in different text splitters, embedding models, and QA models to optimize performance on your specific use case.

Conclusion

Building an AI chat assistant no longer requires huge language models and expensive API access. With the right open-source tools and a bit of glue code, you can create powerful applications that provide relevant, accurate answers from unstructured data.

The approach outlined in this article breaks the problem down into discrete, composable steps – content extraction, text splitting, embedding, semantic search, and question answering. Each of these can be tuned and optimized independently, allowing you to adapt the system to your needs.

Potential use cases are endless – automated customer support, personal research assistants, knowledge management systems, and more. You could even extend it to handle other data types like audio transcripts, scanned images, or social media feeds.

As NLP technology continues to advance and more high-quality models are released, the possibilities will only grow. Hopefully this guide has equipped you with the tools to start building your own powerful AI assistants. Happy coding!

Frequently Asked Questions

Q: How does the all-MiniLM-L6-v2 model work under the hood?

A: all-MiniLM-L6-v2 is a Sentence Transformer model, which means it‘s designed to generate embeddings for sentences and paragraphs. It was trained on a large diverse dataset using a technique called siamese network training.

In siamese networks, two identical models are used to process two different inputs, and the objective is to learn embeddings such that similar inputs are close together in vector space while dissimilar inputs are far apart. This allows the model to learn semantic relationships between different pieces of text.

MiniLM refers to the fact that the model uses a compact architecture based on the Transformer, which allows for efficient computation. The L6-v2 specifies that it‘s a 6-layer model trained with an updated approach. Despite its smaller size, it achieves comparable performance to much larger models on semantic search tasks.

Q: What‘s the difference between semantic search and keyword search?

A: Keyword search looks for exact matches of query terms in a document – for example, if you searched for "dog", it would only return documents that explicitly contain the word "dog". This can miss relevant results that use synonyms or discuss the concept without using that exact term.

In contrast, semantic search tries to understand the meaning behind the query and find documents that are conceptually similar, even if they don‘t contain the exact keywords. It does this by comparing the embeddings of the query and documents, which capture semantic meaning.

So a semantic search for "dog" might also return documents mentioning "pups", "canine", "man‘s best friend", etc. This allows for much more flexible and powerful search capabilities that understand intent rather than just matching strings.

Q: Can I use different models for the question-answering component?

A: Absolutely! The question-answering model is a modular component in this system. You can swap in any model that takes a question and context as input and returns an extracted answer.

Some options to consider:

• BERT or RoBERTa models fine-tuned on SQuAD or other QA datasets
• DistilBERT or other distilled models for faster inference
• More recent architectures like T5, DeBERTa, or ELECTRA
• Domain-specific models trained on data similar to yours

The best model will depend on your specific use case, latency requirements, and available compute resources. It‘s a good idea to experiment with a few different options and evaluate their performance on your task.

Keep in mind you can also fine-tune these models on your own data to improve performance. And as new state-of-the-art models get released, you can swap them in to boost the capabilities of your chat assistant over time.

Q: How can I make this faster for long documents?

A: Handling very long documents like books can be challenging from a compute perspective. Some strategies to make this more efficient:

1. Use a more aggressive text splitter to create smaller chunks. This will result in more chunks to process, but each one will be faster.

2. Pre-compute and cache the embeddings for each chunk. That way you only need to generate the query embedding at runtime. You can save the chunk embeddings to disk or a database for quick loading.

3. Use approximate nearest neighbor search for the semantic search step. Libraries like Faiss can find the top-k most similar embeddings much faster than brute-force pairwise comparisons.

4. Experiment with smaller, more efficient models. For example, you might use a tiny model for embedding and a small distilled model for question answering. The tradeoff in quality may be worth the speedup.

5. Consider a retrieve-then-read approach where you first use semantic search to identify a small subset of potentially relevant chunks, then do a more thorough reading of just those chunks to find the answer. This can be more efficient than running the QA model over all chunks.

Ultimately, there will be some tradeoff between thoroughness and speed. The key is to find the right balance for your use case through experimentation and iteration. With the right optimizations, you can build a system that provides near-instant answers even for large corpora.