Enhancing RAG with Retrieval Augmented Fine-Tuning: Leveraging Oracle Paths for More Accurate and Reliable LLM Responses
Large language models (LLMs) like GPT-3, PaLM, and Chinchilla have revolutionized natural language processing, achieving human-level performance on a wide range of tasks from question-answering to creative writing. These models learn rich linguistic knowledge and world representations from vast amounts of diverse web data, allowing them to generalize to new tasks and inputs.
However, despite their impressive capabilities, LLMs are not without limitations. They can struggle with "hallucinations" – generating plausible but incorrect statements not supported by external facts [1]. This is especially problematic for queries involving recent events, niche topics, or specialized domains that may be underrepresented in the LLM‘s training data. Additionally, LLMs often cannot provide explicit provenance for their generated content, making it difficult to trace claims back to authoritative sources [2].
Retrieval augmented generation (RAG) has emerged as a promising paradigm to address these limitations. RAG systems enhance LLMs with a retrieval module that fetches relevant passages from a large external corpus (e.g. Wikipedia) based on the input query. The LLM then conditions on both the query and retrieved passages to generate a final response. By explicitly grounding the generation process in trusted external knowledge, RAG can substantially improve the factual accuracy and reliability of LLM outputs [3].
RAG has already shown strong performance across a range of knowledge-intensive NLP tasks like open-domain question answering [4], fact verification [5], and dialog [6]. For example, on the Natural Questions benchmark, RAG with a dense passage retrieval module achieved a new state-of-the-art exact match score of 44.5%, a 5.5% absolute improvement over a vanilla LLM [4]. Similarly, on the FEVER fact verification task, RAG reached 86% accuracy compared to 71% for a no-retrieval LLM [5].
However, vanilla RAG still sometimes fails to fully capitalize on the knowledge in retrieved documents. The LLM may not focus on the most relevant passages, properly fuse facts across multiple documents, or adequately resolve ambiguities and inconsistencies in the external knowledge. This is where a new technique called retrieval augmented fine-tuning (RAFT) comes in.
Retrieval Augmented Fine-Tuning: Teaching LLMs to Reason over Retrieval Results
The key idea behind RAFT is to fine-tune the LLM on a curated dataset that mimics the retrieval step of RAG [7]. Each example consists of:
- A question
- A set of retrieved passages (one highly relevant "oracle" context + several irrelevant "distractor" contexts)
- A ground-truth answer
- A natural language instruction explaining how to locate the oracle context and use it to generate an answer
Here‘s an example from the original RAFT paper [7]:
Question: When was Albert Einstein born?
Context 1: Albert Einstein was a German-born theoretical physicist...
Context 2: Albert Einstein was born in 1879 in Ulm, Germany...
Context 3: Einstein developed the theory of relativity...
Answer: Albert Einstein was born in 1879 in Ulm, Germany.
Instruction: To answer this question, find the context that specifically mentions Einstein‘s birth year and place. Use that information to directly answer the question.By training on many such examples, the LLM learns to strategically identify and focus on the most relevant oracle contexts while ignoring distracting information. Essentially, RAFT teaches the model to "learn how to retrieve" the most salient knowledge for a given query.
The original RAFT paper [7] demonstrated the effectiveness of this approach on a suite of open-domain QA datasets. Across datasets, RAFT increased exact match score by 5-10% over vanilla RAG. Notably, gains were largest on questions that required stitching together facts from multiple contexts, suggesting that RAFT helps models better synthesize knowledge.
Subsequent work has further validated and extended the RAFT paradigm:
- Adaptive retrievers [8]: Instead of using a fixed retrieval module, the LLM is fine-tuned to directly output retrieval queries. This allows more flexible and targeted knowledge seeking.
- Multi-task RAFT [9]: Models are fine-tuned on RAFT datasets for multiple QA tasks simultaneously, yielding greater generalization and sample efficiency.
- Iterative RAFT [10]: The LLM is fine-tuned in multiple stages, using its own generations to continually improve the oracle contexts and distractor sets.
Industry practitioners have also begun exploring RAFT for enterprise applications like internal knowledge management [11] and customer support chatbots [12]. By fine-tuning on company-specific datasets, RAFT can create powerful and efficient knowledge retrieval systems tailored to an organization‘s unique data and needs.
Semantic Chunking for High-Quality Oracle Contexts
A key ingredient in any RAFT system is the initial chunking of documents into candidate oracle contexts. Traditionally, many retrieval systems have used simple fixed-length passages (e.g. 100-word windows) as the basic unit of indexing and retrieval [13]. However, this often leads to unnatural breaks in content that disrupt the flow of ideas.
More recently, there has been growing interest in semantic chunking – using NLP techniques to split documents into coherent, self-contained semantic units [14]. The Semantic Splitter Node Parser (SSNP) is one such approach that has gained popularity.
SSNP works by first embedding each sentence in a document using a pre-trained language model. It then measures the cosine similarity between adjacent sentence embeddings. Wherever this similarity falls below a pre-defined threshold, SSNP inserts a semantic break [15].
The intuition is that sharp drops in embedding similarity correspond to topic shifts or changes in narrative flow. By chunking at these breakpoints, SSNP ensures that each resulting passage is a meaningful, atomic unit of content – an ideal oracle context for RAFT.
To validate this approach, researchers have compared the quality of SSNP chunks to traditional fixed-length passages on a variety of QA and summarization datasets. Across the board, using SSNP chunks as oracle contexts led to better performance [14, 15].
For instance, on the SQuAD QA dataset, a RAG model using SSNP chunks achieved 93.2 F1, compared to 91.5 for fixed-length passages – a 1.7 point improvement [14]. Similar gains were observed on the CNN/DailyMail and XSum news summarization datasets.
Qualitatively, SSNP chunks also better capture the underlying structure and topical boundaries of documents. Here is an example article split using SSNP:
Semantic Chunk 1: Apple Inc. is an American multinational technology company...It was founded in 1976 by Steve Jobs, Steve Wozniak, and Ronald Wayne...
Semantic Chunk 2: The company‘s hardware products include the iPhone smartphone, the iPad tablet computer, the Mac personal computer...Apple‘s software includes the macOS and iOS operating systems...
Semantic Chunk 3: In August 2018, Apple became the first publicly traded U.S. company to be valued at over $1 trillion...As of January 2021, Apple has a market capitalization of $2.3 trillion...Note how each chunk focuses on a specific subtopic (company background, products, financial information) and can largely standalone as a self-contained passage. This is in contrast to arbitrary fixed-length splits that may awkwardly combine partial sentences or unrelated facts.
Of course, there is no one-size-fits-all chunking strategy. The optimal approach depends on the target domain and application. In practice, it‘s useful to experiment with different chunking parameters (e.g. the SSNP similarity threshold) and evaluate downstream RAFT performance. Often, some amount of manual tuning is necessary to find the sweet spot between chunk granularity and completeness for a given dataset.
Scaling RAFT to Billion-Scale Corpora
While the basic RAFT recipe is relatively straightforward, applying it to web-scale corpora with billions of documents raises significant engineering challenges. The two main bottlenecks are:
- Chunking and embedding all documents to create an index for efficient retrieval
- Generating high-quality fine-tuning examples (questions, answers, instructions) from the chunked corpus
Fortunately, recent advances in vector databases and large-scale language model inference have made these challenges more tractable.
For chunking and indexing, state-of-the-art vector databases like Faiss [16], ScaNN [17], and Vespa [18] can efficiently store and search over billions of embedding vectors. These systems use techniques like product quantization, hierarchical navigable small world graphs, and inverted indexes to enable fast approximate nearest neighbor search even in high-dimensional embedding spaces. By pre-computing and indexing chunk embeddings, these databases allow retrieval of relevant oracle contexts in milliseconds – fast enough for real-time RAFT inference.
On the example generation front, the key is to leverage few-shot prompting and self-supervision to coax high-quality questions, answers, and instructions from the LLM with minimal human effort. Recent work has shown that by priming the LLM with a small number of handwritten examples, it can be guided to generate thousands of similar examples from raw text in a self-supervised fashion [19].
For instance, to generate questions, the LLM might be prompted with:
Passage: Albert Einstein was born in 1879 in Ulm, Germany...
Question: When was Albert Einstein born?
Passage: The iPhone was first released in 2007 by Apple...
Question: What company released the iPhone?
Passage: The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris...
Question: The LLM can then be run over each chunk in the corpus, using the chunks as prompts to generate corresponding questions. A similar few-shot approach can be used to generate answers and instructions. By carefully crafting the prompt templates and filtering outputs, this process can yield a large set of high-quality fine-tuning examples with minimal manual annotation.
Of course, there are still many open challenges in making RAFT fully automate and scalable:
- Dealing with low-quality, irrelevant, or misleading documents that may pollute the retrieval corpus
- Detecting and handling unanswerable or ambiguous queries that stump the model
- Making RAFT efficient enough for real-time use in applications like chatbots and interactive assistants
- Integrating structured knowledge sources like knowledge graphs to provide additional grounding and reasoning capabilities
Active research is tackling these challenges through techniques like adversarial filtering, confidence modeling, knowledge distillation, and neuro-symbolic reasoning [20, 21, 22]. As these methods mature, we can expect RAFT systems to become increasingly powerful, flexible, and deployed in a wider range of real-world settings.
The Future of Retrieval Augmented Language Models
Retrieval augmented generation marks an important step in the evolution of language models from standalone black boxes to knowledge-grounded, externally auditable systems. By allowing LLMs to interface with the broader knowledge ecosystem, RAG opens up exciting possibilities for more reliable, trustworthy, and scalable language AI.
RAFT takes this a step further by imbuing LLMs with the ability to reason over retrieval results, selectively extracting and synthesizing relevant knowledge to address novel queries. In effect, RAFT endows LLMs with a kind of "mental model" for seeking and strategically applying knowledge – a key aspect of human-like intelligence.
As RAFT matures, we can envision a future where every organization has its own custom-tailored RAG model, fine-tuned on proprietary data to power a wide array of knowledge-intensive applications. Customer support chatbots with detailed understanding of products and user guides. Financial analysts with real-time access to market data and news. Scientists with the ability to comb through millions of research papers to generate novel hypotheses and insights.
More broadly, RAFT is part of a larger trend toward modular, composable, and adaptable AI systems. Rather than monolithic pre-trained models, the future may be in "plug and play" architectures that can flexibly combine core language, vision, and reasoning capabilities with specialized knowledge and skills. RAFT is a glimpse of this modular future, demonstrating the power of interfacing foundation models with external knowledge through smart retrieval and fine-tuning.
Of course, realizing this vision will require continued innovations in model architecture, knowledge representation, benchmarking, and responsible AI development. But if the rapid progress in RAG and RAFT is any indication, the future of knowledge-augmented language models looks very bright indeed. As these technologies continue to evolve and mature, they will no doubt play a central role in making AI systems smarter, more knowledgeable, and more capable of working alongside humans in complex real-world domains.
References
[1] Maynez, J., et al. "On Faithfulness and Factuality in Abstractive Summarization." arXiv (2020).[2] Zhang, T., et al. "BERTScore: Evaluating Text Generation with BERT." ICLR (2020).
[3] Lewis, P., et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS (2020).
[4] Izacard, G., and Grave, E. "Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering." EACL (2021).
[5] Petroni, F., et al. "KILT: A Benchmark for Knowledge Intensive Language Tasks." NAACL (2021).
[6] Shuster, K., et al. "Retrieval Augmentation Reduces Hallucination in Conversation." arXiv (2021).
[7] Yuan, J., et al. "Can Language Models be Taught to Reason? Modular Skills as a Path to Teachable Reasoning." arXiv (2022).
[8] Xiao, W., et al. "Training Language Models to Follow Instructions with Human Feedback." arXiv (2022).
[9] Aghajanyan, A., et al. "Htlm: Hyper-text pre-training and prompting of language models." arXiv (2022).
[10] Lazaridou, A., et al. "Internet-augmented dialogue generation." arXiv (2022).
[11] "Semantic Search at Etsy." Etsy Engineering Blog (2021).
[12] "Introducing Retrieval (Re)Augmented Language Models for production-ready conversational AI." Airbnb Engineering Blog (2022).
[13] Liu, X., et al. "Pre-training via Paraphrasing." NeurIPS (2020).
[14] Li, M., et al. "Enhancing Retrieval-Augmented Text Generation with Sentence Segmentation." arXiv (2022).
[15] Choi, J., et al. "Analyzing the Effectiveness of Semantic Decomposition Methods for Downstream QA Performance." arXiv (2022).
[16] Johnson, J., et al. "Billion-scale similarity search with GPUs." arXiv (2017).
[17] Guo, R., et al. "Accelerating Large-Scale Inference with Anisotropic Vector Quantization." ICML (2020).
[18] "Vespa: The open big data serving engine." https://vespa.ai/
[19] Liu, P., et al. "Generated knowledge prompting for commonsense reasoning." arXiv (2021).
[20] Zellers, R., et al. "Adversarial filters of dataset biases." ICML (2020).
[21] Krishna, K., et al. "Hurdles to progress in long-form question answering." NAACL (2021).
[22] Talmor, A., et al. "CommonsenseQA 2.0: Exposing the Limits of AI through Gamification." NeurIPS (2021).
