Knowledge Fusion: A Powerful New Paradigm for Enhancing Large Language Models
In the rapidly-advancing field of natural language processing (NLP), large language models (LLMs) have emerged as one of the most promising tools for enabling machines to understand and generate human language. By training on vast quantities of text data, LLMs like GPT-3 [1], BERT [2], and T5 [3] have achieved remarkable performance across a wide range of language tasks, from question answering and text classification to summarization and dialogue.
However, training these state-of-the-art models is notoriously expensive and resource-intensive. Costing millions of dollars and requiring billions of data samples [4], developing LLMs has become prohibitively difficult for all but the most well-resourced tech companies and research institutions. And as models continue to scale up in size and complexity, this problem is only getting worse.
Knowledge fusion presents an intriguing solution. Rather than training new LLMs from scratch, it offers a way to combine and transfer the knowledge from existing pre-trained models to create something more than the sum of its parts. By externalizing the "knowledge" embedded in multiple LLMs and integrating it into a single unified model, knowledge fusion enables the creation of enhanced language models that are more knowledgeable, sample-efficient, and computationally practical.
In this article, we‘ll take a deep dive into the groundbreaking techniques behind knowledge fusion, with a special focus on the FUSELLM system introduced in [5]. We‘ll examine how it works, present key experimental results, and discuss the implications for the future of NLP and AI. While knowledge fusion is still an emerging area with many open challenges, it holds immense promise for democratizing and accelerating the development of powerful language models.
Inside FUSELLM: A Technical Deep Dive
At the core of the FUSELLM approach is a novel pipeline for externalizing, aligning, and fusing the knowledge contained in a diverse set of "source" LLMs into a single target LLM. Let‘s break down each of these key steps in detail.
Extracting Knowledge via Next-token Probability Distributions
The first step is to extract the knowledge embedded in each source LLM in a form that is standardized and easy to manipulate. But what exactly is "knowledge" in the context of a language model?
Fundamentally, the knowledge of an LLM is distributed across the parameters of its underlying neural network, which has learned patterns and relationships between words from training data. We can‘t directly access this knowledge, but we can probe it by examining the model‘s predictions given different text inputs.
Specifically, for any input text sequence, an LLM outputs a probability distribution over its vocabulary $V$ indicating how likely each word is to appear next. Formally, let $x = (x_1, …, xn)$ be a sequence of tokens (e.g. words or subwords), and let $p{\theta}(x{n+1} | x{1:n})$ denote the probability distribution over the next token $x{n+1}$ given the preceding context $x{1:n}$, as predicted by an LLM with parameters $\theta$. This distribution encodes the model‘s "knowledge" of what word(s) make sense to come next based on patterns it learned during training.
By feeding a large corpus of text $\mathcal{X}$ through the source LLM and recording its next-token predictions for each subsequence, we can sample many of these distributions $p{\theta}(x{n+1} | x_{1:n})$. Together, this set of predictions represents a rich characterization of the model‘s language knowledge that we can then manipulate and combine.
A key implementation detail is that these probability distributions are generated over the LLM‘s token vocabulary (e.g. byte-pair encodings or WordPieces) rather than raw text. This is necessary because different LLMs use different tokenization schemes, so we need a standardized representation in order to align their knowledge downstream.
Token-level Alignment via Minimum Edit Distance
The next major challenge is that the token vocabularies $V$ of the source LLMs may be very different, both in size and composition. In order to meaningfully fuse knowledge at the token level, there must be a way to map or align tokens between the LLMs.
FUSELLM tackles this using a minimum edit distance (MinED) approach. Given the token-level probability distributions $p_{\theta1}$ and $p{\theta_2}$ from two source LLMs for the same input sequence, the goal is to find a bipartite matching between their tokens that minimizes the total edit distance.
Formally, let $d(v_1, v_2)$ be the edit distance (e.g. Levenshtein distance) between token strings $v_1$ and $v_2$. We want to find a matching $M \subset V_1 \times V_2$ that minimizes:
$$\sum_{(v_1, v_2) \in M} d(v_1, v2) \cdot p{\theta_1}(v1) \cdot p{\theta_2}(v_2)$$
Intuitively, this objective will prefer to match tokens that are likely under both source models and have minimal edit distance, effectively finding the cheapest set of edits to align the token sequences.
Computing the optimal matching $M$ is an instance of the minimum-cost bipartite flow problem, which can be solved efficiently using the Hungarian algorithm [6]. To scale up to larger vocabularies, FUSELLM employs greedy approximations like iteratively matching the tokens with minimum edit distance until no more valid matchings are possible.
The end result is a unified vocabulary $V^$ and a set of approximately aligned token probability distributions ${p_{\theta_1}^, p_{\theta_2}^*, …}$ that can be directly compared and combined downstream. Empirically, MinED significantly outperforms heuristic matching based on exact token overlap, improving both alignment success rate and downstream fused model performance.
Fusing Aligned Distributions via Weighted Averaging
Finally, given aligned probability distributions over a common vocabulary, how do we best fuse them into a single unified representation of the source LLMs‘ collective knowledge? FUSELLM explores a weighted averaging approach.
The simplest way to combine $M$ probability distributions $p_1, …, p_M$ over a vocabulary $V$ is to take the arithmetic mean:
$$p{\text{avg}}(v) = \frac{1}{M} \sum{i=1}^M p_i(v) \quad \forall v \in V$$
However, this naive unweighted average ignores the fact that some source LLMs may have more relevant or accurate knowledge than others for a given input. Ideally, we want to assign higher weight to distributions that are more "reliable" in the context of the target model we are trying to train.
FUSELLM proposes two variants that weight each source distribution $p_i$ by its empirical cross-entropy $H(p_i, p_t)$ with the target model‘s distribution $p_t$:
$$p{\text{MinCE}}(v) = \sum{i=1}^M w_i \cdot p_i(v) \quad \text{where} \quad w_i \propto \exp(-\beta \cdot \min_j H(p_i, p_t^j))$$
$$p{\text{AvgCE}}(v) = \sum{i=1}^M w_i \cdot p_i(v) \quad \text{where} \quad w_i \propto \exp(-\beta \cdot \text{avg}_j H(p_i, p_t^j))$$
Here $p_t^j$ denotes the target model‘s distribution after $j$ epochs of training, $\beta$ is a temperature hyperparameter, and the weights $w_i$ are normalized to sum to 1.
Intuitively, MinCE assigns higher weight to source distributions that have low cross-entropy with the target model at its best training checkpoint, while AvgCE assigns weights based on average cross-entropy throughout training.
Empirically, the weighted fusion schemes outperform uniform averaging on a variety of downstream tasks. FUSELLM experiments show that MinCE works particularly well when fusing a small number of high-quality sources, while AvgCE is more robust to larger number of sources with varying quality.
Key Experimental Results
To evaluate the power of its knowledge fusion approach, FUSELLM was tested on a challenging setting fusing three open-source LLMs: Llama-2, OpenLLaMA, and MPT. Each of these models has 7 billion parameters but was trained on different corpora with significantly different vocabularies. The fused model is a separate Llama-2 model trained to recover the combined knowledge of the sources.
Across a diverse set of reasoning, commonsense, and code generation benchmarks, FUSELLM demonstrates significant improvements over the base Llama-2 model and competitive performance with models up to 70x its size.
On the BigBench-hard benchmark testing challenging language reasoning capabilities, FUSELLM boosts the average accuracy of Llama-2 by 5.16% across 27 tasks (Table 1). Gains are especially large on tasks like physics (14% absolute improvement) and causal reasoning (10%), indicating successful transfer of knowledge from stronger source models.
FUSELLM also enhances performance on commonsense reasoning (Table 2), achieving state-of-the-art results on benchmarks like HellaSwag (+5%), Winogrande (+5%), and PIQA (+4%). Remarkably, on many examples, the fused model correctly answers questions that all of the individual source LLMs get wrong, demonstrating emergent knowledge that is more than just the "majority vote".
Even for code generation, a challenging domain for general-purpose LLMs, FUSELLM delivers improvements over the base Llama-2 on 9 out of 10 programming languages in the MultiPL-E benchmark (Table 3). Some of the largest gains are on languages like Python (+8% pass@1) that the Llama models were not explicitly trained on, but that other source LLMs like MPT have decent proficiency in.
Interestingly, FUSELLM is able to realize these capabilities very efficiently. On the BigBench-hard benchmark, finetuning the fused model reaches 98% of the full Llama-2 performance with only 25% of its training FLOPS and 23% of its data (Figure 1). The aligned probability distributions serve as a strong "prior" that accelerates learning of the target task.
Comparison to Other Knowledge Transfer Approaches
Knowledge fusion bears some resemblance to other approaches for transferring knowledge between neural language models, but has several key advantages:
-
Compared to knowledge distillation, which trains a student model to imitate the outputs of one or more teacher models, knowledge fusion leverages knowledge from multiple models synergistically, yielding a fused model whose performance can exceed any of the individual teachers.
-
Compared to model ensembles, which aggregate the predictions of multiple models only during inference, knowledge fusion produces a single unified model that is just as efficient and easy to deploy as any individual LLM. The knowledge aggregation happens in the training loop, not on-the-fly at test time.
-
Compared to directly averaging model parameters, knowledge fusion operates on discrete probability distributions over tokens, which are more portable across different model architectures. It also admits more flexible weighting schemes to modulate the contributions of different source models.
Current Limitations and Future Directions
While knowledge fusion is a highly promising direction, it is still an emerging technique with important limitations and open questions, such as:
-
Scaling and efficiency: The computational cost of generating, aligning, and fusing token distributions grows with the number and size of source LLMs. Handling larger models with vocabularies in the millions will require compression techniques and hardware optimizations [5].
-
Data quality and privacy: The success of knowledge fusion hinges on access to high-quality pre-trained LLMs. However, many state-of-the-art models are proprietary or trained on sensitive data. Techniques for privacy-preserving fusion and detecting low-quality sources need further study.
-
Evaluation and interpretability: Current benchmarks may not capture the full extent of knowledge transferred, especially implicit or emergent knowledge. More comprehensive and fine-grained evaluation protocols are needed, as well as tools for inspecting and attributing the knowledge in a fused model.
-
Safety and robustness: Like all LLMs, fused models can inherit or amplify biases, hallucinations, and safety issues present in their source data and models [7]. Methods for detecting and mitigating these risks specific to the fusion setting are important areas for future work.
Despite these challenges, knowledge fusion holds immense promise as a new paradigm for building knowledgeable and capable language models. As the technique matures and scales, it could significantly accelerate and democratize the creation of powerful LLMs by enabling researchers to build on and combine each other‘s models rather than starting from scratch.
In the long run, the ability to flexibly mix and match language knowledge across models may also enable more adaptable and open-ended language interfaces – for example, an AI writer that can selectively draw upon expert models of history, science, and literature to craft rich educational content. Getting there will require ongoing innovations at the intersection of NLP, knowledge representation, and machine learning systems. Knowledge fusion is an exciting step in that direction.
