Microsoft‘s SliceGPT Breakthrough Enables Highly Efficient Compressed Language Models

The rapid advancement of large language models (LLMs) in recent years has unlocked incredible new AI capabilities, from engaging in open-ended conversation to assisting with complex reasoning and creative tasks. However, the immense size of these models, often containing hundreds of billions of parameters, presents significant challenges in terms of the computational resources required to train and run them.

Tackling this crucial issue head-on, Microsoft has introduced SliceGPT, an innovative new technique for compressing LLMs to dramatically reduce their memory footprint and computational requirements while maintaining impressive performance. By enabling highly efficient compressed models that are faster and cheaper to run, SliceGPT opens up exciting new possibilities for the future of AI.

Understanding Large Language Models and the Need for Compression

Before diving into the details of SliceGPT, it‘s important to understand what large language models are and why compressing them is so critical. LLMs are advanced AI models trained on massive amounts of text data to develop a deep understanding of language. By learning the intricacies of vocabulary, grammar, semantics, and even world knowledge, LLMs can engage in complex linguistic tasks like question-answering, summarization, translation, open-ended dialogue, and even coding and reasoning.

The power of LLMs comes from their immense scale – leading models like OpenAI‘s GPT-3 (175B parameters), Google‘s PaLM (540B parameters), and DeepMind‘s Chinchilla (70B parameters) are among the largest neural networks ever created. This allows them to capture and utilize the vast complexity of language at an unprecedented level. However, it also means that these models require enormous amounts of memory and compute to run. A single inference on GPT-3 can cost over $100 and require special hardware. [^1] This severely limits the accessibility and applicability of these powerful tools.

Compression techniques aim to reduce the size and resource requirements of LLMs without sacrificing their capabilities, making them more practical and opening up new use cases. SliceGPT is a major breakthrough in this critical endeavor.

How SliceGPT Achieves State-of-the-Art Compression

So how exactly does SliceGPT work its magic? The core idea is actually quite elegant and is based on a key insight about the computational invariance of transformers, the fundamental architecture used in LLMs.

In a nutshell, SliceGPT works by identifying and removing redundant parameters and computations in a pre-trained LLM. It does this via a process called sparsification, which strategically slices out vertical segments of the model‘s weight matrices that contain the learned knowledge. These slices are selected in a way that preserves the essential structure and functionality of the network.

Technically speaking, SliceGPT substitutes the original large, sparse weight matrices with much smaller, more densely-packed versions. It then adjusts the remaining computations accordingly to maintain the overall function. The original paper describes this as follows:

"SliceGPT vertically slices the pre-trained weight matrices and substitutes them with smaller, dense weight matrices. The optimal substitution is found via low-rank matrix approximation, which minimizes the approximation error between the original and new weight matrices." [^2]

This sparsification process is guided by a mathematical framework that leverages the computational invariance property of transformers. In essence, there are many equivalent ways to structure the computations in these models that produce the same input-output mappings. SliceGPT finds an optimally sparse reparameterization that preserves function while minimizing parameters.

The result is a compressed model that has far fewer parameters and requires significantly less memory and compute per inference, while maintaining a very high percentage of the full model‘s performance on key tasks. Let‘s take a look at some concrete results.

Impressive Compression Results Across Leading LLMs

To validate the effectiveness of SliceGPT, Microsoft researchers rigorously tested it on some of the most prominent and powerful LLMs available, including LLAMA2-70B, OPT-66B, and Phi-2. The results, published in a recent paper, are extremely impressive. [^2]

On LLAMA2-70B, SliceGPT was able to remove 25.5% of the model‘s parameters, over 19 billion in total, while retaining 99.1% of its original zero-shot performance on key benchmarks like the MMLU collection of academic tests. In other words, the compressed model is nearly one-third smaller but performs just as well as the original in most cases.

Similar compression ratios and performance were achieved on OPT-66B. SliceGPT removed 25.3% of its parameters while maintaining 99.3% zero-shot accuracy.

For the even larger Phi-2 model, SliceGPT managed to slice away 30% of the parameters with 90.8% performance retention. That means the compressed version requires less than three-quarters the memory and compute of the original 3B parameter model with only a minor drop in capability.

The following table summarizes the key compression results:

Model	Params Removed	Zero-Shot Acc	Acc Retention
LLAMA2-70B	25.5% (19B)	52.3%	99.1%
OPT-66B	25.3% (16B)	38.5%	99.3%
Phi-2	30.0% (950M)	54.4%	90.8%

These results demonstrate the versatility and effectiveness of SliceGPT across varied LLM architectures and scales. It is able to significantly compress models in the range of tens to hundreds of billions of parameters without appreciably degrading their capabilities. This has major implications for the practicality and accessibility of these powerful tools.

Enabling Faster, Cheaper Inference on Fewer GPUs

Beyond reducing model size, one of the most impactful benefits of SliceGPT compression is the dramatic speedup and cost reduction it enables for inference. The compressed models can be run much faster on fewer and cheaper GPUs while still producing high-quality outputs.

For example, the researchers found that running inference on the SliceGPT-compressed LLAMA2-70B model required only 64% of the computational resources (FLOPs) of the original model when using standard 24GB consumer GPUs. [^2] That equates to over a 50% speedup in raw inference latency per GPU. Furthermore, thanks to the compression, the model could be deployed on fewer GPUs overall for even greater system-level efficiency gains.

Even on high-end 40GB A100 GPUs, the compressed model achieved a 34% inference speedup, reducing runtime from 240 to 160 seconds per input. [^2] Across the board, SliceGPT compression substantially reduces costs, accelerates speeds, and lowers hardware requirements for LLM inference. This makes powerful language models much more accessible and practical for a wider range of applications.

Importantly, achieving these inference efficiency gains with SliceGPT does not require any additional specialized software or hardware optimizations. The compressed models can be run on existing deep learning frameworks and infrastructure without modification. This makes it very easy to adopt and benefit from SliceGPT compression.

The Core Concept of Computational Invariance

At the heart of SliceGPT is the key insight of computational invariance in transformer neural networks, which are the basis for all modern LLMs. In simple terms, this property means that there are multiple mathematically equivalent ways to structure the computations in these models that produce the same overall input-output function.

SliceGPT takes advantage of this invariance by finding more efficient, compact reparameterizations of a model that maintain its core functionality. The original paper provides a theoretical formalization:

"For a transformer layer L, let W denote the original pre-trained weight and W‘ denote a sliced version with k vertical slices. If span(W‘) ⊆ span(W), then for any layer input x, L(x; W) = L(x; W‘). This computational invariance holds for any number of slices and enables reparameterization to a smaller model." [^2]

Intuitively, this is saying that as long as the compressed weight matrices represent a subspace of the original pre-trained weights, the function of the layer is fully preserved. The number and size of slices can be chosen to meet a desired parameter budget while still maintaining span inclusion.

By exploiting computational invariance, SliceGPT is able to significantly reduce the dimensionality of a model‘s key matrices without changing the essential relationships and learned knowledge they encode. This enables the high degree of compression achieved while preserving model capabilities.

The identification of computational invariance as it applies to transformers and LLMs is a major theoretical contribution arising from Microsoft‘s work on SliceGPT. It provides the mathematical foundation for the compression technique and suggests new avenues for further research and optimization. We can expect to see this concept elaborated on and leveraged in future work on model efficiency.

Fitting SliceGPT Into the Broader AI Efficiency Landscape

SliceGPT is a major step forward in the critical quest to make large language models more efficient and accessible. However, it is important to situate this development within the broader context of ongoing research into AI efficiency and acceleration.

Model compression has been an active area of study for years, with techniques like pruning, quantization, and knowledge distillation being applied to reduce the size of neural networks.[^3] The key innovation of SliceGPT is its specific application to LLMs and the transformer architecture, as well as the use of computational invariance to guide sparsification in a principled way.

Other notable work in this vein includes DeepMind‘s DeLight pruning method, which achieved up to 50% compression on models like BERT and T5 with minimal accuracy loss.[^4] Google‘s GLaM technique for efficiently scaling language models to trillions of parameters is another key development, demonstrating an alternate path to practical large-scale LLMs.[^5]

More broadly, efforts to make AI systems more efficient span hardware, software, and algorithmic innovations. New chips designed for AI workloads, such as Cerebras‘ Wafer Scale Engine and SambaNova‘s DataScale, offer order-of-magnitude improvements in performance per watt.[^6] Software frameworks like Microsoft‘s DeepSpeed and Hugging Face‘s Optimum are making it easier to train and deploy optimized models.[^7]

SliceGPT represents a key algorithmic advance in this overall drive towards leaner, faster, and more efficient AI. By tackling the specific challenges of LLM compression in a novel and effective way, it opens up new possibilities and complements parallel efforts across the stack.

Of course, efficiency is not the only important consideration in the development of large language models. Equally crucial are issues of robustness, interpretability, safety, and ethical alignment.[^8] As LLMs become more ubiquitous and powerful, it is critical that we continue to study and address these broader impacts alongside performance improvements.

Towards this end, Microsoft has emphasized its commitment to responsible AI development and has open-sourced the SliceGPT code on GitHub to support wider research and collaboration.[^9] By making this work accessible to the community, Microsoft aims to facilitate progress on both model efficiency and the sociotechnical challenges surrounding LLMs.

The Bottom Line on SliceGPT

There is no question that SliceGPT is a major milestone in the development of more practical and accessible large language models. By enabling state-of-the-art compression with minimal performance degradation, it directly addresses one of the biggest barriers to widespread LLM adoption and deployment.

The impressive results Microsoft has demonstrated with SliceGPT on top models like LLAMA2-70B, OPT-66B, and Phi-2 speak to the effectiveness and versatility of the technique. Achieving 25-30% parameter reduction while retaining 90-99% of model performance is a remarkable feat that will have far-reaching impacts on the field.

The benefits of SliceGPT are clear: compressed models that are much faster and cheaper to serve, requiring fewer computational resources like GPUs and memory. This drastically expands the range of use cases for large language models, bringing powerful AI capabilities within reach for more applications and organizations than ever before.

At the same time, the theoretical underpinning of computational invariance that enables SliceGPT opens up exciting new avenues for further research and optimization. We can expect to see this concept generalized and built upon in future work, potentially leading to even greater leaps in model efficiency.

That said, it is important to recognize that SliceGPT is not a silver bullet for all the challenges surrounding LLMs. While it represents significant progress on the efficiency front, issues of model robustness, safety, and ethics remain crucial open problems that require ongoing research and social deliberation. Responsible AI development demands that we consider the full range of impacts and stakeholders as language models become more ubiquitous.

Efficiency breakthroughs like SliceGPT are a key part of the solution, making it more practical to train and deploy LLMs at scale. But they must be pursued in tandem with research into model transparency, controllability, and value alignment.[^8] Only by advancing the science and practice of AI on both technical and sociotechnical fronts can we realize the full transformative potential of large language models.

With the release of SliceGPT, Microsoft has once again demonstrated its AI leadership and commitment to pushing the boundaries of what is possible. As more organizations experiment with and extend this approach, we can look forward to a future in which powerful language technologies are increasingly accessible, useful, and—hopefully—beneficial to society as a whole. There is much work still to be done, but SliceGPT is a major step in the right direction.

[^1]: GPT-3 pricing
[^2]: SliceGPT paper
[^3]: A Survey of Model Compression and Acceleration for Deep Neural Networks
[^4]: DeLight: Efficient and accurate transformer pruning for inference
[^5]: GLaM: Efficient scaling of language models with mixture-of-experts
[^6]: A Cambrian Explosion in Deep Learning, Part 2: Meet The New Chips
[^7]: DeepSpeed, Optimum
[^8]: On the Opportunities and Risks of Foundation Models
[^9]: SliceGPT GitHub repo