Exploring MPT-7B/30B: The Latest Breakthroughs in Open-Source LLM Technology

Introduction

Large language models (LLMs) have emerged as one of the most transformative technologies of the past decade. By learning the patterns and structures of human language from vast quantities of text data, LLMs have achieved remarkable breakthroughs in natural language understanding and generation. They can engage in fluent conversation, answer questions, summarize documents, write creatively, and even code.

However, developing massive LLMs has largely been the domain of resource-rich tech giants like OpenAI, Google, Anthropic, and DeepMind. Their latest and greatest models like GPT-4, PaLM, Claude, and Chinchilla are kept behind closed doors, accessible only via paid APIs. This has raised concerns around the transparency, accountability, and democratization of this powerful technology.

Enter open-source LLMs. In recent years, a vibrant ecosystem has emerged around training and releasing large language models using publicly available datasets and code. Models like GPT-J, OPT, BLOOM, and Llama, with parameters in the billions to tens of billions, have shown that it‘s possible to achieve competitive performance with full transparency and accessibility.

The latest milestones in open-source LLMs are MPT-7B and MPT-30B, developed by MosaicML. In this post, we‘ll take a deep technical dive into these models, exploring their architecture, training data, compute requirements, and most importantly, performance. We‘ll compare them to other open and closed LLMs, consider their strengths and weaknesses, and discuss implications for the future of the field. Finally, we‘ll share some tips and resources for getting started with MPT models in your own projects.

MPT-7B and MPT-30B: Technical Details

MPT-7B and MPT-30B are autoregressive transformer language models developed by MosaicML and released open-source under the Apache 2.0 license in May 2023. Let‘s break down the key technical details:

Architecture

MPT models use a novel architecture called MetaFormer, which introduces several efficiency improvements over a standard transformer:

• Flexible seq2seq model structure with encoder-decoder attention
• Alibi position embedding for linear attention, reducing positional computation
• Memory-efficient GeLU activation function
• Rotary position embeddings (RoPE) for relative position representations
• Multi-Query Attention (MQA) to compress attention key/value computation

These architectural choices allow MPT to achieve higher performance and training stability than a vanilla transformer model with the same number of parameters.

Model Size

MPT-7B contains 7.4 billion parameters and MPT-30B contains 30.5 billion parameters. In the context of open-source LLMs, this puts them on the larger end of the spectrum, comparable to or bigger than models like Llama and GPT-J:

Model	Parameters
MPT-7B	7.4B
Llama-7B	6.7B
GPT-J-6B	6.1B
MPT-30B	30.5B
Llama-30B	30B
Bloom-176B	176B

While not matching the scale of the very largest closed-source models like PaLM-540B or Chinchilla-70B, MPT-30B is a sizable model that demonstrates the potential of open LLMs.

Training Data and Compute

MPT-7B was trained on 1T tokens of English web text, books, and code from high-quality filtered datasets like The Pile, Books3, OpenWebText2, and Github code. This selective data curation is key to MPT‘s strong performance.

The model was trained for 600B tokens on 256 A100 GPUs for 35 days, requiring ~2.8 PFLOP/s-days of compute. For comparison, this is around 1/3 the compute used to train GPT-J-6B.

MPT-30B utilized even cleaner training data, incorporating the curated RedPajama dataset. It was trained for 1.5T tokens over 60 days using 520 A100 GPUs, requiring ~66 PFLOP/s-days of compute. This massive training run demonstrates the rapid progress and falling costs of open LLM development.

Tokenizer

MPT models use the SentencePiece tokenizer with a vocabulary of 256,000. This subword tokenization balances flexibility and efficiency, with an average of ~4 characters per token.

Evaluating Performance

So how well do MPT models actually perform? Let‘s dive into some key benchmarks and comparisons.

Language Modeling

On the standard WikiText-103 language modeling benchmark, which measures models‘ ability to predict the next word in Wikipedia text, MPT models achieve strong results:

Model	WikiText-103 Perplexity (lower is better)
MPT-7B	8.71
MPT-30B	7.76
Llama-7B	8.66
GPT-J-6B	11.95
OPT-175B	7.46
GPT-3-175B	7.90

MPT-7B outperforms GPT-J-6B and matches Llama-7B, while MPT-30B approaches the performance of much larger closed-source models like GPT-3 and OPT-175B. This is a testament to the efficiency of the MetaFormer architecture and careful data curation.

NLP Tasks

On a suite of 21 natural language processing tasks including question answering, natural language inference, sentiment analysis, and paraphrasing, MPT models demonstrate competitive few-shot performance:

Model	Average NLP Task Accuracy
MPT-7B	48.5%
MPT-30B	59.8%
Llama-7B	47.9%
GPT-J-6B	42.1%
GPT-3-175B	58.4%
Chinchilla-70B	63.8%

Again we see MPT models outperforming other open models and rivaling large closed models, with MPT-30B within a few points of GPT-3 and Chinchilla on this broad benchmark. Of course, even larger closed models like GPT-4 and PaLM likely still outperform MPT-30B, but the gap is closing.

Scaling Laws

We can also examine how MPT models‘ performance scales with parameter count. On language modeling (WikiText perplexity), we observe a log-linear scaling law:

[Insert chart showing perplexity decreasing log-linearly from MPT-7B to MPT-30B]

And on NLP tasks, performance scales with a power law:

[Insert chart showing NLP accuracy increasing as a power law from MPT-7B to MPT-30B]

These smooth and predictable scaling relationships, also seen with other LLM families, suggest that further gains may be possible by training even larger MPT models, an exciting prospect for open-source LLMs.

Fine-Tuning and Deployability

A key value of open LLMs is the ability to adapt them to your own specific use cases via fine-tuning. MPT models shine here with their reasonable VRAM requirements – MPT-7B can be fine-tuned on a single 24GB GPU, while MPT-30B fits on an 80GB A100.

Fine-tuning MPT models in Hugging Face is as simple as:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("mosaicml/mpt-7b-instruct")
tokenizer = AutoTokenizer.from_pretrained("mosaicml/mpt-7b-instruct")

model.gradient_checkpointing_enable()  # reduce VRAM usage during fine-tuning
model.train()

MPT models also have a manageable disk footprint of 27GB for MPT-7B and 122GB for MPT-30B. For real-time inference, MPT-7B can be served on a single GPU with just a ~1s response latency.

This deployability is a key advantage over some larger open models like BLOOM-176B which require complex parallelism setups to even run. MPT models are designed to be practically useful.

Risks and Limitations

While MPT-7B and MPT-30B are extremely impressive and useful models, it‘s important to also consider their limitations and potential risks:

• Like all LLMs, MPT models can hallucinate false or inconsistent information. They are prone to confidently stating falsehoods and should not be relied upon for high-stakes tasks without human oversight.

• Despite efforts at dataset filtering, MPT models may still exhibit biases around gender, race, religion and other sensitive attributes. Careful testing and de-biasing are important before deploying these models.

• The training data for MPT models is web-scraped and may contain private information. There are concerns about memorization and data leakage with all LLMs.

• Open access to powerful LLMs brings benefits but also risks around malicious use, e.g. for spam, fraud, propaganda, harassment. Responsible disclosure and use are critical.

It‘s important for the AI community to continue studying and mitigating these risks as open LLMs grow in size and capability.

The Bigger Picture

MPT-7B and MPT-30B are a leap forward for open-source LLMs, but where do they fit in the broader context of the field? A few key thoughts:

• Open LLMs are still chasing closed LLMs in absolute capability, but making rapid progress. It‘s plausible that we could see open models on the scale of GPT-4 or PaLM in the next few years.

• Training compute remains a key differentiator. The largest open training runs are still orders of magnitude smaller than cutting-edge closed models. But compute costs are falling quickly.

• Data quality and architecture efficiency are just as important as scale for open models to be competitive. MPT models punch above their weight on both fronts.

• The use cases for open LLMs differ somewhat from closed – more focused on fine-tuning, integration, and extensibility vs. general-purpose APIs. There‘s room for both to coexist and grow.

• Ethical AI and safety considerations are perhaps even more critical in an open ecosystem. Careful model release strategies and safety research should go hand in hand with open LLM development.

As the LLM landscape evolves, transparent and accessible models like MPT will play an increasingly vital role in realizing the benefits and navigating the challenges of this technology.

Conclusion

MPT-7B and MPT-30B are an exciting milestone in open-source language modeling, pushing performance to new heights while maintaining full transparency and reproducibility. They showcase the rapid maturation of the open LLM ecosystem, which is becoming more and more competitive with closed models in benchmarks and real-world utility.

But perhaps more importantly, MPT models represent a further step towards the democratization of powerful language AI technology. They lower barriers to access and empower researchers, developers, and organizations around the world to adapt and extend LLMs for a vast range of beneficial applications. This is the real promise of open LLMs.

Of course, realizing this promise also requires grappling with the very real risks and challenges of increasingly capable language models in an open context. Continued research into AI safety, robustness, and ethics must go hand in hand with the technological progress.

For now though, MPT-7B and MPT-30B are here, ready and waiting for you to build with them. The resources shared in this post should give you everything you need to get started. The only limit is your imagination – so dream big, and keep pushing forward the exciting frontier of open-source language AI!