Caching Generative LLMs to Save on API Costs

Introduction to Generative LLMs and API Costs

In recent years, large language models (LLMs) have revolutionized the field of natural language processing. Generative LLMs like GPT-3, PaLM, Chinchilla and others are capable of producing human-like text based on a given prompt or context. This has opened up exciting new possibilities for applications like chatbots, content generation, code completion and more.

However, working with these powerful LLMs typically requires sending API requests to a remote service, which can quickly become expensive at high volumes. Every request sent is billed based on the number of input tokens, output tokens, and in some cases compute time. While costs per request are typically quite small (fractions of a cent), they can add up very quickly for popular applications.

Fortunately, a technique called caching can drastically reduce these API costs by storing and reusing results for repeated requests. In this post, we‘ll take an in-depth look at how caching works for LLMs and explore some best practices and tools for implementing it in your own applications.

How Caching Reduces Repeated LLM Requests

The core idea behind caching is to store the results of expensive operations so that they can be quickly retrieved if the same operation is requested again in the future. This is extremely relevant for generative LLMs, where the same prompts and queries are often repeated many times.

For example, let‘s say you are building a customer support chatbot powered by an LLM. There are likely to be many common questions that come up over and over again:

"How do I reset my password?"
"What are your hours of operation?"
"How long does shipping usually take?"

With caching in place, the first time one of these questions is asked, the chatbot sends an API request to the LLM to generate a response. But before returning the response, it also stores a mapping between the exact prompt text and the generated result in a cache. Then, if the same question is ever asked again in the future, instead of sending another API request, the chatbot can simply return the cached result from before.

This can lead to huge savings in both cost and latency. Instead of paying for an API request on every interaction, you only pay for unique requests. And responses can be returned nearly instantly from the cache instead of waiting for a round-trip to the LLM service.

Approaches to Caching LLM Responses

There are a few different ways to implement caching for LLM applications. The right approach depends on your specific needs and constraints.

In-Memory Caching

The simplest option is to use an in-memory cache that lives in the same process as your application. This typically means using a dictionary-like data structure to store a mapping between prompts and results. Many languages have built-in primitives for this, like dictionaries in Python or objects in JavaScript.

Here‘s a stripped-down example of what this might look like in Python:

cache = {}

def get_llm_response(prompt):
  if prompt in cache:
    return cache[prompt]
  else:
    response = send_llm_api_request(prompt)
    cache[prompt] = response
    return response

The biggest advantage of in-memory caching is simplicity. There‘s no external dependency or infrastructure required. As long as your application process stays running, the cache will persist and provide fast lookups.

However, in-memory caches are limited by the memory of a single process. If your application restarts or scales across multiple machines, each process will have its own isolated cache. For large-scale applications, this can lead to duplicated work and inefficient usage of caches.

Database Caching

For a more persistent and scalable caching solution, you can use an external database to store cached LLM responses. This allows the cache to be shared across multiple application processes or even entirely separate services.

A lightweight and file-based database like SQLite is a good choice for simple use cases. You can define a basic schema like:

CREATE TABLE llm_cache (
  prompt TEXT PRIMARY KEY,
  response TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Then, your application code can check the cache by querying the database:

def get_llm_response(prompt):
  cursor = db.cursor()
  cursor.execute(‘SELECT response FROM llm_cache WHERE prompt = ?‘, (prompt,))
  result = cursor.fetchone() 

  if result is not None:
    return result[0]
  else:
    response = send_llm_api_request(prompt)
    cursor.execute(‘INSERT INTO llm_cache (prompt, response) VALUES (?, ?)‘, (prompt, response))
    db.commit()
    return response

With this approach, all of your application instances can share a single cache database. Scaling up becomes much easier. The main downside is the extra complexity of deploying and administering a separate database, as well as the fact that cache lookups are slower than in-memory.

For high-throughput use cases, an in-memory database like Redis or Memcached is often a good middle ground, balancing speed and scalability.

Caching Libraries and Tools

While it‘s not too difficult to implement basic caching yourself, there are also many libraries and tools available that provide more advanced functionality out of the box. The right choice depends on your programming language and environment.

For Python, some popular options include:

• cachetools – Extensible in-memory caching library
• python-cache – Dictionary-like API for multiple backends (in-memory, database, Redis, etc.)
• django-quickcache – Easy caching for Django applications

In the JavaScript ecosystem:

• node-cache – In-memory caching for Node.js
• memory-cache – Simple in-memory cache for Node.js
• lru-cache – In-memory LRU cache for Node.js

And for the JVM:

• Caffeine – High-performance caching library for Java
• Guava Cache – In-memory caching for Java built on Guava collections
• Ehcache – Popular open-source caching library for Java

Using a library provides a good balance of control and abstraction. You get a well-tested and optimized implementation without having to worry about the low-level details yourself. Many libraries also provide helpful features like automatic cache invalidation, size limits, and statistics.

Best Practices for LLM Caching

Regardless of what specific approach or tooling you choose, there are a few caching best practices to keep in mind:

Construct Prompts Deterministically

For caching to be effective, it‘s important that the same prompt always maps to the same cached result. This means being careful about how you construct prompts and making sure they are deterministic.

Include all relevant context in the prompt itself instead of relying on application
state that may change over time. And avoid introducing non-determinism through things like random seeds or
timestamps.

Consider an Expiration Policy

While some prompts and responses may be evergreen, others are more time-sensitive. A response pointing to "this quarter‘s sales numbers" will go out of date as soon as the new quarter rolls over. For these types of prompts, it‘s a good idea to implement a caching expiration policy.

You can either set a global time-to-live (TTL) duration for cached entries, or configure TTLs individually based on the category or domain. When a cached result is requested beyond its expiration time, your application can invalidate the old entry and refresh it with a new API request.

Monitor Cache Hit Rates

It‘s important to keep an eye on your cache hit rates to understand how well your caching strategy is working. Most caching libraries provide a way to inspect statistics like hits, misses, and evictions. Plotting these metrics on a monitoring dashboard can help you spot opportunities for improvement.

As a rule of thumb, you should see a fairly high cache hit rate if your prompts are constructed deterministically and have good cardinality. A low cache hit rate may indicate that you need to restructure prompts or revisit your caching logic.

Use Cache Eviction When Necessary

If your cache is growing unbounded, it may start to consume too many resources and impact application performance. To address this, you can implement a cache eviction strategy that removes old or less frequently used entries when the cache reaches a certain size.

The most common eviction strategy is least recently used (LRU), which discards the entries that were accessed least recently. This is based on the assumption that items accessed frequently in the past are likely to be accessed again in the future. Most caching libraries provide LRU implementations out of the box.

Quantifying the Benefits of Caching

So just how much can you expect to save by implementing LLM caching? The exact amount will depend on the nature of your application and the specific LLM provider you‘re using, but it‘s often substantial.

Let‘s work through an example to illustrate. Imagine you are running a chatbot powered by OpenAI‘s text-davinci-002 model. Each request costs around $0.02 per 1,000 tokens. If your application processes 100,000 requests per day with an average of 500 tokens per request, that works out to:

100,000 requests/day × 500 tokens/request × $0.02 / 1,000 tokens = $1,000 per day

Now let‘s say you implement caching and observe an 80% cache hit rate. That means only 20,000 requests per day are hitting the OpenAI API, with the other 80,000 being served from the cache. Your new cost calculation looks like:

20,000 requests/day × 500 tokens/request × $0.02 / 1,000 tokens = $200 per day

In this scenario, caching is reducing your API costs by 80%, or $800 per day. That‘s nearly $300,000 in savings per year! Of course, this is just a hypothetical example and your actual savings will vary. But it illustrates the powerful economic impact that caching can deliver for LLM-based applications at scale.

In addition to the direct cost savings, caching also provides significant latency and performance benefits. Responses served from a local cache are typically orders of magnitude faster than waiting for a remote API, which can dramatically improve the user experience.

Conclusion

Caching is a powerful technique for optimizing the cost and performance of generative LLM applications. By storing and reusing responses for repeated prompts, caching can dramatically reduce the number of API requests you need to send, saving you significant money and improving response times.

There are a few different approaches to caching, from simple in-memory dictionaries to distributed databases. The right choice depends on your scale, performance needs, and technical environment. Regardless of the specific tools you choose, constructing prompts deterministically, monitoring cache hit rates, and implementing an eviction strategy are all caching best practices.

While the idea of caching may seem straightforward, it can have an outsized impact on the economics of LLM applications. Reducing API requests by even a moderate percentage often translates to eye-popping savings at scale. Implementing caching is one of the highest-leverage optimizations you can make for an LLM-powered application.

If you‘re not already using caching, I highly recommend exploring the tools and techniques covered in this post. Your wallet (and your users) will thank you!