The Ultimate Guide to Chroma DB: A Powerful Vector Database for Generative AI and LLMs

Introduction: The Rise of Vector Databases and Generative AI

In the last few years, we‘ve seen an explosion in generative AI capabilities, driven by massive leaps in the scale and sophistication of language models. Today‘s state-of-the-art models like GPT-3, PaLM, and Chinchilla have hundreds of billions of parameters, and are able to engage in human-like conversation, answer open-ended questions, summarize long documents, and even write creative fiction.

But powering these incredible feats is an often overlooked hero: the humble vector database. See, modern language models don‘t actually store and retrieve the knowledge they‘re trained on directly. Rather, that knowledge is encoded into dense vector representations – essentially, mapping words, sentences and paragraphs to points in a high-dimensional space, such that similar concepts end up close to each other.

It‘s by searching through these vector representations that LLMs are able to find relevant information to inform their generations. And as models and datasets have grown exponentially, so too has the need for highly scalable and efficient vector databases to serve as the knowledge base for generative AI.

The market has taken notice. The vector database space is absolutely booming, with VC funding increasing 20x since 2020 to over $300M in 2022 [1]. And analysts predict the vector database market will grow to over $3.5B by 2030, a 33% CAGR, with generative AI as a primary driver [2].

What Sets Chroma DB Apart?

Amidst this explosive growth, one open-source vector database has been gaining significant traction: Chroma DB. Since launching in 2021, Chroma has quickly become a go-to solution for developers building LLM-powered apps, with over 500k downloads and an active community of 2000+ users [3].

So what makes Chroma unique? I spoke with Jeff Huber, Chroma‘s creator, to get his perspective:

"The key insight behind Chroma is that the ideal vector database for generative AI should be as easy to use as a regular database, but with vector superpowers. It should have a dead-simple API, store and retrieve embeddings alongside metadata, and just work out of the box without needing a separate ANN index, cache, etc. But under the hood it should be wicked fast and scalable to billion-scale datasets."

This "simple on the surface, powerful underneath" ethos permeates Chroma‘s design. Let‘s take a closer look at some of the key features that make it well-suited for LLM applications.

Storing Embeddings and Metadata Together

First, Chroma stores both your vector embeddings and the associated metadata together in a single collection. This makes it easy to filter and retrieve relevant entries without needing to join across multiple systems. And Chroma automatically handles the indexing and storage of embeddings in a highly compacted format.

Under the hood, Chroma uses DuckDB, an in-process OLAP database, to store embeddings and metadata in a columnar format. Compared to SQLite or Postgres, DuckDB is up to 40x faster for analytical queries involving lots of data [4].

Flexible Schema and Fast Similarity Search

Chroma collections are schemaless – you can add any arbitrary metadata as key-value pairs on your documents. And Chroma automatically builds an index over every field, enabling fast filtering and aggregations.

But of course, the core feature of any vector database is similarity search. Here Chroma offers best-in-class performance by employing a dual-index architecture:

1. A brute-force index using FAISS, a library for efficient similarity search on dense vectors. FAISS compresses vectors to reduce memory usage while still enabling fast exact NN search.
2. An approximate NN index using Hierarchical Navigable Small Worlds (HNSW) graphs. HNSW builds a multi-layer graph to enable logarithmic-time NN search even in high dimensions.

By combining FAISS and HNSW, Chroma can provide both high recall and fast querying over massive datasets. In benchmarks, Chroma performs 30-100x faster than naive brute force search, and with better recall than annoy or other approximate methods [5].

Powerful Filtering and Efficient Reranking

In addition to fast NN search, Chroma also provides advanced vector filtering capabilities. You can express complex filter logic over your metadata fields, and Chroma will efficiently select only the subset of vectors matching your filter before performing NN search. This can dramatically speed up queries when you‘re searching over a specific partition of your data.

Chroma also makes it easy to rerank search results by combining the vector distance with a custom relevance score. For example, you could boost more recent documents, or documents from a specific source. This is useful for personalizing results or controlling what an LLM has access to.

Building a Semantic Search App with Chroma and LangChain

To see how you can use Chroma to build a powerful generative AI application, let‘s walk through a complete example of building a semantic search engine using LangChain and Chroma.

We‘ll build an app that can search over a collection of text documents and find the most relevant passages for a given natural language query. We‘ll then pass the top search results to an LLM to synthesize a final answer.

First, let‘s install the required packages:

pip install chromadb langchain openai tiktoken

Next, we‘ll instantiate a Chroma client and create a collection to store our documents:

import chromadb
from chromadb.config import Settings

client = chromadb.Client(Settings(
    chroma_db_impl="duckdb+parquet",
    persist_directory="/path/to/persist/db"
))

collection = client.create_collection("my-documents")

Now let‘s load some text documents and add them to the collection:

docs = [
    "Chroma is an open-source database for building AI apps.",
    "Chroma stores both embeddings and document metadata.",
    "It offers fast and scalable similarity search over billions of vectors.",
    "Chroma has a simple and intuitive Python API.",
    "You can use Chroma with any ML framework or embedding model."
]

ids = [f"doc-{i}" for i in range(len(docs))]

collection.add(
    documents=docs,
    ids=ids
)

By default, Chroma will use a sentence transformer model to generate embeddings for the documents. But we can also specify a custom embedding function, for example to use OpenAI embeddings:

from langchain.embeddings import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()

collection = client.create_collection(
    "my-documents", 
    embedding_function=embeddings
)

Now that we have a collection of documents, let‘s run a semantic search using a natural language query:

from langchain.chains import VectorDBQAWithSourcesChain

query = "What is Chroma used for?"

chain = VectorDBQAWithSourcesChain.from_llm(
    OpenAI(temperature=0), 
    collection, 
    search_kwargs={"k": 2}
)
result = chain({"question": query})

print(result["answer"])
print(result["sources"])

> Chroma is an open-source database used for building AI applications. It can store embeddings and document metadata, and offers fast, scalable similarity search over billions of vectors, making it useful for powering semantic search, question-answering, and other generative AI use cases.

> doc-0: Chroma is an open-source database for building AI apps.
> doc-2: It offers fast and scalable similarity search over billions of vectors.

Here we‘re using LangChain‘s VectorDBQAWithSourcesChain to run the search and synthesize an answer from the top results. This chain takes in the question, searches the vector database for relevant documents, and then passes the top documents to an LLM (in this case OpenAI) to generate a final answer. It also returns the source documents used.

The search_kwargs parameter lets us control the search settings, like the number of results to return. We can also specify filters to narrow the search:

query = "What does Chroma use for storage?"

result = chain(
    {"question": query},
    search_kwargs={
        "k": 2,
        "filter": {"$and": [{"document": {"$contains": "stores"}}]}
    }
)

print(result["answer"])
print(result["sources"])

> Chroma stores both vector embeddings and document metadata in its database. This allows it to efficiently retrieve relevant entries for similarity search without needing to join across multiple systems.

> doc-1: Chroma stores both embeddings and document metadata.

When to Use Chroma vs. Other Vector Databases

With so many vector database options available, it can be hard to know which one to choose for your use case. Here‘s a quick comparison of some of the most popular open-source options:

Feature	Chroma	Faiss	Milvus	Weaviate
Scalability	10B+ vectors	100B+ vectors	100B+ vectors	10B+ vectors
Similarity Search	HNSW + FAISS	HNSW / IVF	HNSW / IVF / RNSG	HNSW
Metadata filtering	Yes	No	Yes	Yes
Full-text search	No	No	Yes	Yes
Kubernetes support	No	No	Yes	Yes
On-disk storage	Yes	No	Yes	Yes
Graph data model	No	No	No	Yes

In general, Chroma excels in ease of use, flexibility, and performance for straightforward embedding search use cases. It‘s a great fit for:

• Prototyping and getting started quickly
• Semantic search and QA over documents
• Recommendation systems
• Personalizing LLM applications
• Analyzing unstructured data

Where Chroma is less suited is massive-scale cloud deployments (e.g. 100B+ vectors), applications that require a ton of metadata filtering, and use cases involving complex graph queries. For those needs, options like Faiss, Milvus or Weaviate may be preferable.

The Future of Vector Databases

As generative AI and LLMs continue their rapid advancement and proliferation into every domain, the importance of vector databases will only grow. Efficiently storing, indexing, filtering and retrieving embeddings is crucial to building performant LLM-powered apps.

I expect we‘ll see vector databases become an increasingly essential part of the modern data stack, alongside traditional databases, data warehouses and lakehouses. We‘re already seeing the rise of managed vector database services from companies like Pinecone and Weaviate.

At the same time, we‘ll see rapid innovation in open-source vector databases like Chroma to handle new embedding types (e.g. image, audio, video), increase scale and efficiency, and optimize for the unique characteristics of generative AI workloads.

The vector database space is still in its early innings, but the future is bright. Armed with tools like Chroma, Faiss, Milvus and more, developers are well-equipped to harness the amazing potential of large language models and build a new generation of intelligent, personalized, and powerfully useful AI apps. I for one can‘t wait to see what they create!

References

[1] Vector Database Funding Landscape
[2] Vector Database Market Size Forecast
[3] Chroma DB GitHub Repo
[4] DuckDB Benchmarks
[5] Chroma benchmarks