Weaviate: Pioneering the New Era of Vector Search Engines

Search technology has come a long way since the early days of the web. From the first primitive keyword search engines to today‘s sophisticated AI-powered relevance and recommendation engines, we‘ve seen search evolve into an indispensable tool for navigating the digital world. And now, a new paradigm shift in search is underway, driven by the rise of vector search engines like Weaviate.

Vector Search 101

To understand what makes vector search engines so revolutionary, we first need to understand the limitations of traditional keyword-based search. Let‘s say you search for "how to bake an apple pie" in a standard search engine. Under the hood, the search engine will parse your query into individual keywords (removing common "stop words" like "how" and "to"), and then look for exact matches of those keywords in its index of web pages. It will return a simple ranked list of pages that contain the keywords "bake", "apple", and "pie".

The problem is that keyword search has no real understanding of the meaning or intent behind the query. It can‘t disambiguate between "apple" the fruit and "Apple" the tech company. It doesn‘t know that "how to make a pie" would also be a highly relevant match, because it uses different words. It‘s constrained by the literal characters and completely misses the semantics.

This is where vector search comes in. Rather than using keywords, vector search engines use machine learning models to encode data into vectors – high-dimensional numeric representations that capture the semantic meaning. With vectors, text, images and other unstructured data can be mathematically analyzed to determine how semantically similar different items are to each other.

For example, the phrase "apple pie recipe" would be encoded into a vector that is very close in vector space to the vector for "how to bake an apple pie", since they express highly similar meanings. A vector search engine can then perform a nearest neighbor search in the vector space to quickly find the most semantically relevant matches to a query vector.

This has huge implications for the quality of search results. Vector search enables true semantic search, where the search engine actually understands the meaning of the query and can surface relevant results even if they use completely different keywords or phrasing. It‘s a much more intuitive and powerful way for humans to express their search intent and discover information.

Weaviate: The Leading Open-Source Vector Search Engine

One of the leading platforms bringing vector search into the mainstream is Weaviate. Weaviate is an open-source vector search engine that allows developers to build semantic search and AI-powered applications with ease. It combines state-of-the-art machine learning models, a cloud-native database architecture, and a GraphQL API into a single cohesive platform for storing and searching vector data at scale.

Since launching in 2019, Weaviate has seen explosive adoption, with thousands of developers and some of the world‘s largest enterprises using it to power next-gen search experiences and AI applications. The global vector database market is expected to grow from $196M in 2021 to $1.7B by 2028, a 44% CAGR according to Verified Market Research. As one of the leading independent platforms in this space, Weaviate is well positioned to ride this wave of growth.

Some of the world‘s most innovative companies have chosen Weaviate to power their vector search use cases, such as:

• Subjekt – This e-commerce product discovery engine uses Weaviate to provide visual semantic search across millions of fashion products. Shoppers can search for items that are visually similar to an uploaded photo and discover new products across brands and retailers.

• Spinque – Spinque leverages Weaviate to enable semantic search on multimedia collections for publishers and archives. It allows journalists to find relevant news footage by searching for specific people, objects, text, or locations within the video content.

• ASAPP – ASAPP‘s AI-native customer experience platform uses Weaviate to power its natural language knowledge base and intelligent search capabilities. Customer support agents can ask questions in plain English and get semantically relevant article suggestions and answers.

Inside Weaviate‘s Architecture

So what makes Weaviate uniquely capable as a vector search engine? Let‘s take a closer look at some of the key components that enable Weaviate‘s scale and performance:

Horizontal Scalability: Weaviate was designed from the ground up to scale horizontally across multiple nodes in a cluster. Data is automatically sharded and balanced across nodes, and the cluster can be dynamically resized to increase capacity and throughput. Each node includes a local vector index and object storage and coordinates with other nodes to fulfill queries. This allows Weaviate to handle massive datasets and high query loads.

Separate Vector and Scalar Indices: To enable efficient retrieval and filtering, Weaviate actually maintains two separate indices for each data class – an approximate k-nearest neighbors (ANN) vector index for finding semantically similar vectors, and a traditional inverted scalar index for filtering and exact matching on property values. At query time, Weaviate uses a pipeline that combines these indices, first finding the relevant vectors and then filtering the results by any specified scalar filters. This allows for the expressiveness of vector search while still supporting more structured queries.

HNSW Vector Index: Weaviate uses the Hierarchical Navigable Small World (HNSW) graph algorithm to build and search its vector indices. HNSW enables finding approximate nearest neighbors in logarithmic time, even in very high dimensional spaces. It builds a multi-layer graph with the full dataset as the lowest layer and each higher layer containing an exponentially smaller number of points. At search time, a greedy algorithm traverses down the graph to quickly find the closest neighbors to the query vector. Weaviate has optimized its HNSW implementation to be extremely fast and memory-efficient.

Data Persistence and Replication: In addition to the vector and scalar indices, Weaviate also persists the underlying data objects using LSM tree-based storage. Multiple copies of each data object are stored on different nodes for redundancy. Weaviate uses a multi-version concurrency control (MVCC) system with point-in-time recovery to ensure strong consistency and avoid data loss in the event of a node failure. All writes are recorded in a write-ahead log that can be replayed to recover an index.

Pluggable ML Models: One of Weaviate‘s most powerful features is the ability to plug in custom machine learning models to vectorize and search your data. Weaviate ships with a number of pre-trained models for common data types and use cases (e.g. BERT, ResNet, CLIP), but users can also train and deploy their own PyTorch or TensorFlow models with a simple containerized interface. This allows optimizing the vector representations and similarity metric for a particular domain or dataset.

Vector Search in Action

To illustrate the power of vector search, let‘s walk through a concrete example using Weaviate. Imagine we are building a news recommendation engine and want to allow semantic search over a large corpus of articles. We can start by defining a simple schema for our Article data class:

Article:
  class: Article
  description: A news article
  vectorizer: text2vec-transformers
  properties:
    title:
      dataType: string
      description: The title of the article
    content:
      dataType: text
      description: The body content of the article
    url:
      dataType: string
      description: The URL of the article

This tells Weaviate that we want to create a class called Article with a title, content, and url property. The text2vec-transformers vectorizer will be used to generate a vector representation of each article based on its content.

We can then import articles into Weaviate using the Python client:

import weaviate
client = weaviate.Client("http://localhost:8080")

article = {
    "title": "Weaviate Introduces New Hybrid Search Capabilities",
    "content": "Weaviate, the leading open-source vector search engine, today announced the release of version 1.8 which introduces new hybrid search capabilities. This allows users to combine traditional keyword matching with semantic vector search for more control over result relevance...",
    "url": "https://weaviate.io/blog/weaviate-1.8-release.html"
}

client.data_object.create(article, "Article")

After importing our articles, we can then perform natural language queries and get semantically relevant results back:

query_result = (
  client.query
  .get(class_name="Article", properties=["title", "url"])  
  .with_near_text({
    "concepts": ["Weaviate releases new version with hybrid search"],
    "certainty": 0.7
  })
  .with_limit(3)
  .do()
)

This query will find the top 3 articles that are most semantically similar to the natural language description "Weaviate releases new version with hybrid search", even if they don‘t contain those exact keywords. The certainty parameter controls the threshold for semantic similarity, with 0.7 requiring moderately related articles.

The query returns the following results:

{
  "data": {
    "Get": {
      "Article": [
        {
          "title": "Weaviate Introduces New Hybrid Search Capabilities",
          "url": "https://weaviate.io/blog/weaviate-1.8-release.html"
        },
        {
          "title": "Weaviate 1.8 Release Notes",
          "url": "https://weaviate.io/developers/weaviate/current/more-resources/release-notes/v1.8.x.html"
        },
        {
          "title": "Weaviate Adds Support for OpenAI‘s CLIP Model",
          "url": "https://weaviate.io/blog/weaviate-adds-openai-clip-support.html"
        }
      ]
    }
  }
}

Note how the top result is exactly what we were looking for, and the other results are also highly relevant to the query, even though they use different phrasing. This is the power of vector search – it can find semantically related information even if it‘s expressed in different ways.

The Future of Vector Search

As we‘ve seen, the shift from keyword-based search to semantic vector search is a major leap forward in information retrieval. It enables search engines and other information access systems to move beyond brittle word matching to actually understand meaning and context. This is critical in a world where unstructured data like text and images are growing exponentially and extracting insights is more challenging than ever.

Vector search is still in its early stages, but it‘s evolving extremely rapidly. Models for generating meaningful vector representations are getting more sophisticated all the time. New techniques are emerging for compressing vectors, building efficient indices, and searching across modalities. And an explosion of startups and open-source projects are translating the latest academic research into practical tools and platforms.

As the underlying technology matures, we can expect to see vector search become the default approach for enterprise search, e-commerce, customer support, and other search and discovery applications. The ability to search semantically over unstructured data will be transformative for knowledge management, decision support, and AI development.

At the same time, vector search introduces new challenges around scalability, relevance tuning, incremental indexing, and efficient filtering. These are active areas of research and development. We will likely see rapid progress on these fronts in the coming years as more resources pour into the space.

Weaviate is well positioned to lead this vector search revolution, with a unique combination of scalability, developer-friendliness, and open-source flexibility. As an open-source project with a vibrant community, Weaviate benefits from a broad range of use cases and contributions. And with a pluggable architecture and state-of-the-art ML integrations, it‘s consistently at the cutting edge of new capabilities.

While it‘s impossible to predict the future, one thing seems clear – vector search will be one of the defining technologies of the decade to come. It has the potential to unlock a massive amount of value from the world‘s information. Those who understand and leverage vector search will have a major advantage in building next-generation intelligent applications. Weaviate will be a key enabler and catalyst in this AI-powered future.