Understanding the Basics and Backend of Amazon Elasticsearch Service

Elasticsearch has rapidly become the most popular open-source search and analytics engine. According to the AWS website, Amazon Elasticsearch Service has been adopted by thousands of customers including Netflix, Lyft, Uber, Airbnb, Comcast, and Tinder to power large-scale search and analytics applications.

As data volumes continue to grow exponentially, organizations need a scalable, resilient, and efficient way to ingest, search, and analyze their data in near real-time. That‘s where Elasticsearch shines. As a distributed search and analytics engine, it can quickly store, search, and analyze huge volumes of data and produce results in milliseconds.

In this comprehensive post, we‘ll dive deep into the fundamental concepts behind Elasticsearch, explore the backend architecture of Amazon Elasticsearch Service, and highlight key features, use cases, and best practices for optimizing your Elasticsearch deployments in the AWS Cloud.

Elasticsearch Primer

At its core, Elasticsearch is an open-source, RESTful, distributed search and analytics engine built on Apache Lucene. It was first released in 2010 by Shay Banon and has since become the most popular enterprise search engine followed by Apache Solr, also based on Lucene.

Some of the key characteristics that have propelled Elasticsearch to widespread adoption include:

• Distributed architecture designed for high availability and horizontal scalability
• Support for diverse data types – structured, semi-structured, geo-spatial, metrics
• Near real-time search and analytics with advanced query and aggregation capabilities
• Schemaless JSON documents for speed and flexibility
• Powerful full-text search using an inverted index and analysis chain
• Multi-tenancy with index and document level security controls

At a high level, an Elasticsearch cluster contains one or more nodes that store full copies of indexed data. Data is organized into indices that are split into shards, which are distributed across nodes for scalability and replicated for high availability. Queries are processed in parallel across shards with results merged and ranked before being returned to the client.

Under the Hood: Inverted Indices and Analysis

To enable fast full-text searches, Elasticsearch uses a data structure called an inverted index, which lists every unique word that appears in any document and identifies all of the documents each word occurs in.

When you index a document, Elasticsearch performs an analysis process that first breaks it down into individual terms in a process called tokenization. The tokens are then lowercased and run through filters that perform further processing such as removing stop words, stemming, etc. The resulting terms are then indexed in the inverted index for fast lookups at search time.

When you submit a search query to Elasticsearch, it performs a similar analysis process on the query string to parse out the individual search terms. It then looks up the search terms in the inverted index to find matching documents, scores them based on relevance algorithms (e.g. TF-IDF), and returns the top results.

Query DSL

Elasticsearch provides a flexible, JSON-based query language called the Query DSL for writing complex queries spanning multiple fields, data types, and logical conditions. It supports a wide variety of query types including:

• Leaf queries (match, term, range) for searching individual fields
• Compound queries (bool) for combining multiple leaf queries
• Full-text queries (match phrase, multi-match, query string) with support for fuzziness, proximity, and relevance tuning
• Term-level queries (term, terms, range, exists) for non-analyzed keyword fields
• Joining queries (nested, has child, has parent) for searching nested objects and parent-child relationships
• Geo queries (geo shape, geo bounding box) for searching geospatial data

Here‘s an example multi-match query that searches for "matrix" in the title and tagline fields:

GET /_search
{
  "query": {
    "multi_match" : {
      "query":    "matrix",
      "fields": [ "title", "tagline" ]
    }
  }
}

Sharding and Replication

An index in Elasticsearch is actually just a logical namespace that points to one or more physical shards. Sharding is key to Elasticsearch‘s scalability as it allows an index to be split into smaller, faster, more manageable parts that can be distributed across multiple nodes.

Elasticsearch automatically distributes shards across all available data nodes in a cluster and can dynamically redistribute shards as nodes are added or removed. This enables the cluster to scale horizontally simply by adding more nodes.

For fault-tolerance and high availability, Elasticsearch allows shards to be replicated to one or more replica nodes. A replica shard is just a copy of the primary shard that‘s available on standby in case the primary fails. If a primary shard does fail, Elasticsearch automatically promotes one of the replica shards to take over as the new primary.

Amazon Elasticsearch Service

Amazon Elasticsearch Service is a fully managed service that makes it easy to deploy, operate, and scale Elasticsearch clusters in the AWS cloud. It offers Elasticsearch APIs, managed Kibana, and built-in integrations with other AWS services like Amazon Kinesis Data Firehose, AWS Lambda, and Amazon CloudWatch.

Advantages Over Self-Managed Elasticsearch

Running your own Elasticsearch cluster is certainly possible but requires deep expertise to properly design, configure, secure, and scale. You‘re also on the hook for time-consuming tasks like hardware provisioning, software patching, backups, and node failure recovery.

In contrast, Amazon ES is a "black box" where all of the underlying infrastructure is taken care of for you. Spinning up a new cluster is as simple as specifying a version, instance type, and number of nodes. Amazon ES will automatically launch and configure the cluster complete with multi-AZ distribution, storage volumes, and networking.

Some of the biggest benefits of using Amazon ES over self-managed Elasticsearch include:

• Easy deployment and management via AWS Management Console, CLI, or SDKs
• No need to manage infrastructure – EC2 instances, EBS volumes, VPCs, etc.
• Automated multi-AZ replication for high availability
• Automated daily snapshots and managed restore process
• Automatic node replacement and rebalancing
• Security with VPC support, IAM, encryption, and fine-grained access control
• Direct Kibana access with single sign-on and Amazon Cognito user pools
• Integrated alerting and monitoring via Amazon CloudWatch

Performance Benchmarks

When evaluating the performance of Amazon ES, it‘s important to understand the impact of instance types, shard counts, and indexing/query loads. The AWS documentation provides detailed benchmark reports that compare query latency, indexing throughput, and data transfer speeds across different instance families.

As a general rule, the instances are the workhorses that usually give you the most bang for your buck. For example, the latest generation R6GD instances with NVMe SSD volumes delivered up to 3X better indexing throughput and 2X faster query latency compared to the previous generation I3 instances according to AWS benchmarks.

Security Deep Dive

Data security is paramount for enterprise search and analytics applications. Amazon ES offers several security capabilities to help you lock down access to your data:

• VPC support – launch domains inside a VPC for isolation and private access
• IAM integration – use AWS identity and access management to control access to the cluster configuration API and Kibana
• Fine-grained access control – define read/write permissions at index, document, and field level
• Encryption at rest – enable AES-256 encryption to protect data on disk
• Encryption in transit – enable TLS encryption for HTTPS access to the cluster
• Node-to-node encryption – enable encryption for communication between nodes
• Audit logging – track user activity for compliance and security analytics

Amazon ES also supports authentication via HTTP basic auth, AWS request signing, SAML single sign-on, Amazon Cognito user pools, and custom headers.

Getting Started

Ready to spin up your first Amazon ES cluster? Here‘s a quick tutorial to launch a cluster, index some sample data, and start running searches and visualizations in Kibana:

1. Open the AWS console and navigate to the Amazon Elasticsearch Service page
2. Click "Create a new domain" and choose "Development and testing"
3. Configure your domain name, Elasticsearch version, and number of data nodes
4. Select an access policy that allows open access (avoid for production!)
5. Review your config and click "Confirm and create"
6. Once the domain status changes to "Active", click on the Kibana URL
7. Run the following bulk command in Dev Tools to index some sample data:

POST _bulk
{ "index": { "_index": "movies", "_id": "2" } }
{ "title": "Interstellar", "director": "Christopher Nolan", "year": 2014 }
{ "index": { "_index": "movies", "_id": "3" } }
{ "title": "Star Trek Beyond", "director": "Justin Lin", "year": 2016 }

8. Run a simple match query to search the data:

GET movies/_search
{  
  "query": {
    "match": {
      "title": "Star"
    }
  }
}

9. Head over to the Visualize tab in Kibana and create a new vertical bar chart showing movie counts by year.

That‘s it! You‘ve successfully launched your first Amazon ES domain, indexed data, ran a search, and visualized the results in Kibana. For a more detailed walkthrough, check out this AWS Database Blog post.

Conclusion

As you can see, Amazon Elasticsearch Service provides a compelling platform for implementing search and analytics workloads in the cloud. By offloading the heavy lifting of managing Elasticsearch infrastructure to AWS, you can focus on innovating for your customers.

Equipped with the fundamentals of how Elasticsearch works under the hood and the core features of Amazon ES, you‘re now ready to start building your own search experiences and deriving insights from your data. Some inspiring use cases to get your creative juices flowing:

• Build a product catalog search engine for your e-commerce application
• Centralize and analyze your application and infrastructure logs
• Monitor and visualize key business metrics in real-time
• Detect and investigate security anomalies across your IT environment
• Implement geospatial search and mapping for a travel or hospitality application
• Analyze and predict customer behavior and purchasing patterns

The possibilities are endless and growing by the day. As the most popular enterprise search engine with 7X more production deployments than Apache Solr according to the DB-Engines rankings, Elasticsearch is not going anywhere anytime soon. Amazon ES sweetens the deal with a set of tools to make deploying and running Elasticsearch a breeze.

Looking to the future, it will be fascinating to watch Amazon ES and the broader Elastic ecosystem evolve. In April 2021, AWS announced OpenSearch in partnership with other industry leaders, a community-driven, open-source fork of Elasticsearch and Kibana. The launch of OpenSearch marks a new chapter of open collaboration and innovation to deliver more choice and flexibility to customers.

As data continues to grow and developers build more intelligent applications, the demand for Elasticsearch and Amazon ES will only accelerate. Buckle up and enjoy the search and analytics journey ahead!