Introduction to Elasticsearch Using Python
Elasticsearch is a powerful open source search and analytics engine that has become extremely popular in recent years for handling large volumes of data. Built on top of Apache Lucene, Elasticsearch provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.
While often referred to as a NoSQL database, Elasticsearch is really more of a search engine and analytics platform. Its primary focus is on providing extremely fast search capabilities, rather than on the transactional and relational aspects commonly associated with NoSQL databases.
So why would you choose to use Elasticsearch over a traditional relational database or another NoSQL solution? Here are a few key benefits:
Fast Full-Text Search
Elasticsearch was built from the ground up to provide near real-time search on large volumes of textual data. It uses an inverted index to quickly look up which documents contain the search terms, allowing it to return results extremely fast even with millions or billions of documents.
Scalability and Resiliency
Elasticsearch makes it easy to scale horizontally by adding more nodes to your cluster. It automatically distributes data and query load across available nodes. Data is also replicated across the cluster to protect against hardware failures. You can add or remove nodes from the cluster as your needs change.
Flexible Data Model
With Elasticsearch, you don‘t have to define your schema upfront. You can just start indexing JSON documents and add new fields on the fly. This flexibility is great for evolving data models and unstructured data. Of course, you can still define mappings to specify how fields should be indexed.
Powerful Analytics Capabilities
In addition to search, Elasticsearch has robust analytics capabilities through its aggregations framework. This allows you to easily group, filter, and extract metrics and statistics from your data in real-time. Popular use cases include things like activity tracking, metrics monitoring, and business intelligence.
Now that we have an idea of what Elasticsearch is good for, let‘s take a look under the hood to see how it actually works.
Elasticsearch Architecture Basics
An Elasticsearch cluster consists of one or more nodes that share the same cluster name. Each node is a single server (physical or virtual machine) that runs the Elasticsearch software. As nodes are added to or removed from the cluster, Elasticsearch automatically reorganizes data to distribute it evenly.
Within the cluster, data is organized into one or more indexes (databases). An index is a collection of JSON documents that have similar characteristics. For example, you might have an index for product data, another for logs, and yet another for customer information.
Indexes are divided into multiple shards for distributing data across nodes. Each shard is an independent Lucene index that can reside on any node in the cluster. By default, Elasticsearch creates 1 shard per index, but for large indexes you will want to increase the shard count for better distribution. Shards can also have one or more replicas for redundancy and fail-over.
When a document is indexed, Elasticsearch first determines which shard it should go to based on its ID using consistent hashing. The document is then indexed and stored on that shard‘s primary copy. If replicas are configured, Elasticsearch will also index the document on the replica shards asynchronously. This architecture allows data to be distributed and replicated automatically across the cluster.
Installing and Running Elasticsearch
The easiest way to get up and running with Elasticsearch is to use Docker. With Docker, we can quickly spin up an Elasticsearch node without having to install and configure the software locally.
First, make sure you have Docker installed. You can download it from the official website: https://www.docker.com/get-started
Once Docker is installed and running, simply run this command to download the latest Elasticsearch Docker image and start a single-node cluster:
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.12.0
This will start an Elasticsearch node and expose the HTTP interface on port 9200 and the transport interface on port 9300. The discovery.type=single-node parameter tells Elasticsearch to run as a single-node cluster.
You can verify Elasticsearch is running by accessing http://localhost:9200 in your browser or via curl:
curl http://localhost:9200
{
"name" : "9205319c95f6",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "H_oi15RbSvyP4QYx_sFXSg",
"version" : {
"number" : "7.12.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
"build_date" : "2021-03-18T06:17:15.410153305Z",
"build_snapshot" : false,
"lucene_version" : "8.8.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
If you see output similar to this, congrats – you have Elasticsearch up and running!
Connecting to Elasticsearch from Python
To interact with Elasticsearch from Python, we‘ll use the official elasticsearch-py client library. This can be installed via pip:
pip install elasticsearch
Once installed, connecting to your local Elasticsearch node is as simple as:
from elasticsearch import Elasticsearch
es = Elasticsearch(‘http://localhost:9200‘)
The Elasticsearch constructor accepts a variety of optional parameters for configuring things like authentication, SSL, connection pooling, etc. See the documentation for details.
Creating an Index and Adding Documents
Before we can search, we need to index some data. With Elasticsearch, data is organized into indexes, which can be thought of like databases in a relational database.
To create an index, we simply make an HTTP PUT request:
es.indices.create(index=‘blog‘)
This will create an index named "blog" with all of the default configuration options. If we want to customize things like the number of shards and replicas, or the field mappings, we can pass additional parameters. See the create index API docs for details.
Once we have an index, we can add JSON documents to it:
doc = {
‘author‘: ‘John Doe‘,
‘text‘: ‘This is the first blog post!‘,
‘timestamp‘: datetime.now()
}
resp = es.index(index="blog", id=1, body=doc)
print(resp[‘result‘])
doc2 = {
‘author‘: ‘Jane Doe‘,
‘text‘: ‘This is the second blog post! How exciting.‘,
‘timestamp‘: datetime.now()
}
resp = es.index(index="blog", id=2, body=doc2)
print(resp[‘result‘])
The index() method adds or updates a JSON document in the specified index. We‘re passing a custom document ID here, but if we leave off the id parameter, Elasticsearch will automatically generate a unique ID for us.
Searching Documents
Now that we have some data indexed, we can start searching! The primary way to search documents is by using the Query DSL (Domain Specific Language). This allows us to express complex queries using a JSON request body.
To perform a basic match query on all fields:
resp = es.search(index="blog", body={"query": {"match_all": {}}})
print("Got %d Hits:" % resp[‘hits‘][‘total‘][‘value‘])
for hit in resp[‘hits‘][‘hits‘]:
print("%(timestamp)s %(author)s: %(text)s" % hit["_source"])
This will return all documents in the index, similar to a SELECT * in SQL. The response contains metadata about the request (how long it took, etc.) as well as the actual search hits.
We can also perform a full-text search on specific fields using the match query:
resp = es.search(index="blog", body={"query": {"match": {"text": "first"}}})
print("Got %d Hits:" % resp[‘hits‘][‘total‘][‘value‘])
for hit in resp[‘hits‘][‘hits‘]:
print("%(timestamp)s %(author)s: %(text)s" % hit["_source"])
This will search the text field of all documents for the word "first" and return any matches. Elasticsearch uses a variant of the TF-IDF similarity algorithm to score matches by relevance.
Filtering Results
In addition to full-text searches, we can also filter documents based on structured data using the filter clause. Filters are like queries, but rather than scoring results by relevance, they simply include or exclude documents based on the specified criteria.
For example, to filter documents by timestamp:
resp = es.search(index="blog", body={
"query": {
"bool": {
"filter": [
{
"range": {
"timestamp": {
"gte": "now-1d/d",
"lt": "now/d"
}
}
}
]
}
}
})
print("Got %d Hits:" % resp[‘hits‘][‘total‘][‘value‘])
for hit in resp[‘hits‘][‘hits‘]:
print("%(timestamp)s %(author)s: %(text)s" % hit["_source"])
This uses a range filter to only include documents from the last day. The bool query allows us to compose multiple query and filter clauses. See the bool query docs for more info.
Aggregations and Analytics
In addition to searching for individual documents, Elasticsearch can also compute aggregations over your data using the aggregations framework.
For example, to get a count of documents by author:
resp = es.search(index="blog", body={
"aggs": {
"group_by_author": {
"terms": {
"field": "author"
}
}
}
})
print("Author counts:")
for bucket in resp[‘aggregations‘][‘group_by_author‘][‘buckets‘]:
print("- %s: %d" % (bucket[‘key‘], bucket[‘doc_count‘]))
The terms aggregation will bucket documents by unique field values and give you a count of documents in each bucket. We could further nest aggregations to get things like average or sum of numeric fields, min/max values, date histograms, geo heatmaps and more. Aggregations are a powerful way to slice, dice, and analyze your data.
Updating and Deleting Documents
In addition to indexing and searching, Elasticsearch supports updating and deleting documents by ID.
To update an existing document:
es.update(index="blog", id=2, body={
"doc": {
"text": "I‘ve updated this blog post."
}
})
And to delete a document:
es.delete(index="blog", id=1)
Bulk Indexing for Improved Performance
If you have a lot of documents to index, making an HTTP request for each one can be slow. To speed things up, Elasticsearch supports bulk indexing, allowing you to batch multiple index/update/delete operations in a single API call.
For example:
from elasticsearch import helpers
actions = [
{
"_index": "blog",
"_id": 3,
"_source": {
"author": "John Doe",
"text": "This is the third blog post.",
"timestamp": datetime.now()
}
},
{
"_index": "blog",
"_id": 4,
"_source": {
"author": "Jane Doe",
"text": "This is the fourth blog post.",
"timestamp": datetime.now()
}
}
]
resp = helpers.bulk(es, actions)
print(resp)
The bulk() helper will batch the actions and send them to Elasticsearch in chunks for improved performance. It handles things like retrying failed operations and logging errors. See the bulk helpers docs for details.
Conclusion
Elasticsearch is a powerful tool for implementing search and analytics in your applications. With its distributed architecture, flexible data model, and near real-time performance, it can handle use cases from the simple to the very complex and large-scale.
In this post, we looked at:
- What Elasticsearch is and some of the key benefits
- How Elasticsearch architecture enables scalability and resiliency
- Installing and running Elasticsearch locally using Docker
- Connecting to Elasticsearch from Python using the elasticsearch-py client library
- Indexing and searching documents
- Filtering results
- Aggregating data for analytics
- Updating and deleting documents
- Improving indexing performance with bulk operations
I hope this has been a helpful introduction to get you started with Elasticsearch in Python! To learn more, I recommend checking out the official Elasticsearch documentation. It‘s very comprehensive with lots of examples. The elasticsearch-py documentation is also a great reference for the Python client specifically.
Thanks for reading, and happy searching!
