Install Elasticsearch on Kubernetes Using Helm Chart

Photo of author

By admin

The Elasticsearch Stack is built to work alongside Kubernetes as a monitoring stack for collecting, storing, and analyzing Kubernetes telemetry data. The easiest approach for establishing and deploying the Elasticsearch stack on Kubernetes is to use helm charts.

The blog contains all the basic and advanced information about Elasticsearch, along with a step-by-step guide for installing it on Kubernetes using Helm charts. You’ll also learn about installing Kibana, which is the data visualization dashboard for Elasticsearch.

What is Elasticsearch?

Elasticsearch can be viewed as a server that can handle JSON requests and return JSON data. Elasticsearch is a Java-based, distributed, and open-source search and analytics platform based on Apache Lucene. It was initially released as a scalable edition of Lucene, but was later designed to expand the indexes horizontally. A user can save, explore, or analyze large amounts of data in near real-time with Elasticsearch, with results arriving in milliseconds.

It is possible to get quick search results with Elasticsearch since it examines the indexes rather than searching the text directly. It has a document-based architecture rather than tables and schema, and it has rich REST APIs for collecting and finding data.

Backend Components

1. Cluster

A cluster is basically a collection of node copies that are linked together. The distribution of jobs, search, and indexing among all nodes inside an Elasticsearch cluster is what gives it its strength.

2. Node

Single servers that are part of a cluster are referred to as a node. A node is a computer that stores information and participates in the cluster’s indexing and search functions. Elasticsearch nodes can be set up in a variety of ways:

  • Master Node: The master node is responsible for managing the Elasticsearch cluster, creating and deleting indexes, and adding and removing nodes.
  • Data Node: Stores information and performs data-related tasks, including searching and aggregating.
  • Client Node: Sends cluster requests to the master node and data requests to the data nodes.

3. Shards

Elasticsearch allows you to split the index into shards, which are smaller parts of the index. Each shard is a completely functional and self-contained “index” that may be stored on any cluster node. Elasticsearch ensures durability by dividing documents in an index over many shards and those shards across numerous nodes, which protects from hardware failures that eventually increase search capacity when nodes are deployed to the clusters.

4. Replicas

Elasticsearch permits users to duplicate one or more of the index’s shards, known as “replica shards” or simply “replicas.” A replica shard is essentially a duplicate of a primary shard. Each page in an index is associated with a single primary shard. Replicas provide multiple copies of your data to safeguard from hardware failure and enhance your capacity to handle and read demands such as document searches and retrieval.

Basic concepts of Elasticsearch

1. Documents

Documents are the simplest type of data that are indexed in Elasticsearch and are represented in JSON, the universal internet data exchange standard. In Elasticsearch, a document can be anything that is structured and encoded in JSON, not just text. Several examples of structured data include strings, numbers, and dates. A document has its own ID and identifies what kind of object it is based on the type of data it contains.

2. Index

A collection of items with similar qualities is referred to as an index. The index may be compared to a database in a relational database schema. In most cases, the documents in an index are logically connected. You can have an index for Customers, one for Products, one for Orders, and so on in the context of an e-commerce website. An index is given a name that is used to refer to it while conducting indexing, searching, updating, and deleting actions on the documents included inside it.

3. Inverted Index

An inverted index is a data structure that keeps a mapping between information (such words or integers) and their positions in a document or group of documents. It’s essentially a hashmap-like data structure that leads from a word to a document. Instead of storing strings explicitly, an inverted index divides each document into separate search phrases (i.e. each word) and then links each search term to the document in which it appears. Elasticsearch swiftly discovers the best matches for full-text searches from even very big data by employing distributed inverted indices.

Benefits of using Elasticsearch

1. Speed

Elasticsearch excels in full-text searching because it is built on top of Lucene. Elasticsearch is also a near-real-time search technology, which means that the time it takes for a document to be indexed and searchable is generally less than a second. As a result, Elasticsearch is ideally suited for time-critical applications like security analytics and infrastructure monitoring.

2. Distributed structure

Elasticsearch documentation is kept in several containers known as shards, that are replicated to offer multiple copies of the data for preventing hardware failures. Elasticsearch’s distributed structure enables it to expand to hundreds (or thousands) of servers and handle petabytes of data easily.

3. Wide features spectrum

Elasticsearch has a variety of strong built-in features that make storing and finding data, and tasks like data roll-ups and index lifecycle management, even more convenient and fast. It also provides better performance, scalability, and robustness.

4. Easy data ingestion, visualization, and reporting

It’s simple to handle data before indexing it in Elasticsearch, thanks to its ability for integrating with several platforms. Kibana, meanwhile, provides real-time visualization of Elasticsearch data as well as user interfaces for easily obtaining information regarding APM, logs, and infrastructure metrics.

5. Analyzing metrics

There are many organizations using the Elasticsearch stack to analyze numerous metrics. This includes a collection of the data through multiple performance parameters that vary as per the use case.

6. Security analytics

Elasticsearch’s security analysis is another popular analytics application. The ELK stack can analyze access logs and other logs related to system security, giving you a more complete view of what’s proceeding in real-time across your systems.

Steps to install Elasticsearch on Kubernetes Using Helm Chart

Given below are four simple steps for installing Elastricsearch on Kubernetes. But before proceeding, make sure that you have a Kubernetes cluster that you can create with minikube for ease in the process. Also, the installation of the kubectl command-line tool and Helm package manager is also very important.

1. Setting up clusters

First of all, you have to set up a Kubernetes cluster that can be done using Minikube. Clusters with multiple Elasticsearch nodes should have sufficient CPU and memory resources, so allocate enough CPU and memory for the –cpus and –memory options:

minikube start –cpus 4 –memory 8192

Run the following command for checking whether your cluster is running or not. The output displayed on the screen will define the status of the Kubernetes control plane and KubeDNS.

kubectl cluster-info

2. Deploying Elasticsearch using Helm (helm install elasticsearch)

For initiating the installation process of Elasticsearch, first, you need to add the elastic repository in Helm:

helm repo add elastic https://helm.elastic.co

Now, execute the curl command for downloading the values.yaml file composed of configured info as defined:

curl -O https://raw.githubusercontent.com/elastic/helm-charts/master/elasticsearch/examples/minikube/values.yaml

Now run the helm install command along with the values.yaml file for installing Elasticsearch helm chart:

helm install elasticsearch elastic/elasticsearch -f ./values.yaml

The -f option allows you to provide the template’s yaml file. Add the -n option, followed by the name of the namespace, for installing Elasticsearch in a specified namespace:

helm install elasticsearch elastic/elasticsearch -n [namespace] -f ./values.yaml

The output received after this will define the status of the application as deployed and offer additional options for testing installations.

Now run the get pods command to verify if all the cluster members are up or not:

kubectl get pods –namespace=default -l app=elasticsearch-master -w

In the displayed output, wait for the READY column to check if it has reached 1/1 entries. If yes, then all the cluster members are up.

Run the following command to check the health of the customers.

helm test elasticsearch

Right after installing Elasticsearch, execute command for forwarding the port to 9200:

kubectl port-forward svc/elasticsearch-master 9200

3. Installing Kibana

Kibana is a free and open-source frontend tool that gets installed over Elasticsearch Stack, allowing users to search and visualize data stored in Elasticsearch. Run the below-mentioned command for installing Kibana over Elasticsearch.

helm install kibana elastic/kibana

Verify whether or not the pods are ready. All the Kibana pods will appear under Elasticsearch pods.

kubectl get pods

Now push Kibana to port 5601 with the help of kubectl:

kubectl port-forward deployment/kibana-kibana 5601

Right after forwarding the port, an individual can access Elasticsearch and Kibana GUI (Graphical User Interface) by typing http://localhost:5601 on the web browser.

4. Installing Metricbeat

Metricbeat is a lightweight package that users can install on their servers for gathering metrics from the operating system and the services that operate on them on a regular basis. The installation of Metricbeat is similar to that of Kibana.

Run the following helm command for install elasticsearch with helm:

helm install metricbeat elastic/metricbeat

Ensure that all the pods related to Metricbeat are up and running:

kubectl get pods

After performing the aforementioned steps, you’ll be able to build the index patterns. For this, you have to visit Kibana and then navigate to the Stack Management -> Index patterns. From there, you can easily create index patterns.

Benefits of installing elastic search by helm

Installing Elasticsearch using Helm in a Kubernetes environment offers several benefits. Here’s a list of some of the advantages:

  1. Simplified Deployment: Helm charts encapsulate the complexity of application configurations into a predefined package, simplifying the deployment of Elasticsearch clusters.
  2. Version Management: Helm helps in managing and versioning the Elasticsearch deployment, allowing users to roll back to previous configurations if needed.
  3. Configurable Parameters: Helm charts come with a values.yaml file where users can customize various parameters like node count, resources limits/requests, and persistent storage options without modifying the main application configurations.
  4. Dependencies Management: Helm can automatically manage dependencies, ensuring that if Elasticsearch requires other services or components to run, they will be installed alongside it.
  5. Scalability: With Helm, scaling the Elasticsearch nodes (like data nodes, master nodes, or client nodes) becomes a matter of changing a parameter and updating the release.
  6. Updates and Upgrades: Helm facilitates easy updates and upgrades of Elasticsearch deployments. This ensures that you can leverage the latest features, improvements, and security patches of Elasticsearch with minimal hassle.
  7. Consistency: Helm ensures consistent deployments. This means that whether you’re deploying in a development, staging, or production environment, the application and its dependencies are deployed in the same way.
  8. Extensibility: Helm charts can be customized and extended to fit specific requirements. This means that if the official Elasticsearch Helm chart doesn’t fulfill a particular need, it can be forked and modified, or a custom chart can be created.
  9. Pre-packaged Configurations: Helm charts often come with best practice configurations out-of-the-box, which reduces the initial setup time and potential configuration errors.
  10. Community Support: The Helm community and broader Kubernetes community often maintain charts, ensuring they are kept up-to-date with the latest best practices and software versions.
  11. Integrated Services: By using Helm in a Kubernetes environment, it becomes easier to integrate Elasticsearch with other services and applications in the same cluster, such as Kibana, Logstash, or Beats, which might also have their Helm charts.

In summary, using Helm to deploy Elasticsearch in a Kubernetes environment streamlines the process, ensures consistency, and leverages the power of both the Helm and Kubernetes communities. It abstracts away many complexities, making the management of Elasticsearch deployments more efficient and reliable.

Conclusion

Based on Apache Lucene, Elasticsearch is a Java-based, distributed, and open-source search and analytics platform. It allows users to save, browse, and analyze massive quantities of data in near real-time, with results arriving in milliseconds. Its back-end component is composed of Clusters, nodes, replicas, and shards.

The advantages of installing Elasticsearch on Kubernetes include ease of data ingestion, visualization, and reporting, as well as speeding up tasks and analyzing metrics. In this blog post, we provide a step-by-step guide to getting Elasticsearch up and running on Kubernetes using Helm charts.

Leave a Comment