Kubernetes continues to lead containerization environments, which are redefining architectures for container-based application development rapidly. According to Forrester’s 2020 Container Utilization Survey, over 65 percent of evaluated businesses use or plan to use container orchestration technologies as part of their IT business transformation. It is almost certain that Kubernetes’ popularity will continue to grow.
Though Kubernetes provides a long-term container solution to boost productivity, usage examples show that relying entirely on Kubernetes services for container-based applications isn’t always the ideal option. In order to make Kubernetes a reliable platform development robust applications and meet regular demands, it is vital to implement best practices and use a customized model.
15+ Kubernetes Best Practices 2021
1. Using the Newest Version
Kubernetes provides new features, bug patches, and platform upgrades with its periodic version updates. You should always utilize the most recent Kubernetes version on clusters as a rule of thumb. This guarantees that your version contains the following features:
- Support for older, outdated versions is included in all the new features, so you don’t have to worry about missing anything.
- Updated security updates that delay the onset of possible attack vectors and also address identified vulnerabilities.
2. Making the Use of RBAC
Kubernetes Role-Based Access Control or RBAC in Kubernetes can assist you in defining the users who are allowed to access the Kubernetes API along with knowing their permissions. On v1.6 and above, RBAC is generally enabled by default. Whenever RBAC gets deployed in Kubernetes, you must also disable the old Attribute-Based Access Control (ABAC) since Kubernetes mixes authorization controllers.
Consider namespace-specific rights when utilizing RBAC instead of cluster-wide rights. Don’t give cluster administrators access, even if you’re troubleshooting. Allow access only when absolutely necessary for your specific and critical scenarios.
3. Run Small-sized Base Images
It is suggested to create packages on top of the lightest, most feasible base image so that you can see exactly what’s inside the image.
Expense is also reduced by using smaller base images. Your application could just be 5 MB, but using the off-the-shelf images, such as Node.js can bring up an extra 600 MB of libraries that aren’t necessary. Other benefits of smaller images include:
- Can be created easily within no time
- Consumes lesser storage space
- Pulling images gets more efficient
- There may be a smaller attack surface
4. Use a Non-root User Inside The Container
Administrators are advised to change users to non-root users when packages are upgraded as root within the containers. The reason for this is that if someone breaks inside the container and you’ve not changed the user from the root, a simple container breakout may provide them root access to your host. When you modify the user’s privileges to non-root, the hacker must make a second effort to get root access.
As a best practice, you should have as many shells as possible along with your infrastructure. You may implement this in Kubernetes by specifying the Security context runAsNonRoot: true, which makes it a cluster-wide policy.
5. Protecting etcd via TLS, Firewall, and Encryption
The fact that etcd holds the cluster’s information and secrets makes it a valuable resource and a tempting target for hackers. Users who gain access to etcd are able to take control of the entire cluster. Access to escalate privileges can be used by malicious individuals, which is harmful. You need to perform the following configuration steps for setting up TLS with etcd:
- cert-file=: SSL/TLS certificate used for communication with etcd
- –key-file=: The key to the certificate
- –client-cert-auth: Declare that etcd should look for a client certificate issued by a reliable Certificate Authority for incoming HTTPS requests
- –trusted-ca-file=<path>: Trusted certification authority
- –auto-tls: For client connections, use a self-signed auto-generated certificate.
Following configuration parameters are used for configuring TLS for etcd for server-to-server communication:
- –peer-cert-file=<path>: SSL/TLS communications between peers require this certificate
- –peer-key-file=<path>: The key to the certificate (not encrypted)
- –peer-client-cert-auth: When this option is enabled, etcd verifies all arriving peer requests for valid signed client certificates.
- –peer-trusted-ca-file=<path>: Trusted certification authority
- –peer-auto-tls: For peer-to-peer connections, use auto-generated self-signed certificates
Install a firewall between API servers and the etcd clusters as well. For instance, run etcd on a different node and establish a firewall on that node using Calico.
Enabling encryption at rest for etcd secrets:
Encryption is required for the security of etcd, although it is not enabled by default. You may activate it without using the –encryption-provider-config option by using the kube-apiserver process by choosing an encryption provider and entering the secret keys in the settings.
1. Using sidecars
The sidecars in a pod are the utilities that are loosely coupled with the main application container. Oftentimes it becomes mandatory to execute a collection of operations for enabling clusters to communicate with one another. The idea of running all of them in one container is not feasible, therefore, every individual runs them in a pod.
Imagine a database on which your operations rely. You wouldn’t put the credentials in each container hardcoded. Instead, you may use the credentials as a proxy in a sidecar to handle the connection safely.
2. Avoid using sidecars for bootstrapping
In spite of the fact that sidecars are perfect for processing requests from both outside and inside a cluster, experts do not recommend using them for bootstrapping. Bootstrapping used to be the only option, but thanks to Kubernetes, ‘init containers’ are now available.
If two containers are interdependent on each other for processing different microservices, the “init containers” program can be used for making both the containers ready. This prevents lots of potential problems caused when processes and microservices are out of sync. The general guideline is to utilize sidecars for recurring events and init containers for one-time occurrences.
3. Isolate Kubernetes Nodes
Kubernetes nodes must be on a dedicated network and must not be connected to the internet. Direct connections to the business network should be avoided whenever possible.
This can be possible only when Kubernetes control and data traffic are separated from each other. Otherwise, both will run through the same pipe, and open data plane access implies open control plane access. Nodes must typically be equipped with an entrance controller that restricts access to the given port to only the master node via the network access control list (ACL).
4. Lock Kubelet
The kubelet is a node-based agent that communicates with the container runtime to deploy pods and provide nodes and pods data. Each kubelet in the cluster has an API that can be used to begin and end pods, among other things. If an unauthorized person obtains entry to this API (on any node) and is able to run code on clusters, the entire cluster can be compromised.
You may use the following configuration settings to lock the kubelet and limit the attack surface:
- Disabling the permissions for unauthorized users by using –anonymous-auth=false, this will show errors to anonymous users. To achieve this, your API server is required to be configured as Kubelet, and for this, add flags -kubelet-clientcertificate and –kubelet-client-key.
- To ensure that requests are permitted, set —authorization mode to a value apart from AlwaysAllow. This is configured as a webhook by default by the kubeadm installation toolset, assuring that kubelet calls SubjectAccessReview on the API server for verification.
- To limit kubelet privileges, add NodeRestriction inside the API server –admission-control option. This restricts the kubelet’s ability to change pods linked towards its node objects.
- Disable read-only ports with —read-only-port=0. Unauthorized users will not be able to gather information about ongoing workloads as a result of this. This port will not allow hackers to take control of the cluster, although it can be exploited for reconnaissance purposes.
- Turn off cAdvisor, which was used to give metrics in previous versions of Kubernetes but has since been superseded with Kubernetes API statistics. To prevent revealing information regarding running workloads, set -cadvisor-port=0. For Kubernetes v1.11, this is the default option. Use a DaemonSet to execute cAdvisor if you need to.
5. Using Namespaces
Kubernetes adoption in larger organizations with many teams using the same cluster demands for a unique resource consumption strategy. Inadequate accessibility provisioning frequently leads to resource usage disputes among groups.
Therefore, make sure to use namespaces to provide team-level isolation for teams seeking to use the same cluster resources at similar instances for solving the problem. Using Namespaces effectively allows you to build various logical cluster divisions, allowing you to allocate different virtual resources to different teams.
7. Setting Limits For Resources
Set demands and limitations for cluster resources, especially for CPU and Memory in order to avoid situations in which a single team or app consumes all of the cluster’s resources. This prevents maximum downtime by limiting excessive resource use by apps and services.
One can use the below-mentioned container standard as a guide to establishing requests and limits on containers:
Copy
containers:
– name: darwin
image: CentOS
resources:
requests:
memory: “256Mi”
cpu: “300m”
limits:
memory: “1024Mi”
cpu: “900m”
The aforementioned container standard effectively provides 256 MiB of RAM and 300 mCPU to resources demands, while restricting the darwin container to 900 mCPU and 1024 MiB.
8. Autoscaling For Applications
It is strongly advised that you take advantage of Kubernetes’ autoscaling features for scaling cluster services automatically when resource demand spikes. Node and pods volumes are automatically changed in real-time using Horizontal Pod Autoscaler and Cluster Autoscaler, ensuring optimal load and preventing capacity downtimes.
9. Enabling Audit Logging Feature
Make sure audit logging is enabled as well as keeping an eye out for any odd or unwanted API requests, specifically authentication failures. If you don’t authorize, it’s possible that an attacker is attempting to use your credentials. The status for such log entries is “Forbidden.”
An individual can also use the –audit-policy-file switch when providing files to Kube-apiserver to enable audit logging and specify what events must be reported. The four logging levels are-
- None
- Metadata only
- Request (which records metadata and requests but not responses)
- RequestResponse (which logs all three)
Managed Kubernetes providers may provide their customers access to this information through their dashboard and set up alerts for permission failures.
10. Using readinessProbe & livenessProbe
Make the use of Kubernetes Check Probes for eliminating any potential pod failures:
- Before permitting traffic on Pods, Kubernetes verifies if the application is prepared for serving traffic using readinessProbe for production applications. It simply shows whether or not a pod is ready to receive traffic and respond to queries.
- Kubernetes runs a health check on the applications using a livenessProbe to verify if it is responsive and operating as expected. If the livenessProbe fails, the kubelet defaults policy restarts the container to get it back up and running.
11. Keep a check on disk usage
High disk utilization is a frequent problem that has an influence on a cluster’s efficiency. You should keep an eye on the following things on a regular basis:
- The root file system
- Cluster’s disk volume linked together.
You may also receive high-disk utilization alerts for unexplained causes; owing to the obscurity of the fundamental cause, such situations might be difficult to resolve. Having alert monitoring in place allows you to take remedial action at the proper moment, such as scaling or clearing disk space.
12. Using Helm Charts
Helm is a repository for Kubernetes configurations that have been bundled. For instance, If you’re looking to set up a MongoDB server, you can make use of a preset Helm chart that includes all of its dependencies and can quickly deploy it to your cluster. Helm charts for commonly used software components may save loads of time and manual efforts.
13. Adopting GitOps
Leverage GitOps, a Git-based workflow, as an ideal paradigm for all operations, including CI/CD pipelines. Using a GitOps infrastructure boosts productivity by allowing you to:
- Shortening deployment times
- Improving the traceability of errors
- Automation of CI/CD workflows
When you use GitOps on the Kubernetes clusters, you may accomplish integrated cluster administration as well as faster application development.
14. Adopt Process Whitelisting
Process whitelisting is a good technique to spot tasks that aren’t supposed to be running. To begin, monitor the application over time in order to identify any processes that are active during typical application behavior. Then, for future application behavior, utilize this list as a whitelist.
At the process level, runtime analysis is challenging. There are a number of commercial security tools that may assist evaluate and discover abnormalities in running processes across clusters.
15. Run single process per container
Although you can run several processes in a container, it is suggested that you only run one. This is due to the orchestrator’s method of operation. Containers are managed by Kubernetes depending on whether or not a process is healthy.
It would be difficult for an orchestrator platform to identify whether a container is healthy or not if it has 20 processes operating inside it. For running multiple processes that can communicate with each other easily, individuals can run them under single Pods instead.
16. Avoid restarting on failures
Instead of restarting containers, one should start them cleanly. Because Kubernetes automatically restarts failing containers, one should crash cleanly via an error code so that they may restart without the involvement of administrators.
Conclusion
Kubernetes is currently the most popular open-source container orchestration platform that helps in the efficient management of Container-based applications. They come with lots of benefits that help organizations to automate and speed up the deployment and management process. However, relying entirely on them and their default configurations is not ideal.
In this way, Kubernetes best practices always provide better operation security, scalability, and management.esses. The blog contains 20 best practices for running Kubernetes services so that organizations can make the most of Kubernetes.