Comparison of the Kubernetes Engines

Many companies use Kubernetes to orchestrate containers nowadays. They have many options for provisioning and managing their clusters. Sometimes they struggle to choose the option that suits their requirements. In this blog post, I will discuss how you can use tools to build your infrastructure frameworks.

You will need the following aspects addressed for a functioning Kubernetes installation:

• Provisioning Servers
• TLS  Management
• Configuration management(Kubernetes upgrade)
• Scaling

You can address these from scratch manually, but it can be hard to provide an automated infrastructure in that case. For a highly-available, automated infrastructure, you need a framework that addresses the aspects above. We will talk about six of these tools today.

Ref: CNCF Survey 2019 

Tools on the Cloud: EKS, AKS, GKE, Kops

There are two methods for orchestration in a cloud environment. 

• Managed for Control Plane layer by Cloud Provider.
• Managed by the Infrastructure Team.

Managed for Control Plane layer by Cloud Provisioner

Their cloud providers fully manage these services. You can provision those services as-code (terraform, etc.).

AWS EKS:

EKS is fully compatible with applications on any vanilla Kubernetes architecture. The EKS provides a single-tenant control plane on each cluster. Control Planes are not shared across clusters. EKS works with AWS Fargate as well; Fargate gives a way to deploy containers with no understanding of server infrastructure for developers.

AZURE AKS:

AKS supports any vanilla Kubernetes architecture, also supports Apache Mesos and Docker Swarm. Windows developers usually prefer AKS. AKS does not support HA cluster provisioning.

Google Cloud Kubernetes Engine (GKE):

GKE is a fully managed service that allows you to set up applications in no time and provides cluster features quicker than other cloud providers. GKE has HA clusters via regional clusters (master and workers replicated.)

Comparison Chart between Cloud Environments:

Managed by the Infrastructure Team

Kops:

Kops is a tool that helps you manage Kubernetes components as systemd services and static pods. You can use kops to create your infrastructure framework to manage your clusters based on as-code principles.

Here is a Kops cluster templating example for framework architecture.

# file cluster.yaml
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
 name: .
spec:
 docker:
   
 additionalPolicies:
   master: |
     
   node: |
     


# File nodes.yaml ~ master.yaml
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
 labels:
 kops.k8s.io/cluster: .
 name: nodes
spec:
 image: coreos.com/CoreOS-stable-1409.6.0-hvm
 kubernetesVersion: 
 machineType: m4.large
 maxPrice: "0.5"
 maxSize: 20
 minSize: 15
 role: Node
 rootVolumeSize: 100
 subnets:
 - a
 - b
 - c


   kops toolbox template \
   --values values.yaml \
   --template templates/cluster/cluster.yaml \
   --template templates/cluster/master.yaml \
   --template templates/cluster/nodes.yaml \
   --format-yaml \
   --output /tmp/cluster.yaml
 
   cat /tmp/cluster.yaml
 
   kops replace --force -f /tmp/cluster.yaml
   

As you can see above, kops manages your cluster nodes and components such as kubelet, api-server and etcd in a cluster and instance groups files as shown below.

Kops Pros/Cons:

Kops is an excellent way to create a Kubernetes cluster in the cloud systems. It allows HA installation and fully customizable infrastructure.

Open issues from GitHub.

Tools for On-Premise Environments: Kubespray, Kubeadm RKE, K3S

Kubespray:

Kubespray is a framework that installs and manages systemd services and static pods. It uses kubeadm in the background. Generally, Kubespray is managed by ansible playbooks. Also, it has Terraform modules too.

Thanks to those Ansible Playbooks, you can edit your cluster nodes and their roles in the inventory.ini file from kubespray GitHub repository. You can also develop cluster components and network configurations in k8s-cluster.yaml.

Kubespray Pros/Cons:

Installation takes a long time to execute your changes because Ansible works sequentially, but if you know Ansible, you can easily manage your clusters with Kubespray.

Open issues from GitHub.

Kubeadm:

kubeadm allows you to install and manage Kubernetes components as systemd services clusters. It is an official installation method suggested by CNCF.

You can develop a framework or use kubespray instead of using kubeadm alone. This configuration is more efficient, especially at HA installations.

kubeadm Pros/Cons:

A lot of Kubernetes installation options(kubespray etc.) implement kubeadm into their process. But if you want to use kubeadm purely, you should consider extra development processes for HA setup. For example, you should design your framework based on your requirements (e.g., cloud-manager, load balancer setup .. etc.)

Github.

RKE:

RKE is a new tool by Rancher that installs and manages kubernetes components as docker containers kubernetes clusters. RKE can be run on the cloud or on-premises.

You can manage RKE easily by a single yaml file, shown below.

nodes:
   - address: #
     internal_address: #
     user: centos
     ssh_key_path: /Users/Desktop/rke.pem
     role:
       - controlplane
       - worker
       - etcd
   - address: #
     internal_address: #
     user: centos
     ssh_key_path: /Users/Desktop/rke.pem
     role:
       - controlplane
       - worker
       - etcd
 
services:
   etcd:
     snapshot: true
     creation: 6h
     retention: 24h
  # Required for external TLS termination with
 # ingress-nginx v0.22+
ingress:
   provider: nginx
   options:
     use-forwarded-headers: "true"
 
kubernetes_version: v1.18.3-rancher2-2

   

RKE Pros/Cons:

RKE is a new solution compared to the other installation solutions. But you can easily manage and automate your cluster management and provisioning steps.

Enterprise support 24/7 give on the Rancher, and you can open issues from Github.

K3S:

K3s is developed for IoT and Edge applications. K3S strips out many legacy features/plugins and substitutes Kubernetes components for lightweight alternatives to achieve a binary size of ~60MB. For example, it uses sqlite instead of etcd. K3S is suggested for local or dev/test/stg installations.

An example of K3S architecture is shown below;

K3S Pros/Cons:

Enterprise support 24/7 give on the Rancher, and you can open issues from Github. 

Conclusion:

In this blog post, I talked about Kubernetes cluster providers, their benefits, and their shortcomings. I spoke about six tools for cloud and on-premise environments. 

GKE is the go-to implementation with easy set-up and regional HA cluster features. AWS has a few distinguishing features, such as support for bare metal nodes, and it has a beneficial IAM service when integrated with RBAC for cluster security. AKS faster than AWS for provisioning clusters, but it has a disadvantage for regional cluster replication than other providers.

When looking at the managed control plane by the infrastructure teams side, RKE has useful features such as easy automation, managing only one YAML and back-up solution for etcd. Kubespray is a practical framework with ansible. Kops helps management clusters if you create your scripts or framework, but it has the disadvantage of only running at AWS as stable. Finally, K3S is a perfect option for local environments or low load production environments.

These tools evolve as our industry expands and new requirements arise. Therefore it is essential to assess new options for your needs always. You can do this by comparing ease of management, installation, and upgrades to pick the best options to manage your Kubernetes environment. We hope that our comparison gave you a headstart in your Kubernetes journey.