KubeVirt on AWS EKS: Unifying Containers and VMs in Kubernetes

Kubernetes is the de facto standard for container orchestration, enabling developers to easily deploy, scale, and manage containerized applications. However, there are scenarios where running virtual machines (VMs) alongside containers is necessary. This is where KubeVirt comes into play. 

KubeVirt is a Kubernetes extension that allows you to manage VMs alongside containers within the same Kubernetes cluster. This blog post will guide you through the process of installing KubeVirt on an AWS Elastic Kubernetes Service (EKS) cluster, complete with examples and best practices.

Use Cases for KubeVirt

KubeVirt is particularly useful in scenarios where:

• Legacy Applications: Many organizations still rely on legacy applications that are not easily containerized. KubeVirt allows these applications to run as VMs within a Kubernetes cluster, enabling modernization without rewriting or refactoring.
• Specialized Workloads: Certain workloads, such as those requiring specific kernel versions or custom drivers, are better suited to VMs. KubeVirt provides the flexibility to run these workloads alongside containerized applications.
• Development and Testing: Developers can use KubeVirt to create isolated environments for testing and development, ensuring consistency across different stages of the software lifecycle.
• Hybrid Cloud: Organizations with hybrid cloud strategies can use KubeVirt to manage

VMs across on-premises and cloud environments, providing a unified management plane.

Integration with Other Kubernetes Tools

KubeVirt can be integrated with other Kubernetes tools to enhance its capabilities:

1. Istio: By integrating KubeVirt with Istio, you can manage traffic between VMs and containers, enabling advanced networking features like service mesh and traffic routing.
2. Prometheus and Grafana: KubeVirt can be monitored using Prometheus and Grafana, providing insights into VM performance and health.
3. Velero: Velero can be used for backup and disaster recovery of VMs, ensuring data integrity and availability.
4. Argo CD: For GitOps workflows, Argo CD can manage the deployment and lifecycle of KubeVirt resources, ensuring consistency and version control.

Challenges and Considerations

While KubeVirt offers numerous benefits, there are challenges and considerations to keep in mind:

• Resource Management: Running VMs alongside containers can lead to resource contention. Proper resource allocation and monitoring are essential to ensure optimal performance.
• Networking Complexity: Configuring networking for VMs, especially in a multi-tenant environment, can be complex. Tools like Multus and CNI plugins can help, but they require careful planning and configuration.
• Security: VMs introduce additional security considerations, such as hypervisor vulnerabilities and VM isolation. Implementing robust security practices, including network policies and RBAC, is crucial.
• Operational Overhead: Managing both containers and VMs within the same cluster can increase operational complexity. Automation and orchestration tools can help mitigate this overhead.
• Community and Ecosystem

KubeVirt is backed by a vibrant community and ecosystem, with contributions from major players in the cloud-native space. The project is part of the Cloud Native Computing Foundation (CNCF), ensuring ongoing development and support. Engaging with the community through forums, mailing lists, and conferences can provide valuable insights and support.

Prerequisites

Before we dive into the installation process, ensure that you have the following prerequisites in place:

• An AWS account with sufficient permissions to create and manage EKS clusters.
• AWS CLI is installed and configured on your local machine.
• kubectl is installed and configured to interact with your EKS cluster.
• Helm is installed for managing Kubernetes applications.
• Virtctl is installed for managing Virtual Machines.
• Setting Up AWS EKS
• Creating an EKS Cluster

To get started, you need to create an EKS cluster. You can do this using the AWS Management Console, AWS CLI, or tools like eksctl. 

For simplicity, I’ll use eksctl in this guide.

Install eksctl: If you haven't already, install eksctl by following the official documentation.
Create an EKS Cluster: Use the following command to create a basic EKS cluster:
eksctl create cluster \ 
--name kubevirt-test \ 
--region eu-west-1 \ 
--nodegroup-name standard-workers \ 
--node-type c5.metal \ 
--nodes 2 \ 
--nodes-min 1 \ 
--nodes-max 4 \ 
--managed

This command creates a cluster named kubevirt-test in the eu-west-1 region with two c5.metal worker nodes.
Verify the Cluster: Once the cluster is created, verify its status using:

eksctl get cluster --region eu-west-1

Installing KubeVirt

With the EKS cluster up and running, the next step is to install the KubeVirt operator.

Deploying KubeVirt Operator

KubeVirt is typically installed using the KubeVirt Operator, which manages the lifecycle of KubeVirt components.
Add the KubeVirt Helm Repository:

helm repo add kubevirt https://kubevirt.io/helm-charts 
helm repo update

Install KubeVirt Operator:

helm install kubevirt kubevirt/kubevirt \ 
--namespace kubevirt \ 
--create-namespace \ 
--set operator.image.tag=v0.54.0 \ 
--set operator.image.pullPolicy=IfNotPresent

Verify the Installation:

kubectl get pods -n kubevirt

You should see the KubeVirt Operator pod running.

Verifying the Installation

To ensure that KubeVirt is installed correctly, deploy a simple VM and check its status.
Create a VirtualMachine Instance:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: test-vm
spec:
  running: false
  template:
    metadata:
      labels:
        kubevirt.io/vm: test-vm
    spec:
      domain:
        devices:
          disks:
          - name: containerdisk
            disk:
              bus: virtio
          - name: cloudinitdisk
            disk:
              bus: virtio
        resources:
          requests:
            memory: 64M
      volumes:
      - name: containerdisk
        containerDisk:
          image: kubevirt/cirros-registry-disk-demo
      - name: cloudinitdisk
        cloudInitNoCloud:
          userData: |
            #cloud-config
            password: password
            chpasswd: { expire: False }
Save this YAML to a file named test-vm.yaml and apply it:

kubectl apply -f test-vm.yaml

Start the VM:

virtctl start test-vm

Check the VM Status:

kubectl get vmi

Managing VM Lifecycle

KubeVirt provides several commands to manage the lifecycle of VMs:
# Access to the Virtual Machine
virtctl console test-vm 

# Stop the Virtual Machine
virtctl stop test-vm
# Restart the Virtual Machine :
virtctl restart test-vm
# Delete the Virtual Machine :
virtctl delete test-vm

Advanced Topics

Live Migration:
KubeVirt supports live migration of VMs, allowing you to move a running VM from one node to another without downtime.
Enable Live Migration:

 Update the KubeVirt CR to enable live migration:

apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
  name: kubevirt
  namespace: kubevirt
spec:
  configuration:
    developerConfiguration:
      featureGates:
        - LiveMigration

Migrate a VM:

 Use the virtctl command to migrate a VM:

virtctl migrate test-vm
Snapshots and Cloning
KubeVirt provides snapshot and cloning capabilities for VMs.
Create a Snapshot:


virtctl snapshot create test-vm --name test-snapshot
Clone a VM:


virtctl clone create test-vm --name test-clone

Best Practices

Running KubeVirt on AWS EKS introduces unique challenges due to the hybrid nature of managing both containers and virtual machines (VMs) in a single Kubernetes cluster. To ensure stability, security, and performance, follow these best practices:
Security Considerations:
Network Security & Isolation
Use Kubernetes Network Policies: Restrict VM-to-VM and VM-to-container communication using NetworkPolicy rules. 
For example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vm-isolation
spec:
  podSelector:
    matchLabels:
      kubevirt.io/vm: ""
  policyTypes:
  - Ingress
  - Egress
  ingress: []
  egress: []

This denies all traffic by default, requiring explicit rules for allowed communication.

Use Service Meshes for Advanced Security: Integrate Istio or Cilium to enforce mTLS between VMs and containers.


Enable Pod Security Policies (PSP) or OPA Gatekeeper: Enforce security policies to prevent unauthorized access to KubeVirt resources.

Isolate Sensitive VMs: Run VMs with sensitive workloads in dedicated namespaces with restricted RBAC permissions.

Authentication & Authorization
Implement RBAC (Role-Based Access Control): Restrict virtctl and KubeVirt API access to only authorized users. 
Example minimal role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: vm-production
rules:
- apiGroups: ["kubevirt.io"]
  resources: ["virtualmachines"]
  verbs: ["get", "list", "start", "stop"]


Use AWS IAM Roles for Service Accounts (IRSA): Ensure secure access to AWS services (EBS, EC2) from KubeVirt-managed VMs.

Enable KubeVirt Audit Logging: Monitor VM creation, modification, and deletion for compliance.

VM Hardening
Use Read-Only Root Disks: Prevent unauthorized modifications to VM images.

Disable Unnecessary Services: Minimize attack surfaces by disabling unused kernel modules and services inside VMs.

Regularly Patch VM Images: Apply security updates to base VM images (e.g., Ubuntu, CentOS) just as you would for containers.

Conclusion

KubeVirt is a powerful tool that extends Kubernetes capabilities to manage virtual machines alongside containers. By following this blogpost, you've learned how to install KubeVirt on an AWS EKS cluster, create and manage VMs, configure networking and storage, and explore advanced features like live migration and snapshots. With these skills, you can leverage KubeVirt to run VM-based workloads in a Kubernetes-native environment, unlocking new possibilities for your cloud infrastructure. As Kubernetes continues to evolve, the integration of virtual machines through tools like KubeVirt is becoming increasingly important. This convergence of containerized and VM-based workloads allows organizations to modernize their infrastructure without abandoning legacy applications. KubeVirt bridges the gap between traditional virtualization and cloud-native technologies, enabling a seamless transition to a hybrid environment.
As you continue to explore KubeVirt, consider experimenting with its advanced features, integrating it with other Kubernetes tools, and engaging with the community. Whether you're running legacy applications, specialized workloads, or building a hybrid cloud environment, KubeVirt on AWS EKS offers a powerful and flexible solution.