AWS Resource Access with
IAM Role from Kubernetes

Today, we will talk about AWS resources access methods from Kubernetes. We will cover two ways to this. These methods are kube2iam and kiam. They provide access to AWS resources without extra AWS’s credentials on your cluster, and they perform with IAM access to nodes of your cluster.

Why do we need Kube2iam or Kiam ?

Many products are developed on AWS and Kubernetes today. They need to access AWS’s resources from the applications running inside a Kubernetes cluster on AWS. However, this isn’t the right way to give access directly to credentials. So we shouldn't use credentials directly on the cluster. Thanks to Kube2iam and Kiam, Kubernetes applications can gain access to AWS resources through the AWS IAM role.

Kube2iam

kube2iam was the first project that was created for accessing AWS resources on Kubernetes clusters with the IAM role. First, let’s explain how it works. kube2iam runs as a daemonset on your cluster. It runs on each worker with hostNetwork: true (in the host network). Kube2iam daemon and iptables rules need to run before all other pods will require access to AWS resources. Iptables prevent pods from directly accessing the EC2 metadata API and unwanted requests for access to AWS resources. So the traffic to the IAM service of AWS must be proxied for pods.

We assume an existing role and create a new role for pods annotations. In this way, our pods access AWS resources by assuming the role.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  labels:
    name: my-app
  annotations:
    iam.amazonaws.com/role: role-arn

Kube2iam supports namespace restrictions. So if we have a specific namespace on our cluster and by using the flag --namespace-restrictions to provide to assume a role for pods, we can enable this mode by an annotation on the pod’s namespace.

Let’s do an example;

We will use kops for creating a k8s cluster and use the python boto3 framework for the AWS s3 buckets list with kube2iam.

First, we install kube2iam with helm chart;

helm install stable/kube2iam namespace kube-system \
  --name kube2iam \
  --set=extraArgs.base-role-arn=arn:aws:iam:<accoun_id>:role/<role_name> \
  --set extraArgs.default-role=kube2iam-default \
  --set host.iptables=true \
  --set host.interface=<our_cluster_virtual_network> \
  --set rbac.create=true

After that, we create the AWS IAM policy to allow nodes to assume different roles;

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": [
                "arn:aws:iam::xxxxxxxxxxxx:role/k8s/*"
            ]
        }
    ]
}

We create the roles that the pods can assume. For example, Thanks to the role and AmazonS3FullAccess policy we created above, we got access to listing s3 bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxxxxxxxxx:role/nodes.k8s.example.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Finally, we create a python script with the boto3 framework to list S3 buckets on AWS.

import boto3

client = boto3.client('s3')

response = client.list_buckets()

for i in response['Buckets']:
    print([i['Name']])

In this script, we take the Buckets in the response variable with boto3. Then we create a deployment for k8s.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kiam-demo
spec:
  template:
    metadata:
      labels:
        app: kiam-demo
      annotations:
        iam.amazonaws.com/role: arn:aws:iam::xxxxxxxxxxxx:role/k8s/kube2iam-demo
    spec:
      containers:
      - name: kiam-demo
        image: halil9/kube2iam-demo

Kiam

Kiam is a project inspired by the kube2iam project. It provides access to AWS resources with an IAM role on our cluster like kube2iam. It is developed to resolve the shortcomings of kube2iam. Kiam works to resolve issues of kube2iam. It has two components (a server and an agent); these components run as a daemonset on our cluster. Kiam agent doesn’t directly communicate with IAM, Kiam agent communicates with IAM through the server component. Server components prefetch credentials from AWS. This optimization reduces response times for SDK clients. We will assume an existing role and will create a new role for pods annotations.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  labels:
    name: my-app
  annotations:
    iam.amazonaws.com/role: role-arn

kiam has a server and agents as shown below. The server communicates with IAM with k8s master role for apps on the agent.

Kiam deployment is more complicated than kube2iam. We must define IAM resources for communication between Kiam components on our cluster and AWS. We can provision IAM resources with terraform modules at Kiam GitHub repository and before when we install Kiam, we must define a certificate (helm chart generates a self-signed TLS certificate by default) for communication between Kiam components(server and agent). If you want to create and install your own, you can create TLS certificates and private keys as described here. (Show details)

If we want to install Kiam, we can use helm charts deployment ;

helm install stable/kiam namespace kube-system --name kiam \ 
  --set=extraArgs.base-role-arn=arn:aws:iam::<account_id>:role/<role_name> \
  --set extraArgs.default-role=kube2iam-default \
  --set host.iptables=true,host.interface=<our_cluster_virtual_network> \
  --set rbac.create=true

And Then;

We can follow the same steps I outlined above for the s3 buckets listing example.

Conclusion

Here is a table that compares the two solutions.

If we evaluate the two projects in similar situations;

• Installation:

  kube2iam can be installed easily with the helm chart, but kiam agent can be tricky during installation.

• Security: 

  Kiam is better than kube2iam because Kiam has two components (server and agent) and nodes can’t directly communicate with AWS through this feature, only the server communicates with AWS.

• Contribution: 

  Kiam GitHub's contribution is more frequent than kube2iam GitHub's contribution.

• Performance: 

  This depends on your choices. For example, if security or up-to-dateness is important for your case, you should use Kiam. But if the easy installation is important for you, you should use kube2iam. Kiam is better than kube2iam with respect to technical performance.

Resources;

https://github.com/jtblin/kube2iam

https://github.com/uswitch/kiam