In this post, I will talk about the background of debug, a new alpha feature in the kubectl.
In Kubernetes, each pod runs as a docker container and these containers are isolated through the Linux namespace isolation feature.
Each container has its own process filesystem, as shown below.
$ docker exec nginx ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.1 0.0 10632 5344 ? Ss 13:53 0:00 nginx: master process nginx -g daemon off;
nginx 6 0.0 0.0 11088 2664 ? S 13:53 0:00 nginx: worker process
root 18 0.1 0.0 2388 696 pts/0 Ss 13:54 0:00 sh
This is the output of a healthy process and everything looks good and we can attach pods via exec command .
As you can see in the output, nginx process works as PID 1. If this process crashes, it is impossible to attach to this container via kubectl exec.
So how can we debug our applications when their main process crashes?
Solution of Kubernetes
Kubernetes 1.17 supports ephemeral containers, and this capability makes it possible to share PID namespaces.
Ephemeral containers allow injections to IPC namespaces and you can run debug commands on your crashed pods.
To use this capability, you need an ephemeral container definition, which is shown below:
{
"apiVersion": "v1",
"kind": "EphemeralContainers",
"metadata": {
"name": "example-pod"
},
"ephemeralContainers": [{
"command": [
"sh"
],
"image": "busybox",
"imagePullPolicy": "IfNotPresent",
"name": "debugger",
"stdin": true,
"tty": true,
"terminationMessagePolicy": "File"
}]
}
Then, you need to attach to the IPC namespace of your container with this command:
kubectl replace --raw /api/v1/namespaces/default/pods/java-app/ephemeralcontainers -f ephemeral.json
What does `kubectl debug` do ?
The new debug feature automates ephemeral container attaching and namespace sharing with a single command and without any manifest files. Even if your cluster version is 1.15, 1.16 or 1.17, you should enable the ephemeral container feature. But with version 1.18, this feature comes by default.
I have a created sample cluster configuration file for kubeadm that enables feature-gates on Kubernetes clusters.
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
featureGates:
EphemeralContainers: true
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.18.0
apiServer:
featureGates:
EphemeralContainers: true
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.18.0
controllerManager:
featureGates:
EphemeralContainers: true
Summary
As you can see at above, you need to enable EphemeralContainer and write another JSON manifest file to use debug containers in previous versions. But on version 1.18 kubectl, you can officially complete this flow with only one CLI argument.
kubectl alpha debug -it java-app-failing --image=busybox --target=failing-java-app
Kubernetes environment is improving itself day by day. These new debug capabilities make debugging easy on your pods. We are excited for the upcoming new features in Kubernetes.
Have a nice day, stay at home and support the Kubernetes community!