Managing Kubernetes Clusters with the GitOps

Kubernetes is a container management platform that includes many components. There are many problems to be solved under different headings such as installation, configuration, maintenance, and observability in K8S management. In this article, we will talk about how the necessary components such as add-ons, tools, etc. can be managed with GitOps management for the cluster to become production-ready after installation.

What is GitOps?

GitOps uses Git repositories as a single source of truth to deliver applications. Submitted code checks the CI process. All code changes are tracked, making updates easy while also providing version control should a rollback be needed. 

GitOps delivers:

• A standard workflow for application development
• Increased security for setting application requirements upfront
• Improved reliability with visibility and version control through Git
• Consistency across any cluster, any cloud, and any on-premise environment

GitOps tools constantly check the git repos you define and ensure that the relevant environments are synchronized with the git repos.

Why Do We Need to Manage Kubernetes with GitOps Patterns?

If the scale you are working on is small, you can choose any method for kubernetes add-on installation, maintenance, and configuration. You can even manage them manually. However, when the scale grows and the systems you have to manage start to be expressed in 10s and 100s, things start to change. At this point, you can start to benefit from the blessings provided by GitOps. 

• Thanks to the GitOps tool, you can simultaneously deploy the same code or configuration to n clusters. 
• Environment provisioning operations can be done with just a few code changes.
• You can follow the status of the systems you work with and all changes via git.
• Since the GitOps tool will constantly synchronize, you will be able to make sure that the production configuration is stable/reliable.
• Since you can create all components with bootstrap logic, you can manage all installation and maintenance processes automatically.

Setting up GitOps Repository Directory Structure

First of all, ArgoCD was preferred as a GitOps tool. ArgoCD is a Kubernetes-native continuous deployment (CD) tool. Unlike other CD tools that only enable push-based deployments, ArgoCD can pull updated code from Git repositories and deploy it directly to Kubernetes resources. ArgoCD was preferred for many reasons such as having a user interface, easy GitOps implementation, being a scalable and stable product, etc.

Two issues need to be decided to determine the repository structure. These are: How to position GitOps tool? Monorepo or multirepo?

1. How to Position GitOps Tool? 

          2 different positions can be used when using ArgoCD:

One ArgoCD To Rule Them All Clusters

With this architecture, it is possible to control all clusters with a central ArgoCD. 

Advantages: 

• Single view for deployment activity across all clusters.
• Single control plane, simplifying the installation and maintenance.
• Single server for easy API/CLI integration.
• Great integration with the ApplicationSet cluster generator.

Disadvantages: 

• Scaling requires tuning of the individual components.
• Single point of failure for deployments.
• Admin credentials for all clusters in one place.
• Significant network traffic between Argo CD and clusters.

ArgoCD Instance Per Cluster (The Separation Of Concern)

In this architecture, a new ArgoCD is configured on each cluster to be managed.

Advantages:

• Distributes load per cluster.
• No direct external access is required.
• Eliminates the Argo CD traffic leaving the cluster.
• An outage in one cluster won't affect other clusters.
• Credentials are scoped per cluster

Disadvantages:

• Requires maintaining multiple instances and duplicating configuration.
• API/CLI integrations need to specify which instance.
• Limited integration with the ApplicationSet cluster generator.

2. Monorepo and Multirepo

Monorepo 

Monorepo means using a single git repository for clusters or environments. It becomes difficult to manage when systems are scaled. When you want to make changes to any environment, ArgoCD will re-render the entire structure, which can cause performance problems. On the other hand,  the advantage is that this pattern provides a centralized location for all your configuration changes and deployment.

Multirepo

In the multirepo model, different git repos are used for clusters or environments. In the meantime, repositories can be divided for each cluster and environment, as well as for the separation of concerns and organizational boundaries. In addition, the multirepo approach can be preferred in multi-tenant structures.

The main drawback of using this pattern is that it creates a large number of Git repositories, with each having its release process that needs to be coordinated. This makes it a challenge to manage, and deployments can become complex. However, this pattern is flexible and scales incredibly well. 

As a result, the answer to the question of what type of git repo to choose is simple: it depends. Solutions can be produced in mono-repo or multi-repo structures by considering criteria such as customer, business, performance, management, etc.

In our study, each cluster is configured to be managed by the ArgoCD instances on it. In addition, a single git repo and directory structure is created for clusters in different environments using the mono-repo approach.

Implementation

In the implementation to be made, the installation and management of cluster tools will be done declaratively using Helm templates and Kustomize. ArgoCD support Helm and Kustomize. Standardized deployments of a wide range of Kubernetes add-ons and tools will be made using Helm charts. With the use of Kustomize, the structure created by the DrY (do not repeat yourself) principles can be easily implemented for each environment/cluster.

In this context, it becomes important to set up the correct directory structure in the git repo when managing Kubernetes with ArgoCD. 

Directory Structure

In the mono-repo approach, when creating a directory structure using Kustomize, two main folders are created. These are the bootstrap and components files.

.
├── README.md
├── bootstrap
│   ├── app-of-apps
│   └── initial
└── components
    ├── argocd
    ├── backing-services
    ├── cluster-certificates
    ├── cluster-core
    ├── cluster-logging
    ├── cluster-repo-creds
    └── namespaces

The app-of-apps folder contains the resources that will trigger the installations like dominoes, so to speak, and then continuously monitor and synchronize them.

For example, let's consider the argocd folder to examine the folder structure of a component. 

.
├── base
│   ├── kustomization.yaml
│   └── patches
│       └── argocd-cm.yaml
└── overlays
    └── prod
        ├── ingress.yaml
        ├── kustomization.yaml
        ├── patches
        │   ├── argocd-cm.yaml
        │   ├── argocd-cmd-params-cm.yaml
        │   └── argocd-rbac-cm.yaml
        └── service-monitors.yaml

The component structure is completely organized according to Kustomize’s declarative folder structure. While the resources are defined in the base folder, all in YAML format, only the fields that will patch according to the environment/cluster can be defined under the overlays folder. With Kustomize, files can be referenced at the directory level paths as well as URL-based. The kustomization.yaml file under the base file in the structure above shows this example.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: argocd
resources:
- https://raw.githubusercontent.com/argoproj/argo-cd/v2.11.0/manifests/install.yaml

patches:
- path: patches/argocd-cm.yaml

The ArgoCD installation is defined in the structure above. Since ArgoCD can be customized with configmaps, the resource to be customized while installing ArgoCD is defined in the patches section.

By opening a file according to the environment or cluster to be established under overlays, the patches to be made specific to the relevant environment are added to the files under the patches directory. For example, the patches to be made for ArgoCD to be deployed to the production environment are defined in overlays/prod/kustomization.yaml.

# components/argocd/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: argocd
resources:
  - ../../base/
  - ingress.yaml
  - service-monitors.yaml

patches:
  - path: patches/argocd-cm.yaml
  - path: patches/argocd-cmd-params-cm.yaml
  - path: patches/argocd-rbac-cm.yaml

Thus, the relevant component can be deployed by making customizations to n environments with the folder structure created for the component to be installed.

Bootstrapping

Installations begin with a one-time manual installation of ArgoCD and dependent components that will manage everything with the initial folder located under the bootstrap folder.

.
├── README.md
├── bootstrap
│   └── initial
│       ├── 00-namespace
│       │   └── kustomization.yaml
│       ├── 01-argocd
│       │   └── kustomization.yaml
│       ├── 02-cluster-core
│       │   └── kustomization.yaml
│       ├── 03-cluster-repo-creds
│       │   └── kustomization.yaml
│       └── 04-app-of-apps
│           └── kustomization.yaml

We have to do the installations here manually, one-time. Because ArgoCD and the tools it requires, which will later manage everything, must first be deployed once. Installations will be carried out one by one according to the numbered order in the structure shown above. For example, firstly, kustomization.yaml under 00-namespace folder will be deployed to create the relevant namespaces.

# bootstrap/initial/00-namespace/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../../components/namespaces/base
--- 
# components/namespaces/base/kustomization.yaml.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - argocd-ns.yaml
  - external-secrets-ns.yaml
  - ingress-nginx-ns.yaml

The first kustomization file above references the namespace resources in the namespace folder under components. By applying this file, the namespaces will be installed.

Kubectl apply -k bootstrap/initial/00-namespace 

Note: Kustomization resources can be deployed using kubectl with the “-k” parameter without the need for a different tool.

All resources in the initial folder are deployed in order and the installation of the requirements required to automate the structure is completed.

kubectl apply -k bootstrap/initial/00-namespace
kubectl apply -k bootstrap/initial/01-argocd
kubectl apply -k bootstrap/initial/02-cluster-core
Kubectl apply -k bootstrap/initial/03-cluster-repo-creds

In summary, to install ArgoCD and make the created git repository accessible to argocd,

• Setting up namespaces 
• Installing ArgoCD (in its most basic form)
• The external-secret operator will generate the secret that ArgoCD will use to access the private git repo.
• cluster-repo-creds and external-secret templates that will generate private git repo secrets for ArgoCD.

After manual installations, app-of-apps deployment should be done which will automate everything. Let's examine the app-of-apps structure.

# bootstrap/app-of-apps
.
├── base
│   ├── argocd.yaml
│   ├── backing-services.yaml
│   ├── bootstrap.yaml
│   ├── cluster-certificates.yaml
│   ├── cluster-core.yaml
│   ├── cluster-logging.yaml
│   ├── cluster-repo-creds.yaml
│   ├── cluster-traffic.yaml
│   ├── kustomization.yaml
│   └── namespaces.yaml
└── overlays
    ├── preprod
    │   └── kustomization.yaml
    └── prod
        └── kustomization.yaml

A directory structure was created under the app-of-apps file using kustomize. Under the base file, ArgoCD applications are defined for each component to be deployed in the cluster. For example, let's examine the cluster-core.yaml file.

# bootstrap/app-of-apps/base/argocd.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-core
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-1000"
spec:
  destination:
    namespace: argocd
    server: "https://kubernetes.default.svc"
  source:
    repoURL: "https://github.com/kloia/platform-gitops.git"
    targetRevision: "HEAD"
    path: "components/cluster-core/PATCH_ME"
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
     - CreateNamespace=true

With the ArgoCD application defined in this file, the cluster-core components in the components/cluster-core path in the platform-gitops repo are deployed and then continuously monitored by ArgoCD. Thus, the relevant component will be continuously synchronized with the repo. In a similar logic, an ArgoCD application is defined for each component under the base folder that will manage them.

Another application has been defined that will manage all applications of the system (app of apps) and also control itself. This application is the ArgoCD bootstrap application and is defined in bootstrap.yaml under the base directory. 

# base/bootstrap.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: bootstrap
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-997"
spec:
  destination:
    namespace: argocd
    server: "https://kubernetes.default.svc"
  source:
    repoURL: "https://github.com/kloia/platform-gitops.git"
    targetRevision: "HEAD"
    path: "bootstrap/app-of-apps/PATCH_ME"
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

This application will ensure that all applications located under the bootstrap/app-of-apps directory are deployed and then automatically synchronized with the auto sync feature. Since this application is located in the same directory, it also synchronizes itself.

A mechanism has been developed here to be able to deploy applications to multiple environments/clusters without repeating code. To explain with an example, for the cluster-core component to be deployed to the pre-prod environment, the expression indicating the component path in the form of "components/cluster-core/PATCH_ME" in the ArgoCD application must be customized specifically for each environment/cluster under the overlay folder. In the application to be deployed to the preprod environment, the path must be "components/cluster-core/preprod". To perform the mentioned changes in the application yaml files, the bootstrap/app-of-apps/overlays/preprod/customization.yaml file was created. Let's examine this file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: argocd
resources:
  - ../../base/

commonAnnotations:
  gitops.kloia/overlay-path: "overlays/preprod"

replacements:
  - source:
      group: argoproj.io
      version: v1alpha1
      kind: Application
      name: bootstrap
      namespace: argocd
      fieldPath: "metadata.annotations.[gitops.kloia/overlay-path]"
    targets:
      - select:
          group: argoproj.io
          version: v1alpha1
          kind: Application
        fieldPaths:
          - "spec.source.path"
        options:
          delimiter: "/"
          index: 2

In this file, “commonAnnotation” is added as an annotation to each application located under base directory and annotation defines the environment (for example overlays/preprod). Afterward, using the “replacements” feature of kustomize, the last part of the “spec.source.path” field of the bootstraper application is replaced with the “overlay/preprod” expression in the previously added annotation.

As a result, the expression "bootstrap/app-of-apps/PATCH_ME" in bootstrapper.yaml as the path where the applications will be deployed becomes "bootstrap/app-of-apps/overlays/preprod". Thus, we can trigger the environment you will initialize with a single command.

After making all these adjustments, for example, to start the configuration of the preprod cluster, it will be sufficient to set the context to be preprod with kubectl and apply the bootstrapper application with a single command.

kubectl apply -k bootstrap/app-of-apps/overlays/preprod

To trigger the installation of the prod environment, apply the prod overlay with kubectl after running previously mentioned manual steps.

kubectl apply -k bootstrap/app-of-apps/overlays/prod

The Bootstrapper application will deploy all applications in order. And the triggered applications will trigger the installation of the components they are responsible for. In a short time, the entire structure will rise to govern itself. Once the installation is complete, the applications will appear on the ArgoCD UI as follows.

For example, other applications installed with the cluster-core application will look like this.

Conclusion

In this study, how Kubernetes addon and tool management can be done using the GitOps method, how GitOps tools can be positioned in the application architecture and git repository structures are discussed. Afterwards, it was shown how a self-initiating and self-managing structure can be implemented on ArgoCD using Helm and Kustomize with the created directory structure. You can access all the codes used in this article at https://github.com/kloia/platform-gitops.

References

1. Gitops Best Practices
2. App of Apps