Developing and maintaining services on kubernetes can be quite cumbersome when you surpass an (arbitrary) threshold of total services deployed. Although it can still be manageable even after you surpass that threshold of services, having a kustomizable templating solution that you can apply to different projects or even just for simple staging and production environments is a must!

Thankfully, several solutions developed by multiple parties exist to allow you to customize your kubernetes deployments:

• Ansible Kubernetes module
• Forge by Datawire.io
• Helm Charts, initially mainly by Bitnami and Deis (now: Microsoft)
• k8comp by cststack
• kedge by Red Hat
• kdeploy by Flexiant
• kpm by Coreos (note: no longer developed or maintained)
• ksonnet, kubecfg by Bitnami and Heptio
• ktmpl by InQuicker
• kubeapps.com by the Kubernetes community
• Kubestack
• kubegen by Ilya Dmitrichenko of Weaveworks
• kustomize by Kubernetes Authors
• OpenCompose by Red Hat
• OpenShift Templates by Red Hat
• Puppet Kubernetes module by Gareth Rushgrove of Puppet
• Terraform
• Yipee.io by Yipee.io

There are more solutions for customizing k8s manifests, but, from these, the most popular ones certainly are Helm charts and kustomize. Both help you to customize your kubernetes resources, but they do it in different ways and they are more complementary than self-excluding: helm charts are template based while kustomize uses generators and transformers to change the configurations of resource’s yaml manifests via inheritance.

In this post, we will dig into kustomize only and explain what makes it great for customizing and managing your services’ deployments onto kubernetes.

Kustomize comes bundled with kubectl as of 1.14, but some features used here require you to have kustomize standalone installed on your system.

What is kustomize?

Kustomize is a tool (now bundled with kubectl) to customize Kubernetes objects through a kustomization.yaml file. It has some really nice features to manage application configuration, which we can bundled them in the following three core functionalities:

• generating resources from other sources (e.g., generate configmaps or secrets directly from files)
• composing and customizing collections of resources (e.g., lets bundle a database with a connection pool and a db management tool)
• setting fields for resources (e.g., change the default value of an environment variable)

By using a mix of generators, transformers and meta information, we can pretty much customize any key and/or value of any manifest and compose multiple resources into a single deployment with a single command. More on this later.

What makes kustomize so special with respect to the other solutions?

The biggest advantage of kustomize (in my opinion) is that it comes bundled with the latest releases of kubectl. While this may not be its strongest point for some, having it bundled with kubectl means that you and your team can modify and generate resources with ease and quickly deploy them while developing a solution with little to no friction.

This is great when iterating on the development of the infrastructure with different stakeholders (DevOps, Software Engineers, Data Scientists, etc.), mainly because it lowers the bar of entry for less knowledgeable members of your team to be able to contribute to the development of the infrastructure of an ongoing project.

But this is just one of its main strengths. With kustomize, you can easily compose and bundle different resources like databases, monitoring and logging services, web-apps, ETL processes, etc., that may be common to multiple projects and are hosted on their own repos, and modify any configuration you want like with ease. For example, set a pod’s resource limits to use on a test environment or setting the secrets used in staging and producting requires only a patch to the original configuration for those specific fields, leaving all the other fields unchanged. Also, maintaining a sane organization of these resources in a verifiable way with a single source of truth is one of kustomize’s major wins. Moreover, all you need to know is how to write yaml / json configuration files!

The ease of use and ease of modification is what makes kustomize so powerful for projects or any size.

A small warning

Although kustomize comes bundled with kubectl since 1.14+, the current version bundled as the time of writting is an older version of kustomize (v2.0.3), so some stuff may not work as expected with respect to the latest versions available. However, this is, in my opinion, not a huge deal breaker because most of the functionality you need comes with the version bundled with kubectl and it can get you pretty far without having to use the latest features, unless you have a more complex setup. For some more complex use cases, you will have to use the latest versions of kustomize, but this is being tackled by the development team.

Therefore, with this in mind, depending on what tool you are using, you should check the corresponding documentation resource:

• if you are using kustomize bundled with kubectl, I suggest you to check the kubectl book, more specifically the reference section;
• if, instead, you are using the standalone kustomize, check here for the latest, most up-to-date information of features and usage examples.

Using kustomize: a simple example

Lets see how we can kustomize to manage a project’s infrastructure. One of its great features is its composability capabilities which allows you to start simple but also allow you to keep on adding more and more resources into a project without getting in your way.

Say you are looking to deploy a single website using nginx.

# app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.19
        ports:
        - containerPort: 80

To deploy this, all you need to do is run kubectl create -f app.yaml and you are set. You probably would add a service on top of this deployment to make it easier to access this service with an internal dns.

# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80

Again, kubectl create -f service.yaml is all you need to deploy it on your kubernetes cluster. Simple.

So far, adding kustomize to the mix would not be much of an improvement, but it does not take you that much more effort to use it. For that, we need to add a kustomization.yaml file to the mix to allow us to combine these two resources into a single resource manifest:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- app.yaml
- service.yaml

This config file is grouping these two resources effectively into a single once you compile the resultign manifst with kubectl kustomize > deploy.yaml. The resulting deploy.yaml file contains the following output of the compilation:

# deploy.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.19
        ports:
        - containerPort: 80

Not big of an improvement so far, but at least you gained a bit in code organization. But lets now start using some of the features that makes kustomize worth while the effort.

For example, say you want to change the image version of your deployment. You could do it by: 1. changing the image tag in app.yaml 2. define the image tag in kustomization.yaml

For this simple example, this is not much relevant. But for cases where deployments have multiple images and you want to specify a tag for each one, it becomes more readable if they are defined in a single place compared to multiple locations on multiple files. To accomplish this, we’ll add the following to our kustomization.yaml.

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- app.yaml
- service.yaml

images:
- name: nginx
  newTag: 1.19.1

Quite readable to be honest. Furthermore, other features at our disposable in kustomize like number of replicas, configmap and secret generatores, patches and overlays, among others, provides us with some neat quality of life features that makes kustomize so appealing (at least for me). It makes big deployments with lots of resources and dependencies more manageable at the end of the day without losing flexibility or having to jump through too many hoops (it is not common to face one, but is manegeable without much fuss).

Some issues with kustomize

There are a few issues I’ve had with kustomize that are not a big of a deal but fixing them would help improve its usage.

One issue I had with configMapGenerator was that you can’t give a file a different name on the generated configMap. By default, kustomize’s configMapGenerator uses the name of the file in the configmap. But say you have different files with the same name but organized in a different dir structure:

configMapGenerator:
- name: db
  files:
  - sql/database/sensor/create.sql
  - sql/database/metrics/create.sql

Because both have the same name, the second one will override the first one. Kustomize does not take this into account when creating the configMap, nor does it allow you to give them different names. A quick workaround you can use is just to use different configmap names to solve this:

configMapGenerator:
- name: db-sensor
  files:
  - sql/database/sensor/create.sql
- name: db-metrics
  files:
  - sql/database/metrics/create.sql

This is not the only issue you may encounter when using kustomize, but usually you can get away relatively easily like as in the previous example.

However, one issue that cannot be worked around is that the implicitness declarative style of kustomize is defined may make it harder to figure out what is being combined at a glance. This, though, is not really a big issue because, by compiling the kustomization.yaml file and manually checking what is being generated, you will know for sure what is being bundled as resources. If this issue is too much of an issue for you, I suggest maybe going with Helm3 which is a great solution for explicit definition of kubernetes resources.

Note: Always take a look at the generated manifest so you don’t get surprised if something is missing or misconfigured before deploying it. It will save you some white hairs.

Conclusion

Kustomize offers a nice way to structure your kubernetes resources. It provides you with plenty of functionality, while allowing you to use vanila kubernetes yaml config files you already know. If you are considering using kustomize, please take into account that, depending on your type of usage, you may very well just use the one bundled with kubectl and not have to install an additional tool. However, if you have a more complex use-case, or you need the latest features, you will have to use the latest installment of kustomize. Hopefully, this will be solved in a future version of kubectl where a more recent version of kustomize gets bundled together.