Recently, we switched to using Kubernetes for deploying the new version of our application at work. I love Kubernetes. It can handle so many different scenarios. And you can customize it to boot. The customization can take a few different forms, but the one I was most intrigued by were Custom Resource Definitions and Custom Controllers. There are few helpful tools and tutorials out there, but I thought I document my own progress here. I used Kubebuilder to help create the CRD and controller.

The Custom Resource and controller I decided to make was a configuration that would be used by a mutating webhook to inject a sidecar container into a pod. The primary use case I see for this (at least at my current place of employment) is to inject another container to watch the main container and expose metrics or alerts if errors are logged. Therefore, I cleverly called this resource a Watcher.

Run the following command in your GOPATHand project folder to initialize a new kubebuilder project. Replace my.domain with your project’s domain.

kubebuilder init --domain my.domain

This will scaffold out an entire project that will house a lot of boilerplate code for our project. The next step is to create a new API.

kubebuilder create api --group watch --version v1alpha1 --kind Watcher

In the above command, we pass in a Group (watch) and version (v1alpha1) and finally a name (Watcher) for our new resource. It will give you two prompts, one to create a resource and one to create a bare controller. Answer yes to both.

CRD Specification

Now comes the fun part, actually designing the resource specification.

My goal for the Watcher resource is to act as configuration template for injecting a sidecar container into a Pod (more on that in future posts). Therefore, we will need the following Spec object for the CRD. Kubebuilder allows us to define a CRD specification in the file ./api/v1alpha1/watcher_types.go. Find the struct WatcherSpec and edit like so.

// WatcherSpec defines the desired state of Watcher
type WatcherSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// Container definition for Watcher Sidecar container
	Container corev1.Container `json:"container"`

	// Optional Volume to add to Pod when injecting sidecar
	// +optional
	Volume corev1.Volume `json:"volume,omitempty"`

	// Optional ConfigMap Data to use with sidecar container
	// +optional
	ConfigData map[string]string `json:"configData,omitempty"`

	// Optional ConfigMap Binary Data to use with sidecar container
	// +optional
	ConfigBinaryData map[string][]byte `json:"configBinaryData,omitempty"`

	// Optional ImagePullSecrets for sidecar container
	// +optional
	ImagePullSecrets corev1.LocalObjectReference `json:"imagePullSecrets,omitempty"`
}

Let me explain what all this is. Note the json tags at the end of the line are important and tell the controller how to serialize/deserialize the struct.

• Container describes the configuration of the actual container to be injected. It is the exact same type as in a Pod Spec. This is the only required field in the WatcherSpec.
• Volume describes any volumes that need to be add to the Pod when the Watcher is being injected.
• ConfigData and ConfigBinaryData are any data that gets used to create a managed ConfigMap that the Watcher container will get env variables from.
• ImagePullSecrets is the same as when used in a Pod Spec.

That describes the Spec subresource of the Watcher CRD. But we must also define a Status subresource. We will only need to track the name of the Watcher Controller generated ConfigMap.

// WatcherStatus defines the observed state of Watcher
type WatcherStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	ConfigMapName string `json:"configMapName"`
}

Implementing the Controller

When we kubectl apply a Watcher manifest, the object as described in the YAML file gets stored by the Kubernetes API Server, and that essentially “creates” the watcher. However, we do want to have some additional functionality occur wheneve a Watcher resource is created/updated/deleted. Specifically, we would like to have a ConfigMap resource created and managed by our controller with the data specified in the Watcher Spec fields ConfigData and ConfigBinaryData.

Here is the code in the Reconcile function for the controller

func (r *WatcherReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
	ctx := context.Background()
	log := r.Log.WithValues("watcher", req.NamespacedName)

	var watcher watchv1alpha1.Watcher
	if err := r.Get(ctx, req.NamespacedName, &watcher); err != nil {
		log.Error(err, "unable to find Watcher")
		return ctrl.Result{}, ignoreNotFound(err)
	}

	// Get the previously created Watcher ConfigMap if it exists
	cfgKey := types.NamespacedName{
		Name:      fmt.Sprintf("%s-config", watcher.Name),
		Namespace: watcher.Namespace,
	}

	constructConfigMap := func(w *watchv1alpha1.Watcher, cfgName types.NamespacedName) (*corev1.ConfigMap, error) {
		cfg := &corev1.ConfigMap{
			ObjectMeta: metav1.ObjectMeta{
				Name:      cfgName.Name,
				Namespace: cfgName.Namespace,
			},
			Data:       w.Spec.ConfigData,
			BinaryData: w.Spec.ConfigBinaryData,
		}

		if err := ctrl.SetControllerReference(w, cfg, r.Scheme); err != nil {
			return nil, err
		}

		return cfg, nil
	}
	cfg, err := constructConfigMap(&watcher, cfgKey)
	if err != nil {
		log.Error(err, "unable to create ConfigMap from Watcher")
		return ctrl.Result{}, err
	}

	shouldUpdate := false
	createErr := r.Create(ctx, cfg)
	if apierrs.IsAlreadyExists(createErr) {
		shouldUpdate = true
	}

	if shouldUpdate {
		var configMap corev1.ConfigMap
		if err := r.Get(ctx, cfgKey, &configMap); err != nil {
			log.Error(err, "unable to get ConfigMap")
			return ctrl.Result{}, ignoreNotFound(err)
		}

		if !reflect.DeepEqual(configMap, cfg) {
			retryErr := retry.RetryOnConflict(retry.DefaultRetry, func() error {
				var result corev1.ConfigMap
				if err := r.Get(ctx, cfgKey, &result); err != nil {
					return err
				}
				result.Data = cfg.Data
				result.BinaryData = cfg.BinaryData
				updateErr := r.Update(ctx, &result)
				return updateErr
			})
			if retryErr != nil {
				return ctrl.Result{}, retryErr
			}
		}
	}

	watcher.Status.ConfigMapName = cfgKey.Name
	if err := r.Status().Update(ctx, &watcher); err != nil {
		log.Error(err, "unable to update Watcher status")
		return ctrl.Result{}, err
	}

	return ctrl.Result{}, nil
}

Basically we just check if a ConfigMap exists, if not create one else update it. Then we set the name of the ConfigMap to the WatcherStatus and we are done.

Next part will be the MutatingWebhook to inject the WatcherSidecar container.