ALL THINGS KUBERNETES

Custom Resources and Custom Resource Definitions (CRDs) in Kubernetes

Among the best features of Kubernetes are its infinite “abilities:” pluggability; configurability; extensibility.

Kubernetes users have multiple options for extending K8s with their vendor APIs, custom resources, and infrastructure solutions via useful extension points, including:

  • kubectl plugins that extend the kubectl binary.
  • Storage plugins. Kubernetes supports Flex Volumes and Container Storage Interface (CSI), which allow users to mount volume types without built-in support in Kubernetes. With CSI, for instance, users can use a cluster-agnostic storage interface suitable for all major container orchestration platforms.
  • API access extensions. Users can utilize extension points in the kube-apiserver to customize authentication of user requests.
  • Scheduler extensions. Users can completely replace the K8s scheduler with their own or extend the behavior of the default scheduler. One example of the scheduler extension implementation is the Portworx STORK storage-aware scheduling that adds storage hyperconvergence, health monitoring, snapshot-lifecycle features, etc. to the default scheduling behavior.
  • Network Plugins. Network plugins like Cilium, Weave Net, Flannel etc. allow for different implementations of pod networking.
  • Custom Resources. Users can extend Kubernetes API with their custom resources. These resources can be managed by custom controllers similarly to Pods, Deployments, and other built-in K8s API resources.

We’ll focus in this post on the concept of custom resources in Kubernetes and how you can use them to extend Kubernetes with the user-defined declarative APIs and controllers, and we’ll also show how to define and use a new custom resource in the K8s API. Let’s get started!

What Are Custom Resources?

Custom resources let users define new application configuration objects and controllers that extend the Kubernetes API and which can be managed by client tools like kubectl  and use many native Kubernetes features.

In particular, with custom resources you can:

  • Use K8s watch operations via HTTP.
  • List, display and edit new objects using kubectl and/or Kubernetes dashboard.
  • Use HTTPS with the new API endpoints.
  • Leverage built-authentication and reuse the authorization used by the kube-apiserver (e.g., RBAC).
  • Use finalizers to block deletion of resources until the cleanup.
  • Use admission webhooks. Users can validate extension resources during any CRUD operations.
  • Use common metadata such as labels and annotations.

When to Use Custom Resources?

Although K8s custom resources are extremely powerful, you should not always use them.

For example, it’s preferable to use a ConfigMap  when there exists a well-documented configuration file format like ( httpd.conf ) or when the configuration you want to create is to be consumed by a program running in the container in your Pod.

Custom Resources are a good fit, however, when you want to use a declarative API. This type of API allows defining a desired state of an application and create controllers for maintaining it. Kubernetes native API resources like Deployments and StatefulSets are implemented with this API style. The following prerequisites are met to make the API declarative:

  • API objects define application configuration.
  • They are updated relatively infrequently.
  • Transactions across objects are not required.
  • The main operations on your objects are CRUD-y (creating, reading, updating, deleting).

The above-described approach contrasts with the Imperative API where the client says “do this,” and then gets a synchronous response back when it is done.

Custom resources that use declarative API are also useful in case you:

  • Want to use kubectl or view new resources in the K8s UI.
  • Want to use Kubernetes API conventions like .spec , .status , and .metadata .
  • Want your objects to be an abstraction over a collection of controlled resources.
  • Your API resources are naturally scoped to a cluster or to namespaces of a cluster.

However, a custom resource by itself only stores and retrieves structured data. A controller that maintains the desired state of custom resources is needed to make an API extension fully declarative. K8s native resources like Deployments and Pods use such controllers. (Using custom controllers in Kubernetes will be discussed in a subsequent article.)

You can create custom resources in Kubernetes using:

  • Custom Resource Definitions (CRDs).
  • API aggregations (Aggregate API). Aggregated APIs are user-defined APIServers that sit behind the primary kube-apiserver, which acts as a proxy.

In the following, we’ll discuss how to create a custom resource using a CRD manifest.

Tutorial: Creating a Custom Resource via CRD

To complete examples in this tutorial, you’ll need:

  • Kubernetes >= 1.7.0 (CRDs were introduced in K8s 1.7.0). See Supergiant documentation for more information about deploying a Kubernetes cluster with Supergiant. As an alternative, you can install a single-node Kubernetes cluster on a local system using Minikube.
  • A kubectl command line tool installed and configured to communicate with the cluster. See how to install kubectl here.

Note: you can find all manifests and code snippets used in this tutorial here.

CustomResourceDefinition (CRD) is an API object that allows users to create custom resources in Kubernetes, and they were introduced in K8s 1.7. Below is the example of the CustomResourceDefinition  that extends the K8s API with a new resource kind named MyResource  and a new API group named stable.example.com

There are several fields to discuss here:

metadata.name  — a CRD’s name. It must match the spec fields below, and be in the following form: ..

spec.group  — the group the custom resource belongs to. The group will be reflected in the REST API as /apis/stable.example.com/v1 .

spec.versions  — users can list all supported versions of the custom resource in this field. Each version you specify can be enabled/disabled by the served  flag. In our example, we use only one version ( v1 ). You could use spec.version  field if you have only one version of your API but it’s now deprecated.

spec.names  — the names used to describe a custom resource. You can use plural, singular, and short name forms to manage a custom resource via kubectl. For example, kubectl get myresources  , kubectl get myresource , or kubectl get mr .

spec.names.kind — is the kind of custom resource. For example, Deployment , CronJob , or CustomResourceDefinition  are the kinds of Kubernetes API resources.

spec.scope — defines whether a custom resource is cluster- or namespace- scoped. Default: namespaced

spec.validation  — describes custom validation methods for your custom resource. You can block the creation of resources using validation if certain fields are left empty. In this example, we used OpenAPIV3Schema validation conventions to check the type of some fields in our custom resource. We ensure that spec , spec.cert , spec.key , and spec.domain  fields of the custom resource do exist and that they are of a String type. Users can also use validatingadmissionwebhook as a validation schema. You can find more about restrictions for using this field in the official documentation.

Now that you understand what this CRD does, save the manifest to crd.yaml  and run:

This will create a new API endpoint at:

Let’s describe the new custom resource:

You can now access the API endpoint of the new resource on the kube-apiserver.

First, turn on the kubectl proxy  mode:

Then, curl  the API group we’ve just created:

You can see that the API serves the v1  of our custom resource.

Create an Instance of the Custom Resource

After the CRD object has been created, you can create custom API resources of the MyResource  kind. Custom objects can contain any custom fields. In their turn, these fields can contain any arbitrary JSON as long as the validation rules specified in the CRD are met.

Below is the example of the MyResource  manifest:

Let’s save this spec to myresource.yaml  and try to create it:

Oops! The output above indicates that the new MyResource  is invalid. That’s because spec.domain  did not pass validation. That is, it must be a String instead of the Integer. Let’s change the domain value to “mydomain.com” and try again:

Now it works. You can now manage a new resource with kubectl:

Or:

Also, let’s try to access the new API resource via the K8s API:

Then:

Conclusion

In this tutorial, you’ve learned the concept of K8s custom resources and how to create them using CRDs. However, a CRD we’ve created in this tutorial is just a structural representation of arbitrary data. Although it’s managed by native K8s controller loops, we have not yet added custom controllers to manage the new resource. This topic will be addressed in a subsequent tutorial.

References

 

Subscribe to our newsletter