Skip to main content

Custom Resource Definitions

Course Reading

Learning objectives

  • Grow available Kubernetes objects
  • Deploy a custom resource definition (CRD)
  • Deploy a new resource and API endpoints
  • Discuss aggregates APIs

Custom Resources

Kubernetes allows the flexibility to add your own dynamic resources to the cluster, outside the scope of those built-in. Once a custom resource has been added, it can be created through the same methods of any other built-in resource, like with kubectl.

For a custom resource to be made part of the API, it needs a controller to receive the specification for that object and work to maintain the object in that state. These custom controllers should handle all the tasks a human would have to take if deploying the application outside of Kubernetes.

There are two methods for adding custom resource to the cluster that offer different levels of flexibility. The less flexible option is to add a custom resource definition (CRD). The more flexible option is to add aggregated APIs (AA), which uses a new API server to be written and deployed in the cluster. Both options still manage custom resources outside the scope of the built-ins.

With RBAC, you will need to allow access to the new CRD resources and controller. When using an AA, you can use the typical auth methods or a different one.

Custom resource definitions

When a new API object and controller is added via CRDs, the existing kube-apiserver can be used to monitor and control the object's state. A CRD will add a new path to the API, under the apiextensions.k8s.io/v1 group.

CRDs are the easiest way to add a new type of object to the cluster, although it is less flexible. Only the existing API functionality can be used, objects must respond to RESTful requests, and their state must be validated and stored in the same way built-in API objects do.

CRDs also allow the new resources to be deployed in a namespace or cluster wide. The manifest uses the scope field and takes the value of either Namespaced or Cluster depending on how it should be deployed.

Example

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: backups.stable.linux.com
spec:
  group: stable.linux.com
  version: v1
  scope: Namespaced
  names:
    plural: backups
    singular: backup
    shortNames:
    - bks
    kind: BackUp

Let's walk through each part of this example

Like any manifest, we begin with the apiVersion, which for a CRD must be apiextensions.k8s.io/v1.

Next is the kind, which for a CRD is CustomResourceDefinition.

In the metadata field, the name must match the later defined spec. It must have the structure of <spec.names.plural>.<spec.group>.

Next is the group which is used to determine the endpoint in the API which is in the structure /apis/<group>/<version>, which for this example specifically is /apis/stable/v1.

The scope determines if the objects must exist in a namespace or be cluster-wide.

The plural field is what determines the end of the API url, so in this case it would be /apis/stable/v1/backups.

The singular and shortName fields are used to make CLI easier. This is the same ideas of being able to retrieve pods with pods, pod or po.

Finally, the spec.kind field determines the camel cased name that would be used when writing a manifest for one of these objects.

Defining the new object

To then define the object the CRD creates would look something like this.

apiVersion: "stable.linux.com/v1"
kind: BackUp
metadata:
  name: new-backup-object
spec:
  timeSpec: "* * * * */5"
  image: linux-backup-image
replicas: 5

The apiVersion and the kind must match what is specified in the CRD. The fields in spec all depend on the associated controller. If the controller has validation, it would check the values to be what is expected and error if it does not match. With no validation, only the existence of the needed fields is checked.

Optional hooks

Finalizer

Just like built-in objects, an asynchronous pre-delete hook called a Finalizer can be used. When the API receives a delete request, the metadata.deletionTimestamp field is updated and then the configured finalizer is run. When a finalizer completes, it is removed from the list and the next one is processed, until the list is empty.

metadata:
  finalizer:
  - finalizer.stable.linux.com

Validation

Validation for custom objects can also be done using the OpenAPI v3 schema. This will check the properties being passed in a manifest defining the resource.

validation:
  openAPIV3Schema:
    properties:
      spec:
        properties:
          timeSpec:
            type: string
            pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
          replicas:
            type: integer
            minimum: 1
            maximum: 10

Here timeSpec field must match a certain string pattern, and the replicas field must be an integer between 1 and 10.

Understanding Aggregated APIs

Aggregated APIs allow adding more Kubernetes-like API servers to the cluster. The additional servers are subordinate to the kube-apiserver, which will run the aggregation layer. Then when a new custom resource is requested, the aggregation layer will listen for any URLs and proxy them to the new API if it is responsible for that object.

The aggregation layer can be enabled by adding the enable-aggregator-routing=true flag to the kube-apiserver start-up commands.

Configuring TLS between components and RBAC rules is necessary.

Lab Exercises

Lab 14.1 - Create a Custom Resource Definition

First view any existing CRDs on the cluster.

ubuntu@cp:~ $ kubectl get crd --all-namespaces

You should see they are all for the Calico network plugin we used when building the cluster as well as for linkerd. If you were to look at the calico.yaml file we used when first building the cluster, you would see the CustomResourceDefinitions that where added.

ubuntu@cp:~ $ cat calico.yaml

Having looked at some exampled, let's build our own.

ubuntu@cp:~ $ vim crd.yaml

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              cronSpec:
                type: string
              image:
                type: string
              replicas:
                type: integer
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontabs
    shortNames:
    - ct
    kind: CronTab

Now create the CRD, view it among the others and describe it.

ubuntu@cp:~ $ kubectl create -f crd.yaml

ubuntu@cp:~ $ kubectl get crd

ubuntu@cp:~ $ kubectl describe crd crontabs.stable.example.com

Now, let's create a new CronTab object. Note that it will not do anything since there is no associated controller with this object.

ubuntu@cp:~ $ vim crontab.yaml

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: new-crontab
spec:
  cronSpec: "* */4 * * *"
  # This image doesn't need to exist in this case
  image: some-image

ubuntu@cp:~ $ kubectl create -f crontab.yaml

ubuntu@cp:~ $ kubectl get crontabs

ubuntu@cp:~ $ kubectl get ct

ubuntu@cp:~ $ kubectl describe ct

You can then delete the CRD, which should also remove the API endpoints and all objects.

ubuntu@cp:~ $ kubectl delete -f crd.yaml

ubuntu@cp:~ $ kubectl get ct

Knowledge check

  • When adding a new API object to the kube-apiserver, we use a Custom Resource Definition
  • When we add a new API server that is subordinate to the kube-apiserver, we use Aggregated APIs
  • CRDs are not required to live in a namespace