ALL THINGS KUBERNETES

Creating Stateful Apps with Kubernetes StatefulSets

In previous tutorials we discussed ways to enable and manage persistent storage in Kubernetes. In particular, we saw how to use PersistentVolumes  to bind various external storage options like Amazon EBS to the pods, introducing statefulness to your Kubernetes workloads.

However, exposing persistent storage to pods is only one component of statefulness. One limitation of this approach is that we use PersistentVolumes  and associated PersistentVolumeClaims  within a standard deployment API designed primarily to manage stateless apps. However, as Kubernetes users and administrators, we want to manage pods in stateful apps in a controlled and predictive way, combining persistent storage with a sticky identity of the pods.

Fortunately, Kubernetes developers have addressed our concerns with a StatefulSet  workload API object designed to create and manage applications that are stateful by design. In this article, we discuss the architecture and key concepts of the StatefulSet  API and walk you through the process of creating a working StatefulSet . In upcoming tutorials, we’ll move on to the discussion of scaling, rolling updates and roll outs, and other methods for managing applications created with StatefulSets . Let’s start!

What Are StatefulSets?

A StatefulSet  is an alternative to deployments when it comes to managing a set of pods based on an identical container spec. Unlike a deployment, however, a StatefulSet  maintains a sticky identity for its pods. As we remember, each time the pod is restarted in the Deployment, it gets a new identity, including a new UID and IP address. Under these circumstances, deployments are more suitable for stateless applications where the pod’s identity does not matter and where some service can load-balance client requests between pods that perform identical tasks.

StatefulSets  take a radically different approach. Each pod in a StatefulSet  gets a persistent identifier (UID) maintained across any rescheduling and restart. Moreover, since StatefulSets  are currently required to use headless services to manage network identities for pods, each pod gets a sticky network sub-domain and CRV records that do not change across the pod’s rescheduling.

The question arises: why do we actually need all these features for pods? The answer is quite simple: maintaining a state and network identity across pod restarts and rescheduling is one of the major requirements for stateful apps, which are not currently supported in deployments and other stateless controllers in Kubernetes. Therefore, StatefulSets  are extremely useful if your application needs the following features:

  • Stable and unique network identifiers
  • Stable, persistent storage
  • Ordered deployment and scaling
  • Ordered deletion and termination
  • Ordered, automated rolling updates

Note: StatefulSets  are not available in Kubernetes prior to 1.5 and were a beta feature until the 1.9 release. You should consider using StatefulSets  only with the latest releases of the Kubernetes platform.

Creating a StatefulSet

Unlike deployments which only require a deployment API object to be defined, StatefulSets  come with a number of prerequisites. Before using them, you must:

  • provision the storage using PersistentVolumes Provisioner  based on the requested StorageClass  or use the storage pre-provisioned by the Kubernetes administrator.
  • define a headless service. Such services are not assigned a Cluster IP and do not support load-balancing, which reduces coupling to the Kubernetes system and allows freedom for users to do service discovery their custom way.
  • finally, create a StatefulSet API object.

In what follows, we walk you through this process, discussing key StatefulSet  concepts and methods along the way.

Tutorial

In this tutorial, we create a StatefulSet  for Apache HTTP server using a pre-provisioned PersistentVolume  based on the pre-defined StorageClass  and give network identities to the pods in the StatefulSet  using a headless service.

To complete this tutorial, we need:

  • a running Kubernetes cluster. We use a local single-node cluster with Kubernetes 1.9 deployed with Minikube.
  • a kubectl command line tool installed and configured to communicate with the cluster. See how to install kubectl here.

Step #1 Create a Headless Service

As you remember, a service defines a set of pods that perform the same task and some policy to access them. A headless service defined below is used for the DNS lookups between Apache HTTP pods and clients within the Kubernetes cluster.

This service spec:

  • creates a service named “httpd-service” and assigns pods labeled “httpd” to it. All future pods with that label will be associated to this headless service.
  • specifies that a service is headless by setting clusterIP  field to None . For more details about headless services, see the official documentation.
  • creates a named port “web” that maps to the port:80  used by the service.

Let’s save this spec in the httpd-service.yaml  file and create it using the following command:

Now, check whether this headless service was successfully created:

The response should be:

That’s it! Now we have a headless service ready to assign network identities for our future pods. Let’s move on to the second step.

Step #2 Create a StorageClass to Enable Dynamic Provisioning of PersistentVolumes for the StatefulSet

As we remember from the previous tutorial, Kubernetes supports dynamic provisioning of volumes based on StorageClasses  defined by the administrator. A StorageClass  is a convenient way to describe storage types available in the cluster and their specific backup, reclaim, and mounting policies. If a PersistentVolumeClaim  (PVC) refers to a particular StorageClass , Kubernetes can dynamically provision the requested amount of resources from the underlying storage provider like AWS EBS or Azure disk. For this tutorial, we create a StorageClass  (aka storage ‘profile’) that can dynamically provision hostPath volumes that can mount a directory from the host node’s filesystem. This is how our spec looks:

In this spec, we:

  • create a StorageClass  named “fast” provisioned by the k8s.io/minikube-hostpath  volume provisioner. The provisioner is responsible for the interaction with the volume plugin ( hostPath  plugin in our case) needed to provision the volume.
  • set a “retain” reclaim policy for the volumes provisioned by this StorageClass .See more about reclaimPolicy  parameter in our recent article.

Let’s save this spec in the fast-sc.yaml  and create the StorageClass  using the following command:

Then, let’s check whether our new StorageClass was created:

which should return something like this:

So far, so good! Now we have a StorageClass  that can dynamically provision hostPath volumes for the pods in our StatefulSet . Let’s move on!

Step # 3 Creating a StatefulSet

Now, as we have both a headless service and a custom StorageClass  defined and created, we can do the same with our StatefulSet .

Let’s analyze key parameters of this spec:

  • spec.selector.matchLabels  — as in the deployment object spec, this field defines a set of pods to be managed by the StatefulSet. In our case, the StatefulSet  will select pods with the “httpd” label. This field’s value should match the pod’s label defined in the spec.template.metadata.labels .
  • spec.serviceName  — the name of the headless service for this StatefulSet . As you see, we have specified the name of the headless service created in the first step of this tutorial.
  • spec.replicas  — a number of pod replicas run by the StatefulSet . Default value is 1.
  • spec.template.spec.terminationGracePeriodSeconds  — a lapse of time before a given pod is terminated (graceful termination). Force termination of pods can be achieved by setting the value to   , but  this is strongly discouraged. For more information, see this article.
  • spec.template.spec.containers  – container images with their corresponding settings. See our article about Kubernetes Pods to find out more.
  • volumeClaimTemplates.storageClassName  — a StorageClass  that manages dynamic volume provisioning for the pods running in this StatefulSet. Here, we are using our “fastStorageClass  created in the second step of this tutorial.
  • volumeClaimTemplates.resources.requests.storage  — the request for storage resources from our “fast” StorageClass. This request is similar to how PersistentVolumeClaims  (PVC) claim resources from their respective PersistentVolumes  (PVs).

Now, let’s open two terminal windows to see how this StatefulSet works. In the first terminal window, type kubectl get pods -w -l app=httpd  to watch the creation of the StatefulSet’s pods.

Then, save the spec above in the httpd-ss.yaml  , and create a StatefulSet  running the following command in the second terminal:

Then, in the first terminal, you’ll notice that the StatefulSet’s pods are created sequentially in order from {0..N-1}. For example, the pod apache-http-2  will not be launched until the pod apache-http-1  is running and ready. This approach is known as the ordered pod creation.

Once all pods are created, the same command will return the following response:

As you might have noticed, all pods in our StatefulSet have a sticky, unique identity that is based on the unique ordinal index appended to the StatefulSet  name, so that eventually a pod takes the form <statefulset name>-<ordinal index> . This approach is different from how pods are created in deployments, for example. As you remember, pod names in the deployment are created by appending the deployment name by the random hash value generated for each pod:

After a pod in the deployment is terminated, a rescheduled pods get a new UID. In contrast, a StatefulSet  allows maintaining the pod’s identity across terminations and restarts, so that the users always know with which pods they are interacting.

Step #4 Checking the Pods Network in the StatefulSet

Each pod in a StatefulSet also has a stable network ID that persists across the pod’s restarts. The hostname of each pod is derived from the name of the StatefulSet  and the ordinal of the pod. In our example, the pods’ hostnames will be apache-http-0 , apache-http-1 , and so on, depending on the number of replicas in the StatefulSet .

In its turn, the domain of pods in our StatefulSet  is controlled by the headless service “httpd-service” defined above. The domain managed by the service takes the form of $(service name).$(namespace).svc.cluster.local , where “cluster.local” is the cluster domain. Pods in the StatefulSet  get a matching DNS subdomain within this domain that takes the form $(podname).$(governing service domain) , where the governing service domain is defined by the serviceName  field of the StatefulSet .

To verify what domains and sub-domains were created for the pods in our StatefulSet , we can use a kubectl run  command that executes a container providing the nslookup  command from the dnsutils  package.

As you see, the headless service has created a domain named httpd-service.default.svc.cluster.local  and three sub-domains for the pods using the DNS conventions described above. Each pod was assigned a unique IP address, sub-domain, and hostname. The CNAME  of our headless service points to SRV  records (one for each running pod), which in their turn point to the A record entries that contain the pod’s IP address. That’s how our pods can be discovered by clients.

Now, if you delete the pods using kubectl delete pod -l app=httpd , you’ll notice that once the pods are recreated they have the same ordinals, hostnames, SRV records, and A records. However, the IP addresses associated with the pods might have changed. That’s why it’s important to configure other applications to access pods in a StatefulSet  the right way.

In particular, if you need to find the active member of a StatefulSet, you should query the CNAME of the headless service (e.g httpd-service.default.svc.cluster.local ). The SRV records associated with this CNAME will contain only the Pods belonging to the StatefulSet . Alternatively, you can reach the Pods using their SRV records directly (e.g apache-http-0.httpd-service.default.svc.cluster.local ) because these records are stable ( they do not change across the Pod restarts).

Step #5 Checking Persistent Storage in the StatefulSet

Kubernetes will create one PersistentVolume  for each VolumeClaimTemplate  specified in a StatefulSet  definition. In our case, each pod will receive a single PV with a StorageClass  “fast” and 2 Gib of a provisioned storage. Please, note that PersistentVolumes  associated with the pod’s PersistentVolumeClaims  are not deleted when the pods or StatefulSet  are deleted. The volumes deletion should be done manually.

Let’s check what volumes were attached to the pods in our StatefulSet :

Since we have specified the StorageClass  for our PVCs, the PersistentVolumes  associated with this class ( hostPath  SSD) will be dynamically provisioned and automatically bound to our claims. Also, the volumeMounts  field in the StatefulSet  spec will ensure that the /usr/local/apache2/htdocs  directory of the Apache HTTP server is backed by a PersistentVolume . The volume will persist any Apache HTTP server data across the pod’s restarts and terminations, ensuring that our application is stateful.

Step #6 Deleting the StatefulSet

StatefulSet supports two types of deletion: non-cascading and cascading. In the first one, the StatefulSet’s pods are not deleted when the StatefulSet is deleted. In a cascading delete, both the pods and the StatefulSet  are deleted.

To clean up after the tutorial is completed, we’ll need to do a cascading delete like this:

This command will terminate the Pods in your StatefulSet  in reverse order {N-1..0}. Also, before a given Pod is terminated, all of its successors must be completely shutdown. Note that this operation will just delete the StatefulSet  and its Pods but not the Headless Service associated with your StatefulSet .

Delete our ‘fast‘ StorageClass:

Delete our httpd-service  Service manually:

Finally, remember that any PVs associated with your StatefulSet  should be deleted manually because Kubernetes prioritizes data safety over the automatic purge of all StatefulSet resources. Also, you’ll need to delete spec files we created if you don’t need them anymore.

Conclusion

That’s it! We have demonstrated how to use StatefulSets  to enable persistent storage, stable UID and network identity of your pods. Leveraging ordered pod creation and sticky network identity managed by the headless services, we can ensure that each pod of our StatefulSet  maintains its state and identity across application restarts and rescheduling. These features make StatefulSets  the first choice for stateful applications.

In this article, however, we primarily focused on key concepts and the basic steps to create a StatefulSet  from scratch. In the continuation of this article, we will discuss a StatefulSet’s management options including scaling, rolling updates, and roll outs, which will give you a deeper understanding of managing stateful apps in Kubernetes.

Subscribe to our newsletter