ALL THINGS KUBERNETES

Working with Kubernetes Secrets

As a Kubernetes user or administrator, you may sometimes need to include sensitive information, such as username, passwords, or ssh keys in your pods. However, putting these types of data in your pod specs verbatim might compromise the security of your application. You need to avoid situations when the sensitive data accidentally ends up in the hands of bad actors.

Fortunately, Kubernetes ships with a developed Secrets API designed specifically to solve this problem for you. Kubernetes Secrets are essentially API objects that encode sensitive data and expose it to your pods in a controlled way, enabling encapsulating secrets by specific containers or sharing them.

In this tutorial, we introduce you to this powerful API and show you several options for creating and using secrets in your Kubernetes applications. Let’s get started!

Why Do Secrets Matter?

Kubernetes Secrets come with numerous benefits compared to exposing sensitive data in your pods verbatim:

  • Since Secret objects can be created independently of the pods that use them, there is less risk of your Secret being viewed if somebody gets access to your pod spec.
  • Secrets are not written to disk (they are stored in a tmpfs ), and they are sent only to nodes that need them. Also, Secrets are deleted when the pod that is dependent on them is deleted.
  • On most native Kubernetes distributions, communication between users and the apiserver is protected by SSL/TLS. Therefore, Secrets transmitted over these channels are properly protected.
  • Any given pod does not have access to the Secrets used by another pod, which facilitates encapsulation of sensitive data across different pods.
  • Each container in a pod has to request a Secret volume in its volumeMounts  for it to be visible inside the container. This feature can be used to construct security partitions at the pod level.

You now understand the benefits of Secrets, so let’s move on to the examples of using them in your Kubernetes applications.

Tutorial

To complete examples in this tutorial, you’ll need:

  • a running Kubernetes cluster. See Supergiant GitHub wiki for more information about deploying a Kubernetes cluster with Supergiant. As an alternative, you can install a single-node Kubernetes cluster on a local system using Minikube.
  • akubectl command line tool installed and configured to communicate with the cluster. See how to install kubectl here.

Different Options for Creating Secrets

Secrets can be created in one of the following ways:

  • from local files using kubectl  tool
  • from literal values using kubectl  tool
  • using a manifest file of kind:secret

Creating Secrets from Files using Kubectl

You can create local files with sensitive data like passwords and ssh keys and then convert them into a Secret stored and managed by the Kubernetes API. Kubernetes will create key-value pairs using the filenames and their contents and encode the sensitive data in base64  format.

To illustrate how this works, let’s create a secret that stores username and password needed to access some application (e.g., database). First, let’s create the files with the username and password on your local machine:

Next, we can use kubectl create secret  command to package these files into a Secret and create a Secret API object on the API server:

Let’s see if the Secret was successfully created by running the following command:

You should get the following response:

For more detailed information about the secret, we can use kubectl describe :

The type: Opaque  of the Secret means that from Kubernetes’ perspective the contents of the Secret is unstructured (i.e., the Secret can contain arbitrary key-value pairs). Opaque Secrets contrast with Secrets storing ServiceAccount  credentials that have constrained contents.

Also, as you see, kubectl get secrets  and kubectl describe secrets/db-auth  do not output information stored in the Secret to the console. However, you can still retrieve your Secret data in the base64  encoding by running the following command:

This should return something like this:

Rather than returning the username and password in their plaintext form, the above command returns them in base64  encoding. You can easily decode the Secret if needed using native console tools:

Creating Secrets from Literal Values using Kubectl

The second option is to create Secrets from literal values. In this case, we manually provide kubectl with our credentials like this:

When creating Secrets using literals, you need to make sure that special characters such as $ , \* , and !  are escaped using the \  character. If your actual password is N!d$jjj! , for example, you should escape !  and $  characters like this:

Now, let’s check and see if those characters were escaped correctly:

Get the base64 value of the password and decode it by running the following command:

Awesome! The decoded value matches our original plaintext password.

Note: you do not need to escape characters if you create secrets using --from-file .

Creating a Secret Spec

The third option is to create Secrets using a Secret spec. In this case, we need to convert the username and password data into base64 encoding manually. Let’s do it like this:

Now, we can safely use the base64-encoded credentials in the Secret spec:

Save this spec in the test-secret.yaml  file, and create the Secret by running the following command:

Format Note: Using newlines in your Secret data is not permitted. Also, beware that when using the base64  utility on Darwin/OS X users should avoid using the -b  option in order to split long lines. In their turn, Linux users should add the option -w 0  to base64  commands or the pipeline base64 | tr -d '\n'  if -w  option is not available.

Security Note: In this example, you include the base64-encoded Secret in the JSON or YAML manifest. This implies that sharing this file or sending it to a source repository will compromise the Secret. That is because base64  encoding is not an encryption method and is regarded the same as plain text.

Using Secrets in Pods

So far, we have learned how to create Kubernetes secrets using kubectl  and Secret spec. It’s time to show how existing Secrets can be consumed by containers in your pods. The most common option is mounting Secrets as data volumes at some location within your container. Let’s create a pod to see how this works:

Let’s describe key secret-related fields of this spec:

  • .spec.volumes[]  — a volume to store the Secret
  • .spec.volumes[].secret.secretName  — a Secret object to store in the volume
  • .spec.containers[].volumeMounts[]  — a volume to be mounted into container and the mountPath  where we want our Secret to appear. Once the pod is created, each key in the Secret data map (i.e ‘username’ and ‘password’) will become the filename under mountPath .
  • .spec.containers[].volumeMounts[].readOnly = true  — sets the volume access rights to readOnly . This prevents from changing the Secret data stored in the volume.

Also, when referencing Secrets in your pods, pay attention to certain restrictions:

  • a Secret needs to exist before any pods depending on it are created. The pod won’t be started if the Secret does not exist. If the Secret cannot be retrieved for some reason, kubelet  will periodically retry, and once the Secret is fetched, it will create and mount a volume with it.
  • Secrets can be referenced only by pods in the same namespace as the Secret object.
  • individual Secrets are limited to 1MB in size.

Our Secret 'test-secret'  meets all of these requirements, so let’s save the spec in secret-volume.yaml  and create the pod:

Next, let’s check if the volume mounted at the /etc/secrets  path of our MongoDb container actually contains the Secret.

First, get a shell to the container running the following command:

When inside the container, list the /etc/secrets  folder:

That’s it! Our Secret object with the username and password was projected to two files in /etc/secrets  folder. All Secret values are base-64  encoded inside these files. To verify that, within the container, run:

In this example, the filenames for our Secret contents were constructed from the Secret keys (i.e.,  ‘username’ and ‘password’). However, Kubernetes allows changing the default filenames and paths to which to project the Secret’s contents using volumes.secret.items  parameter. For example, in order to mount your username at /etc/secrets/admin-group/admin-username  and password at /etc/secrets/admin-group/admin-password , you need to make the following changes to the volume spec.

The updated spec will project username and password values of the Secret to the paths specified in the items  list. If our Secret had another key and we had not specified it, the default path /etc/secrets/<key>  would be used. Also, take note that if a wrong Secret key is specified, the volume will not be created.

Update Note: as soon as our Secret is consumed by a volume, Kubernetes will be running periodic checks on the Secret. If the Secret is updated, the Kubernetes will ensure that the projected keys are updated as well. The update may take some time though depending on the kubelet  sync period. However, if your container uses a Secret as a subPath  volume mount, the Secret will not be updated.

Using Secrets as Environmental Variables

As you might remember from the previous article about Working with Kubernetes Containers, environmental variables offer a convenient way for exposing some arbitrary data to your application without overcrowding the execution context. Environmental variables can be used for storing Secret keys as well. We can store Secrets in environmental variables using the spec.containers.env  field with a set of name-values in it:

The spec above creates two environmental variables: SECRET_USERNAME  and SECRET_PASSWORD , which take their values from the username and password keys of the test-secret .

Let’s save the spec in secret-env.yaml  and create the pod as usual:

Now, you can access components of the Secret as simple environmental variables inside the container.

First, get a shell to the running container:

Once inside the container, run:

That’s it! The components of your Secret are available as environmental variables inside the MongoDB container.

Note: In case if you are using envFrom  instead of env  to create environmental variables in the container, the environmental names will be created from the Secret’s keys. If a Secret key has invalid environment variable name, it will be skipped, but the pod will be allowed to start. Kubernetes uses the same conventions as POSIX for checking the validity of environmental variables but that might change. According to POSIX:

Environment variable names used by the utilities in the Shell and Utilities volume of IEEE Std 1003.1-2001 consist solely of uppercase letters, digits, and the ‘_’ (underscore) from the characters defined in Portable Character Set and do not begin with a digit. Other characters may be permitted by an implementation; applications shall tolerate the presence of such names.

If the environmental variable name does not pass the check, the InvalidVariableNames  event will be fired and the message with the list of invalid keys that were skipped will be generated.

Conclusion

As you have already learned, Kubernetes Secrets are extremely powerful in protecting sensitive data in your pods from ending up in wrong hands. We’ve learned several options for configuring Secrets such as using kubectl  --from-file  and defining a secret’s spec. You also saw how to use Secrets in pods both as data volumes and environmental variables. When working with Kubernetes Secrets, however, users should be aware of the following risks and best practices for using them:

  • Since Secret data is stored in the API server in the etcd  as plaintext, Kubernetes administrators should limit access to etcd  to admin users and delete disks used by etcd  when no longer in use.
  • Secrets used as data volumes should be protected at the application level from accidental logging or transmitting them to an untrusted party.
  • You should remember that users who can create a pod with a Secret may also see the value of that Secret. Additional security measures are needed to avoid exposing your Secrets to wrong users.
  • If the cluster runs multiple replicas of etcd , the Secrets will be shared between them. By default, the peer-to-peer communication with SSL/TLS is not secured by etcd . However, this can be configured.

It’s crucial to pay attention to the risks outlined above when designing your applications with Kubernetes Secrets. However, once you get using Secrets right, the security of your Kubernetes applications will be dramatically enhanced.