ALL THINGS KUBERNETES

Kubernetes Networking Explained: Introduction

Kubernetes is a powerful platform for managing containerized applications. It supports their deployment, scheduling, replication, updating, monitoring, and much more. Kubernetes has become a complex system due to the addition of new abstractions, resource types, cloud integrations, and add-ons. Further, Kubernetes cluster networking is perhaps one of the most complex components of the Kubernetes infrastructure because it involves so many layers and parts (e.g., container-to-container networking, Pod networking, services, ingress, load balancers), and many users are struggling to make sense of it all.

The goal of Kubernetes networking is to turn containers and Pods into bona fide “virtual hosts” that can communicate with each other across nodes while combining the benefits of VMs with a microservices architecture and containerization. Kubernetes networking is based on several layers, all serving this ultimate purpose:

  • Container-to-container communication using localhost  and the Pod’s network namespace. This networking level enables the container network interfaces for tightly coupled containers that can communicate with each other on specified ports much like the conventional applications communicate via localhost .
  • Pod-to-pod communication that enables communication of Pods across Nodes. If you want to learn more about Pods, see our recent article).
  • Services. A Service abstraction defines a policy (microservice) for accessing Pods by other applications.
  • Ingress, load balancing, and DNS.

Sounds like a lot of stuff, doesn’t it? It is. That’s why we decided to create a series of articles explaining Kubernetes networking from the bottom (container-to-container communication) to the top (pod networking, services, DNS, and load balancing). In the first part of the series, we discuss container-to-container and pod-to-pod networking. We demonstrate how Kubernetes networking is different from the “normal” Docker approach, what requirements for networking implementations it imposes, and how it achieves a homogeneous networking system that allows Pods communication across nodes. We think that by the end of this article you’ll have a better understanding of Kubernetes networking that will prepare you for the deployment of the full-fledged microservices applications using Kubernetes services, DNS, and load balancing.

Fundamentals of Kubernetes Networking

Kubernetes platform aims to simplify cluster networking by creating a flat network structure that frees users from setting up dynamic port allocation to coordinate ports, designing custom routing rules and sub-nets, and using Network Address Translation (NAT) to move packets across different network segments. To achieve this, Kubernetes prohibits networking implementations involving any intentional network segmentation policy. In other words, Kubernetes aims to keep the networking architecture as simple as possible for the end user. The Kubernetes platform sets the following networking rules:

  • All containers should communicate with each other without NAT.
  • All nodes should communicate with all containers without NAT.
  • The IP as seen by one container is the same as seen by the other container (in other words, Kubernetes bars any IP masquerading).
  • Pods can communicate regardless of what Node they sit on.

To understand how Kubernetes implements these rules, let’s first discuss the Docker model that serves as a point of reference for Kubernetes networking.

Overview of the Docker Networking Model

As you might know, Docker supports numerous network architectures like overlay networks and Macvlan networks, but its default networking solution is based on host-private networking implemented by the bridge  networking driver. To clarify the terms, as with any other private network, Docker’s host-private networking model is based on a private IP address space that can be freely used by anybody without the approval of the Internet registry but that has to be translated using NAT or a proxy server if the network needs to connect to the Internet. A host-private network is a private network that lives on one host as opposed to a multi-host private network that covers multiple hosts.

Governed by this model, Docker’s bridge  driver implements the following:

  • First, Docker creates a virtual bridge ( docker0 ) and allocates a subnet from one of the private address blocks for that bridge. A network bridge is a device that creates a single merged network from multiple networks or network segments. By the same token, a virtual bridge is an analogy of a physical network bridge used in the virtual networking. Virtual network bridges like docker0  allow connecting virtual machines (VMs) or containers into a single virtual network. This is precisely what the Docker’s bridge driver is designed for.
  • To connect containers to the virtual network, Docker allocates a virtual ethernet device called veth  attached to the bridge. Similarly to a virtual bridge, veth  is a virtual analogy of the ethernet technology used to connect hosts to LAN or Internet or package and to pass data using a wide variety of protocols. The veth  is mapped to eth0  network interface, which is Linux’s Ethernet interface that manages Ethernet device and connection between the host and the network. In Docker, each in-container eth0  is provided with an IP address from the bridge’s address range. In this way, each container gets its own IP address from that range.

The above-described architecture is schematically represented in the image below.

 

Docker network design

In this image, we see that both Container 1 and Container 2 are part of the virtual private network created by the docker0  bridge. Each of the containers has a veth  interface connected to the docker0  bridge. Since both containers and their veth  interfaces are on the same logical network, they can easily communicate if they manage to discover each other’s IP addresses. However, since both containers are allocated a unique veth , there is no shared network interface between them, which hinders coordinated communication, container isolation, and ability to encapsulate them in a single abstraction like pod. Docker allows solving this problem by allocating ports, which then can be forwarded or proxied to other containers. This has a limitation that containers should coordinate the ports usage very carefully or allocate them dynamically.

Kubernetes Solution

Kubernetes bypasses the above-mentioned limitation by providing a shared network interface for containers. Using the analogy from the Docker model, Kubernetes allows containers to share a single veth  interface like in the image below.

K8s Networking Solution

As a result, Kubernetes model augments the default host-private networking approach in the following way:

  • Allows both containers to be addressable on veth0  (e.g., 172.17.02  in the image above).
  • Allows containers to access each other via allocated ports on localhost . Practically speaking, this is the same as running applications on a host with added benefits of container isolation and design of tightly coupled container architectures.

To implement this model, Kubernetes creates a special container for each pod that provides a network interface for other containers. This container is started with a “pause” command that provides a virtual network interface for all containers, allowing them to communicate with each other.

By now, you have a better understanding of how container-to-container networking works in Kubernetes. As we have seen, it is largely based on the augmented version of the bridge  driver but with an added benefit of a shared network interface that provides better isolation and communication for containerized applications.

Tutorial

Now, let’s illustrate a possible scenario of the communication between two containers running in a single pod. One of the most common examples of the multi-container communication via localhost  is when one container like Apache HTTP server or NGINX is configured as a reverse proxy that proxies requests to a web application running in another container.

Elaborating upon this case, we are going to discuss a situation when the NGINX container is configured to proxy request from its default port ( :80 ) to the Ghost publishing platform accessible on some port (e.g port:2368 )

To complete this example, we’ll need the following prerequisites:

  • A running Kubernetes cluster. See Supergiant GitHub wiki for more information about deploying a Kubernetes cluster with Supergiant. As an alternative, you can install a single-node Kubernetes cluster on a local system using Minikube.
  • A kubectl command line tool installed and configured to communicate with the cluster. See how to install kubectl here.

Step #1: Define a ConfigMap

ConfigMaps are Kubernetes objects that allow decoupling the app’s configuration from the Pod’s spec enabling better modularity of your settings. In the example below, we are defining a ConfigMap for NGINX server that includes a basic reverse proxy configuration.

In brief, this ConfigMap tells NGINX to proxy requests from its default port localhost:80  to localhost:2368  on which the Ghost is listening to requests.

This ConfigMap should be first passed to Kubernetes before we can deploy a Pod. Save the ConfigMap in a file (e.g.,  nginx-config.yaml ), and then run the following command:

Step #2: Create a Deployment

The next thing we need to do is to create a Deployment for our two-container pod (see our recent article for the review of the Pod deployment options in Kubernetes).

These deployment specs:

  • Define a deployment named ‘tut-deployment‘ ( metadata.name ) and assign a label ‘tut‘ to all pods of this deployment ( metadata.labels.app ).
  • Sets desired state of the deployment to 2 replicas ( spec.replicas ).
  • Define two containers: ‘ghost‘ that uses ghost Docker container image and ‘nginx‘ container that uses the nginx image from Docker repository.
  • Open a container port:80  for the ‘nginx‘ container ( spec.containers.name.image ).
  • Create a volume ‘proxy-config‘ and one volume of a type configMap  named ‘nginx-config‘ that will be used by containers to access a ConfigMap  resource titled “nginx-config” created in the previous step.
  • Mounts proxy-config  volume to the path /etc/nginx/nginx.conf  to enable the container’s access to NGINX configuration.

To create this deployment, save the above manifest in the tut-deployment.yaml  file and run the following command:

If everything is OK, you will be able to see the running deployment using kubectl get deployment tut-deployment:

Step #3: Exposing a NodePort

Now, as our Pods are running, we should expose the NGINX port:80  to the public Internet to see if the reverse proxy works. This can be done by exposing the Deployment as a Service (in the next tutorial, we are going to cover Kubernetes services in more detail):

After our deployment is exposed, we need to find a NodePort  dynamically assigned to it:

This command will produce an output similar to this:

We need a NodePort  value, which is 30234  in our case. Now you can access the ghost publishing platform through NGINX using http://YOURHOST:30234 .

That’s it! Now you see how containers can easily communicate via localhost  using built-in Pod’s virtual network. As such, a container-to-container networking is a building block of the next layer, which is a pod-to-pod networking discussed in the next section.

From Container-to-Container to Pod-to-Pod Communication

One of the most exciting features in Kubernetes is that pods and containers within pods can communicate with each other even if they land on different nodes. This feature is something that is not implemented in Docker by default (Note: Docker supports multi-host connectivity as a custom solution available via overlay  driver). Before delving deeper into how Kubernetes implements pod-to-pod networking, let’s first discuss how networking works on a pod level.

As we remember from the previous tutorial, pods are abstractions that encapsulate containers to provide Kubernetes services like shared storage, networking interfaces, deployment, and updates to them. When Kubernetes creates a pod, it allocates an IP address to it. This IP is shared by all containers in that pod and allows them to communicate with each other using localhost  (as we saw in the example above). This is known as “IP-per-Pod” model. It is an extremely convenient model where pods can be treated much like physical hosts or VMs from the standpoint of port allocation, service discovery, load balancing, migration, and more.

So far, so good! But what if we want our pods to be able to communicate across nodes? This becomes a little more complicated.

Referring to the example above, let’s assume that we now have two nodes hosting two containers each. All these containers are connected using docker0  bridges and have shared veth0  network interfaces. However, on both nodes a Docker bridge ( docker0 ) and a virtual ethrenet interface ( veth0  ) are now likely to have the same IP address because they were both created by the same default Docker function. Even if veth  IPs are different, we still do not avoid a problem of an individual node being unaware of private network address space created on another node, which makes it difficult to reach pods on it.

How Does Kubernetes Solve this Problem?

Let’s see how Kubernetes elegantly solves this problem. As we see in the image below, veth0 , custom bridge , eth0 , and a gateway that connects two nodes are now parts of the shared private network namespace centered around the gateway ( 10.100.01 ). This configuration implies that Kubernetes has somehow managed to create a separate network that covers two nodes. You may also notice that addresses to bridges are now assigned depending on what node a bridge is living on. So for example, we now have a 10.0.1...  address space shared by a custom bridge and veth0  on Node 1 and a 10.0.2...  address space shared by the same components on Node 2. At the same time, however, eth0 on both nodes share the address space of the common gateway, which allows both nodes to communicate ( 10.100.0.0  address space).

Pod-to-Pod networking

The design of this network is similar to an overlay network. (In a nutshell, an overlay network is a network built on top of another low-level network.) For example, the internet was originally built as an overlay over the telephone network. A pod network in Kubernetes is an example of an overlay network that takes individual private networks within each node and transforms them into a new software-defined network (SDN) with a shared namespace, which allows pods to communicate across nodes. That’s how the Kubernetes magic works!

Kubernetes ships with this model by default, but there are several networking solutions that achieve the same result. Remember that any network implementation that violates Kubernetes networking principles (mentioned in the Intro) will not work with Kubernetes. Some of the most popular networking implementations supported by Kubernetes are the following:

  • Cisco Application Centric Infrastructure — an integrated overlay and underlay SDN solution with the support for containers, virtual machines, and bare metal servers.
  • Cilium — open source software for container applications with a strong security model.
  • Flannel — a simple overlay network that satisfies all Kubernetes requirements while being one of the most easiest to install and run.

For more available networking solutions, see the official Kubernetes documentation.

Conclusion

In this article, we covered two basic components of the Kubernetes networking architecture: container-to-container networking and pod-to-pod networking. We have seen that Kubernetes uses overlay networking to create a flat network structure where containers and pods can communicate with each other across nodes. All routing rules and IP namespaces are managed by Kubernetes by default, so there is no need to bother creating subnets and using dynamic port allocation. In fact, there are several out-of-the-box overlay network implementations to get you started. Kubernetes networking enables an easy migration of applications from VMs to pods, which can be treated as “virtual hosts” with the functionality of VMs but with an added benefit of container isolation and microservices architecture. In our following tutorial, we discuss the next layer of the Kubernetes networking: services, which are abstractions that implement microservices and service discovery for pods, enabling highly available applications accessible from the outside of a Kubernetes cluster.

Subscribe to our newsletter