TLS Certificate Creation

TLS Certificates Creation

In this tutorial, we are going to discuss the TLS certificate creation for the Kubernetes cluster.

To generate certificates there are different tools available such as easyrsa, openssl or cfssl etc. In this tutorial we will use OpenSSL tool to generate the certificates.

We will start with the CA certificates. First we create a private key using the openssl command

openssl genrsa -out ca.key 2048

Then we use the Open SSL requests command along with the key that we just created to generate a certificate signing request.

openssl req -new -key ca.key -subj "CN=KUBERNETES-CA" -out ca.csr

The certificate signing request is like a certificate with all of your details but with no signature.

In the certificate signing request with specified the name of the component this certificate is for in the common name or CN field. In this case, since we are creating a certificate for the Kubernetes CA, we name it Kubernetes-CA.

Finally, we sign the certificate using the openssl x509 command and by specifying the certificate signing request we generated in the previous command.

openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt

Since this is for the CA itself, it is self-signed by the CA using its own private key that it generated in the first step. Going forward for all other certificates we will use this ca key pair to sign them. The CA now has its private key and root certificate file.

Let’s now look at generating the client’s certificates. We start with the admin user.

Admin User

We follow the same process where we create a private key for the admin user using the openssl command.

openssl genrsa -out admin.key 2048

We then generate a CSR and that is where we specify the name of the admin user, which is kube-admin.

openssl req -new -key admin.key -subj "CN=kube-admin" -out admin.csr

A quick note about the name. It doesn’t really have to be kube-admin. It could be anything. But remember this is the name that the kubectl client authenticates with when you run kubectl commands. So in the audit logs and elsewhere this is the name that you will see. So provide a relevant name in this field.

Finally, generate a signed certificate using the openssl x509 command.

openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -out admin.crt

But this time, you specify the CA certificate and the CA key. You are signing your certificate with the CA key pair. That makes this a valid certificate within your cluster.

The signed certificate is then output to admin.crt file. That is the certificate that the admin user will use to authenticate to Kubernetes cluster.

If you look at it, this whole process of generating a key and a certificate pair is similar to creating a user account for a new user. The certificate is the validated user I.D. and the key is like the password. It’s just that it’s much more secure than a simple user name and password.

So this is for the admin user. How do you differentiate this user from any other users. The user account needs to be identified as an admin user and not just another basic user.

You can do that by adding the group details for the user in the certificate. In this case a group named SYSTEM:MASTERS exists on Kubernetes with administrative privileges.

We will discuss about groups later. But for now its important to know that you must mention this information in your certificate signing request.

Once it’s signed we now have our certificate for the admin user with admin privileges.

We follow the same process to generate client certificates for all other components that access the Kube API server.

Kube Scheduler

Now the Kube Scheduler is a system component part of the Kubernetes control plane. So it’s name must be prefixed with the key word “system”.

Kube Controller Manager

Kube controller manager is again a system component. So its name must be prefixed with the keyword system.

Kube Proxy

Kube proxy is again a system component. So its name must be prefixed with the keyword system.

So far we have created CA certificates, then all of the client certificates including the admin user, scheduler, controller manager, kube proxy.

We will follow the same procedure to create the remaining 3 client certificates for the API Servers and Kubelets.

When we create the service certificates for them so we will set them aside for now. Now what do you do with these certificates.

Take the admin certificate for instance to manage the cluster. You can use the certificate instead of a user and password in a REST API call you make to the Kube API server.

You specify the key the certificate and the ca certificate as options

curl https://kube-apiserver:6443/api/v1/pods --key admin.key --cert admin.crt --cacert ca.crt

That’s one simple way. The other way is to move all of these parameters into a configuration file called Kube config. Within that specify the API server endpoint details the certificates to use etc.

This is what most of the Kubernetes clients use. We will discuss more about Kube Config in depth in one of upcoming tutorials.

So we are now left with the server side certificates but before we proceed. One more thing, remember in the previous tutorial we mentioned that for the clients to validate the certificate sent by the server and vice versa.

They all need a copy of the certificate authorities public certificate, the one that we said is already installed within the user’s browsers in case of a web application.

Similarly, in Kubernetes for these various components to verify each other, they all need a copy of the CA’s root certificate.

So whenever you configure a server or a client with certificates you will need to specify the CA root certificate as well. Let’s look at the service side certificates now.

ETCD Servers

Let’s start with the ETCD server. We follow the same procedure as before to generate a certificate for ETCD. We will name it ETCD-SERVER. ETCD Server can be deployed as a cluster across multiple servers as in high availability environment.

In that case to secure communication between the different members in the cluster, we must generate additional peer certificates. Once the certificates are generated specify them while starting the ETCD server.

There are key and cert file options where you specify the ETCD server keys. There are other options available for specifying the peer certificates.

And finally as we discussed earlier it requires the CA root certificate to verify the clients connecting to the ETCD server are valid.

Kube API Server

Let’s talk about the Kube API Server now. We generate a certificate for the API server like before. But wait, the API server is the most popular of all components within the cluster.

Everyone talks to the Kube API Server. Every operation goes through the Kube API Server. Anything moves within the cluster the API server knows about it.

You need information, you talk to the API server and so it goes by many names and aliases within the cluster.

It’s real name is Kube API Server. But some call it Kubernetes. Because for a lot of people who don’t really know what goes under the hoods of Kubernetes, the Kube API Server is Kubernetes.

Others like like to call it Kubernetes.default, while some refer to it as Kubernetes.default.svc. and some like to call it by its full name, kubernetes.default.svc.cluster.local.

Finally, it is also referred to, in some places, simply by its IP address. The IP address of the host running the Kube API Server or the POD running on it.

So all of these names must be present in the certificate generated for the Kube API server. Only then those referring to Kubernetes by these names will be able to establish a valid connection.

So we use the same set of commands as earlier to generate a key. In the certificate signing request you specify the name KubeAPI Server.

But how do you specify all the alternate names. For that you must create an openssl config file. Create an openssl.cnf file and specify the alternate names in the alt name section of the file.

[req]
req_extensions = v3_req 
distinguished_name = req_distinguished_name [req_distinguished_name] 
[ v3_req ] 
basicConstraints = CA:FALSE 
keyUsage = nonRepudiation, digitalSignature, keyEncipherment 
subjectAltName = @alt_names 
[alt_names] 
DNS.1 = kubernetes 
DNS.2 = kubernetes.default 
DNS.3 = kubernetes.default.svc 
DNS.4 = kubernetes.default.svc.cluster.local 
IP.1 = ${K8S_SERVICE_IP} 
IP.2 = ${MASTER_HOST}

Include all the DNS names the API server goes by as well as the IP address. Pass this config file as an option while generating the certificate signing request.

Finally, sign the certificate using the CA certificate and key. You then have the Kube API Server certificate.

Specifying keys

Now it is time to look at where we are going to specify these keys.

Remember to consider the API Server client certificates that are used by the API server while communicating as a client to the ETCD and Kubelet servers.

The location of these certificates are passed in to the Kube API servers executable or service configuration file.

First the CA file needs to be passed in. Remember every component needs the CA certificate to verify its clients.

Then we provide API server certificates under the tls-cert options. We then specify the client certificates used by the Kube API server to connect to the ETCD server, again with the CA file.

And finally the Kube API server client certificate to connect to the kubelets.

Kubelet server

Next comes the kubelet server. The kubelet server is an HTTPS API server that runs on each node, responsible for managing the node.

That’s where the API server talks to to monitor the node as well as any information regarding what PODs to schedule on this node. As such, you need a key-certificate pair for each node in the cluster.

Now what do you need these certificates. Are they all going to be named kubelets? answer is No. They will be named after their nodes. Node01, node02 and node03.

Once the certificates are created. use them in the kubelet-config file.

As always you specify the root CA certificate. And then provide the kubelet node certificates. You must do this for each node in the cluster.

We also discussed about a set of client certificates that will be used by the kubelet to communicate with the Kube API Server. These are used by the kubelet to authenticate into the Kube API Server. They need to be generated as well.

What do you name these certificates? The API server needs to know which node is authenticated and give it the right set of permissions. So it requires the nodes to have the right names in the right format.

Since the nodes are system components, like the kube-scheduler, and controller-manger we discussed about earlier the format starts with the system keyword. Followed by node and then the node name.

system:node:node01
system:node:node02
system:node:node03

And how would the API server give it the right set of permissions?

Remember we specified a group name for the admin user so the admin user gets administrative privileges. Similarly, the nodes must be added to a group named system nodes.

Once the certificates are generated they go into the kube-config files as we discussed earlier.