Cloud Native Authentication on Kubernetes

In the daily job routines, the developers are mainly focusing on gathering, implementing and testing the functional requirements so much needed by the business. However, there is a lot more to it, than the functionality, to make it a successfull production solution. One important aspect is the authentication side of the story.

Cloudless sky of our Milkyway

In this article we’ll do a hands-on exercise on securing a simple Kubernetes application using a well known open source project, based on OAuth2 standard, namely oauth2-proxy.

• First we will install a k8s cluster on the local machine and configure it’s ingress controller,
• Then we are going to deploy a sample application in the cluster, making sure we publicly expose its UI,
• Next we will provide a small overview of the oauth2-proxy project and its critical configuration options,
• Following that, we will install and configure the oauth2-proxy container in our Kubernetes cluster in order to secure the previously installed UI,
• Finally we will look at some advanced configuration options and their benefits.

Evidently since this exercise is leveraging a Kubernetes cluster, first we need to have access to an instance.

These days you have a tremendous variety of options like:

• running a small k8s instance on your local machine (through kind, k3s, Microk8s, Minikube or other options),
• using an as-a-service flavour from IBM, AWS, Azure, Google or any other cloud services vendors,
• leveraging an existing k8s instance on-prem or in cloud.

Note: For the scope of this exercise we don’t need a multi-node Kubernetes cluster, so even a single-node cluster is sufficient.

Installing a Kubernetes cluster on the local machine

Next, we will install a kubernetes cluster on the local machine using kind. You can follow these instructions:

1. Install kind locally, instructions found here or just run:

curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.9.0/kind-darwin-amd64

2. Now that we have kind tool installed locally we need to create a kubernetes cluster. This will create a single node cluster and add ingress-ready label, preparing it for the ingress-controller installation. Open a terminal and paste the following snippet (which should take 1–2 minutes to create and start the kubernetes cluster).

It takes about 2 minutes for all the cluster’s pods to start, so before moving to the next step make sure they are running:

Cluster’s pods accross all namespaces

3. Install ingress nginx in the cluster by running:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/kind/deploy.yaml

This will create a new namespace ingress-nginx and then deploy the nginx ingress controller container. Make sure the nginx controller is running, you might wait a minute for the pod to be ready.

Updating the cluster’s core-dns

This step is necessary only for our local cluster use-case since we will define specific domains like oauth2-proxy.localtest.me and dex.localtest.me . When moving to a public accessible kubernetes cluster you will have a DNS that is created by the cloud provider.

This is done in two steps, first we update the core-dns configmap, by adding the two hosts, and then restarting the core-dns deployment to make sure the new configurations are loaded up. To change the host configuration, just run:

kubectl apply -f https://raw.githubusercontent.com/morarucostel/cloud-native-auth/main/custom-dns.yaml

and then force the core-dns deployment refresh by deleting the kube-dns pods:

kubectl delete pod -n kube-system -l k8s-app=kube-dns

Deploying a sample application

Next we want to install an application in our cluster which later we will secure the access to it. Clearly you can use any hello-world applications out there, but for our exercise we’ll install a small http-echo container exposed through an ingress. The kubernetes resources for the sample application are: a conservis/hello-world container that will echo the request body, a service and an ingress.

hello-world application deployment

To install it just simply run the following command in the terminal:

kubectl apply -f https://raw.githubusercontent.com/morarucostel/cloud-native-auth/main/hello-world.yaml

The hello-world application should start running immediately and be accessible from your local browser. You can now open your browser and just enter http://hello-world.localtest.me . If everything is OK you should see a similar response in your browser.

Unsecured hellow world application in browser

Now that we have our application running and accessible from the browser, we need to introduce a bit of context about oauth2-proxy, dex and how they will secure our application with the help of auth-signin and auth-urlingress annotation.

An overview of oauth2-proxy and dex

OAuth2-Proxy is a open source project that provides authentication using various providers like Google, GitHub, Azure, Facebook and many others as well as any provider that implements OpenID Connect protocol which means … everyone. It is written in go, making it very fast in a kubernetes environment and has a mature community behind it.

Dex is a federated OpenID Connect provider which acts as a portal to other identity providers — IdP through connectors. In this exercise we are going to use it as an local identity provider, with a static user/password.

In a real scenario you will typically use dex with one or more of IBM, Google, GitHub, Azure, Facebook or your own company’s IdP. However, today since we are installing it locally on our machine, we can’t register a callback url in a major IdP since they requires a public accessible DNS endpoint and we don’t have one on our local machine.

Note: In the last section of this article, I will add the link to another hands-on article that will walk you through the configuration using of a couple of major public IdPs.

It’s important to emphasize that in this exercise we are using only a very small part of dex’s capabilities and as such we are detailing only the important configuration values for a local identity provider.

The staticClients section register the oauth2-proxy framework, that we’ll have a look imediatly after, as a oauth2 client. As in any public identity provider when you register an application, at a minimum you need to define a name, client-id, client-secret and a callback url. This is what is defined in the staticClients section. In the staticPasswords section we define a user/password entry as part of the dex local identity provider. Part of the authentication, we will validate our own credentials against the ones defined in this section. Just note that the hash corresponds to the password password.

Next, lets have a look at the oauth2-proxy configuration.

The first section describe the details of the identity provider, that oauth2-proxy needs to use for authentication process, in our case the local dex instance. The values from this section have to match the ones defined in the dex identity provider so that the two systems trust each other.

The second section describes the behaviour of the oauth2-proxy framework in relation to the cookie it sets and the upstream systems.

Note: There are more advanced oauth2-proxy configuration options which are going to be described later in this article.

Installing oauth2-proxy and dex

Now that we have a brief understanding of how this should work, lets install the two frameworks in our cluster. We have defined the necessary kubernetes resources like: ingress definitions, services, configmaps, secrets and deployments for the two frameworks to run, however the most important configuration was described earlier. To install it just simply run the following in the terminal:

kubectl apply -f https://raw.githubusercontent.com/morarucostel/cloud-native-auth/main/oauth2-proxy-dex.yaml

We’ll give it a minute for the containers to be downloaded and started after which you should see the following pods running:

Running oauth2-proxy and dex pods

Protecting our ingress using oauth2-proxy

We have our hello-world application running in our cluster and we were able to access it through the UI. The only thing remaining is to instruct the ingress controller that we want to secure this resource with an external authentication provider. The ingress annotations auth-url and auth-signin allow us to use external authentication providers and to protect the ingress resources, also detailed here.

The hello-world ingress is currently looking like:

Unsecured hello-world ingress definition

and we need to add the annotations section, to look like:

Securing the ingress via oauth2-proxy service

In order to update the ingress definition just run:

kubectl apply -f https://raw.githubusercontent.com/morarucostel/cloud-native-auth/main/hello-world-secured-ingress.yaml

Now that the ingress was updated, let’s try again the hello-world application http://hello-world.localtest.me . Now that the ingress is configured to protect the access, it will redirect the browser to the identity provider, which in this case is dex (check the url in the brower, is now dex.localtest.me). Just use the credentials that we configured back at the An overview of oauth2-proxy and dex section which were:

email: admin@example.com
password: password

Login screen when accessing the hello-world application

After the successful authentication you will be redirected back to the initial requested page, in our case http://hello-world.localtest.me .

Note: If you want to invalidate the oauth2-proxy session, basically to sign-out from the application, you can invoke http://oauth2-proxy.localtest.me/oauth2/sign_out?rd=http://hello-world.localtest.me endpoint.

Additional configuration of oauth2-proxy

Let’s have a look at the oauth2-proxy framework in more details and how it works.

Whenever you need to protect the access to a resourcevia the ingress, you annotate it with auth-signin just like we did before.

nginx.ingress.kubernetes.io/auth-signin: http://oauth2-proxy.localtest.me/oauth2/start

The value of the annotation is the oauth2-proxy‘s endpoint that start the authentication process called /start. This will instruct the browser through an HTTP/302 to redirect it to the configured identity provider, in our case the local installed dex or any configured public identity provider. After the end-user is authenticating successfully to the identity provider, the browser is being instructed again to do a redirect to the oauth2-proxy’s /callback endpoint together with some parameters issued by the identity provider. Part of the last step of the authentication, the oauth2-proxy has to validate that the information received is comming from the real identity provider and create a cookie acknowledging that the user was authenticated. This is very, very briefly how the OAuth2 protocol is being implemented by this oauth2-proxy.

At the moment of this writing, the oauth2-proxy is supporting the following identity providers: Google, Azure, Facebook, GitHub, Keycloak, GitLab, LinkedIn, Microsoft Azure AD, login.gov, Nextcloud, DigitalOcean, Bitbucket, Gitea and any generic OpenID Connect provider.

In addition it brings additional capabilities like: persisting the client session in an internal redis database or in the cookie, basic authentication form via a htpasswd file, setting and passing user headers, cookie configuration and others. If you want to check the details on how to configure a specific identity provider or to configure the oauth2-proxy to a more detailed level, I suggest to go through their public documentation.

Introducing the oauth2-proxy helm chart

In our exercise we’ve installed the kubernetes resources using the kubectl cli. If we would to write the resources by hand, it will be prone to errors due to the duplication of many kubernetes specific sections. oauth2-proxy framework offers a helm chart to ease up with the process of installing and managing the kubernetes resources.

The helm chart is being maintained in the oauth2-proxy github repository found here and it can be used as simple as:

helm install stable/oauth2-proxy --name my-release -f my-values.yaml

Conclusions and what’s next…

In this article we went through quite a few concepts, i.we started by installing on our local machine a single-node kubernetes cluster, ii.we then installed and exposed through an ingress a hello-world application, iii.we introduced and installed oauth2-proxy and dex frameworks and then iv.secured the access to hello-world application.

Note: Making an end-to-end exercise on a local environment is a very good start with the concepts and technologies. Stay tuned for another hands-on exercise using OpenShift on IBM Cloud cluster.

Big shout out to my colleagues from boomerang-io team for all their hard work and also to the OAuth2-Proxy maintainers for the doing an awsome work.

This one goes to nea Țuca.