Releasing kubectl support in Access
Starting today, you can use Cloudflare Access and Argo Tunnel to securely manage your Kubernetes cluster with the kubectl command-line tool.
We built this to address one of the edge cases that stopped all of Cloudflare, as well as some of our customers, from disabling the VPN. With this workflow, you can add SSO requirements and a zero-trust model to your Kubernetes management in under 30 minutes.
Once deployed, you can migrate to Cloudflare Access for controlling Kubernetes clusters without disrupting your current
kubectl workflow, a lesson we learned the hard way from dogfooding here at Cloudflare.
What is kubectl?
A Kubernetes deployment consists of a cluster that contains nodes, which run the containers, as well as a control plane that can be used to manage those nodes. Central to that control plane is the Kubernetes API server, which interacts with components like the scheduler and manager.
kubectl is the Kubernetes command-line tool that developers can use to interact with that API server. Users run
kubectl commands to perform actions like starting and stopping the nodes, or modifying other elements of the control plane.
In most deployments, users connect to a VPN that allows them to run commands against that API server by addressing it over the same local network. In that architecture, user traffic to run these commands must be backhauled through a physical or virtual VPN appliance. More concerning, in most cases the user connecting to the API server will also be able to connect to other addresses and ports in the private network where the cluster runs.
How does Cloudflare Access apply?
Cloudflare Access can secure web applications as well as non-HTTP connections like SSH, RDP, and the commands sent over
kubectl. Access deploys Cloudflare’s network in front of all of these resources. Every time a request is made to one of these destinations, Cloudflare’s network checks for identity like a bouncer in front of each door.
If the request lacks identity, we send the user to your team’s SSO provider, like Okta, AzureAD, and G Suite, where the user can login. Once they login, they are redirected to Cloudflare where we check their identity against a list of users who are allowed to connect. If the user is permitted, we let their request reach the destination.
In most cases, those granular checks on every request would slow down the experience. However, Cloudflare Access completes the entire check in just a few milliseconds. The authentication flow relies on Cloudflare’s serverless product, Workers, and runs in every one of our data centers in 200 cities around the world. With that distribution, we can improve performance for your applications while also authenticating every request.
How does it work with kubectl?
To replace your VPN with Cloudflare Access for
kubectl, you need to complete two steps:
- Connect your cluster to Cloudflare with Argo Tunnel
- Connect from a client machine to that cluster with Argo Tunnel
Connecting the cluster to Cloudflare
On the cluster side, Cloudflare Argo Tunnel connects those resources to our network by creating a secure tunnel with the Cloudflare daemon,
cloudflared. As an administrator, you can run
cloudflared in any space that can connect to the kubectl API server over TCP.
Once installed, an administrator authenticates the instance of
cloudflared by logging in to a browser with their Cloudflare account and choosing a hostname to use. Once selected, Cloudflare will issue a certificate to
cloudflared that can be used to create a subdomain for the cluster.
Next, an administrator starts the tunnel. In the example below, the
hostname value can be any subdomain of the hostname selected in Cloudflare; the
url value should be the API server for the cluster.
cloudflared tunnel --hostname cluster.site.com --url tcp://kubernetes.docker.internal:6443 --socks5=true
This should be run as a
systemd process to ensure the tunnel reconnects if the resource restarts.
Connecting as an end user
End users do not need an agent or client application to connect to web applications secured by Cloudflare Access. They can authenticate to on-premise applications through a browser, without a VPN, like they would for SaaS tools. When we apply that same security model to non-HTTP protocols, we need to establish that secure connection from the client with an alternative to the web browser.
Unlike our SSH flow, end users cannot modify
kubeconfig to proxy requests through
cloudflared. Pull requests have been submitted to add this functionality to
kubeconfig, but in the meantime users can set an alias to serve a similar function.
First, users need to download the same
cloudflared tool that administrators deploy on the cluster. Once downloaded, they will need to run a corresponding command to create a local SOCKS proxy. When the user runs the command,
cloudflared will launch a browser window to prompt them to login with their SSO and check that they are allowed to reach this hostname.
$ cloudflared access tcp --hostname cluster.site.com url 220.127.116.11:1234
The proxy allows your local kubectl tool to connect to
cloudflared via a SOCKS5 proxy, which helps avoid issues with TLS handshakes to the cluster itself. In this model, TLS verification can still be exchanged with the
kubectl API server without disabling or modifying that flow for end users.
Users can then create an alias to save time when connecting. The example below aliases all of the steps required to connect in a single command. This can be added to the user’s bash profile so that it persists between restarts.
$ alias kubeone=”env HTTPS_PROXY=socks5://127.0.0.3:1234 kubectl
A (hard) lesson when dogfooding
When we build products at Cloudflare, we release them to our own organization first. The entire company becomes a feature’s first customer, and we ask them to submit feedback in a candid way.
Cloudflare Access began as a product we built to solve our own challenges with security and connectivity. The product impacts every user in our team, so as we’ve grown, we’ve been able to gather more expansive feedback and catch more edge cases.
kubectl release was no different. At Cloudflare, we have a team that manages our own Kubernetes deployments and we went to them to discuss the prototype. However, they had more than just some casual feedback and notes for us.
They told us to stop.
We had started down an implementation path that was technically sound and solved the use case, but did so in a way that engineers who spend all day working with pods and containers would find to be a real irritant. The flow required a small change in presenting certificates, which did not feel cumbersome when we tested it, but we do not use it all day. That grain of sand would cause real blisters as a new requirement in the workflow.
With their input, we stopped the release, and changed that step significantly. We worked through ideas, iterated with them, and made sure the Kubernetes team at Cloudflare felt this was not just good enough, but better.
kubectl is available in the latest release of the
cloudflared tool. You can begin using it today, on any plan. More detailed instructions are available to get started.
New to Cloudflare for Teams? You can use all of the Teams products for free through September. You can learn more about the program, and request a dedicated onboarding session, here.