Preface
On March 4, I reported a security vulnerability in kubectl to the Kubernetes and OpenShift security teams, which was assigned CVE-2019-1002101. This post explains the discovery process, the vulnerability details and its impact and exploitation methods. Thanks to Brandon Phillips Red Hat for coordinating the disclosure process. The announcement made today by the Kubernetes team can be found here.
Vulnerability discovery
I was exploring Kubernetes commands when a particular notice drew my immediate attention.
This note refers to the kubectl cp command, which allows copying files between containers and the user machine. To copy files from and to containers, Kubernetes calls the `tar` binary inside the container, to either create or unpack a tar archive with the requested files. This has always been the way the cp command worked, but I was surprised that the same design was still in use after a vulnerability in the cp code was found about a year ago.
Some of the readers might recall the directory traversal vulnerability that was found in the cp command last March. It was assigned CVE-2018-1002100, though it was mostly overshadowed by a similar vulnerability in OpenShift that was announced in the same post.
This vulnerability was ultimately a “classic” directory traversal – paths include directory climbing using ../ (dot dot slash) were not sanitized, allowing malicious containers to write any file to any path on the user machine when copied from.
Before we dig in to the vulnerability I found, let’s examine the kubectl cp architecture in more detail.
To copy files from a container, kubectl creates a tar with the source files inside the container and unpacks it on the user machine. To copy files from the user machine to a container, kubectl creates a tar with the files and unpacks it inside the container.
For creating and unpacking a tar archive on the user machine, kubectl relies on its tar parsing code in cmd/cp/cp.go. However, for containers kubectl executes the `tar` binary inside the container for these actions. For that reason, a `tar` binary is required to be in the container for the cp command to work, as specified in the documentation notice.
If the `tar` binary on the container is malicious, it could run any code and output unexpected, malicious results. This is a prerequisite for both the previous vulnerability and the new one to be exploited.
The function `copyFromPod` in cp.go implements the process of copying files from the container. It calls tar in the container through remote exec (`&exec.DefaultRemoteExecutor`) and then untars the result on the user machine in the function `untarAll`. This function uses the “archive/tar” Go package for tar parsing and finally writes the files to the destination directory based on the result tar headers.
Since the fix to CVE-2018-1002100, the untarring function calls the `cp.go:clean` function to strip path traversals.
However, after reading its code I realised that the function can both create and follow symbolic links from the tar headers. An attacker could craft a malicious tar that has a header with a symbolic link to virtually any path, and a subsequent header to a file inside a directory with the same name as the symlink. When extracted by the cp untar function, the link would result in creating or modifying the desired file in the path from the symbolic link.
Impact
This vulnerability can be dangerous in one of two scenarios:
- A user unknowingly downloads a malicious container image with a bad tar. The attacker can push such an image to any registry (e.g. Docker Hub) for a popular image he has control of or rely on typosquatting.
- An attacker compromises a running container by exploiting another vulnerability or in some cases he may have legitimate access to a container. The attacker then plants a malicious tar replacing the original tar of the image.
A sophisticated attacker will also conceal the exploit by making the malicious tar include the originally requested files in its output, or make the symlink file a dotfile, so that the attack is not immediately apparent.
So far I have only mentioned directory traversal, which is dangerous by itself. Kubectl users are generally developers, devops staff, or administrators, whose environment contains sensitive configuration and deployment files. Targeted exploitation of this vulnerability can thus lead to leakage and exfiltration of potentially highly sensitive information. Nevertheless, you may be wondering how straightforward it is for an attacker to get full code execution on the user machine through this vulnerability.
First, if kubectl is run as root, achieving remote code execution becomes trivial through the modification of one of many configuration or system files. Files like /etc/profile or cron files are good targets, but virtually any startup file are easy targets to get immediate and persistent code execution.
Yet kubectl is more commonly run as a user. This makes it harder for an attacker to deterministically get code execution. It eventually sums up to the creativity of the attacker or how much information about the user filesystem may be known, but for the sake of this post I present a simple idea that may work in many cases.
In the proof of concept found below, I get code execution by writing to the `.bashrc` file in the directory kubectl is run from. The path /proc/self/cwd links to that directory. If kubectl is ran from the user’s home directory, the bash loading script is replaced with my payload which is run the next time a bash shell is spawned by the user.
Proof of Concept
The following infographic goes through the steps of this exploit.
Suggestion for redesign
This vulnerability can be closed by fixing the `untarAll` function and writing regular files instead of symbolic links. Within the original advisory, I’ve also reported another non-security out-of-bounds access bug in the same function. The maintainers can probably close both issues with a simple PR, the CVE will be marked as fixed and everybody updates and move on.
The way I see it though, kubectl cp, or for that matter any Kubernetes command, should not be calling tar or execute any binary inside the container. Any output from inside a container should not be considered safe or reliable, since it can be controlled by an attacker if compromised. Even without a vulnerability, the binary could output to the user terminal any misleading text (in the tar case, through stderr) that could confuse even an experienced user. Or potentially attempt to exploit terminal vulnerabilities to do much more. (1)
An issue has already been opened for getting rid of the tar requirement in kubectl cp, although on the basis of usability and not security.
Twistlock protection
Twistlock can alert or prevent a change of a tar binary inside a running container, alerting on a breach and preventing the exploitation of this vulnerability. Twistlock also has a CIS benchmark that can be enforced for all images to run with a readonly rootfs, which prevents hijacking tar or any binary inside a running container altogether.
Twistlock also allows you to define Trusted Images, a feature introduced 2 years ago in Twistlock 1.7, that enables marking a specific set of registries, repositories, images or base layers that are allowed within your organization.
Finally, with the new Kubernetes Auditing feature you could gain visibility over use of the cp command and easily pin down the events where a user may have been affected by a compromised container.
To monitor all cp events, use the following rule:
jpath("stage") = "ResponseComplete" and jpath("objectRef.resource") = "pods" and jpath("objectRef.subresource") = "exec" and jpath("verb") = "create" and jpath("requestURI") contains "command=tar"
Thanks to Misha and Omri for providing it.
Summary
This vulnerability only affects the user client for Kubernetes, kubectl (and the OpenShift equivalent, oc). All kubectl versions from v1.9.0-alpha.2 to 1.14.0 are vulnerable (1.11.9, 1.12.7, 1.13.5 also have the fix).
If you like our advisories, follow us on Twitter and be the first to read our posts.