IL-0052 JW Player · 2019-03-19 (published)kubernetesawssecurity
JW Player's DevOps team found a Monero miner, named gcc, running on every node of its development and staging Kubernetes clusters. Production was not affected, no customer data was accessed and service was not interrupted. The clusters had recently been rebuilt on Kubernetes 1.10 with different instance sizes, so some unusual behaviour was expected, and noisy load alerts from a legitimate service delayed the investigation.
The miner's parent process was Weave Scope, a tool for watching a Kubernetes cluster in real time. No CVE applied and its image hash matched the published Docker Hub image. Rebuilding an isolated cluster with the same deployment method showed the entry point: the Weave Scope service was a LoadBalancer without the annotation that makes the load balancer internal, so its dashboard was reachable from the internet with no login. Weave Scope can open a command line in any container, and the scope-agent container, which runs on every node, was started with --privileged, mounted files from the host's root filesystem and ran as root. The team reproduced the attack by starting a script on a host from a scope-agent shell.
AWS CloudTrail explained why an internal dashboard was public. When Kubernetes created the load balancer, it attached a security group open to 0.0.0.0/0 on port 80. Someone then restricted it to allow-listed IPs by hand in the AWS console, but Kubernetes owns that load balancer and reverted the edit shortly afterwards.
The damage was one CPU core at 100% on each node in development and staging. Access logs showed no outside logins, the Kubernetes API was not used by the intruder, and RBAC limited Weave Scope to the weave namespace, so a request for secrets would have been denied. The team concluded the crawler's only purpose was mining.
Datadog alerted the DevOps team to a high normalised load average, first on the staging cluster (Day 0, 21:06 EST) and the next day on the development cluster (Day 1, 16:53 EST). The load was first put down to a legitimate service that kept scaling up and down in both environments, and because of the volume of identical alerts they were muted until things settled.
On Day 3 at 17:40 EST, after four days of raised load, an engineer logged onto a node over SSH and ran top. A process named gcc was using 100% of a CPU, and the same process was running on every machine in both clusters, master nodes included.
The first guess was a faulty container launched by the internal deployment tool, but that did not explain why master nodes were affected, and the infrastructure-as-code repositories contained nothing that ran gcc. Inspecting the process showed that its parent was a Weave Scope container, that the binary ran from the host's root directory rather than inside a container, and that it was in fact a cryptocurrency miner. Weave Scope's own view showed the process holding an outbound connection to a mining pool; a search for the name XMRig led to a Monero miner.
Symptom ? Sustained high node CPUUnauthorised access or a foreign workload
Root cause ? Configuration change
Datadog alerts on high normalised load average, muted at first as noise from a legitimate service; on Day 3 an engineer ran top on a node and found the gcc process at 100% CPU.
Load average was not a useful signal for spotting a problem in the clusters, and muted noisy alerts nearly hid the miner. A load balancer that Kubernetes creates belongs to Kubernetes: a manual security group edit in the AWS console is reverted, so internal exposure must be declared in the Kubernetes YAML. A dashboard that can open shells, in front of privileged agents that mount the host's root filesystem, gives whoever reaches it root on every node.
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer
same root cause · shared Cloud load balancer