IL-0189 Adevinta · 2019-10-22 (published)kubernetesaws
Adevinta was moving a Spring JVM microservice from EC2 to its central platform with a Kubernetes runtime, as a blueprint for about 150 more services behind marketplaces in Spain such as Infojobs and Fotocasa. A Zuul API gateway sent requests either to the EC2 instances or through the NGINX ingress controller to the Kubernetes pods. For each request the service queried an AWS Elasticsearch domain one to three times, and the AWS Java SDK authorised those calls with temporary IAM credentials taken from the instance metadata API.
On Kubernetes, the instance metadata API is answered by KIAM, an agent on every node that intercepts metadata calls and serves the IAM role credentials assigned to the pod from its own cache. KIAM issues short-lived credentials, 15 minutes by default. The AWS Java SDK refreshes credentials when their expiry is within EXPIRATION_THRESHOLD, hardcoded to 15 minutes, or when the last refresh is older than REFRESH_THRESHOLD, hardcoded to 60 minutes. Every credential KIAM handed out was therefore already due for refresh, so before each Elasticsearch call the SDK fetched new credentials with two metadata calls, which added most of the extra latency. DNS lookups in the container were also slower than on EC2, but that was a smaller part.
The team found the SDK behaviour in AWS Java SDK GitHub issue #1921 and later a feature request describing the same problem. KIAM was reconfigured to request credentials with a longer expiration period. After that, requests no longer touched the metadata service and latency dropped below what it had been on EC2. The author stresses that neither Kubernetes, KIAM, the SDK nor the service had a bug: two defaults that each make sense on their own produced the problem together.
After a Spring-based JVM microservice was deployed to Kubernetes and part of production traffic was routed to it, request latency there was up to 10 times higher than on EC2. Behind the NGINX ingress controller the backend took 100 to 200 ms to answer, where the EC2 instances took about 20 ms.
The team ruled out the usual suspects. The JVM version was the same, the application already ran in containers on EC2, GC pauses were negligible, and latency was high even at one request per second, so load was not the cause. A Kubernetes admin suggested DNS, which had caused similar trouble before. In a shell inside the container, dig for the Elasticsearch domain took 21 to 43 ms per query ("Query time: 22 msec", "Query time: 43 msec"), while on an EC2 instance the queries after the first returned in 0 ms. The JVM DNS cache was set to networkaddress.cache.ttl = 10. Mapping the Elasticsearch domain to its IP in /etc/hosts improved latency somewhat but nowhere near the target.
A tcpdump capture taken in the container and read in Wireshark showed where the time went. Accepting the incoming request took 1 ms and the Elasticsearch call took 18 ms, but between them the service spent 86 ms on two HTTP calls to the AWS instance metadata endpoint on one connection: GET /latest/meta-data/iam/security-credentials and then GET /latest/meta-data/iam/security-credentials/arn:... These two calls appeared on every request in the capture.
Fetching the credentials by hand with curl, from the container and from an EC2 instance, showed the difference: the credentials served inside the container expired after exactly 15 minutes.
Symptom ? High latency
Root cause ? Configuration change
After part of production traffic was routed to the Kubernetes deployment, its request latency was seen to be up to 10 times higher than on EC2.
–
KIAM's short credential lifetime for pods and the AWS Java SDK's eager refresh of any credential within 15 minutes of expiry are each reasonable, but together they force a metadata round trip on every request. When latency rises after a move to Kubernetes, the cause is often the interaction of components that never ran together before rather than Kubernetes itself. Here a packet capture from inside the container found what latency metrics and DNS tests had not.
–