At my current client, we recently switched from Kiam to a recently introduced feature from AWS to associate IAM roles with Kubernetes service accounts: EKS IAM Roles for Service Accounts. Although the switch was pretty straight forward, many things from EKS Roles for Service Accounts seem pretty magic to me, so I decided to dig a little bit deeper to understand what is going on behind the screen.
Disclaimer: a lot of the things described here are from my own research through different sources and may not be exactly how they are implemented. Feel free to drop me a message if I miss out anything.
The flow is summarised in the diagram above. There are a few components involved in this flow:
As instructed from official AWS guide.
This is so that EKS OIDC identity provider can act as the token issuer for your pods’ service accounts. For clusters that are manually managed by yourself, you will need to provide the url to OIDC identity provider as
--service-account-issuer flag when starting up Kubernetes API server.
For EKS, this should be done automatically for you once you enable EKS OIDC identity provider.
And annotate service account with the annotation required by EKS (
When this pod definition is submitted to Kubernetes, EKS Pod Identity Webhook intercepts the request before the pod spec is saved into etcd. This webhook is an admission controller and it modifies the pod spec to add in a few things:
- Environment variables:
AWS_ROLE_ARN: to be used by AWS SDK later to assume role
AWS_WEB_IDENTITY_TOKEN_FILE: location of service account token (identity token) to be saved by kubelet
- A service acount token volume projection This volume projection is used to project/save service account token to the location mounted by the volume.
Nothing special here, just regular Kubernetes work
Kubernetes used to leave service account token fixed without any expiration. Recent change in Kubernetes makes kubelet be responsible for regularly refreshing service account token before its expiration. Before the token expires, kubelet will contact tokens controller (one of the controllers defined in Kubernetes controller manager) to generate a new token.
The tokens controller will contact service account issuer defined in step 0 to get a new token for given service account. In EKS, this is the EKS OIDC identity provider.
Since the pod has the service account token volume projection defined, the token returned from tokens controller will be projected automatically in the specified location inside the pod.
When application inside pod makes any call to any AWS resource (through AWS SDK), AWS SDK will look up its credentials chain (list of classes that can provide AWS temporary credentials to the request) and use
AssumeRoleWithWebIdentityCredentials to assume role (defined in environment variable
AWS_ROLE_ARN above) and exchange identity token for temporary credentials.
This is also the reason why we need to grant permission for the OIDC provider to perform
And that’s it.
Overall this has been a fun experience for me to trace through different resources to understand the complete flow.