Kubernetes Series: Managing ServiceAccounts Access to the k8s API

10 min readJun 6, 2023

Introduction

This article covers the k8s API Access Control . Section 1 gives an overview of the Kubernetes API and Access Control functioning. Section 2 presents the kubeconfig file structure. Furthermore, Section 3 illustrates the creation of serviceAccounts and their associated tokens on k8s. Finally, this article ends with a conclusion.

1. API Server Overview

Kubernetes (k8s) in an API driven client/server architecture where the API Server exposes an HTTP API that lets users, different parts of the cluster, and external components communicate with one another. [1]

For a better understanding of this article, basic knowledge of k8s architecture is required.

It is worth remembering that the API server is the single entry to the k8s cluster. It is the only component responsible for:

Receiving requests
Forwarding requests to the rest of k8s components
Responding to requests.

All k8s clusters have two categories of users: service accounts managed by k8s, and normal users. [2]

Kubernetes does not have objects which represent normal user accounts. Normal users cannot be added to a cluster through an API call.[2]
In contrast, service accounts are users managed by the Kubernetes API. They are bound to specific namespaces, and created automatically by the API server or manually through API calls. Service accounts are tied to a set of credentials stored as Secrets, which are mounted into pods allowing in-cluster processes to talk to the Kubernetes API.[2]

API requests are tied to either a normal user or a service account, or are treated as anonymous requests. This means every process inside or outside the cluster, from a human user typing kubectl on a workstation, to kubelets on nodes, to members of the control plane, must authenticate when making requests to the API server, or be treated as an anonymous user.[2]

Anonymous requests are given a username of system:anonymous and a group of system:unauthenticated. [2]

For a communication to be established with the API-server. The API-server performs authentication , authorization and admission control operations for each API request.

1.1 Authentication

authentication can be established via multiple methods : bearer tokens, basic credentials (i.e username and password) or certificates. These methods are provided via plugins that try to associate the following attributes with the request:

Username: a string which identifies the end user. Common values might be kube-admin or jane@example.com. [2]
UID: a string which identifies the end user and attempts to be more consistent and unique than username.[2]
Groups: a set of strings, each of which indicates the user’s membership in a named logical collection of users. Common values might be system:masters or devops-team.[2]
Extra fields: a map of strings to list of strings which holds additional information authorizers may find useful. [2]

You can enable multiple authentication methods at once. You should usually use at least two methods: [2]

service account tokens for service accounts
at least one other method for user authentication.

For example:

client certificate authentication method is enabled by passing the --client-ca-file=SOMEFILE option to API server manifest available under /etc/kubernetes/manifests/kube-apiserver.yaml .

cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep client-ca
# OUTPUT:
- --client-ca-file=/etc/kubernetes/pki/ca.crt

Even though a normal user cannot be added via an API call, any user that presents a valid certificate signed by the cluster’s certificate authority (CA) is considered authenticated. In this configuration, the API Server determines the username from the common name field in the subject of the certificate for example CN=Alice . It determines the group from the organization field of the certificate for example O=developers.From there, the role based access control (RBAC) sub-system would determine whether the user is authorized to perform a specific operation on a resource. [2]

Kubernetes Caution [6]

The API server reads bearer tokens from a file when given the --token-auth-file=SOMEFILE option. When using bearer token authentication from an http client, the API server expects an Authorization header with a value of Bearer <token>. [2]

Currently, tokens last indefinitely, and the token list cannot be changed without restarting the API server. [2]
The token file is a csv file with a minimum of 3 columns: token, user name, user uid, followed by optional group names. [2]

token,user,uid,"group1,group2,group3"

A service account is an automatically enabled authenticator that uses signed bearer tokens to verify requests. [2]

Service accounts are usually created automatically by the API server and associated with pods running in the cluster through the ServiceAccount Admission Controller. Bearer tokens are mounted into pods at well-known locations, and allow in-cluster processes to talk to the API server. Accounts may be explicitly associated with pods using the serviceAccountName field of a PodSpec. [2]
Service account bearer tokens are perfectly valid to use outside the cluster and can be used to create identities for long standing jobs that wish to talk to the Kubernetes API. [2]

1.2 Authorization

authorization is established through RBAC (Role, RoleBinding, ClusterRole, ClusterRoleBinding) that specifies if the user can or cannot perform CRUD operations (verbs) on certain k8s objects.

1.3 Admission control

admission control verifies if the CRUD operationperformed by the client on the k8s object is legitimate. For example, when the cluster has a ResourceQuota object that limits the number of podsthat can be created in a namespace dev to 2. If the user tries to create 3d pod then its request will be rejected.

2. The kubeconfig File

When kubectl is used to communicate with the API Server, It uses a config file that contains normal users certificates and the cluster’s CA.

This file should be available under $HOME/.kube/config . It is accessible via the following commands: [5]

# command 1: 
cat $HOME/.kube/config
# command 2: 
kubectl config view

The $HOME/.kube/config file looks as follows:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: xxxx
    server: https://x.x.x.x:6443
  name: test-cluster
contexts:
- context:
    cluster: test-cluster
    user: k8s-admin
  name: k8s-admin@test-cluster
current-context: k8s-admin@test-cluster
kind: Config
users:
- name: k8s-admin
  user:
    client-certificate-data: xx
    client-key-data: xxx

It consists of the following attributes:

clusters groups information about clusters. A cluster is defined by its name , its API endpoint (server) and its CA (Certificate Authority) certificate (certificate-authority-data) .
contexts group context information. A contextis used to group access parameters under a convenient name. Each contexthas three parameters: cluster, namespace, and user .

By default, the kubectl command-line tool uses parameters from the current contextto communicate with the API Server. current context indicates which cluster and user to use.

# To choose the current context:
kubectl config use-context k8s-admin@dev-cluster

users define the clusters users. It defines the user name (name) along with his credentials: certificate (client-certificate-data) and client key (client-key-data) . Clients certificates should be signed by the Cluster CA in order for authentication to be established.

3. Service Accounts

A service account provides an identity for processes that run in a Pod, and maps to a ServiceAccount object.[6]

ServiceAccounts have names prefixed with system:serviceaccount:, and belong to groups that have names prefixed with system:serviceaccounts:.

ServiceAccounts complete names follow the underlying pattern:

system:serviceaccount:namespace:serviceaccountname

# For example, a serviceaccount named prod-sa deployed
# within the namespace production will be named as follows:

system:serviceaccount:production:prod-sa

When Pods contact the API server, Pods authenticate as a particular ServiceAccount.[6]

Every Kubernetes namespace contains at least one ServiceAccount.the default ServiceAccount for each namespace is named default.

# Option 1: List service accounts within the default namespace
kubectl get serviceaccounts

# Option 2: List service accounts within the default namespace
kubectl get sa 
#OUTPUT
NAME      SECRETS   AGE
default   0         1d

If you do not specify a ServiceAccount when you create a Pod, Kubernetes automatically assigns the default ServiceAccount to the Pod.

# create a pod  
kubectl run nginx --image=nginx

# checkout the auto assigned service account 
kubeclt get pod nginx -o yaml  | grep serviceAccount
#OUTPUT
  serviceAccount: default
  serviceAccountName: default
      - serviceAccountToken:
 ...

An application running inside a Pod can access the Kubernetes API using automatically mounted service account credentials: mounted into /var/run/secrets/kubernetes.io/serviceaccount/token. Its its level of access however, depends on the RBAC policies set for the cluster.

# display the nginx auto mounted serviceAccount token 
kubectl exec -it nginx -- /bin/bash -c "cat /var/run/secrets/kubernetes.io/serviceaccount/token"

If you don’t want the kubelet to automatically mount a ServiceAccount’s API credentials, you can specify automountServiceAccountToken: falseon

The Pod’s .spec

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: build-robot
  automountServiceAccountToken: false
  ...

The ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: build-robot
automountServiceAccountToken: false
...

If both the ServiceAccount and the Pod’s .spec specify a value for automountServiceAccountToken, the Pod spec takes precedence.

4. Configure Service Accounts for Pods

4.1 With an auto mounted API Token

First, let’s create our ServiceAccount object

# Create the nginx-sa.yaml file 
kubectl create sa nginx-sa --dry-run=client -o yaml > nginx-sa.yaml

#OUTPUT: content of nginx-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-sa
# Apply the yaml file to create the serviceAccount object
kubectl apply -f nginx-sa.yaml
# Check if the serviceAccount has been created
kubectl get sa
#OUTPUT:
NAME       SECRETS   AGE
default    0         1d
nginx-sa   0         3s

Next, let’s create our Pod object

# Create the nginx-pod.yaml file 
kubectl run nginx --image=nginx --dry-run=client -o yaml > nginx-pod.yaml

# OUTPUT: content of nginx-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
  restartPolicy: Always

Let’s edit the nginx-pod.yamlfile to attach the created nginx-saServiceAccount to the nginx Pod

# edit nginx-pod.yaml: add serviceAccountName: nginx-sa

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx
  name: nginx
spec:
  serviceAccountName: nginx-sa
  containers:
  - image: nginx
    name: nginx
  restartPolicy: Always

Let’s create the Pod

kubectl apply -f nginx-pod.yaml

Let’s verify if the ServiceAccount has been attached to the Pod

kubectl get pod nginx -o yaml | grep serviceAccount

#OUTPUT:
  serviceAccount: nginx-sa
  serviceAccountName: nginx-sa
      - serviceAccountToken:
  ...

Let’s display the ServiceAccount token that has been auto mounted to the Pod

# display the auto-mounted serviceAccount token 
kubectl exec -it nginx -- /bin/bash -c "cat /var/run/secrets/kubernetes.io/serviceaccount/token"

Finally, Let’s use this auto-mounted token to communicate with the API Server

# bash into the pod 
kubectl exec -it nginx -- /bin/bash

# export the token as an environment variable within the pod 
export token=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
# curl the api server using the auto mounted token 
curl -k https://kubernetes.default/api/v1/pods --oauth2-bearer $token
# OUTPUT 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:default:nginx-sa\" cannot list resource \"pods\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}

We can see from the output error message that the nginx has been successfully authenticated but the authorization operation was not successful because the user nginx Pod, identified as user : system:serviceaccount:default:nginx-sa doesn’t have permissions to query k8s objects. Permissions are managed in k8s using RBAC. They’ll be discussed in the next article.

4.2 Without an auto mounted API Token

First, let’s create our ServiceAccount object

# Create the nginx-sa.yaml file 
kubectl create sa nginx-sa --dry-run=client -o yaml > nginx-sa.yaml

#OUTPUT: content of nginx-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-sa

Next, let’s edit nginx-sa.yaml file to add the setting automountServiceAccountToken: false

#OUTPUT: content of nginx-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-sa
automountServiceAccountToken: false

Let’s apply the ServiceAccount object

# Apply the yaml file to create the serviceAccount object
kubectl apply -f nginx-sa.yaml

# Check if the serviceAccount has been created
kubectl get sa
#OUTPUT:
NAME       SECRETS   AGE
default    0         1d
nginx-sa   0         3s

Next, let’s create our Pod object

# Create the nginx-pod.yaml file 
kubectl run nginx --image=nginx --dry-run=client -o yaml > nginx-pod.yaml

# OUTPUT: content of nginx-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
  restartPolicy: Always

Let’s edit the nginx-pod.yamlfile to attach the created nginx-saServiceAccount to the nginx Pod

# edit nginx-pod.yaml: add serviceAccountName: nginx-sa

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx
  name: nginx
spec:
  serviceAccountName: nginx-sa
  containers:
  - image: nginx
    name: nginx
  restartPolicy: Always

Let’s create the Pod

kubectl apply -f nginx-pod.yaml

Let’s verify if the ServiceAccount has been attached to the Pod

kubectl get pod nginx -o yaml | grep serviceAccount

#OUTPUT:
  serviceAccount: nginx-sa
  serviceAccountName: nginx-sa
      - serviceAccountToken:
  ...

Let’s display the ServiceAccount token that has been auto mounted to the Pod

# display the auto-mounted serviceAccount token 
kubectl exec -it nginx -- /bin/bash -c "cat /var/run/secrets/kubernetes.io/serviceaccount/token"

# OUTPUT
cat: /var/run/secrets/kubernetes.io/serviceaccount/token: No such file or directory

We can see that due to the usage of automountServiceAccountToken: false no token has been created and mounted to the Pod.

4.2.1 Manually create an API token for a ServiceAccount

Let’s create a time-limited API token for the ServiceAccount nginx-sausing kubectl:

kubectl create token nginx-sa --duration 1h

This token can be copied and mounted to any pod under /var/run/secrets/kubernetes.io/serviceaccount/token, It will allow the pod to authenticate as nginx-sa ServiceAccount

Conclusion

The primary goal of this article was to give a deeper understanding of the Kubernetes API Access Control functioning through practical examples. This has been accomplished through giving a general overview about the k8s API then getting more practical through different examples using the curl and kubectlcommands.

References

[1] The Kubernetes Api (2023) Kubernetes. Available at: https://kubernetes.io/docs/concepts/overview/kubernetes-api/ (Accessed: 05 June 2023).

[2] Authenticating (2023) Kubernetes. Available at: https://kubernetes.io/docs/reference/access-authn-authz/authentication/ (Accessed: 05 June 2023).

[3] Controlling access to the Kubernetes Api (2023) Kubernetes. Available at: https://kubernetes.io/docs/concepts/security/controlling-access/ (Accessed: 05 June 2023).

[4] Generate certificates manually (2023) Kubernetes. Available at: https://kubernetes.io/docs/tasks/administer-cluster/certificates/ (Accessed: 05 June 2023).

[5] Configure access to multiple clusters (2023) Kubernetes. Available at: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/ (Accessed: 05 June 2023).

[6] Configure service accounts for Pods (2023) Kubernetes. Available at: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/ (Accessed: 06 June 2023).

[7] Using RBAC authorization (2022) Kubernetes. Available at: https://kubernetes.io/docs/reference/access-authn-authz/rbac/ (Accessed: 07 June 2023).