Kubernetes Series: Managing Normal Users Access to the k8s API

9 min readJun 5, 2023

Introduction

This article covers the k8s API Access Control . Section 1 gives an overview of the Kubernetes API and Access Control functioning. Section 2 presents the kubeconfig file structure. Furthermore, Section 3 illustrates the creation of normal users on k8s. Finally, this article ends with a conclusion.

1. API Server Overview

Kubernetes (k8s) in an API driven client/server architecture where the API Server exposes an HTTP API that lets users, different parts of the cluster, and external components communicate with one another. [1]

For a better understanding of this article, basic knowledge of k8s architecture is required.

It is worth remembering that the API server is the single entry to the k8s cluster. It is the only component responsible for:

Receiving requests
Forwarding requests to the rest of k8s components
Responding to requests.

All k8s clusters have two categories of users: service accounts managed by k8s, and normal users. [2]

Kubernetes does not have objects which represent normal user accounts. Normal users cannot be added to a cluster through an API call.[2]
In contrast, service accounts are users managed by the Kubernetes API. They are bound to specific namespaces, and created automatically by the API server or manually through API calls. Service accounts are tied to a set of credentials stored as Secrets, which are mounted into pods allowing in-cluster processes to talk to the Kubernetes API.[2]

API requests are tied to either a normal user or a service account, or are treated as anonymous requests. This means every process inside or outside the cluster, from a human user typing kubectl on a workstation, to kubelets on nodes, to members of the control plane, must authenticate when making requests to the API server, or be treated as an anonymous user.[2]

Anonymous requests are given a username of system:anonymous and a group of system:unauthenticated. [2]

For a communication to be established with the API-server. The API-server performs authentication , authorization and admission control operations for each API request.

1.1 Authentication

authentication can be established via multiple methods : bearer tokens, basic credentials (i.e username and password) or certificates. These methods are provided via plugins that try to associate the following attributes with the request:

Username: a string which identifies the end user. Common values might be kube-admin or jane@example.com. [2]
UID: a string which identifies the end user and attempts to be more consistent and unique than username.[2]
Groups: a set of strings, each of which indicates the user’s membership in a named logical collection of users. Common values might be system:masters or devops-team.[2]
Extra fields: a map of strings to list of strings which holds additional information authorizers may find useful. [2]

You can enable multiple authentication methods at once. You should usually use at least two methods: [2]

service account tokens for service accounts
at least one other method for user authentication.

For example:

client certificate authentication method is enabled by passing the --client-ca-file=SOMEFILE option to API server manifest available under /etc/kubernetes/manifests/kube-apiserver.yaml .

cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep client-ca

# OUTPUT:
- --client-ca-file=/etc/kubernetes/pki/ca.crt

Even though a normal user cannot be added via an API call, any user that presents a valid certificate signed by the cluster’s certificate authority (CA) is considered authenticated. In this configuration, the API Server determines the username from the common name field in the subject of the certificate for example CN=Alice . It determines the group from the organization field of the certificate for example O=developers.From there, the role based access control (RBAC) sub-system would determine whether the user is authorized to perform a specific operation on a resource. [2]

The API server reads bearer tokens from a file when given the --token-auth-file=SOMEFILE option. When using bearer token authentication from an http client, the API server expects an Authorization header with a value of Bearer <token>. [2]

Currently, tokens last indefinitely, and the token list cannot be changed without restarting the API server. [2]

The token file is a csv file with a minimum of 3 columns: token, user name, user uid, followed by optional group names. [2]

token,user,uid,"group1,group2,group3"

A service account is an automatically enabled authenticator that uses signed bearer tokens to verify requests. [2]

Service accounts are usually created automatically by the API server and associated with pods running in the cluster through the ServiceAccount Admission Controller. Bearer tokens are mounted into pods at well-known locations, and allow in-cluster processes to talk to the API server. Accounts may be explicitly associated with pods using the serviceAccountName field of a PodSpec. [2]
Service account bearer tokens are perfectly valid to use outside the cluster and can be used to create identities for long standing jobs that wish to talk to the Kubernetes API. [2]

1.2 Authorization

authorization is established through RBAC (Role, RoleBinding, ClusterRole, ClusterRoleBinding) that specifies if the user can or cannot perform CRUD operations (verbs) on certain k8s objects.

1.3 Admission control

admission control verifies if the CRUD operationperformed by the client on the k8s object is legitimate. For example, when the cluster has a ResourceQuota object that limits the number of podsthat can be created in a namespace dev to 2. If the user tries to create 3d pod then its request will be rejected.

2. The kubeconfig File

When kubectl is used to communicate with the API Server, It uses a config file that contains normal users certificates and the cluster’s CA.

This file should be available under $HOME/.kube/config . It is accessible via the following commands: [5]

# command 1: 
cat $HOME/.kube/config
# command 2: 
kubectl config view

The $HOME/.kube/config file looks as follows:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: xxxx
    server: https://x.x.x.x:6443
  name: test-cluster
contexts:
- context:
    cluster: test-cluster
    user: k8s-admin
  name: k8s-admin@test-cluster
current-context: k8s-admin@test-cluster
kind: Config
users:
- name: k8s-admin
  user:
    client-certificate-data: xx
    client-key-data: xxx

It consists of the following attributes:

clusters groups information about clusters. A cluster is defined by its name , its API endpoint (server) and its CA (Certificate Authority) certificate (certificate-authority-data) .
contexts group context information. A contextis used to group access parameters under a convenient name. Each contexthas three parameters: cluster, namespace, and user .

By default, the kubectl command-line tool uses parameters from the current contextto communicate with the API Server. current context indicates which cluster and user to use.

# To choose the current context:
kubectl config use-context k8s-admin@dev-cluster

users define the clusters users. It defines the user name (name) along with his credentials: certificate (client-certificate-data) and client key (client-key-data) . Clients certificates should be signed by the Cluster CA in order for authentication to be established.

3. Manage Access to the API Server: Normal Users

Let’s create a normal user and grant him access to the k8s API Server [4] [5]

# login as sudo 
sudo su

# Let's create an ubuntu user
adduser test

# Generate a private key for test.
openssl genrsa -out test.key 2048

# Generate a Certificate Signing Request (CSR) for test
openssl req -new -key test.key -out test.csr -subj "/CN=test/O=devops"

# Generate a Certificate signed by the Cluster CA for the user test with a validity period of 30 days
openssl x509 -req -in test.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out test.crt -days 30

# Move the generated credentials to test user home
mv test.key /home/test/
mv test.crt /home/test/
chown test:test /home/test/test.crt /home/test/test.key

3.1 Curl the API Server

When curl is used to communicate with the API Server . a user passes his certificates and the cluster’s CA certificate on command line as follows:

# login as root
sudo su

# export the API-Server endpoint
export server=https://x.x.x.x:6443

# Curl the API-Server endpoint with test user credentials 
curl $server/api/v1/pods/ --cacert /etc/kubernetes/pki/ca.crt --cert /home/test/test.crt --key /home/test/test.key

# OUTPUT:
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"test\" cannot list resource \"pods\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}

We can see from the output error message that the user test has been successfully authenticated but the authorization operation was not successful because the user doesn’t have permissions to query k8s objects.

Permissions are managed in k8s using RBAC.

3.2 Configure the kubeconfig File

When kubectl is used, It uses a config file which contains user certificates and the cluster’s CA. This file should be available under $HOME/.kube/config

Let’s create a userwithin $HOME/.kube/config file.

# specify credentials for user test within the config file : users attribute

kubectl config set-credentials test \
--client-certificate=/home/test/test.crt \
--client-key=/home/test/test.key

Let’s check the configured user lines in $HOME/.kube/config file

# Since we're performing operations as root, the file will be located under /root/.kube/config

apiVersion: v1
clusters: null
contexts: null
current-context: ""
kind: Config
preferences: {}
users:
- name: test
  user:
    client-certificate: /home/test/test.crt
    client-key: /home/test/test.key

Let’s create a contextwithin $HOME/.kube/config file.

kubectl config set-context test@test-cluster \
--cluster=test-cluster \
--namespace=test \
--user=test

Let’s check the configured context lines in $HOME/.kube/config file

# Since we're performing operations as root, the file will be located under /root/.kube/config
apiVersion: v1
clusters: null
contexts:
- context:
    cluster: test-cluster
    namespace: test
    user: test
  name: test@test-cluster
current-context: ""
kind: Config
preferences: {}
users:
- name: test
  user:
    client-certificate: /home/test/test.crt
    client-key: /home/test/test.key

Finally, let’s set information about the k8s cluster we want to connect to:

kubectl config set-cluster test-cluster \
--server=https://x.x.x.x:6443 \
--embed-certs \
--certificate-authority=/etc/kubernetes/pki/ca.crt

Let’s check the configured clusterlines in $HOME/.kube/config file

# Since we're performing operations as root, the file will be located under /root/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: xxxxx
    server: https://x.x.x.x:6443
  name: test-cluster
contexts:
- context:
    cluster: test-cluster
    namespace: test
    user: test
  name: test@test-cluster
current-context: ""
kind: Config
preferences: {}
users:
- name: test
  user:
    client-certificate: /home/test/test.crt
    client-key: /home/test/test.key

Let’s Display configured information within the $HOME/.kube/config file

# move the configured config file under the home of test user

mv /root/.kube/ /home/test/
sudo chown -R test:test /home/test/

# login as test user
sudo su test

kubectl config get-contexts
kubectl config set current-context "test@test-cluster"
kubectl config current-context

List pods using kubectl command that uses $HOME/.kube/config file.

kubectl --context=test@test-cluster get pod

#Output: 
Error from server (Forbidden): pods is forbidden: User "test" cannot list resource "pods" in API group "" in the namespace "test"

Permissions are managed in k8s using RBAC.

Conclusion

The primary goal of this article was to give a deeper understanding of the Kubernetes API Access Control functioning through practical examples. This has been accomplished through giving a general overview about the k8s API then getting more practical through different examples using the curl and kubectlcommands.

References

[1] The Kubernetes Api (2023) Kubernetes. Available at: https://kubernetes.io/docs/concepts/overview/kubernetes-api/ (Accessed: 05 June 2023).

[2] Authenticating (2023) Kubernetes. Available at: https://kubernetes.io/docs/reference/access-authn-authz/authentication/ (Accessed: 05 June 2023).

[3] Controlling access to the Kubernetes Api (2023) Kubernetes. Available at: https://kubernetes.io/docs/concepts/security/controlling-access/ (Accessed: 05 June 2023).

[4] Generate certificates manually (2023) Kubernetes. Available at: https://kubernetes.io/docs/tasks/administer-cluster/certificates/ (Accessed: 05 June 2023).

[5] Configure access to multiple clusters (2023) Kubernetes. Available at: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/ (Accessed: 05 June 2023).

[6] Using RBAC authorization (2022) Kubernetes. Available at: https://kubernetes.io/docs/reference/access-authn-authz/rbac/ (Accessed: 07 June 2023).