• November 10, 2023
• By Cyberarch Admin

KUBERNETES

Kubernetes is an open-source container orchestration engine for automating deployment, scaling, and management of containerized applications. The open-source project is hosted by the Cloud Native Computing Foundation (CNCF).When you deploy Kubernetes, you get a cluster. A Kubernetes cluster consists of a set of worker machines, called nodes that run containerized applications. The control plane manages the worker nodes and the Pods in the cluster.

Control Plane Components

The control plane’s components make global decisions about the cluster, as well as detecting and responding to cluster events. It consists of components such as kube-apiserver, etcd, kube-scheduler, kube-controller-manager and cloud-controller-manager.

kube-apiserver – kube-apiserver exposes the Kubernetes API. The API server is the front end for the Kubernetes control plane.

etcd – etcd is a consistent and highly-available key-value store used as Kubernetes’ backing store for all cluster data.

kube-scheduler – kube-scheduler watches for newly created Pods with no assigned node, and selects a node for them to run on.

kube-controller  – manager kube-controller-manager runs controller processes. Logically, each controller is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process.

cloud-controller-manager – The cloud controller manager lets you link your cluster into your cloud provider’s API, and separates out the components that interact with that cloud platform from components that just interact with your cluster.

Node Components

Node components run on every node, maintaining running pods and providing the Kubernetes runtime environment. It consists of components such as kubelet, kube-proxy and container runtime.

kubelet –  kubelet is an agent that runs on each node in the cluster. It makes sure that containers are running in a Pod.

kube-proxy –  kube-proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept.

Container runtime –  The container runtime is the software that is responsible for running containers.

 

Source: https://cheatsheetseries.owasp.org/cheatsheets/Kubernetes_Security_Cheat_Sheet.html

 

ARCHITECTURE

Source: https://hacktricks.boitatech.com.br/pentesting/pentesting-kubernetes

• NODE: Operating Systems with pod or pods.

Pod: Wrapper around a container or multiple containers with. A pod should only contain one application (so usually, a pod run just 1 container). The pod is the way Kubernetes abstracts the container technology running.

Service: Each pod has 1 internal IP address from the internal range of the node. However, it can be also exposed via a service. The service has also an IP address and its goal is to maintain the communication between pods so if one dies the new replacement (with a different internal IP) will be accessible exposed in the same IP of the service. It can be configured as internal or external. The service also actuates as a load balancer when 2 pods are connected to the same service. When a service is created you can find the endpoints of each service running kubectl get endpoints.

Kubelet: Primary node agent. The component that establishes communication between node and kubectl, and only can run pods (through API server). The kubelet doesn’t manage containers that were not created by Kubernetes.

Kube-proxy: is the service in charge of the communications (services) between the apiserver and the node. The base is an IP tables for nodes. Most experienced users could install other kube-proxies from other vendors.

Sidecar container: Sidecar containers are the containers that should run along with the main container in the pod. This sidecar pattern extends and enhances the functionality of current containers without changing them. Nowadays, we know that we use container technology to wrap all the dependencies for the application to run anywhere. A container does only one thing and does that thing very well.

•  Master process:

Api Server: Is the way the users and the pods use to communicate with the master process. Only authenticated request should be allowed.

Scheduler: Scheduling refers to making sure that Pods are matched to Nodes so that Kubelet can run them. It has enough intelligence to decide which node has more available resources the assign the new pod to it. Note that the scheduler doesn’t start new pods, it just communicates with the Kubelet process running inside the node, which will launch the new pod.

Kube Controller manager: It checks resources like replica sets or deployments to check if, for example, the correct number of pods or nodes are running. In case a pod is missing, it will communicate with the scheduler to start a new one. It controls replication, tokens, and account services to the API.

etcd: Data storage, persistent, consistent, and distributed. Is Kubernetes’s database and the key-value storage where it keeps the complete state of the clusters (each change is logged here). Components like the Scheduler or the Controller manager depends on this date to know which changes have occurred (available resourced of the nodes, number of pods running…)

Cloud Controller Manager: Is the specific controller for flow controls and applications, i.e: if you have clusters in AWS or OpenStack.Note that as the might be several nodes (running several pods), there might also be several master processes which their access to the Api server load balanced and their etcd synchronized.

Volumes: When a pod creates data that shouldn’t be lost when the pod disappear it should be stored in a physical volume. Kubernetes allow to attach a volume to a pod to persist the data. The volume can be in the local machine or in a remote storage. If you are running pods in different physical nodes, you should use a remote storage so all the pods can access it.

• Other Configurations:

ConfigMap: You can configure URLs to access services. The pod will obtain data from here to know how to communicate with the rest of the services (pods). Note that this is not the recommended place to save credentials!

Secret: This is the place to store secret data like passwords, API keys… encoded in B64. The pod will be able to access this data to use the required credentials.

Deployments: This is where the components to be run by kubernetes are indicated. A user usually won’t work directly with pods, pods are abstracted in ReplicaSets (number of same pods replicated), which are run via deployments. Note that deployments are for stateless applications. The minimum configuration for a deployment is the name and the image to run.

StatefulSet: This component is meant specifically for applications like databases which needs to access the same storage.

Ingress: This is the configuration that is use to expose the application publicly with an URL. Note that this can also be done using external services, but this is the correct way to expose the application. If you implement an Ingress you will need to create Ingress Controllers. The Ingress Controller is a pod that will be the endpoint that will receive the requests and check and will load balance them to the services. the ingress controller will send the request based on the ingress rules configured. Note that the ingress rules can point to different paths or even subdomains to different internal kubernetes services.

A better security practice would be to use a cloud load balancer or a proxy server as entrypoint to don’t have any part of the Kubernetes cluster exposed. When request that doesn’t match any ingress rule is received, the ingress controller will direct it to the “Default backend”. You can describe the ingress controller to get the address of this parameter. minikube addons enable ingress

 

PENTESTING PHASE

Kubernetes is a popular container orchestration platform used for managing and scaling containerized applications. When performing penetration testing on a Kubernetes environment, you’re essentially assessing the security of the Kubernetes cluster and the containerized applications running within it. Here are some key points to consider when conducting Kubernetes penetration testing:

1.Reconnaissance

Determine the IP address and port of the Kubernetes API server (typically 6443). Perform a scan of Kubernetes-related services and ports using tools like Nmap or kube-scan. Gather information about namespaces, pods, services, and nodes using tools like kubectl, kubectx, and kubens.

• Basic Recon

nslookup kubernetes.default

nmap -p- <Kubernetes API IP>

kubectlget namespaces

kubectl get pods -n <namespace>

• Check your Kubernetes server version

kubectl versionThe kubectl version command is used to check the client and server versions of Kubernetes. It provides information about the version of the kubectl command-line tool you are using (the client version) and the version of the Kubernetes server (the server version) to which kubectl is configured to connect. This information is helpful for ensuring that your kubectl client is compatible with the Kubernetes server, and it can also be useful when troubleshooting issues or coordinating updates within a Kubernetes environment.

• Dump All

Based On: https://github.com/ProfessionallyEvil/harpoon/blob/master/fingerprinting/k8s_dump.sh

for res in $(kubectl api-resources -o name);do kubectl get “${res}” -A -o yaml > ${res}.yaml; done

• Check Anonymous Access

#Kube API
curl -k https://<master_ip>:<port> #Check etcdctl (binary on https://github.com/etcd-io/etcd/releases)
etcdctl –endpoints=http://<MASTER-IP>:2379 get / –prefix –keys-only #Kubelet
curl http://<external-IP>:10255/pods

These resources may allow us to retrieve many information from the cluster or even run commands anonymously.

Etcdctl Cheat sheet: https://lzone.de/cheat-sheet/etcd

• Check Images Version

kubectl get deployments -A -o yaml | grep “image:”

This will output all the images deployed on the cluster and their version, then check out if some of them are vulnerable. This helped me to discover a vulnerable blackbox image that allows SSRF into the cluster, so do it!

• Check Services

kubectl get services -A -o wide

You can take this output as an nmap result, there’re services exposed (or not, maybe they’re only reachable inside the cluster) just check and try to exploit those services as a “legacy” system.

• Check Deployments

We can use deployments.yaml extracted on section b (DUMP ALL), or:

kubectl get deployments -A -o yaml > deployments.yaml

Deployments are the main recipe, here’s where the DevOps tell to kubernetes what they want to run and how everything should work. We recommend this tool that is not mentioned in the previous article but it works pretty fine, let me introduce you to checkov: https://github.com/bridgecrewio/checkov

checkov -f deployments.yaml

• Check Permissions

kubectl auth can-i –list (-n <namespace)

Check what you can do, the best thing you can find here is * which means ALL resources (pods, secrets, deployments…) and/or ALL actions (get, list, create, patch….).

• Check Risky Roles

If you have enough permissions just check the role binding permissions with this:

https://github.com/cyberark/kubernetes-rbac-audit

• Pod Capabilities

Enter to any pod:

kubectl exec -ti <pod_name> -n <namespace> bash

Some images have no bash or PATH variable is not defined, so you can also run:

kubectl exec -ti <pod_name> -n <namespace> /bin/sh

Then, when you’re in a pod check what it can do:

capsh –print

• Sensitive Informations

Pods are interesting not only by its capabilities, they may contain sensitive information so get in and harvest some intel.

• Tokens

Every pod as a token (JWT) to talk to API Server, so retrieve them from inside the pod:

#Dump tokens from inside the pod
kubectl exec -ti <pod> -n <namespace> cat /run/secrets/kubernetes.io/serviceaccount/token#Dump all tokens from secrets
kubectl get secrets -A -o yaml | grep ” token:” | sort | uniq > alltokens.txt

To check if those tokens can reach other namespaces or can read sensitive information:

#Standard query:
curl -v -H “Authorization: Bearer <jwt_token>” https://<master_ip>:#<port>/api/v1/namespaces/<namespace>/<resource>/ #Example:
curl -v -H “Authorization: Bearer <jwt_token>” https://<master_ip>:<port>/api/v1/namespaces/default/pods/

• Secrets

With the previous tokens, check out if they can read secrets from other namespaces, you may obtain passwords, ssh keys, OAuth tokens…:

curl -v -H “Authorization: Bearer <jwt_token>” https://<master_ip>:<port>/api/v1/namespaces/<namespace>/secrets/

• Breakout Pods, Reach the Node

We got several techniques on how to get out the pod:

Abusing pod capabilities or privileged pods: We’ve already discussed some of them on capabilities section.

Scan your default gateway: From pod POV the default gateway (x.x.x.1) is part of the host, try to scan it and discover some vulnerable services reachable from the pod shell.

Deploy malicious pods: If you reach the API Server and you’re allowed to create new pods, deploy a malicious one and mount the root folder / from the node. We’ve already discusses on cap_sys_admin how to abuse this scenario.

Abuse the already mounted insecure hostPath: Check here: https://blog.aquasec.com/kubernetes-security-pod-escape-log-mounts

2.Authorization & Authentication

Review RBAC configurations, identifying weaknesses in permissions using tools like kubectl, kubectx, and kubens. Attempt to gain access to the Kubernetes API server without authentication or anonymously, using tools like curl or kubectl. Try to bypass authentication mechanisms, e.g., by altering kubeconfig files.

kubectl get roles,rolebindings,clusterroles,clusterrolebindings

curl https://<Kubernetes API IP>:6443

3.Network Security Policies

Review and test network policies to ensure proper segmentation of pods, using tools like kube-hunter or kube-scan. Attempt to bypass network policies and communicate with restricted pods.

kubectl get networkpolicies –all-namespaces

tcpdump -i <interface> host <pod-ip>

4.Container Security

Review container images for security vulnerabilities using vulnerability scanning tools like Trivy, Clair, or Anchore. Test for privilege escalation or container breakout vulnerabilities using tools like kubeletctl or kube-bench. Examine misconfigured security contexts in pods using tools like kubectl and kube-hunter.

trivy <image-name>

kubectl get pods -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,CONTAINERS:.spec.containers[*].name

5. API Server Security

Investigate the security of the API server by examining for weak configurations using tools like kube-hunter, kubei, or kubeletctl. Inspect API server certificates for vulnerabilities using SSL/TLS scanning tools such as sslyze or openssl.

kubectl cluster-info

kubectl get endpoints kube-apiserver -n kube-system -o json

6.POD Security Policies

Assess PodSecurityPolicies (PSPs) for overly permissive policies using kubectl. Exploit misconfigurations to run privileged containers or escalate privileges.

kubectl get psp

kubectl run -i –tty –image=<image-name> <pod-name> –restart=Never — bash

7. Node Security

Scan nodes for vulnerabilities or misconfigurations using security scanning tools. Check for open ports, exposed Docker APIs, and other potential security issues.

8.ETCD Security

Investigate the etcd cluster for security misconfigurations using etcdctl. Verify etcd access control with etcd-rbac.

Check if etcd is publicly accessible or misconfigured.

etcdctl –endpoints=<etcd-server> version

etcdctl –endpoints=<etcd-server> get <key>

etcdctl –endpoints=<etcd-server> put <key> <value>

etcdctl –endpoints=<etcd-server> watch <key>

9.Secrets Managements

Search for exposed secrets in pods using kubectl or tools like kube-hunter. Exploit vulnerabilities in applications or configurations to gain access to sensitive information.

kubectl get secretskubectl describe secret <secret-name>

10.Ingress and Service Security

Check for insecure Ingress configurations that may expose internal services using tools like kube-scan. Attempt to exploit insecure Services to gain unauthorized access.

11.Logging & Monitoring

Attempt to disrupt or evade logging and monitoring solutions, such as manipulating logs, to avoid detection. Check if security events are being correctly monitored and logged.

12.Pod-to-Pod Communication

Intercept and analyze pod-to-pod communication using packet capture tools like Wireshark. Attempt Man-in-the-Middle (MITM) attacks to exploit weak encryption or authentication mechanisms.

wireshark

tcpdump -i <interface> host <pod-IP>

kubectl get networkpolicies –all-namespaces

13.API Authentication

Systematically probe and exploit potential security gaps in authentication protocols, including OIDC (OpenID Connect) and JWT (JSON Web Tokens) tokens. Employ specialized tools for the manipulation of tokens to assess the resilience of authentication mechanisms. This meticulous evaluation ensures that the authentication components are robust and capable of withstanding sophisticated attacks, fortifying the overall security posture of the system against unauthorized access.

kubectl –token <JWT Token> get pods

14.Exploitation and Privilege Escalation

Employ identified vulnerabilities for unauthorized access or privilege escalation within the cluster, utilizing targeted exploitation methods and frameworks, such as Metasploit, Kubesploit, or kubectl tools as appropriate.

15.Kubernetes Cluster Hardening

In order to fortify the security posture of the Kubernetes cluster, it is imperative to rigorously implement and adhere to industry-leading security practices and policies. This process involves judiciously limiting the attack surface while minimizing the presence of superfluous and potentially vulnerable components within the cluster’s ecosystem. These measures aim to bolster the robustness of the Kubernetes environment against security threats and vulnerabilities.

Two critical dimensions of this security strategy include:

Backup & Disaster Recovery: The establishment of robust backup and disaster recovery mechanisms is of paramount importance. This entails the implementation of comprehensive data backup protocols and contingency plans to safeguard critical cluster data and configurations. It is vital to ensure swift recovery in the event of unforeseen incidents, including system failures, data corruption, or malicious breaches. This safeguards the integrity and availability of the Kubernetes environment, preventing prolonged disruptions and data loss.

Compliance: Ensuring strict compliance with relevant industry standards and regulatory requirements is a pivotal component of Kubernetes security. Adhering to frameworks such as the CIS Kubernetes Benchmarks or adhering to data protection regulations like GDPR is essential. Compliance not only showcases an organization’s commitment to data privacy and security but also minimizes legal and regulatory risks. By adhering to these standards, Kubernetes clusters can maintain a high level of security and demonstrate due diligence in safeguarding sensitive data and operations.

Incorporating these best practices, along with a proactive security mindset, will contribute to the resilience and trustworthiness of the Kubernetes cluster, fostering a robust and secure environment for container orchestration and application deployment. 

TOOLS

• kubebox: Perfect to navigate through pods and namespaces, fast access to shell, metrics (if they’re available) and logs – https://github.com/astefanutti/kubebox
• checkov: You know, SAST your yaml deployments.- https://github.com/bridgecrewio/checkov
• kube-bench: Run this on your nodes and check the CIS Benchmark recommendations for kubernetes. – https://github.com/aquasecurity/kube-bench
• kube-goat: A deliberately vulnerable Kubernetes cluster, a training lab.- https://github.com/ksoclabs/kube-goat
• kube-hunter: Offensive suite that hunts for security weaknesses in Kubernetes clusters.- https://github.com/aquasecurity/kube-hunter

OTHER SOURCES:

• MITRE matrix for Kubernetes: https://www.microsoft.com/en-us/security/blog/2020/04/02/attack-matrix-kubernetes/
• Kubernetes Pentest Methodology:
• Part 1: https://www.cyberark.com/resources/threat-research-blog/kubernetes-pentest-methodology-part-1
• Part 2: https://www.cyberark.com/resources/threat-research-blog/kubernetes-pentest-methodology-part-2
• Part 3: https://www.cyberark.com/resources/threat-research-blog/kubernetes-pentest-methodology-part-3
• 11 Way not to get hacked (kubernetes): https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hacked/
• Recon Checklist: https://lobuhisec.medium.com/kubernetes-pentest-recon-checklist-tools-and-resources-30d8e4b69463
• Pentesting Kubernetes: https://hacktricks.boitatech.com.br/pentesting/pentesting-kubernetes 

MORE REFERENCES:

• Kubernetes-attack-vectors: https://github.com/ilcapone/Hack-Comands/blob/master/Kubernetes/kubernetes-attack-vectors
• OWASP Kubernetes Security Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Kubernetes_Security_Cheat_Sheet.html

 

Written By : Rajnish Kumar