KUBERNETES SECURITY & TOOLS

KUBERNETES SECURITY & TOOLS

KUBERNETES

Kubernetes (K8s) is a production grade portable, extensible, open source container orchestration system.  Kubernetes helps in creating clusters, automating application deployment, managing containerized workloads/services and scaling. Kubernetes supports automated roll outs and roll backs, routing of service traffic, secret & configuration management, batch execution, horizontal scaling, storage orchestration, automatic bin packing, self healing, service discovery and load balancing, which makes it the most widely used container orchestration system in production environments.

KUBERNETES SECURITY

Cyber threats are ever increasing! It is very important to understand potential risks of deploying your workloads in a distributed K8s cluster and the best practices to secure it. Security requirements and approaches are different at different stages of the application life cycle (like build, deploy and run time).

K8s Innate Security Measures

Kubernetes provides a set of built-in security features, configurations and best practices which can be used to secure your K8s clusters.

• Application containers are always replaced and not updated/patched. This enforces version control and rollback, when required
• Configure security contexts to limit pod access.
• Enable Roles Based Access Control (RBAC) and grant only necessary permissions.
• Use name spaces and apply necessary security and network policies to name spaces.
• Define and enable pod security policies to control how workloads run in a cluster.
• Always use the latest stable versions, with the most recent patches.
• Turn on audit logs

How to enforce the 4Cs of Cloud Native Security in K8s?

• Cloud or Data Center Facility Security: Make sure the infrastructure used for building the K8s cluster is safe and secure.
• Cluster Security: This is all about ensuring security of configurable components like Kubernetes API and Applications within the cluster.
• Container Security: Follow the best practices while building containers. Scan the images before building, grant only required privileges (deny all other), install and use only bare minimum required components and modules.
• Code Security: Use encrypted communications, do not expose any unused ports, scan and test the code regularly to improve code security and minimize possible attacks.

K8s Security during Build, Deployment and Runtime

Below are some of the useful security notes to observe, while you build, deploy and run your containers in Kubernetes Clusters.

Build Time

• Do not use code from untrusted registries. This could lead to vulnerabilities and attacks.
• Use only minimal and up-to-date images.
• Do install only required packages, libraries and shells. Lesser the components, lesser the vulnerabilities and better the security.
• Scan the image to identify any potential vulnerabilities and remove it.
• Integrate validations using scanning etc. in to the CI/CD pipeline. Fail the process and generate alerts to notify any failures.

Deployment Time

• Grant only required (least) privileges to the containers to perform the intended functions.
• Limit access to secrets – mount only required secrets, which will reduce exposure to potential attacks.
• Use name spaces for separation of concerns, for both resources and teams. Name spaces can isolate sensitive workloads and provide a reference for network policies, access control and other security controls. This helps to contain the attacks and limit the impacts.
• Use Kubernetes network policies to control traffic between pods and clusters.
• Segment the network and use network segmentation to prevent lateral entry of attackers with in a cluster.
• Enforce Kubernetes Roles Based Access Control (RBAC) to resources.
• Use the best practices while implementing Pod Security Policies.

Run Time

• Continuously monitor attacks on infrastructure elements like API Server, etcd, controllers, etc. which are exposed. The dynamic structure of K8s require, intelligent tools to detect any issues in containers very quickly and then isolate and replace immediately to eliminate chances of any potential vulnerabilities.
• Compare build and deploy time information to find expected versus observed activities during runtime which will help to detect any suspicious activities.
• Perform vulnerability scanning in running containers.
• Continuously monitor network traffic.

KUBERNETES SECURITY TOOLS

Containerization is the new norm and containerized apps are the future. Traditional and legacy security tools are not enough to secure containers in production at large scale. Moreover, security standards are continuously changing based on changing cybersecurity threats. In this scenario, it is essential to have adequate and up to date security measures to realize the full potential of Kubernetes and containers in production. Find below some of the most common Kubernetes Security Tools used by enterprises to tighten security of K8s clusters.

• Project Calico: is an open source tool that connects and secures containers. It creates a micro firewall for every workload and secures it. Calico provides high level of scalability.

• Kube-Bench: open source Kubernetes security tool which supports the benchmark tests provided by CIS (Center for Internet Security). It checks authentication, authorization and encryption (in transit and at rest) levels and make sure privileges assigned are the least. It lists out the issues as well as solutions to fix the issues. Kube-Bench is a Go application distributed as a container. In Kube-Bench, any Benchmark changes can be easily updated using a yaml file.

• Kube-Hunter: is an open source tool which scans for security threats in Kubernetes by discovery and penetration testing. Kube-Hunter enhances the CIS validation provided by Kube-Bench.

• Kube-Audit: is a command line tool used to audit clusters by checking them against benchmarked security checks.

• Twistlock: provides full life cycle container and cloud-native cyber security solutions. It comes with more than 200 built-in checks for the Kubernetes CIS benchmarks. Twistlock takes care of image scanning, vulnerability management, firewalls, articulate a policy on a user-by-user basis, etc.

• Aqua Security: is one of the leading Kubernetes security tool available today. It provides transparency, automated container security profiles, tight controls on privileged user access, and real-time enforcement of security policies Aqua has powerful automation and works well across popular cloud-native platforms. It also provides full-stack development to production security across your CI/CD pipeline and runtime environment.

• Kops: is an official production grade open source Kubernetes security project. IT is also known as kubectl for clusters. Kubectl supports creating clusters, updating settings/changes, scaling up of nodes/clusters, running Kubernetes on AWS, infrastructure changes, deploying clusters to VPC, building new VPC and heterogeneous clusters.

• NeuVector: provides end-to-end container security with compliance testing, vulnerability scanning and run time protection. It is delivered as a container and comes with 7 layer container firewall. NeuVector is the only next-generation firewall for containers with packet-level interrogation and enforcement.

• Istio: is an open source service mesh that allows to control, connect and secure services on Kubernetes. It provides, automatic load balancing, fine grained traffic control, automatic metrics, logs, collections and secure service to service communications within a cluster.

• Whitesource: is the simplest tool to manage open source in containers. It has automated policy enforcement to block problematic components from entering your software, continuous integration by streamlining your development with native integration to registries and orchestration platforms and real time security alerts.

• Docker-bench: is a script that checks for dozens of common best practices around deploying Docker containers in production. All these tests are automated and is followed by the CIS Docker community benchmarks. This tool can be used as an easy way to self-assess hosts and Docker containers against CIS benchmark.

• Sonobuoy: is a diagnostic tool that makes it easier to understand the state of Kubernetes cluster by running a choice of configuration tests in an accessible and non-destructive manner. It does conformance testing to ensure that a cluster is properly configured and that it’s behaviour conforms to official Kubernetes specifications, workload debugging which generates diagnostics for troublesome workloads that are difficult to debug and custom tests and data collection. Plugin support allows developers and operators to extend system with additional tests.

• Snyk: is a platform to build cloud native applications securely. This tools provides security features like, open source security (automatically find and fix vulnerabilities in open source dependencies), code security (find and fix vulnerabilities in application code in real time), container security (find and fix vulnerabilities in containers and Kubernetes applications) and infrastructure as code security (find and fix Kubernetes and Terraform infrastructure as code issues while development).  

• StackHack: helps to find and fix application security bugs before they hit production. It scans the app (finds security bugs in your app), triage and fix security bugs (fix key bugs, add to back log, mark entries with do-not-fix, etc.) and helps to automate build pipeline and prevents security bugs from hitting production. The tool comes with features like security bug scanner, bug triage, YAML configuration, bug history, pre-production scans, scan history, fix guides and is CI/CD ready.    

KUBERNETES SECURITY – BEST PRACTICES

Find below some of the key points and industry best practices to enforce Kubernetes Security.

• Provide as many as hurdles (security layers) between threats and assets.
• Attackers can shift their focus, targets and time of attack, so make sure all your assets and interfaces are secure enough all the time.
• Setup and use layered (redundant) security architecture as an effective defence mechanism.
• Limit the attack surface by limiting the options for an attacker by using only the minimum required software.
• Enable least privilege. Provide only as much as permission/privilege as is absolutely necessary for an application/user/account to run.
• Use RBAC (Role Based Access Control). Roles given to the users can only perform operations permitted by that assigned role. RBAC settings can be applied to a name spaces and to the respective users in that name space.
• Firewall the services and API server to prevent attackers from connecting to it. Use regular firewalling rules like restrict access from certain IP address, etc. or GKE rules like master authorize network feature, etc.
• Set network policies to restrict access to pods (which pods can send request to and receive request from). For example, restrict access to databases to only the pods that require it. Use network policy enforcement mechanisms like Calico, Weave, etc.
• Make sure your etcd is secured. Make sure etcd requires authentication, firewall is set up to restrict access and data in etcd is encrypted.
• Prevent attackers from getting access to the host, which may help them to get access to different other containers, kubelet, secrets, etc. Best practices to prevent such attacks include, run application in the container as a non-root-user, make your root file system read-only and allow no new privileges (AllowPrivilegeEscalation=false).
• Use sandboxed pods, which provides an extra layer of isolation between container and host system.
• Use gVisor, which intercepts and implements sys calls in user space, restricting access to host.
• Add more layers like seccomp/AppArmor/SELinux and its policies to isolate and protect containers.
• Restrict Kubelet permissions by adding RBAC for Kubelet and by setting rotate option for Kubelet certificates.
• If multiple people are using the cluster, make sure everyone follows the best practices and there is no unsecured pods in the cluster. Use Open Policy Agent for Admission Review Request and Response policies to block such unsecured pods being created.
• Listen and monitor traffic using service mesh like istio. Istio puts a proxy between services, applies a policy, encrypts traffic and enable rolling certificates.
• Smaller the safer, use the minimum packages, applications, etc. Always use the latest stable versions, with the most recent patches. Use minimal OS and read only partitions/mounts.
• Integrate image scanning and security tests to the CI/CD pipeline. Scan the OS and applications for any vulnerabilities and fix it before building.
• Minimize administrative access to Kubernetes nodes.
• Continuously monitor the cluster using proper security tools.

SUMMARY

Cyber Security Incidents are ever increasing. Vulnerabilities and security incidents can be minimised, if we use the best practices and right security tools. Some of the key points to keep in mind are, update Kubernetes early and often, don’t use admin for day-to-day work, try benchmarking tools like kube-bench and use managed services like GKE.