Managing multi-tenancy in Kubernetes – What you need to know

Introduction to Kubernetes multi-tenancy

There is a steady rise of developers sharing a single instance of resources in the cloud environment. That said, administrators and developers favor adopting a similar practice to their Kubernetes clusters. In Kubernetes, multi-tenancy allows several users to share the cluster’s resources, such as CPU, control plane, storage, and networking.  This architectural model is beneficial to companies as it saves costs while improving productivity. Since resources are shared, developers can also have more scalable container-based deployments. 

This article will focus on the three aspects of multi-tenancy in Kubernetes – use cases, considerations, and best practices.  

Types of multi-tenancy

Multi-tenancy in Kubernetes can be done in soft and hard isolation. Soft multi-tenancy, which doesn’t have strict isolation between tenants, is aimed at preventing accidental interference and is suitable for trusted tenants. In this, we have a single enterprise with different teams accessing the same cluster, this requires less amount of security overhead as users can trust each other. Hard multi-tenancy, which assumes tenants cannot be trusted, enforces stricter isolation to protect tenants from malicious interference and is suitable for both trusted and untrusted tenants. This is required when Kubernetes is exposed to multiple enterprises with independent and untrusted users.

• Control planes as a Service – This model has virtual clusters where every tenant has a separate Kubernetes control plane. However, they will continue to share worker node resources. This model gives the user the freedom to manage their own resources within the cluster. 
• Clusters as a Service – In this model, each tenant is given their own cluster. Users can have their own versions of cluster-wide resources with complete isolation of the Kubernetes control plane. This type of multi-tenancy will have a tenant administrator who focuses on restricting centrally managed services and other cluster information. 
• Namespaces as a Service – Considered one of the popular models of multi-tenancy, tenants are expected to share their cluster and tenant workload as assigned. Main control plane resources and worker node resources are to be used by all tenants. To ensure the workloads are isolated, each namespace has role bindings, network policies, and resource quotes. 

5 Things to consider 

Implementing a multi-tenancy Kubernetes, while advantageous, can also limit the existing tenants as they do not have complete control. To ensure effective implementation of Kubernetes multi-tenancy, here are a few points to consider. 

• 1. Resource Limits – It is an important component in multi-tenancy as it can disrupt the performance of other tenants. Developers must set quotas based on memory usage, CPU, and object count. 
  2. Audit Logging – The audit log has the relevant records in chronological set within a cluster that can be referenced when issues arise. Configuring audit logs ensures all the data to track user, control plane, and application activities is readily available. 
  3. Virtual Clusters – Developers looking for isolated environments cannot benefit from namespaces as it limits the use of resources. This is where virtual clusters come into play. Virtual clusters give the freedom to use the resources present in a standard Kubernetes cluster. 
  4. Pod Security – While working with multi-tenancy, it is important to set up Pod security standards. There are three standards, namely privileged, baseline, and restricted. Policy levels apply to namespace resources but cannot be applied to pods that already exist within the cluster. 
  5. Policy Engines – If the organization caters to a global audience, it is important to comply with various regulatory standards. To ensure compliance and security, having predefined policies in the policy engine, with the freedom to modify dynamically is imperative. 

Best practices 

Multi-tenancy is a great solution within Kubernetes as it helps manage cluster costs, reduces operational overheads, enables maximum utilization of clusters as one can host both related and unrelated applications together. However, in order to make multi-tenancy model a success, it is important to adopt the best practices. Here are our important recommendations. 

• Role-based Access Control (RBAC) – Security is non-negotiable when it comes to Kubernetes. While handling multiple tenants, implement RBAC with necessary authorization policies to restrict unwarranted entry and handle user behavior in the cluster. 
• Namespace – If you are using namespace as a service, it is important to keep three things in mind – creating a namespace template to enforce isolated operation, have a scalable naming convention that makes it easy to understand, and attach relevate metadata or labels to namespace for easy retrieval. 
• Resource distribution – While sharing clusters, giving unlimited access to containerized applications can lead to performance issues, node constraints, team conflicts. To that end, ensure there is a defined resource quota that focuses on storage, memory, CPU and other resources that will be shared by teams. 
• Networking – Since all pods communicate by default, there is a security isolation risk within a single cluster. However, this can be mitigated by introducing network security policies as it helps create a secure network within the cluster by categorizing pod traffic from the network. Developers can customize network policies to their needs by changing pod selector, ingress and egress, and from/to rules. 

Optimizing the Kubernetes cluster for multi-tenancy demands careful planning and meticulously adopting the best practices. However, that is not enough. Consistently work on monitoring workload, cluster logs, Kubernetes events, and core metrics. 

Key takeaways 

Before investing in multi-tenancy in Kubernetes, it is important to be aware of the organization’s requirements and current infrastructure. While multi-tenancy offers several advantages, it can become a challenge by itself for organizations that have high traffic and have several security and reliability requirements. You need to have a good understanding of the essentials of multi-tenancy before making the call.