Container Platforms are hot – many organizations nowadays explore the options to acquire a “ready-made” solution or build one themselves.
In the previous article, we explored a long list of the organizational challenges as well as some business-related aspects that are important. As a follow up we now present some of the technical challenges that need to have an answer. Container Platforms and governance – the complete picture
DevOps teams
The DevOps teams that create, maintain, and now also host their own application (components) are in fact the end-users of your Container Platform. For them it’s important to be aware of the following aspects:
- Know the SPOC – these teams need to know Who is the owner of which aspect of the Container Platform.
- Since they are end-responsible for their applications, they need to be able to understand every major aspect of the Container Platform. Not all of the nitty-gritty details but at least the things that make them help themselves in case of problems. Furthermore, they should not rely (too much) on the core platform team or anyone else. That slows CI/CD processes down.
- Every DevOps team that wants to land on the Container Platform should have sufficient knowledge about containers and Kubernetes. Without this knowledge, there is too much effort needed to train them. It’s important that they do not pose a big risk for the organization, since the technology is complex and demanding.
- Bring in feature requests and other feedback to the core team. During the first phase, it’s important to have some pilot teams willing to try the MVP of the platform and gather feedback. This requires some knowledgeable DevOps teams with a representative application. A team that is willing to invest time to test out the platform and accept its incompleteness.
Container Communities
Your Container Platform needs to be promoted, you need to get a feeling of the thoughts of others as well as other enthusiasts to help each other. Only then you can gain enough support to push forward. A large and professional community can help to drive your initiatives. Perhaps you require a (new) container community in which members share knowledge, bring in external experts that tell their stories & experience as well as common pitfalls from which they have learned.
You might use the community to spread the word of your work in order to convince difficult-to-reach stakeholders. For this to be successful, you need to team up with the community manager to streamline the processes around it. In addition to that, it’s vital to regularly present your progress and be actively involved in what’s happening.
CI/CD tools and pipelines
Every individual and independent component of the Container Platform requires its own source code repository and lifecycle. Since there are so many “moving parts” it’s crucial to keep things separated and very well organized.
CI/CD pipelines can’t be missed when it comes to the workflows that are needed to establish this goal. Simply speaking you require (standard) pipelines for at least the following components:
- A container image pipeline that can build and store (standard) container images. Every other DevOps team should also benefit from it as well.
- An IaC template pipeline for your cloud resources (such as a Terraform or Cloud Formation Template pipeline).
- A pipeline that actually provisions the main components of the Container Platform itself. Don’t forget the unit, component, and integration tests.
- A pipeline that handles secrets and other sensitive configuration items that your Container Platform requires.
- A pipeline that onboards new DevOps teams. It’s vital to build checks, controls, and validations into this pipeline so that all teams are onboarded in a similar way and no data pollution creeps in.
The very first versions of these pipelines can be created by the core platform team that needs them immediately. But in the end, when things grow bigger, there should be a dedicated pipeline team that make those pipelines more professional. Besides, they should also standardize them to make them applicable for other DevOps teams. This greatly helps to speed up and work in a uniform way. Enterprise Architects see that people act by the standards and guidelines that they designed. And the Legal & Compliance department has (more) confidence that everyone plays by the rules they set.
CI/CD tools
These CI/CD pipelines require a CI/CD orchestration tool like Gitlab, Jenkins, or Azure DevOps to just name a few. Container images are stored in private (and hopefully trusted) container registries or OCI compatible artifact repositories. Besides these functions, the CI/CD tools also require integrations with the Container Platform as well as ticketing systems to generate tickets for critical issues that break the pipelines. Define the owners for these tickets and be sure that you are able to have a process in place to follow these tickets.
Most large companies employ a CI/CD tooling team to maintain and extend these CI/CD related tools. Therefore it’s also an important stakeholder for the core platform team. They need to collaborate on various aspects such as: support for container runtimes, storage of (shared and public) container images, IaC tooling such as Terraform Enterprise (think about managing state of the templates) as well as handling secrets or non-personal accounts that are needed to provision (parts) of the Container Platform.
Secrets management
Speaking of secrets, you can’t miss the secret management team in the list of partners with which you need to collaborate.
Every DevOps team that wants to deploy their containerized applications on the Container Platform requires (externally) managed to operate them in a secure way. Think of managing secrets for the following use cases:
- Private container registries which host container images that can only be pulled using proper authentication.
- Service principals, tokens, and other cloud-enabled authentication mechanisms to access cloud (native) resources such as databases, storage, message queues, etc.
- The provisioning of certificates for applications, websites, and other systems which require encrypted traffic (basically all systems nowadays).
- Security tokens, non-personal accounts to access CI/CD tools, security systems as well as release orchestration tools.
- The offering of cryptographic solutions that DevOps teams need to secure their applications and workloads.
- Secrets detection and prevention in the build- and runtime phase of application (components).
Besides these practical use cases, the secrets management team should facilitate training and education for the DevOps teams and/or CISO teams to handle secrets the best possible way.
Security teams
Container Platforms need to be secured on all levels since they can carry any workload including the most sensitive ones.
Here the following security teams come into play with these key topics:
- CISO should (help) to sign off the entire solution to mark it as “safe enough” for the organization and the applications it intends to host.
- Security standards for build-time and runtime systems should be established, perhaps together with specific security expert teams.
- Projects from DevOps teams as well as the cloud-native services and the components of the Container Platform should be risk assessed and approved by CISO.
- The Security Operations Center (SOC), as well as CISO representatives, are involved into follow-ups that arise from security issues in workloads. This applies to vulnerabilities, security misconfigurations as well as compliance issues, malware detection, etc.
- A RED team may perform a PEN test to find weak spots in the Container Platform. Perhaps they can help to conduct threat modeling sessions.
Security and especially the governance of security is critical in large organizations. Therefore, these aspects should be put very high on the agenda.
Test experts
Every aspect of the Container Platform that is automated should also be tested. You require test experts that can define and build unit tests for the smallest components. In addition to that, you need to create component tests that can test individual components, perhaps using stubs. And last but not least, integration tests are necessary to test the platform from end to end.
Some of the challenges you can encounter when designing and building test(cases) are the following:
- Writing unit & component tests for IaC components is relatively new. It’s not always clear where the boundaries are between a unit- and component test. Test frameworks such as Terratest can help here.
- Another problem is that some tests can only be performed while a component is actually deployed. In case of the Container Platform, that might mean you require a completely working Kubernetes environment. This can take a while on specific cloud providers, thus spinning up and tearing down a cluster increases the time you need to wait for your test to pass or fail. You might use pre-provisioned clusters, but that also requires maintenance and overhead. Adding yet another team to the list might be a bit too much, but good to keep in mind.
- Integration tests require other tools or stubs. To complicate things even more, integrations with cloud (native) resources need to be established as well. Including valuable and reliable test data.
Test experts can also help to define tests based on perceived risk and impact. Given the fact that there are a large number of tests needed for all of your components, this is very helpful to prioritize your work.
Other experts
As you can see, the list of departments and teams grows rapidly. We’re not done there yet, but we have to stop somewhere. Keep in mind that you also require the following experts/teams to help you with the following topics:
- Data storage teams need to define persistent and nonpersistent storage for containerized workloads as well as help to integrate containers with cloud-native database solutions such as DynamoDB or Azure Cosmos DB.
- Networking teams need to help secure the networks which are in scope: host networks, container networks including overlay networks.
- Logging and monitoring experts. They can be the ones that define the standards and guidelines on how and what to log and to which system. Perhaps it’s a centralized Splunk environment or other cloud-native logging systems. Attached to this topic are the dashboards and KPIs. You might require a metric team as well to build those for you.
External help
If you need advice or practical help to implement external tools you may need to consult external consultants. These experts should also come up with industry best practices, help you create the functional and/or technical design document(s). Ideally, they should also guide you to prevent common pitfalls. And last but not least, they can help convince the management or other budget owners based on their “track record” or “success stories”.
Conclusion
Implementing a Container Platform is not easy. It does not only contain technical challenges, but it also surprises a lot of organizations once they realize how many departments and teams are involved to make it a success story. In this article, I presented a huge list of organizational aspects, especially governance-related ones to keep in the back of your mind when you plan to buy, build a Container Platform for your valuable applications. Good luck.
If you have questions related to this topic, feel free to book a meeting with one of our solutions experts, mail to sales@amazic.com.
