Containers remain a top deployment method for companies all across the world. Nearly all organizations have containers running in production nowadays. Whether it be on-premises or in the public cloud, containers are there to stay. Since these containers are now treated as “regular infrastructure” services, container security becomes as important as the container technology itself. There are a lot of tools already on the market which enable organizations to adopt container security topics for container images as well as container runtime security. From a business and process perspective, it’s important to keep track of what happens with the sheer amount of vulnerabilities, security misconfigurations, and other compliance-related issues. Container security requirements to include in your list.
Context and problem area
Container security tools tend to produce a lot of seemingly overwhelming information, especially in the case of a big number of Kubernetes clusters. Properly setting up filters and a decent process around the handling of the results is vital to keep the (ever-ongoing) stream of issues manageable.
Two examples
This is especially true if DevOps teams are creating and destroying containerized infrastructure and applications on the fly. A vulnerability might exist now but be gone just a day later. Or a high-risk issue occurs “out of the blue” in a long-running containerized process. To react to these issues, it’s wise to define process-related requirements which need to be implemented in your organization. Often, this leads to different departments and/or teams collaborating beyond the scope of the container security tool itself. In this article, I’ll share a couple of those requirements which you can use to evaluate your current use cases.
Keep track of vulnerabilities
Vulnerabilities in existing images and/or runtime environments are not static. Container images are based on layers and each layer can have a different owner. This is especially the case in base/parent images which act as the “solution building blocks” for DevOps teams.
Different responsibilities
Suppose a new vulnerability is discovered in one of the base images for which an Ops team is responsible. Manually finding out who (which team) and what (which IT product) is using the base image in charge is a tedious effort. As soon as you have them all in scope, things change and you can start all over. It would be much more convenient to publish a message to a system that can be accessed by all the teams. Once a new message arrives, the teams in charge would be notified. This new piece of information is then automatically picked up by them and the Business Owner can decide if the team continues or not.
Risk score
Another example would be the change in the CVSS (Common Vulnerability Scoring System). Suppose the initial risk score is 6 and this number is accepted for not critical workloads and after a new scan, the CVSS turns to 9, company policy dictates that the issue is fixed before going to production (even for not critical workloads). The owner of the image would like to be informed of the change in the risk score to trigger the process that needs to be followed to handle it. Just having the static information of the CVSS is not enough here.
Detect changes in runtime
Container images should be immutable and running containers as well. No one should mess with a running container. A lot of organizations violated this rule when the log4j issue popped up and all of a sudden, they had to roll out new versions of this important Java library. Some teams decide to manually patch running containers by copying over the patched version of the library to replace the infected one. You would like to have a process in place that notifies anyone who has an interest in the running container once it changes after it has been deployed.
So you need to include a requirement that basically captures the integrity or “container model” as it is prior to its deployment. Anything that leads to a change of this, should trigger a security rule and immediately report it to your container security tool.
Three examples
Powerful examples of what to monitor and detect are:
- Images that are updated “on the fly” (think of the kubectl apply command like this one: “kubectl set image deployment/nginx-deployment nginx=nginx:1.20.0”). This image might not have been checked and your runtime environment changes without following the golden rules about immutability.
- Don’t include or change packages in the running container. For example, don’t do an “apt-get update” in the running container. You won’t know what you would get.
- Be keen on images that “all of a sudden” contain vulnerabilities while they have been marked as “compliant” in the CI/CD pipeline(s).
For all of the items mentioned above, integration with ticketing or other monitoring tools should be a top priority since they are all relevant to be analyzed by the respective teams. Without a tight integration, configuration drift often becomes unnoticed and is only looked at in the following build iteration – which can be too late.
Reports to reveal trends
Every container security tool should have a feature to conduct reporting on the findings and other security-related aspects. While it’s nice to have a global overview of the “security hygiene of the organization”, it’s much more interesting to see which trends are hidden behind the raw data. The following statistics can help to find trends that could (or perhaps should) lead to a company-wide initiative.
- An aggregation of the level of risk for the infrastructure as well the application level of an IT system. This can be combined with the data classification. When combined, you get a great trend of applications with infrastructure components that demand extra attention. This is much more valuable than just risk levels alone without the contextual information of the affected system.
- Reporting on the same CVE for multiple applications or components. Track the open/fixed ratio of them to reveal trends on how fast common CVEs are fixed. Focus on the ones which are not fixed within a certain amount of time so you can build a knowledge database to help teams fix them.
Keeping track of the numbers
- Report on the number of issues (perhaps apply a specific filter if you want to zoom in or narrow down the results) both in production as well as non-production environments. This helps to pinpoint security efforts and to “shift them left”.
- The number of tickets created on board X. Suppose you integrate your tool with Jira or Qradar to track tickets based on security issues that require deep forensic investigation. If the number of tickets increases over time without actually adding more applications to the scanning software, a negative trend kicks in. Reporting on this and having a process ready to tackle it helps to reduce the risks and also helps to avoid too many “ad hoc” decisions.
Every security/risk officer can come up with more trends and reports, especially when working together with the business owners to narrow down the amount of time and effort to deal with so many issues. Often templates to generate predefined reports are of great help and are available in the tool itself.
Final words
Container security tools offer a lot of features to detect security-related issues, misconfiguration, vulnerabilities, etc. Your work is not finished if you have not set up requirements to implement process-related features. These often span multiple teams and/or departments. However, being able to collaborate together in a dedicated process helps to streamline the organizational efforts to tackle a lot of issues that would otherwise take a long time to fix. Looking back, we saw process-related requirements for the handling of vulnerabilities, runtime issues as well as reporting on trends.
