In one of my previous articles I wrote about the concepts of container security from a business point of view. One of the topics to start with is to secure the container base images. Easier said than done, since it involves a lot. Let’s get it started in a meaningful way: container security – the static part.
Be consistent
To write good software code, it’s good to be consistent. That’s why developer teams agree on coding standards. For container images it’s the same. You may decide to check your base images every two weeks. New versions of the Image are deployed very frequently, so being consistent helps to speed up fixing vulnerabilities. This statement becomes even more true since you might have a lot of base images which all consist of a lot of packages.
Use a Linting tool
Most of the scripting languages have a Lint tool to check for best practices and problems in static source code. For Docker container images, there is Hadolint. This open source tool has a lot of rules – maybe too much. It’s good to define which rules are relevant and which are less relevant since it’s difficult to comply to both the Lint rules and to other best practices.
Useful examples of linting
Check out the following Lint rules which are easy to implement. These rules are selected since they make sense from the perspective of (re)usability and other security principles.
- Use a non-root user in your container images. It helps to keep the damage limited when there is a breakout of the container. Drawback is this: if a developer wants to extend a base image (written by someone else) he needs the root user to install or update new packages. It means he need to switch from a regular (less privileged user) to the root user and back again. This violates with this Lint rule.
- Pin versions in apt-get install. You need to know which version of a package you are about to use and install. If you leave the version out, you don’t know the version (maybe latest?) and you don’t know if this package is vulnerable or not.
- Use only (private) trusted container image registries in the FROM statement of your image. It’s the root of all trust. Do not blindly trust images coming from non-private registries like Dockerhub or Quay. Even official Images can have vulnerabilities. For example the Calico/CNI image, since it is based on a rather old version of Alpine.
- Do not use apk upgrade. When you execute this without specifying a package, it upgrades a whole bunch of (unwanted) packages. Those packages increase the size of the Docker image, they increase the attack surface. Logically the more packages the bigger risk for vulnerabilities. Sometimes you really need to upgrade a bunch of related packages to avoid a single package from being upgraded (think of dependencies). If this is the case, upgrade wins over the Hadolint rule.
Dependency scan
Once finished with Lint, it’s time to scan your container image with another tool to check the dependencies for CVEs. Think of third party dependencies as part of your application. For example: the Spring boot dependencies in a typical Java application. But also operating system level dependencies which are already part of the existing container image. For example: curl, netbase, zlib.
When CVEs are found it is wise to patch the dependencies of your application and/or the operating system level dependencies. Bother interact with each other, so there is a need to test the interdependence in advance. This is exactly one of the benefits of container images, so that won’t be a big hassle.
Tools
Scanning for these dependencies can be done using one of the available tools. Some open source tools like Anchore can do the work. Other commercial tools like Twistlock, Aquasec or Sysdig can do the same. All tools use a very extended and updated CVE database list. Most of these tool provide a nice dashboard which shows this information.
Docker Enterprise contains a Universal Control Plane in which you can view the security scans of dependencies. It is possible to “promote” a container image from one environment to another. For example: promote an image from TEST to ACCEPTANCE if there are no high risk vulnerabilities found. This also hooks in into the complete image life-cycle in which security can/should be traced from code to production (another big topic).
Once a CVE is found, the tools give advice on how to fix it or to upgrade to a version which has the CVE fixed.
It is important to know the risk factor of any vulnerability. Sometimes you can’t upgrade a package since there is no new version available. Or even the latest version of the package also has CVEs. For these kind of reasons it’s important to know the risk.
Remediation
So it’s good to know where to start. Some best practices get you up to speed:
- Use another base image: why should you patch it if you have one or more alternatives? Think from the business perspective: if the features are the same, there is no need to patch it.
- Upgrade the packages which have one or more vulnerabilities. Not very difficult as long as there are not many dependencies which depend on each other. Don’t forget to scan your new version of your container image.
- Remove the component which causes trouble. For example delete a private key (yep, private keys are packaged inside of container images), delete a package which you don’t need. This resolves the problem, but always make sure your image still works as intended.
- A lot of vendors deliver their software as container images. If there is an issue in the image, ask them to fix it. In case you obtain an image from the “community”, you yourself can also help to fix the problem.
- Accept the risk: this is not a real remediation but sometimes you need to accept a risk to move on. Only consider it if you are certain that the risk is acceptable (think of this up front) and the probability of an exploit is low. Remember: accepting a risk is not just a technical decision. Involve the business owner. At least you now know what the risk is.
- And the final tip: move away from containers and find a completely different solution.
Wrap up
As you can see there’s a lot of knowledge needed to check and fix container images. Since there are a lot of components and dependencies involved, container security requires a lot of time and knowledge. This article cannot cover all cases, so stay tuned for the next article which is all about container run-time security aspects.
