Tips and tricks to secure containers for run-time environments

One of the core concepts of container security is to protect any system which runs containers. Especially when using a container orchestration tool and when running containers in production. In the previous articles we talked about the first phases of the container life-cycle: the concepts and the static part of securing container images. We now focus on the dynamic part: container run-time security.

A lot of topics

As it’s always the case when starting something new: where to start? Container security for run-time environments is a large topic which spans multiple perspectives and a lot of different components. Think of the following 5 perspectives which come into play:

1. The containers itself: how are those configured to run-time.
2. Container engine: the software which is used to run the actual container.
3. Orchestration: Docker Swarm, Kubernetes, Mesosphere, etc.
4. Operating system: various flavors of Linux systems, but also windows can run containers. And think of server-less which runs some kind of container to execute your code.
5. Networking: container to container communication but also container to VM communication and to and from the container engine.

The technical part requires specialized people to understand the topics and to implement things on the right level. So it is from the organizational aspect. A lot of it comes down to responsibilities and governance. A lot of large enterprises struggle with the governance of this topic. One of the reasons why this is difficult is because containers act as “middleware“.

Start with the containers itself

To align with the previous articles it’s nice to start with the containers itself and the way they interact with the container engine. Let’s zoom in into that. Here are some first steps to get you started:

• Do not run containers in privileged mode. Running a container in privileged mode means: switch off any security. Avoid this at all costs, unless you want to hack around and destroy your system afterwards.
• Use a non-root user inside of your container. Sounds obvious but the default is the root user. Some containers are forced to use the root user in order to function. These cases make stuff a lot harder.
• Do mount sensitive host directories inside a container. For example: do not mount /root read-write inside the container itself. It’s easy to modify the root filesystem of the host when you do this. Containers should run in isolation, so this is undesired.
• Avoid containers from getting additional privileges. It means: a container should not get more privileges than explicitly specified and controlled up front. At run-time things should be predictable.
• Set memory and CPU limits when defining the container definitions. By setting these flags you prevent system from crashing since containers cannot overload your system. A lot of factors come into play how to define those limits. It’s not just the container itself, it’s also the application level and the resource limits of the VMs on which the container runs.
• Don’t assume official base images are secure by default. While browsing through registries like Quay you can see a lot of official images which have a high number of critical and high vulnerabilities. Let alone all of the other bad practices which do do not see at the first overview screen.

CIS compliance

Probably every security expert knows about the CIS and NIST security standards which are available online. CIS security standards and recommendations for Virtual Machines and other systems are common. For containers there are also security rules and standards. Nearly all container run-time security tools check your system for CIS compliance. Commercial tools like Sysdig, Twistlock and Aquasec show the results in a nice GUI. Other tools like Docker benchmark show the command line output in text format.

Some rules which are relevant here:

• Audit your sensitive container configuration files on your host. Once there is an (unintended) change, you know your system is at risk.
• Store container logs in a central place to keep them “as evidence” when there is an attack. The container can already be deleted, your logs should tell you what happened.
• Use apparmor and selinux profiles to “harden” your container system. Both of the methods provide an extra layer of protection to avoid containers doing malicious stuff on your system.
• Restrict commands which can run inside a container. For example: deny creating files at sensitive places and avoid unintended and unneeded packages from being installed.
• Scan for malware, cryptominers and viruses. Looks like old school ways to protect your system, but anyone can put anything in a container image, so better check it in the static phase and in the run-time phase.

Learning behavior

One of the big advantages of the commercial tools is the learning mode of container (images). In short, the system learns the “typical” commands of which are executed as part of the running container. It learns the command right after the container has been started. When the learning mode is finished, only these commands in the container are allowed, all others are blocked. With this feature, the tool prevents the container (processes) from running any command which is “unrecorded” and thus being blocked. Security operators don’t need to manually tweak the rules for all new situations since the system “learns” automatically.

Kubernetes

Kubernetes is by far the most popular container orchestration platform in the world. Security within this context is a huge topic on itself. Some highlights to get you started:

• The CIS compliance rules apply also for Kubernetes – even in a managed environment like AKS or EKS.
• Enable Role Based Access Control and use the least privilege principle. This helps you keeping the damage limited in case there is an attack.
• Split your Kubernetes cluster into useful namespaces (sections). It helps to get your applications (and containers) organized and isolated. It is also possible to define access rights per namespace which adheres to the least privilege principle mentioned above.
• Make the update & upgrade process as easy as possible to patch your system. It really helps you when you need to apply a critical vulnerability.
• Enable and use network policies. With these policies in mind, containers are run in more isolation and this reduces the risk and damage when a container is compromised.
• Use secure and encrypted communication everywhere. This should be the default for every component of your Kubernetes cluster and the way you connect to it.

If you want to get more detailed (very technical) information, check the 11 ways to prevent you from being hacked.

Concluding

There is so much more to explore and it is difficult to give good advice on which rules apply for your situation. A lot of them are context specific and are different for (cloud) managed and self-managed environments. I hope these considerations guide you in your container security journey.