Many enterprises have successfully shifted to the cloud because they embraced the adoption of DevOps. Running their IT services in the cloud made it possible to implement DevOps processes in a greenfield environment. The tools used are also frequently built for cloud environments. While developers and operations staff found common goals like “fail early and fail fast” to increase the throughput of the software delivery pipeline they were at the same time out of touch with the security team. In this article, we will look at why DevSecOps, or embedding Continuous Security into the delivery pipeline, is important and how to start such workflows.
Security came late to the DevOps party
The inclusion of security into DevOps processes has been lacking from the start. The first-ever DevOpsDays conference which is seen as the official birth of DevOps movement was held in 2009. The first mention of DevSecOps on Twitter dates from March 2012. It took to mid-2018 to get some traction according to Google Trends. At the beginning of the year, the godfather of DevOps Patrick Debois even joked about it: #devsecops was late to the #DevOps party because they didn’t trust the invitation.
We haven’t seen any real evidence of why security came late to the DevOps party. Maybe because security is hard? Or was it, unfortunately, an afterthought? What certainly did not help was that info-security teams are a silo in the organization. Busy with verifying controls manually so they can fill in yet another spreadsheet. In How to do DevSecOps with Chef InSpec and the DevSec.io Policy-as-Code framework we already showed that ensuring compliance can be done in an automated way. The next step is to embed these kinds of checks in the software delivery pipeline.
What is the value of security in the development pipeline?
The DORA State of DevOps reports from previous years have always shown that high achieving team heavily rely on continuous integration and continuous deployment/delivery. The outcomes of these approaches are shorter development cycles, reduced time to recover, lower change fail rates, higher deployment frequency, and higher system availability.
Security processes outside of DevOps workflows will undo the improvements listed above. An example is putting security controls at the end of the development pipeline. Deployment can’t be done unless the security team has given their approval. If the security team makes heavy use of manual processes this prolongs the time to market even more. In the end, this leads to long development cycles that were tried to be avoided in the first place.
Securing your infrastructure or application should be seen as a way to deliver value. Unfortunately, security has never had an easy task to convince the business of this. Bridging the gap between key performance indicators used by management and those by security teams is challenging. In the worst case, management and security teams only find each other when it is too late: after a data breach.
The following example clearly shows that developers need to be aware of the risks when using Docker images from publicly available sources. K. Wist et al conclude in the paper Vulnerability Analysis of 2500 Docker Hub Images the following:
We found that as many as 82% of certified images contain at least one high or critical vulnerability, […] Official images came out as the most secure image type with 45.9% of them containing at least one critical or high rated vulnerability. Only 17.8% of the images did not contain any vulnerabilities, and we found that the community images are the most exposed as 8 out of the top 10 most vulnerable images are community images.
Another example is the hacking of the servers hosting the Linux Mint ISO files. The official ISO file was replaced with an ISO containing a backdoor. Last but not least, there also exist rogue AMIs in the Amazon Marketplace containing malware and other nasty things.
The DevOps Handbook by G. Kim et al, define three principles to follow when implementing DevOps:
- First Way: Work always flows in one direction – downstream.
- Second Way: Create, shorten, and amplify feedback loops.
- Third Way: Continued experimentation, in order to learn from mistakes, and achieve mastery.
Embedding Continuous Security into the delivery pipeline is an implementation of the second way. By introducing the right tools that automatically verify your security controls in the pipeline at the correct time will help you prevent which vulnerabilities are introduced into your environment. Secondly, everybody who is impacted (the committer and/or his team) will be notified immediately. This increases the awareness and stimulates a response of the team involved when the event happens and not at the end of the development cycle. This shortens and amplifies the feedback loop.
Embedding security in the virtual machine image build pipeline
In my previous article How to do DevSecOps with Chef InSpec and the DevSec.io Policy-as-Code framework I primarily focussed on ad-hoc verification of security baselines. Instead of focussing on verifying after a system has gone to production it’s more appropriate to do these verifications when building your image. A frequently used tool for this is Packer. We recently published an introduction to Packer article too.
In a DevSecOps workflow one could provide the necessary Inspec controls in a separate repository. The CI pipeline building the image can add the Inspec provisioner to their packer configuration file as shown in the documentation.
The advantage of this workflow is that the building of an image will fail as soon as Inspec fails a test. This will prevent a non-compliant image from being inserted into your environment. The person who introduced the faulty build will notice it immediately. It is then up to her to fix the issue if she wants to continue. A perfect example of shortening and amplifying the feedback loops.
The Inspec code doesn’t need to live in the same repository as the code used to build the image. This means separate teams can be the owner and can focus on what matters most. Of course, the teams can contribute fixes to each other.
Another approach is using a CIS hardened image. These images are CIS approved and will save you time too. They aren’t free, unfortunately.
One thing to be aware of is that the above workflow is only a hundred percent correct from a security point of view if you are using a complete immutable workflow for your compute instances. As soon as you install software or change configuration after the image has booted you risk being not compliant. For many DevOps engineers, immutable infrastructure is a panacea, but few reach that level of maturity. Like always, it’s up to you to make the correct tradeoffs.
Embedding security in your container image build pipeline
With the shift to containers achieving immutable infrastructure has become easier for application developers. We can embed security in the container build pipeline too.
Gitlab’s Ultimate and Gold versions have the following features built-in:
- Container Scanning: scan Docker containers for known vulnerabilities.
- Dependency List: view your project’s dependencies and their known vulnerabilities.
- Dependency Scanning: analyze your dependencies for known vulnerabilities.
- Dynamic Application Security Testing (DAST): analyze running web applications for known vulnerabilities.
- Secret Detection: analyze Git history for leaked secrets.
- Security Dashboard: view vulnerabilities in all your projects and groups.
- Static Application Security Testing (SAST): analyze source code for known vulnerabilities.
Another option is to use a tool like Snyk in your pipeline. If you can’t choose one of the paying options it is good to look at the tools available that are used by Gitlab and include them yourself in the pipeline.
DevSecOps forces us to rethink traditional security models
Enterprises are forced by the market into tech transformation. Tech transformation depends on quick, reliable, and safe software delivery. If security teams don’t rethink their strategies they won’t be able to cope: they will always come late. It’s time for management to facilitate bridging the gap between developers and operations. Using the tools above isn’t the answer, but introducing workflows that do something similar is.
DevSecOps initiatives for embedding Continuous Security into the delivery pipeline will make sure information security is built in the application delivery process from day zero and that a plan for security automation is in place. It will help developers code with security in mind. And when they don’t they will be notified immediately. Which shortens and amplifies the feedback loop. It will save money because less rework will need to be done.
The built-in security will need to take into account risk tolerance. Some applications will need less strict controls as they don’t pose big threats or time to market is more important. In the end, trade-offs will need to be made. It’s up to everyone involved to start talking.
