GitOps quickly becomes the de-facto standard for DevOps teams to build, test and release software. Previously, there were some articles about GitOps itself published on Amazic. Therefore, please refer to the article about the GitOps pipeline and GitOps adoption by cloud vendors. With the rise of GitOps and “pushing to production” becoming much more common, security becomes a “must-have” for everyone to reduce the risk of data loss, identity theft, and other damage due to security flaws and vulnerabilities. In this article, we’ll present the most common GitOps security topics you must address.
Focus area
First of all, it’s vital to understand that GitOps focuses on “single processes” to build, test, and verify source code as well as target environments. The time for multiple scans for multiple sets of changes, environments, and tests is over. Everything is tied to the full history which is linked to every commit. This makes tracking and tracing changes much more predictable. However, it also introduces risks, since all of the steps in every single process are carried out automatically. No human intervention in anything.
Right at the heart: the Git Repository
Collaboration is king when it comes to (secure) GitOps operations. The aim of GitOps is to let Operators and Developers work together from the same source code repository. So the interplay of “both camps” should be implemented with the correct balance between usability and (security) regulations.
Think of the following measures to implement in your CI/CD environment:
- Branch policies (and by preventing force pushes) determine who can push to a certain branch (f.e. feature branches, release branches, and master/main branches) and should always be enforced. This is to prevent problems in the first place since changes “further down the line” can only be propagated by certain people or processes (backed by technical accounts).
- Peer reviews are required with a mix of junior and senior developers. Having this in place ensures a “four-eye principle” early on in the development process. Catching bugs early and spotting security issues before the code is merged saves time and effort to let tools find them afterward.
- SSH keys should be rotated regularly and this should apply for personal as well as non-personal accounts. “Read-only” accounts (except for the CI/CD pipelines) should be disabled or removed completely.
- Apply the least privilege principle for everyone interacting with the Git repositories. Audit access levels and trigger alerts for anything which looks critically suspicious.
Besides these security measures, it’s advised to regularly back up your Git repositories so external tools can be used to verify the integrity of those repositories. This measure helps to prevent someone from “rewriting history” to bypass security restrictions. Furthermore, only allow signed commits so the authenticity of the change is guaranteed.
IaC and the extension of GitOps
GitOps processes are similar to application development and deployment processes. Therefore, the operating model is the same. It shouldn’t be a surprise that the IaC principles also apply here. Security professionals should determine the threat actors for any GitOps related piece of source code. Regular security tools to scan your IaC templates and scripts as well as prevention to inject secrets are the norm here. Everything is stored in Git repositories and applied automatically to production, so it’s also crucial to have proper quality gates in place.
Among the components mentioned earlier, other common attack surfaces which need to be protected in a professional GitOps environment are:
- Everything that is needed to deploy, update, replace and remove workloads from Kubernetes clusters as well as other cloud-native services. Think of config files, secrets, config maps, cron jobs, secret files, etc.
- The CI/CD orchestration tool as well as the CI/CD pipelines. This includes access to the tool itself, securing access to source code repositories which can be accessed from within the tool, and also (container) registries which are located at the heart of the GitOps task chain.
- Any runtime such as a running container and other key assets on which those containers depend. Pay special attention to side-car containers as they are fired and stopped once a new deployment occurs. Sometimes, these containers have more permissions compared to the application workload container, thus the damage can be greater if compromised.
And last but not least, secure the third-party components that the applications and/or infrastructure-related parts use.
Help by the GitOps tools
GitOps uses source code, stored in Git as the single source of truth. Tools help to adhere to the shift left principle. Universal tools are best here to avoid a complete tool sprawl.
Lint
Every common programming or scripting language has its own Linting tools nowadays. Dockerfiles can be checked using Hadolint, Python can utilize pylint, and so on. Megalinter is a tool that supports over 51 languages and 22 formats for all of your source code. Add this to your programming tasks and in your CI/CD pipeline to make sure your syntax is correct and consistent for every change you perform. It’s a first step towards detecting (security) issues while you code.
Policy as Code
Every piece of the software puzzle is source code nowadays. Therefore, a universal tool to define and enforce your security efforts is required as well. OPA – Open Policy Agent can apply policies to your (Kubernetes) manifests and other configuration files pretty easily. The policies are written in the Rego scripting language, which also gains traction nowadays. Please check out the article that showcases how to prevent a workload to be deployed to the default namespace in Kubernetes. It’s a great example of what OPA can do for you.
Kubernetes test clusters
Many tools require a fully functional Kubernetes cluster to check out the deployment of your workload. Due to the amount of time that is needed to set up a working and compliant cluster, and to test your workload in isolation, it’s also possible to provide your own test cluster on the fly. If you are working under Linux, you have the option to utilize KinD (Kubernetes in Docker). Ideal for simple tests on your local machine or in your CI/CD pipeline. There is no need to spin up a complete cluster with all the bells and whistles.
Treat Pull requests with care
GitOps workflows are centered around change requests that propagate source code changes from a feature branch to a development branch and further down the line to the main/master branch. Keep the following tips in mind when reviewing pull requests to enhance your security levels.
- Careful choose who to add as a required reviewer and who as an optional one. Suppose you want to push your changes from ACCEPTANCE to PRODUCTION, then a Product Owner might be the only person who is allowed to accept a change. This might be based on the positive outcome of all previous steps (including linting, unit testing, component tests, security tests, etc) as well as an agreement from the end user. Therefore another developer can be an optional reviewer but the Product Owner has the final vote at this stage. Be sure to always provide an escape in case the Product Owner is on holiday and thus unavailable.
- Just blindly accepting pull requests is not advised. Developers need to make sure they add useful information in the approval message box to be aware there is no “Yes, approve” automatism.
- In case you do refactor a lot (adding or deleting a lot of files, say 10+), it’s advised to assign multiple personals to approve the change, so everyone is notified of an impactful change. Problems can be spotted early on since a lot of people can check on it before the change is actually merged.
- Include security experts in the approval list. Don’t just add developers who work on the code on a daily base. Be sure to add (external) non-biased security experts as well when there is a major (security) change. This leads to new insights and potential problems are quickly revealed. It also helps to steer the discussions between DEV and SEC.
Wrap up
As seen in this article, there are many security-related challenges to take care of when you apply a GitOps approach to your organization. While DevOps tends to focus on better collaboration between Developers and Operators, GitOps focuses on timely deliveries straight from (Git) source code repositories to the target deployment platform. Since the speed to delivery counts, additional security measures should be taken. In this article, common and more advanced techniques are covered to make both your workflows and application workloads more secure.
If you have questions related to this topic, feel free to book a meeting with one of our solutions experts, mail to sales@amazic.com.
