Organizations constantly feel the pressure to deliver security software. Auditors can show up at unexpected moments. Traceability of software components is king to stay compliant. CI/CD and DevOps best practices help them a lot, as well as adding security early on in the processes. The integrity of source code and “ready to ship” software packages is an important aspect. Many companies now explore the concept of signing their source code and/or other deliverables. This is especially relevant in the light of the recent Executive Order of the US government. So far so good, but where to start? In this article we will explore code signing tips and tricks for organizations.
What and why
Code signing helps to establish and verify the authenticity and integrity of a software application. As source code travels through the CI/CD pipeline during every stage of the entire Software Development Life Cycle, many things can go wrong. It’s about the things that have a negative impact on the trustfulness of the author of the source code and/or application component.
In other words, you don’t know if the software can be trusted or not. If this is the case, the data that the application processes and other systems are at risk. Signing helps to prevent this, so organizations want to make sure they can trust the source code and/or the application before they deploy or run it. Signing software source code is like using a digital fingerprint which cannot be tampered with.
How does it work
Signing software components is a complex process that consists of many steps and a lot of aspects which need to be carefully implemented and secured. The following steps give you a brief overview of what the main aspects of a typical architecture are about.
- Code Sign System: receives the source code, validates the submitter and generates a so called “signature”.
- Authenticate the signers using an appropriate CA (Certificate Authority) and issue certificates. An alternative would be to use a Time Stamp Authority.
- Verifiers validate the above-mentioned certificates and timestamps.
- Trust anchors validate signatures and code signing certificates.
There is a lot more detail to this list, tools can really help to implement and safeguard the processes that are relevant here.
Common use cases
To get a better understanding of typical use cases, see the following topics:
- Driver signing: Operating System drivers that control hardware such as soundcards, WLAN adapters and other devices need to work correctly at all times. It is extremely undesirable for vendors to ship drivers which could harm the computers of their end-users. Therefore these vendors often have the software of their drivers signed by the OS vendor or another trustful partner.
- Signing firmware: this is also critical to the working of the hardware and software of computer systems. Often developers send out their source code to a centralized authority which then signs the entire software package. Once it meets the security standards, it is allowed to be shipped.
- Application software signing: Any (other) application that developers create and ship to their end-users. Basically it could be every application that the software vendor creates. It can be assumed that application that process critical (sensitive and/or privacy related) information needs to be signed first.
The last use case acts as an example for the rest of this article since this is one is rather generic.
Real world threats
One of the best best examples of what the impact is if your source code signing methods are compromised, can be read on the website of Solarwinds. This company faced a huge problem and they had to review a lot of processes and technical aspects that were part of their software signing architecture.
Two other threats require your attention.
Signing the wrong source code, thus unauthorized or even malicious source code. Since this topic is rather complex and pretty new to a lot of companies, things can go wrong in many ways. Think about errors and other flaws in the implementation of the signing processes. If this is the case or if there is no proper governance around the processes which are involved, you could end up with signed software that is dangerous to your organization and/or your customers which are using it.
Every source code signing solutions requires various “keys” to sign source code and signatures. Without a proper protection of these keys, they could be stolen.
By then your entire signing solution is at risk and it’s difficult to recover from it. That’s one of the reasons some companies protect their most important keys using so called HSM (Hardware Security Module) to physically store them there.
Best practices
To overcome the threats mentioned earlier, there are a number of best practices to follow.
- Sign the version (as part of the applications’ meta-data) of the source code which is about to be rolled out. This prevents the roll-out of outdated and thus invalid software packages.
- Protect the code signing solution: isolate it from external networks as well as from production environments. It is a highly critical asset in your application landscape, just like your secrets management solution for example. Needless to say: carefully monitor it and detect anything that is suspicious that threatens it’s integrity.
- Carefully, systematically and regularly check the (audit) logs of the signing keys. You need to detect anomalies as early as possible to avoid misuse of your keys.
- Revoke and recreate certificates at the right time. Here a secrets management solution also comes into play.
- A trustworthy Certificate Authority (CA) is desirable since it’s also at the heart of your source code signing solution. Your entire architecture is compromised if you don’t have one and you cannot protect your keys in a secure way.
- Cryptography should follow international standards (such as NIST approved algorithms). Never create it yourself, other will do a better job here!
Helpful tools and solutions
By now there a number of companies and organizations that jumped on this topic to help you and your DevOps teams with ready to use solutions.
Notary
First of all and one of the most well known open source solutions to sign software is the Notary Project. It’s based on The Update Framework. Recently, it has been officially accepted by the Cloud Native Computing Foundation (CNCF). Their primary goal is to let users sign their content (not limited to source code) using offline stored keys. Once done they can upload their signed content to a so called Notary Server. Based on this, you can also sign Docker images using Docker Content Trust. If you enable content trust and when using a so called “trusted registry”, you can force consumers of your Docker images to only use signed images. Proposed benefit of this mechanism is to prevent the usage of malicious Docker images.
KeyFactor
KeyFactor claims to solve “your biggest code signing challenges”. This vendor aims for a smooth use experience for developers. To make this a reality, they offer a platform to enable secure processes and to sign code across all applications, DevOps environments as well as all the DevOps teams. Interesting to note is that their code signing solution works through an API, the console or using a remote utility. It also audits everything and developers do not have to handle sensitive keys. Policy-based access controls and workflows help to prevent unauthorized signing. This is a key aspect since in large organizations, the developers might not be the only ones that should sign their source code. What about the CISO department that also has a say on it, for example when commonly shared software packages are on the table. Those are not the sole responsibility of the developers. Again: governance is important.
Encryption Consulting
Speaking about developers, the company named “Encryption Consulting” shares an informative video in which they explain code signing from the perspective of developers. This really helps to understand the different steps and techniques that make up a complete solution. Shifting security left means that developers need to understand this to fully utilize this capability. Another non functional requirement that is important here.
Their platform offers some differentiated features compared to other platforms: support different workflows for source code signing across DevOps teams and support multi-tier support for approvals. Use the command-line signing tool (only) to sign requests in bulk. Use client side hashing and avoid copying files to and from different system in your network. Both aspects make it a fast and very secure solution.
Conclusion
Signing software source code and other artifacts becomes pretty common nowadays. A number of events trigger the need for this discipline. Your organization requires a decent software signing solution that needs to protected very very well. There are a lot of companies and custom made solutions to make the life of developers and other key stakeholders easier. My advice would be to explore the topic, be informed about the most important aspects and get your hands dirty. All helps to get familiar with what is needed now and especially in the near future.
If you have questions related to this topic, feel free to book a meeting with one of our solutions experts, mail to sales@amazic.com.
