Modern DevOps teams employ professionals which master multiple skills. Strong software development skills are one of the most important. Your team members also need to have a security mindset. And last but not least, they need to master different testing skills. Finding and fixing an issue early on in the Software Development LifeCycle is much cheaper compared to fixing it in production. This is a no-go in this DevOps era. DevOps requires early feedback and to move fast, you need to ensure that you reduce the business risks when it comes to problems in production. Test automation helps and continuous testing helps even more. In this article, we will discuss what is continuous testing and why it matters.
What is Continuous Testing?
Especially for core applications, business continuity is a top requirement. Once a broken system interrupts an application that is used by high demanding customers, all hell breaks loose. Continuous Testing is all about getting feedback about business risks. To fill in this requirement, your CI/CD pipeline demands automated tests as part of the underlying processes to make sure the next version of your application reduces risks as much as possible. Besides this, continuous testing addresses increased complexity and supports modern application development and delivery methodologies.
Business Risks
First of all, what are business risks? Anything that poses a threat to your business can be a risk for it. A security vulnerability can be a risk to get hacked and someone stealing your valuable information. Lack of functional tests can lead to a risk of creating the wrong feature. In case your end-user wants something else, you wasted your time and money to build the wrong thing. Perhaps you missed “the right momentum” and you are too late for your marketing campaign. A missed opportunity is also a business risk.
Proceed or not?
Changes to an application that result in unmet requirements create business risks and should not proceed to the next stage of the SDLC. To determine if you want to proceed, you need to set a certain baseline for every type of risk. Think of a baseline for unit test coverage. To continuously increase your testing experience, you need to raise the bar bit by bit.
Once DevOps teams gain more security-related knowledge they can improve the quality of their solutions. A security baseline states something like: only applications without critical and high-level risks are allowed to proceed to production. Any application which meets this level and has no significant residual risks is approved by the business.
Therefore it is important that DevOps teams see Continuous Testing as a safety net to make sure they stop before they deploy anything that could harm the business. The before-mentioned safety net should be implemented in the early stages of the SDLC, preferable in the development phase. But don’t forget about the later stages as well.
Shift left or shift right?
It’s interesting to note here that some types of tests should be “shifted left” and others should be “shifted right”. Unit tests, component tests, and code coverage tests should be executed as early as possible while performance tests, load tests, etc should be tested in production. The latter is to gain speed but only do this if you know exactly what your system can handle and how to react to interruptions. Don’t test in production without taking resilience seriously.
Testing early on in the SDLC means you need to have a stable and reliable test environment with actual data. Every time you run your CI/CD pipeline, you require a fresh test environment. To achieve this, you can use disposable/visualized environments based on containers.
Shifting right here means you capture valuable data from the actual running system itself. No other system is more reliable than a system that tries to mimic the actual production environment. Even based on IaC, external factors influence the result. Think of the number of users who are logged in and use an application and an uneven distribution of records being read from a database. Deployments that pose a threat to stability, security, or other business-related risks should be rolled back (automatically).
Don’t create bottlenecks
While the previous aspects intend to differentiate tests in various stages of the SDLC, it’s evident that you should not create any bottlenecks in your CI/CD pipeline. In practice, this means you should carefully select which tests to execute in which stage of the pipeline. We’ll leave out the infrastructure requirements of your CI/CD tools.
Unit tests & components tests
Unit tests are fast and should run after each code change. Run them early on in the pipeline and run them in parallel since they do not depend on other components. Proceed to the next step and run your component tests. It might take a little while to set them up, try to run them in parallel as well to speed things up. If a component test requires another component for its core functionality, consider mimicking this component or set the other component up before your test the actual component.
Smoke tests & integration tests
After deployment runs smoke tests to quickly catch failures in the deployed package. As soon as the deployment script discovers a problem, it’s time to roll back. Other target platforms offer similar options as the example in Azure DevOps. After successful deployment, it’s time to run integration tests. These types of tests require external dependencies and services to be online. There are also cases in which you can use stubs to mimic actual services. Tests are more limited since they can not cover exactly the same situation as with real-life services, but they are fast and you have virtually unlimited control over them.
Collect feedback
One of the most important aspects of continuous testing is to collect valuable feedback from your systems. Brittle test scripts which are not reliable every time you execute them should be replaced by modern and flexible testing frameworks. Some examples of them are the RobotFramework (for python applications) or Cypress (great of testing APIs) or Carina (for web applications). A good testing framework should also help to minimize false positives since these are true speed-killers. They put a burden on the persons who actually interpret the test results. Filtering out false positives is a key requirement, just like filtering vulnerabilities in other security-related tools.
Context helps
Valuable feedback adds the right context to tests such as “Feature X failed because of reason Y so that has an impact on business application component Z”. On top of that, feedback should enable a developer to define actions such as where to look in the source code to fix things. Without the proper context and actionable output, tests do not add so much value to an organization.
Proper actions and valuable feedback about an application help to reduce business risks since issues can be fixed fast and with the right priority. Seen from a business perspective, this really helps to push the right things forward and spend fewer hours on less important things. Therefore it’s important to keep these aspects in mind when selecting the right test automation frameworks and develop the tests themselves.
End to end
To meet even more realistic feedback it’s important to design tests that cover both front-end and back-end related services. It’s about the complete experience of an end-user who explores an application using a browser but is impacted by a connected back-end or an external service. If your tests fire an API directly, you would miss out on the user experience which takes into account browser-related aspects.
An ever-changing scope
It’s always a challenge to keep tests up to date, especially with the ever-changing features of modern applications. From a business perspective, you would need to focus on the most impactful features when it comes to writing tests. This is not a one-time effort, business features change every time and thus you also need to change your tests. Having them aligned requires a proper plan.
Classify your business features (just like data classification) gives you guidance on where to put the most effort in. Highly classified features demand a high unit test coverage as well as a lot of integration tests. You should not only test the so-called “happy flows” but also cover “nonhappy flows” which simulate situations and end-user scenarios that you might not think of directly.
When the scope of your application changes, you need to adjust the number of tests and the actual tests themselves. It’s important to keep your tests fast, without a scope reduction, it’s impossible to gather adequate feedback on time. Forget the times in which you run an ever-growing number of tests overnight (the so-called “nightly tests”). As soon as the tests are all finished, there are already new commits to be validated. Cloud technology such as auto-scaling can help on the technical aspects, but it’s about the business impact here.
Other challenges
Organizations that implement continuous testing as part of their DevOps initiatives face other challenges such as:
- Instabilities of their test environments. Think of environments that are set up inconsistently, which require dependencies that are not stable, which do not have access to the right sources of test data, slow data connections which cause time outs, etc. Therefore create a proper testing strategy or test plan (short!) which helps DevOps teams across the organization to set these kinds of environments up in a similar way. In larger organizations, you have the so-called “Test Environment Support Teams (TEST). It should not be necessary to employ such teams, that’s considered a lot of overhead.
- High maintenance. Especially User Interface tests require a lot of effort to set up and maintenance from the perspective of frequent changes. It’s best to align the efforts of UI tests with the release processes so both processes do not slow each other down. If test automation efforts become painful and require too much effort with every minor release, developers abandon this topic and move on. It results in more business risks and thus impacts the business continuity of your organization.
- No shift left possible. Sometimes it’s not possible to actually practice the shift left philosophy. Back-end APIs and other features which take a long time to build are not ready yet when the sprint is finished. Therefore tests are not written at all or these are created in a later sprint. Without discipline, other business features eat up precious sprint time, and then the original tests are forgotten.
Final words
As seen in this article continuous testing helps to reduce business risks when things are carefully considered and carried out. It’s a vital discipline when it comes to implementing DevOps and practicing the shift left and shift right approach. There are several challenges you need to overcome, but I hope my considerations helped you to state your goals to improve and speed up your software development activities. All without creating too many risks for your business.
