This is the first post on productionalizing your Terraform environments with Terratest.
This builds on our original set of articles that show how to build out a traditional LAMP stack using terraform as the underlying Infrastructure as code provider. For a refresher on this series review, start with How simple Terraform plans make hybrid and multi-cloud a reality: an introduction.
As you move to productionalizing and expanding your automated builds you will want to introduce some confidence in your deployments. In the traditional development world, this is done by testing. Now it is true to say that no amount of testing can cover every possible issue that may or may not be apparent in your code, it is about managing risk and increasing confidence that outages will be minimized and current capabilities or any newly introduced capabilities will not be subject your current environment to an outage on deployment. I have seen a large number of environments that do not do any form of formal testing of their Terraform code, believing that the response that is received from a successful apply when coupled with a plan is sufficient, but the reality is that is only confirmation of deployment it is not a confirmation of functionality and the plan can be difficult to spot mistakes in your code.
What is Testing?
In traditional testing, you have unit tests, integration tests, and end-to-end testing. each of these tests increases the testing attack surface.
Unit Test
This form of testing is used against single pieces of code or functions. In traditional coding it would be used to test a function or a class. These tests are quick verification tests and can and do give instant feedback and build confidence that your small pieces of code are up to the task at hand. However, this does not mean that they will work when coupled with other pieces of the jigsaw that is your deployment. This leads to the need for integration tests.
Integration Test
An integration test moves the testing scope out and in a traditional coding environment will deal with the integration of several functions or classes. In an IaC environment it would be your Load Balancer deployment for example or testing your database. Once these tests have completed successfully you can feel confident in large parts of your deployment, but you are still not finished as there are still questions about how everything will fit together. This is where the End-to-End tests come into play.
End-to-End Tests
As the name suggests, this is a complete test of the whole service, verifying access to storage, databases, and end-user usage tests. Once these are complete you can have confidence in your entire deployment stack. Note I do not say complete confidence as your test results are only a good as test plan and users are the mother of ingenuity where application usage is concerned, but you should be confident that many of your issues and bugs should have been caught as each form of test serves a different purpose and can and does catch different bugs.
Manual Testing
When you start to automate your tests, you need to think about what you physically do when you manually test your environment. If, for example you are testing a database, you would test logging in as a user, creating a database, creating a table, writing to a table, modifying a table, deleting a table, if it is a clustered database you would most likely add tests to verify a write is in both locations. Test database failover if it is an active/passive node cluster. If it is multi-node active/active; you would probably add tests to verify “leader/follower” elections. Tests with networking functionality, dropping paths to storage to test loss of replicated datastore etc.
Starting to Automate your Tests
The journey to test automation tends to start with a growing perception that your code is poor, there is no longer confidence in your deployments as something always fails causing expensive outages in production. This is only exacerbated as your deployments become more complex.
The first problem in automating IaC code is that it is almost untestable, the only reliable test that could be done was to deploy the code and see if it was successful, but as we have already stated this just test the deployment not any functionality. Terraform has a plugin that can extend your code with the ability to undertake basic unit testing and uses the “GO” language to build out the tests. However, the basic Terraform plugin is a little basic and to get decent results it requires deep understanding of the GO language. Therefore we will utilize Terratest from Terragrunt, this open-souce product still uses the Go Language but provides several methods to streamline your testing rather than having to re-create the proverbial wheel. Another advantage of using Terratest is that you can also integrate it into your Packer builds too, but it also has packages that integrate with Consul, Vault, Nomad, Docker, Kubernetes, AWS GCP and much more.
Installing Terratest
Before you can start to use Terratest you must install the GO Libraries to do this navigate to GoLang site there are options for Windows, Linux and Mac so it is fairly cross-platform. We are going to install in our WSL build which is currently running 20.04 Ubuntu.
To download the binary we use the following command:
wget https://golang.org/dl/go1.15.6.linux-amd64.tar.gz
if it successfully downloads you should receive an output similar to below
Next you will need to untar the download, we are going to extract the file to /usr/local, the build will add a go directory.
tar -C /usr/local -xzf go1.15.6.linux-amd64.tar.gz
To verify the installation move to /usr/local and verify that the “go” directory has been created
cd /usr/local ls
next to make sure that your Terratest testing code will run you will need to add the go libraries to your path statement to do this use the following command:
export PATH=$PATH:/usr/local/go/bin
Finally, verify that the installation has worked as expected by issuing:
go version
you will receive the following response dependent on the version installed:
Next we need to install Dep, which is a dependency manager for Go this is a simple as issuing the following command in Debian and Ubuntu
Next create the following folders in your home directory /go. /go/src and go/src/terraform and create the GOPATH environment variable
Export GOPATH=$HOME/go
Finally move to $GOPATH/src/terraform and issue the following command
dep init
this will create the following files and folder
As a final sanity check to verify that your configuration is correctly configured create the following file: Sanity_test.go and add the following contents:
package test
import (
“fmt”
“testing”
)
func TestGoIsWorking(t *testing.T){
fmt.Println()
fmt.Println(“if you can see this, everything is working as expected – Yay You”)
fmt.Println()
}
Then run the test using the following command
Go test -v
If it has been successfully installed you will receive the following:
NOTE: Go is very finicky regarding spacing and line feeds. The first time this was written the test failed because there was a leading space on the final closing curly bracket.
So what do we want for our first test?
Remember when we said when automating your test, think about what you would do manually,
- Create a generic standalone module
- Create an easy to deploy example for the module
- Run terraform apply to deploy the example in a real environment
- Validate the result is as expected
- Run terraform destroy to remove the infrastructure at the end of the test
But that is for the next article
Summary
Testing traditional code is difficult, you have to understand the stories that are used to define the needs of the function or program, but traditional development environments have been designed with testing in mind and the concept is fully understood. Testing infrastructure as Code is still evolving and testing is not as well understood as evidenced by environments that have no real formal testing strategy outside of Terraform plan, terraform fmt, etc, the situation is changing and as deployments get more complicated, companies are starting to look at a proper validated testing strategy. As we move through this new series we will learn how to create the necessary test for the already created LAMP stack.
