Continuing on with our series of posts about Terraform and the vagaries about language usage. This post is a result of a question we were asked regarding overriding variables in Terraform from the defaults that are set in the variables.tf or variables.tfvars files that are stored in the terraform deployment directory. For a quick refresh of variable principles in Terraform have a look review of the post on the difference between variable.tf and variable.tfvars files.
Whitepaper: Increasing Developer Velocity in the Cloud Operating Model
This whitepaper discusses how HashiCorp tools and the GitHub platform work together to enable organizations to adopt a strong CI/CD workflow and increase developer velocity.
During a normal deployment, Terraform will load all files with the extension of tf, tfvar, and tf.json that are found in the directory that the deployment is being run from and it quite rightly expects that each file will contain unique objects. In fact, if you have duplication Terraform will scream and shout at you during the planning or validation stages.
However, in some use cases, there may be a need to override the values that are contained in your variable files. For example, you want to test out a build in a different location. Or to check the efficacy of a new image build, perhaps increase or decrease the size of the image deployed for performance testing.
Fortunately, Hashicorp has already thought of this and produced a number of methods that can override previously defined variables.
Overriding variables in Terraform via the command line
Obviously the best method of using variables to prevent human error is to have them correctly defined in the first place, but as already alluded too there is value in allowing overrides and ad-hoc changes to build definitions.
The first method we will look at is to use an input variable at the command line this is the simplest of methods and most commonly used for ad-hoc overrides, here we simply add a -var ‘variable_name=”value” as an option for the terraform plan or apply command.
terraform apply -var region=”eu-west-1”
This command would override the region variable defined in the variable.tf or variable.tfvars file. This option is not confined to a single override as you can define multiple -var options at the command line
terraform apply -var region=”eu-west-1” -var size=”t3.large” -var ami=”ami123”
This will obviously get very, very complicated the more variables there are that need to be changed.
There is also the order of precedence to be considered here too. Variables on the command line are applied in the order that they are written so as an example, If we write:
terraform apply -var region=”eu-west-2” -var region=”eu-west-1”
The value of region will be set to “eu-west-1” as that is the last option provided and as such the last one applied. However, If you meant for these options to actually be part of a list rather than a single input; then the syntax for the option changes to:
terraform apply -var region=[”eu-west-2”,”eu-west-1”]
The resulting variable will be
region = [ eu-west-1, eu-west-2, ]
To pass a map variable use the following syntax:
terraform apply -var ‘mymap={foo=”bar”}’ -var ‘mymap={boo=”hoo”}’
This will result in the following values
mymap ={
foo =”bar”
boo =”hoo”
}
There is however major a caveat here, if you are attempting to override a map variable definition from the command line it will fail, as is there is currently no method of replacing map inputs from the command line and they are just merged into the current definition (perhaps this is an area Hashicorp need to investigate).
So your resultant map would be something like this:
mymap ={
another=”variable”
yet_another=”variable_too”
foo =”bar”
boo =”hoo”
}
If you are replacing the values to map options already assigned in the map then those values are changed.
mymap ={
another=”variable”
yet_another=”variable_too”
}
|
mymap ={
another=”different_variable”
yet_another=”this_is_different_too”
}
|
As you can see if you are attempting to change a significant number of variables massive potential for human error.
This leads on to another command-line option for overriding variables the option “-var-file”.
Overriding variables in Terraform using a new variable file
If you are finding that you need to override multiple variables it can obviously become very tedious with a potential for human error to keep inputting “-var” at the command line. Instead, you can override the entire variable file by replacing it with a new one.
terraform apply -var-file=newvariable.tf
This option can be used multiple times as well and can be intermixed with the “-var” option, but it is important to remember the precedence order that command line options are applied. This does not solve all the problems (it suffers from the same issues as with maps and undeclared options).
Overriding variables in Terraform using an Override file
Another method for changing variables is the Override file. This is a file that sits outside of the normal terraform method of loading files and assigning values.
The standard method of reading *.tf and *.tfvar files is in alphabetical order. Terraform will, during the initial loading of the configuration files, initially skip any file with the name “_override.tf” or “override.tf.json” and after the completion of the files revert back to parse them; thereby overriding any options already defined.
NOTE: there is an underscore (_) at the start of the override file, this is very important. If a file called “override.tf” was in the directory folder this file would be parsed in alphabetical order and any later named files would override the changes.
One thing to be aware of with the use of override file is that they will muddy the waters surrounding code readability as they are outside of the normal. It is highly recommended that any portions of code that can be overridden are annotated to highlight this in the relevant “*.tf” files.
The override files are parsed last and unlike a variable file the format of override files are in either HCL or JSON, so for example, if we wished to change the size of an EC2 instance
In the original *.tf file we would have a stanza that looked somewhat similar to the following:
resource "aws_instance" "web" {
instance_type = "t2.micro"
ami = "ami-408c7f28"
}
In the override file we would have the following stanza
resource "aws_instance" "web" {
instance_type = "t3.large"
}
After parsing out the file Terraform will override or merge the original output with the new one so the configuration will now read:
resource "aws_instance" "web" {
instance_type = "t3.large"
ami = "ami-408c7f28"
}
As a general rule overriding or what is actually happening, merging will follow these rules
- A top-level block in an override file merges with a block in a normal configuration file that has the same block header. The block header is the block type and any quoted labels that follow it.
- Within a top-level block, an attribute argument within an override block replaces any argument of the same name in the original block.
- Within a top-level block, any nested blocks within an override block replace all blocks of the same type in the original block. Any block types that do not appear in the override block remain from the original block.
- The contents of nested configuration blocks are not merged. Nested configuration blocks are replaced.
- The resulting merged block must still comply with any validation rules that apply to the given block type.
Exceptions
Now that we have the general rules what are the exceptions because there are always some.
Overriding resource and data blocks
When you are using nested blocks the contents are merged on an argument by argument basis. So if you override the create_before_destory argument then all ignore_changes options will not be altered. Provisioner blocks are ignored. Connection blocks changes will completely override pre-existing connection blocks. You cannot use the depends_on meta-argument in an override block this will result in an error.
Overriding variable blocks
The vast majority of variable block overrides work according to the general rules outlined above, with the exception of type and default arguments.
When a default value is declared any attempt to use override to change the variable type (ie string to Boolean) will result in an error as the conversion is not possible. There any attempt to change a default value must be made using a value of the same type as that originally declared.
Overriding output blocks
You cannot use the meta-argument depend_on as with merging resource and data block any use of this will result in an error.
Overriding locals blocks
Any locals variable block change is applied on a value by value basis, the blocks persay are ignored.
Overriding terraform blocks
The setting within terraform blocks are individually considered during the merging process. If the required_providers argument has been defined its value is merged on a element by element basis, this allows the override block to adjust the values for a single provider rather than affecting all the providers defined.
However, if the required_version and required_provider setting is set. Each override constraint will entirely replace the settings for the same component.
Summary
As you can see the capabilities and possibilities of overriding variables in Terraform are quite broad and powerful, but with some gotchas and complexity. If you follow the rules carefully there is no reason why you should not be able to successfully modify objects without burning the house down. Terraform does a good job of protecting you from yourself. Remember that as long as you do not use the “-autoapprove” option on your command-line you can verify that the output is as you expect before you run the final deploy.
