HashiCorp are unpacking packer

As you know HashiCorp Packer is an awesome tool for building repeatable Windows and Linux based VM or Docker images, be those, built to run on VMware, Azure, AWS, GCP or pick your own poison. Personally it is one of my favourite tools, and a key part of any deployment that involves Virtual Machines and Containers. It is simple to learn and very powerful. But it had become unwieldly due to HashiCorp bundling popular plugins with the installation routine, as the number of supported and bundled packages grew, Hashicorp were backed into a corner over maintenance; because the plug-ins were tied to a release, in fact that is not entirely true, they were actually backed into the packer binary, any bug that was found in a plugin required a pull request to be made on Packer Core and could not be released until a new version of Packer was released. What you say! That is not how plugins are supposed to work. They are supposed to have their own dev cycle outside of the core product. Didn’t Armon the CTO of HashiCorp state something along the lines of “each tool should do one thing and do that thing well”? Yes, he did and it looks like they forgot that for a while with Packer.

It also caused other issues too, as the binary was growing due to the increasing size of bundled plugins, binary memory usage was more than it should have been. Also, if you are just deploying to a VMware or Azure environment, why would you ever need the plugins for qemu, AWS or GCP? But you never had that choice as they were bundled whether you neded them or not.

Another downside was that it was very difficult for a community-backed plugin to gain any traction within Packer. Any community led plugin’s release cycle would have to dovetail with Packer’s release cycle. For a community led initiative that would have been an onerous albatross around your neck.

So what is HashiCorp doing to fix this Packer Mistake?

HashiCorp decided to fix that mistake and starting with version 1.7 they started the process of unbundling and separating plugins from Packer Core into multi-component plugin binaries. One of the key things that was that with version 1.7 HCL2 became the preferred method of writing Packer files (this was introduced as an experimental feature in 1.5), HCL2 is Hashicorp Configuration language version 2. It was the introduction of HCL2 that made this split possible, third party plugins were the first to be removed from the core binary; this was made possible by the introduction of the “required_plugins” block when using third-party builders, provisioners, or post-processors in your HCL2 configuration. The “required_plugins” block defines the plugins that your configuration depends on and their source location. This may look familiar to terraform users and you would be correct, it looks very similar to the “required_providers” block, in fact the syntax is very similar too.

packer {
  required_plugins {
    my-plugin = {
      version = ">= 1.0.0"
      source  = "github.com/my-org/my-plugin"
    }
  }
}

This block is read when you run “packer init”, which initialises an packer environment, the command will download the required plugins and store them in a “.packer.d” folder in your home directory, not in the build directory; this is not that dissimilar to the running of terraform init. The “.packer.d” folder contains the plugins and their versions, allowing you to use multiple versions of a plugin for different projects.

There is also a “packer plugins” subcommand too, this command is used to interact with packer plugins. Allowing you to install, and remove plugins, list currently installed plugins, and interrogate the necessary plugins required to run a build. Running packer plugins -h will show the following:

OK but json still works and the core components are still in the binary, so why?

Well yes, it is true that the following core plugins are still curently bundled with the binary:

This will not be the case when version 1.10 is released which is when Hashicorp are aiming to have the unbundling complete. As a precursor to this, starting with 1.9.2 which was released in July 2023, HashiCorp, the company initiated a warning trigger, to inform packer users that a script had a reliance on a bundled script and that you should start to rewrite your script to remove the reliance. As you can see from the links above all the necessary plugins have now been released as a multi-component plugin. So there really is no excuse.

Hidden meanings

Although this move makes total sense from a development point of view, it simplifies the binary, shinking it and making it more memory efficient, because you are not loading unnecessary plugins with the binary into memory. It does again raise the question about Hashicorp’s recent license changes. Was this change really due to increasing binary sizes and memory management or a piece of precursor work to the license change; how long have HashiCorp been planning the change to licencing to the core code. HashiCorp stated that only the core products were to be moved to the BSL, however Plugins have remained GPLv2, keeping the plugins in the binary and having differing licensing assigned to core code and plugins would have really muddied the waters. All that said, it was with the release of Packer 1.5 in December 2019 that HashiCorp introduced experimental support for “required_plugin” block, that was almost 4 years ago, so I think position is unlikely. That said, the change has not hindered the license change.

Summary

This is a massive change to how packer is used, the fact that it has taken four years from inception to completion is a testament to that fact. A core tenant of open-source software is to do one thing per binary and do it well and bundling plug-ins in the binary just felt wrong. Also, I have found the move from a JSON format file to one written in HCL2 has dramatically improved the functionality of packer. Personally, I believe that this is a welcome change any period of transition is difficult. The release of 1.10 is earmarked for November 2023 so a little under two months away, I would seriously recommend looking at your packer build scripts and start your migration to the multi-components plug-ins of the last of the integrated plug-ins. Things will stop functioning after the 1.10 version release and you really do not want to be left behind.