AwesomeOps Part 7: Terraform Lockdown

Well hello there AwesomeOps fans!! We are back at it this week with a quick post to help teams lockdown all aspects of their Terraform implementation. Hopefully this post will help teams avoid security blunders like this poor defensive back missing a tackle :( Writing and

maintaining secure Terraform is like the ability of the defense to tackle. If you cannot tackle effectively, it is game over. If you cannot secure your Terraform environment, it is also game over. So, AwesomeOps is here to review ALL of the things you need to lockdown your automation. We start off by explaining the importance of pre-commit hooks, and then we move on to pre-flight check steps to run before you even run a Plan. Next, we discuss marking variables sensitive, and using the sensitive function. Finally we will talk about securing your statefiles, and what resources and data calls to avoid to prevent leakage of truly secret values into your state. Hopefully we can enable your team to read all plays pre-snap and crash through the offensive line to sack the quarterback like the guy below.

You might be thinking "Hey Mentat! What is the deal with all of these US football references?" The answer is multifaceted really: 1. Engineers are complex creatures with diverse interests, 2. It is US football season, 3. Defensive programming is a necessity, so relating coding to defensive football plays was natural. With all of that said, let's get into it!!

Secure Before Commit:

If you have never heard of pre-commit hooks go download and install it now on your system. Seriously, we will wait!... All set? Great! Pre-commit is an important part of the automation workflow. The tool is used to inspect the snapshot that’s about to be committed for anything and everything you add to your pre-commit config file. Below is an example of a pre-commit config we frequently use:

---
repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.76.0
    hooks:
      - id: terragrunt_fmt
      - id: terragrunt_validate
      - id: terraform_tflint
      - id: terraform_docs
      - id: terraform_tfsec
  - repo: https://github.com/jumanjihouse/pre-commit-hook-yamlfmt
    rev: 0.2.2 # or specific tag
    hooks:
      - id: yamlfmt
        args: [--mapping, '2', --sequence, '4', --offset, '2']
  - repo: https://github.com/zricethezav/gitleaks
    rev: v8.18.0
    hooks:
      - id: gitleaks
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.3.0
    hooks:
      - id: check-merge-conflict
      - id: trailing-whitespace
      - id: check-added-large-files
      - id: check-executables-have-shebangs
  - repo: https://github.com/psf/black
    rev: 22.10.0
    hooks:
      - id: black

The config file above is our boiler plate starter template that we start with and then update as necessary based on needs. The first block titled pre-commit-terraform targets Terraform specific configs. Let's see what is going on here:

1. id: terragrunt_fmt --> This checks your code to make sure it is well formatted

2. id: terragrunt_validate --> This runs a Terraform validate

3. id: terraform_tflint --> This runs Terraform lint

4. id: terraform_docs --> This runs terraform docs and updates the repo README file with all of your needed and optional variables, and other details that take a while to type out.

5. id: terraform_tfsec --> This runs tfsec against your Terraform code to check against a set of best practices and security policies.

I am sure some of our readers are thinking, "I get it, but what is the big deal if the code is cleaned up before committing and pushing, when it could easily be cleaned up later? Also, this is going to slow me down from delivering value. Why do I need this?" Great questions. There are 3 big reasons why we urge engineers to use pre-commit hooks:

1. Humans are fallible

2. It is much easier to cleanup code locally before it gets distributed to other developer systems

3. Things that will "get done later" usually do NOT get done later

All of the hooks defined above are used to secure environments. Properly formatted code ensures readability, long term maintenance, and mitigates the introduction of vulnerabilities due to hidden logic flaws. Terraform validate and TFLint allows for early detection of errors, validates resource references and configuration files, and helps maintain code quality, consistency, and reliability. Terraform Docs helps all future developers easily understand the module you created, which lends to supportability and mitigates risks of security mistakes caused by confusion about the module. Lastly, TFSec is a great opensource tool that runs a set of best practices and security policies against your code to point out possible issues. All of these security measures are completed automatically before your code is actually committed locally and before the code is pushed up to your Git repo.

The final pre-commit hook we will cover in this blog is Gitleaks. Gitleaks is a great opensource tool that identifies common secrets that are committed to code. The other nice thing about Gitleaks is that you can define a TOML file to add more regex patterns that Gitleaks can search and identify. While all of the previously discussed pre-commit hooks are fantastic, this one might be the most important. Adding Gitleaks to your pre-commit hooks will prevent common secrets from ever making their way into your codebase. If you have ever had to cleanup accidentally or purposefully exposed secrets in Git, then you know the difficulty of actually purging the secret. You have to use something like BFG that can REDACT secrets from all branches, but then you have to notify all devs that have pulled the codebase to purge or delete and re-clone the repository from their local systems. Long story short, it is not fun or easy to remove secrets from code in Git, which is why you want to identify and remediate the secret leaks before they every make it into a commit.

Section Cheat Sheet: Install the list below on your system to get started quickly with pre-commit hooks

• python

• pre-commit

• gitleaks

• tfsec

• terraform-docs

• tflint

• terraform

• VSCode OR Atom

• Install the following plugins in your text editor of choice: Ansible, HashiCorp HCL, GitLens, Markdown Preview, Prettify JSON

Terraform Sensitive Variables & Function

Now that you have the basics of your security automation tooling in place protecting you and your team from security risks, we can move onto some features built into Terraform that will protect your code from exposing secrets. First up on our list is marking variables as sensitive. This is very easy to do and highly recommended. The code example below shows how to do this:

variable "username" {
  type        = string
  description = "Your Default Administrator Username"
  sensitive   = true
}

That is neat! All we needed to do is add the key "sensitive" with a value of "true" to tell Terraform that this particular secret value should not be exposed in the output. It is highly recommended that you add this sensitive key:value pair to all variables that are considered secret and semi-secret. Once you have done this, your Terraform output should look similar to this:

username = (sensitive value)

Next up is the sensitive function. The Terraform sensitive function performs the same work and has the same result as marking variables as sensitive, except it is a function so it can be used in most areas of your code. The implementation is slightly more complicated than marking variables as sensitive, so let's take a closer look:

sensitive(1)

sensitive(var.sample_variable

locals {
    protected = sensitive(data.your_data_call.id)
}

Above are three examples of how you could use the sensitive function in Terraform. The result is the same as marking variables as sensitive. The output displayed in your pipeline or terminal will show "(sensitive value)". This is particularly helpful when the sensitive data is retrieved or generated by a resource declaration.

That was easy and cool! However, you must always remember that just because you marked something sensitive, that does NOT mean the values are marked sensitive in your statefiles! Marking values as sensitive will only prevent Terraform from exposing the secrets in outputs of pipelines and logs while Terraform is running. Once Terraform has completed the Apply phase, any value marked as sensitive will be written to the statefile in plain text.

While this sounds scary, it is not. Statefiles need to be treated as secrets and secured accordingly. This means that it is imperative that all necessary controls are put in place to protect your statefiles. In order to do this you will need to leverage a well known and supported remote backend. We also recommend using one of the big 4 backends to secure your statefiles: Terraform Enterprise, Amazon S3, Azure Blob Storage, Google Cloud Storage. With these backend types you will be able to create a layered security approach to who has access to your secret statefiles. Here are our basic recommendations:

• Identity Access Management (IAM). Layer 1 in securing your statefiles. Think Azure Active Directory (AAD --> Soon Entra ID) and using your company provided identity to access the platform that manages the service hosting your statefiles.

• Multi-Factor Authentication (MFA). Layer 2 is paired with IAM to add another layer of security.

• Role Based Access Controls (RBAC). Layer 3. Role access is granted once your identity has been validated and you have successfully logged into the statefile backend. Roles dictate the types of actions your validated identity is able to perform.

• Remove public access to your service. Layer 4. This can be done in many ways. Depending on the backend, you can add a known set of your company public IPs to the service that are allowed access, or present your backend into a private network (VPC or VNET) so that your backend gets an RFC 1918 IPv4 address.

• Only allow non-human identities access to statefiles. Layer 5. These are things like Active Directory service accounts, Service Principal Names (SPN), Machine Identities

Next Level Security With Terraform Lockdown

While statefiles should be treated as secrets, the reality is that there are going to be cases where your business simply will not allow secrets of a certain type to be written to state.

So, what to do? Glad you asked. :) Let's cover a few techniques to prevent secrets from ever be stored in code, or passed into TFVARS files, and yes even prevent them from being exposed in statefiles.

Store & Retrieve Credentials From a Secret Service:

- Do not ever store credentials locally

- Never write them to files

- Access the secrets programmatically from a secret service like Azure Key Vault

- ONLY use the secrets retrieved in memory of the runtime of your pipeline service. When designing and implementing your Terraform hosting platform make sure that you define the acceptable secret services supported by your company, and make sure that the service has a modern API so that you can programmatically get and set secrets. Here is a practical example that we use frequently.

Azure DevOps --> Pipeline Executed --> Request picked up by AKS cluster --> secrets are retrieved from Azure Key Vault and stored in the memory runtime of the container --> Terraform references the in memory secrets for the duration of the run. This type of workflow will prevent secret leakage in logs and outputs and code. That said, if you are using data declarations to get the secrets, then those secrets will get written to your state! So, what to do? Our recommendation is to replace data calls to secrets providers with in-memory alternatives that pass the secret to the memory runtime of your pipeline context. Example of a swap we have done to prevent secrets from being written to state:

# Before
data "azurerm_key_vault_secret" "this" {
    name       = "secret-squirrel"
    key_vault_id = var.key_vault_id
}

# After
export USER=$(az keyvault secret show --vault-name $KV --name "secret-squirrel" -o json | jq -r .value)

That was easy enough! We replaced the Terraform data call to Azure Key Vault with some basic Azure CLI and jq to store the secret temporarily in the runtime memory context of the pipeline run. This allows Terraform to reference the secret without anyone ever knowing it, without it being written to disk, without it being stored in code, AND without it being persisted into the statefile!!

Now you may be asking, but what about secrets that are already persisted in statefiles? Well that is a topic for another blog. We have created a simple and easy to implement automated harness to purge those super secret secrets from statefiles, while allowing Terraform to continue to successfully execute Plan and Apply phases.

The End

Well that was a quick and fun journey through Terraform Lockdown. We covered the necessity of pre-commit hooks to protect your code, Terraform sensitive variable marking and Terraform sensitive function use and execution, and reviewed how to secure your entire Terraform environment by moving secrets to in-memory Terraform references, and finally preventing super secret data from ever persisting into your statefiles.

We hope you enjoyed this AwesomeOps post! We will be back in a few weeks with another edition of AwesomeOps.