Terraform for Fullstack Developers

The more complex your infrastructure the more can go wrong.

Three important error sources are cloud provider outages, application errors, and human errors.

The level of resilience means the type and amount of failures with which a service still fulfills its purpose.

How to Improve Your Application Resilience

Here's how we can improve our own service resilience in response to each error class:

Cloud provider outages are when services your application architecture relies on fail, like the Amazon S3 outage or the Azure AD outage. But it's not just whole services that go down. Your database server or application server might also fail. You can improve resilience against these failures by working on your application architecture. This encompasses many small things such as request retries or database reconnections, and also bigger improvements such as using multiple availability zones or a multi-cloud setup.
Application errors are probably the best-known errors, and preventing them (or handling them appropriately) is part of your standard work as a full stack developer. Counter measures include most code-improvements such as proper error handling, request retries, using a linter, handling edge-cases and always checking for null-values.
Human errors can be devastating, and they a common reason for widespread outages like the AWS S3 outage above. Guarding your application against them is harder, but some counter-measures exist.
- Improve the way you work. Use code reviews. Pair program critical stuff.
- Automatically validate application configuration (the environment variables or config files) upon application startup.
- Automate your infrastructure setup and changes. Keep your infrastructure as code. This makes it easy to review and to keep in sync with your application needs.

The last point is where Terraform comes into play.

What is Terraform?

Terraform is a declarative tool for infrastructure as code (IaC). It's written in Go and developed with Open Source by Hashicorp.

This allows you to keep track of your infrastructure and configuration in a repository, to review it, and to spin everything up whenever you need it.

What I like about Terraform is that it is declarative. This means you write how the infrastructure should look like, not which changes you want.

The basic Terraform workflow is the following:

Write - Describe your infrastructure in the terraform configuration language
Plan - Get a preview of all planned changes from terraform
Apply - let Terraform apply all changes

How does this look in a real-world architecture?

A Real-World Workflow

A real-world workflow can look like this:

The terraform configuration lives in a repository separate to our application code (we have a multi-repo Microservices architecture)
Terraform's state lives in Azure Blob Storage (this allows for easy locking of that state when in use, which terraform automatically does)
When we need to change something with the infrastructure, someone on the team creates a branch with the infrastructure changes. This person makes sure that the terraform plan command shows a reasonable plan. They then create a pull request.
Someone reviews that PR. When everything looks good, the PR is merged and the repository's deploy pipeline creates a plan with the changes. This plan must be approved by a team member before it will be applied to our infrastructure.

Let's look into the three steps write, plan, and apply.

Preparing The Tutorial

We're going to release an official Node.js example application to Heroku.

We begin by installing terraform. Check the terraform downloads, or use brew install terraform if you're on macOS.

After installing you should be able to run terraform -v and get a result like Terraform v0.14.7. If the terraform command is not available, restart your terminal and check your PATH environment variable.

Additionally, you'll need the following Heroku credentials:

Your Heroku account's email address
Your Heroku API key, which you can find in your Heroku account settings

Write - How The Terraform Configuration Language Works

For our goal of creating a Heroku deployment with Terraform we need two terraform files.

One file that describes our variables (called variables.tf), and the main.tf file where we describe all resources we need.

Let's look at the variables.tf:

variable "heroku_email" {
  type    = string
  default = "simon@mannes.tech"
}

variable "heroku_api_key" {
  type    = string
}

This is how we define variables in terraform.

To set values for these variables, set the corresponding environment variables: TF_VAR_heroku_emailand TF_VAR_heroku_api_key. Envivonment variables for Terraforn need to be prefixed with TF_VAR_.

If you don't set the environment variables, Terraform will ask you for their values when you run terraform apply.

Now let's look at the main.tf:

# Enable the heroku provider
terraform {
  required_providers {
    heroku = {
      source  = "heroku/heroku"
      version = "~> 3.2"
    }
  }
}

# Configure the Heroku provider
provider "heroku" {
  email   = "simon"
  api_key = var.heroku_api_key
}

# Create the app
resource "heroku_app" "simple_app" {
  name   = "node-example-mannes-tech"
  region = "eu"
}

# Create a heroku build
resource "heroku_build" "simple_app" {
  app        = heroku_app.simple_app.id

  source = {
    # This app uses a community buildpack, set it in `buildpacks` above.
    url     = "https://github.com/heroku/node-js-getting-started/archive/1.tar.gz"
    version = "1"
  }
}

# Launch the app's web process by scaling-up
resource "heroku_formation" "simple_app" {
  app        = heroku_app.simple_app.id
  type       = "web"
  quantity   = 1
  size       = "Free"
}

At first we need to enable the heroku provider (provider requirements documentation). This is necessary because the heroku provider is not part of the hashicorp namespace, but the heroku namespace.

After that we initialize the provider (provider "heroku" {) with email and api_key. The way to reference variables in terraform is by var.variable_name.

Then follow three Heroku-specific blocks. The resources you need will depend on your project, and the available resource definitions vary with your Terraform provider.

For deploying our app, we first create the app, then create a Heroku build, and finally tell Heroku to spin up one free web dyno for that app.

Terraform automatically detects the dependencies between our resources when we reference one resource within another. If you don't reference a resource but still need to wait for it, you can use the depends_on property to tell Terraform to wait.

Plan - Init and Plan Before You Apply

Before we run terraform plan we must initiate terraform, to install all configure providers.

Start with the command terraform init, which should give an output like this:

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding heroku/heroku versions matching "~> 3.2"...
- Installing heroku/heroku v3.2.0...
terra- Installed heroku/heroku v3.2.0 (signed by a HashiCorp partner, key ID 49ACC74D80C7B012)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

The init command creates a local directory (.terraform) with the used providers, as well as a local terraform lock file (.terraform.lock.hcl). the lock file makes sure that no two terraform apply commands mess with each other. In an automated CI/CD environment you would put the state and the lock into a blob storage of the cloud provider of your choice.

Now is the time to run terraform plan:

terraform plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # heroku_app.simple_app will be created
  + resource "heroku_app" "simple_app" {
      + acm                   = (known after apply)
      + all_config_vars       = (sensitive value)
      + buildpacks            = (known after apply)
      + config_vars           = (known after apply)
      + git_url               = (known after apply)
      + heroku_hostname       = (known after apply)
      + id                    = (known after apply)
      + internal_routing      = (known after apply)
      + name                  = "node-example-mannes-tech"
      + region                = "eu"
      + sensitive_config_vars = (sensitive value)
      + stack                 = (known after apply)
      + uuid                  = (known after apply)
      + web_url               = (known after apply)
    }

  # heroku_build.simple_app will be created
  + resource "heroku_build" "simple_app" {
      + app               = (known after apply)
      + buildpacks        = (known after apply)
      + id                = (known after apply)
      + local_checksum    = (known after apply)
      + output_stream_url = (known after apply)
      + release_id        = (known after apply)
      + slug_id           = (known after apply)
      + source            = {
          + "url"     = "https://github.com/heroku/node-js-getting-started/archive/1.tar.gz"
          + "version" = "1"
        }
      + stack             = (known after apply)
      + status            = (known after apply)
      + user              = (known after apply)
      + uuid              = (known after apply)
    }

  # heroku_formation.simple_app will be created
  + resource "heroku_formation" "simple_app" {
      + app      = (known after apply)
      + id       = (known after apply)
      + quantity = 1
      + size     = "Free"
      + type     = "web"
    }

Plan: 3 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

Our duty is now to look at the changes terraform plans to make, and determine if that sounds reasonable and is what we want.

You can specify an -out parameter like terraform proposes to store the plan and run that exact plan later.

Apply - Make Your Plan Reality

After checking the plan, it's finally time to run terraform apply:

terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # heroku_app.simple_app will be created
  + resource "heroku_app" "simple_app" {
      + acm                   = (known after apply)
      + all_config_vars       = (sensitive value)
      + buildpacks            = (known after apply)
      + config_vars           = (known after apply)
      + git_url               = (known after apply)
      + heroku_hostname       = (known after apply)
      + id                    = (known after apply)
      + internal_routing      = (known after apply)
      + name                  = "node-example-mannes-tech"
      + region                = "eu"
      + sensitive_config_vars = (sensitive value)
      + stack                 = (known after apply)
      + uuid                  = (known after apply)
      + web_url               = (known after apply)
    }

  # heroku_build.simple_app will be created
  + resource "heroku_build" "simple_app" {
      + app               = (known after apply)
      + buildpacks        = (known after apply)
      + id                = (known after apply)
      + local_checksum    = (known after apply)
      + output_stream_url = (known after apply)
      + release_id        = (known after apply)
      + slug_id           = (known after apply)
      + source            = {
          + "url"     = "https://github.com/heroku/node-js-getting-started/archive/1.tar.gz"
          + "version" = "1"
        }
      + stack             = (known after apply)
      + status            = (known after apply)
      + user              = (known after apply)
      + uuid              = (known after apply)
    }

  # heroku_formation.simple_app will be created
  + resource "heroku_formation" "simple_app" {
      + app      = (known after apply)
      + id       = (known after apply)
      + quantity = 1
      + size     = "Free"
      + type     = "web"
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

heroku_app.simple_app: Creating...
heroku_app.simple_app: Creation complete after 7s [id=node-example-mannes-tech]
heroku_build.simple_app: Creating...
heroku_build.simple_app: Still creating... [10s elapsed]
heroku_build.simple_app: Still creating... [20s elapsed]
heroku_build.simple_app: Creation complete after 27s [id=fe559e8a-0210-40b8-1111-aaaaaaaaaaaa]
heroku_formation.simple_app: Creating...
heroku_formation.simple_app: Creation complete after 1s [id=a472a2a0-26dd-49c5-1111-bbbbbbbbbbbb]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

After generating a plan, terraform asks us to accept it. When running Terraform in CI/CD you can tell Terraform to always accept changes and to use a pre-generated plan.

When the command finishes, you'll have a terraform.tfstate file with the current terraform state. If you open it you can find the URL for your newly deployed Heroku app.

When you make changes to your infrastructure, like adding more apps or databases, or increasing dyno sizes/numbers, you update your *.tf files and then run terraform init, terraform plan, and terraform apply again.

In real-world projects you will probably automate this process with your CI/CD infrastructure.

Next Steps - Should You Use Terraform?

Terraform is just a tool.

So, when is Terraform the right tool? Here are some reasonable use-cases:

Your infrastructure is more complex than simple web server + database
You use, e.g., a microservices architecture on Kubernetes together with various cloud services
You need to spin up your infrastructure multiple times, e.g. for multiple customers

Additional Resources: