Migrate state from S3 to HCP Terraform

Terraform backends define how to perform Terraform operations and where to store Terraform state. Terraform supports multiple remote backends, including AWS S3 for state storage and DynamoDB for state locking to prevent concurrent operations. HCP Terraform also offers both of these features, as well as built-in version control (VCS) integration, remote execution environments for safe and uninterrupted runs, and a private module and provider registry.

In this tutorial, you will migrate state from S3 to HCP Terraform. To do so, you will initially use an AWS S3 bucket to store your state, and then migrate your state to HCP Terraform. Finally, you will trigger a plan and apply through HCP Terraform.

Prerequisites

This tutorial assumes that you are familiar with the Terraform and HCP Terraform plan and apply workflows. If you are new to Terraform itself, refer first to the Getting Started tutorials. If you are new to HCP Terraform, refer to the Get Started - HCP Terraform tutorials.

For this tutorial, you will need:

• the Terraform v1.1.0+ CLI installed locally.
• an HCP Terraform account.
• an AWS account with AWS Credentials configured for use with Terraform.

Create variable set for AWS credentials

You will need to configure your HCP Terraform workspace with AWS provider credentials for provider authentication. The most convenient way to do so is with an HCP Terraform variable set, which allows you to centrally manage provider credentials and reuse them across workspaces.

If you do not have a variable set containing your AWS credentials, follow the steps in the Create and Use a Variable Set tutorial to create one. Select Apply to specific workspaces for the variable set scope.

Create resources for S3 remote backend

In your terminal, clone the example repository. This repository contains Terraform configuration to deploy an S3 bucket and a DynamoDB table, which you will use as a remote backend for your EC2 instance configuration before you migrate your state to HCP Terraform.

$ git clone https://github.com/hashicorp-education/learn-terraform-s3-remote-state

Navigate to the cloned repository.

$ cd learn-terraform-s3-remote-state

Initialize the configuration.

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.65.0"...
- Installing hashicorp/aws v3.65.0...
- Installed hashicorp/aws v3.65.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Next, apply the configuration. Respond yes to the prompt to confirm.

$ terraform apply

## ...

Plan: 2 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + dynamodb_endpoint = "terraform-state-lock-dynamo"
  + s3_bucket_name    = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_dynamodb_table.terraform_state_lock: Creating...
aws_s3_bucket.terraform_state: Creating...
aws_s3_bucket.terraform_state: Creation complete after 3s [id=learn-s3-remote-backend-20210720075828379700000001]
aws_dynamodb_table.terraform_state_lock: Creation complete after 8s [id=terraform-state-lock-dynamo]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

dynamodb_endpoint = "terraform-state-lock-dynamo"
s3_bucket_name = "learn-s3-remote-backend-20210720075828379700000001"

Now that you have an S3 bucket and DynamoDB table, you are ready to use them to store your Terraform state.

Notice that Terraform displays the dynamodb_endpoint and s3_bucket_name outputs. You will use these values to configure your remote backend in the next section.

Clone example repository

Navigate out of your first repository directory.

$ cd ..

Clone the second example repository from GitHub. This repository contains the configuration to deploy an Ubuntu EC2 instance in the US West 1 region.

$ git clone https://github.com/hashicorp-education/learn-terraform-migrate-s3-tfc

Navigate to the cloned repository directory.

$ cd learn-terraform-migrate-s3-tfc

Update and review Terraform configuration

Open main.tf. The terraform block defines the S3 remote backend configuration, instructing Terraform to store your state in the S3 bucket you provisioned in the last step. The configuration also uses the DynamoDB table for state locking.

Update the bucket name to the s3_bucket_name output value from the previous step.

terraform {
   backend "s3" {
     encrypt        = true
     bucket         = "learn-s3-remote-backend-20210720075828379700000001"
     dynamodb_table = "terraform-state-lock-dynamo"
     key            = "learn-terraform-s3-migrate-tfc/terraform.tfstate"
     region         = "us-west-1"
   }

   ## ...
}

Find the aws_instance.web resource, which defines an Ubuntu EC2 instance. Notice that the instance’s workspace tag is set to terraform.workspace, a special value that resolves to the configuration’s workspace name.

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t2.micro"

  tags = {
    Name      = "HelloWorld"
    workspace = terraform.workspace
  }
}

Deploy EC2 instance

Initialize the Terraform configuration.

$ terraform init

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.65.0"...
- Installing hashicorp/aws v3.65.0...
- Installed hashicorp/aws v3.65.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Next, apply the configuration. Respond yes to the prompt to confirm.

$ terraform apply

## ...

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_instance.web: Creating...
aws_instance.web: Still creating... [10s elapsed]
aws_instance.web: Still creating... [20s elapsed]
aws_instance.web: Still creating... [30s elapsed]
aws_instance.web: Creation complete after 33s [id=i-0497f64b5bb340b1b]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Visit the AWS S3 Dashboard and select the S3 bucket containing your remote state. You will find a Terraform state file in the bucket.

Verify EC2 instance

Open the AWS EC2 dashboard and find your recently deployed EC2 instance.

Notice the EC2 instance’s workspace tag resolves to default.

Since you ran Terraform in the default workspace, the terraform.workspace variable resolves to default.

Deploy development EC2 instance

Terraform CLI workspaces enable you to associate multiple states with a single configuration. Some organizations use CLI workspaces to provision similar infrastructure across multiple environments (for example: development, test, production). Though modules are now the best practice way to reuse configuration, many Terraform configurations still use CLI workspaces.

HCP Terraform workspaces are different from Terraform CLI workspaces. HCP Terraform workspaces manage collections of infrastructure; separate workspaces function like completely separate working directories.

This tutorial guides you through setting up Terraform CLI workspaces to show you how to effectively migrate configuration organized in workspaces to HCP Terraform.

Create a new Terraform workspace named dev to deploy another EC2 instance using the same Terraform configuration.

$ terraform workspace new dev
Created and switched to workspace "dev"!

You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.

Next, apply the configuration. Respond yes to the prompt to confirm.

$ terraform apply

## ...

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions in workspace "dev"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_instance.web: Creating...
aws_instance.web: Still creating... [10s elapsed]
aws_instance.web: Still creating... [20s elapsed]
aws_instance.web: Still creating... [30s elapsed]
aws_instance.web: Creation complete after 33s [id=i-005787460b2252e42]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Navigate to the S3 bucket containing your remote state. Find the directory named env:/. Select the directory, then select dev/. This directory contains your Terraform state file for the dev workspace.

Replace remote backend with HCP Terraform

Using AWS services as a remote backend requires managing disparate services to handle your Terraform state. You need an S3 bucket for state storage, and a DynamoDB instance for state locking. You must manage the security and lifecycle of these resources for each of your Terraform projects. Instead, you can use HCP Terraform for state storage, locking, and remote execution for all of your Terraform projects.

To migrate your state from S3 to HCP Terraform, you need to replace the backend configuration.

Still working in your learn-terraform-migrate-s3-tfc directory, replace the backend “s3" block in main.tf with the following cloud block, replacing <YOUR-ORG-NAME> with your HCP Terraform organization’s name.

cloud {
  hostname = "app.terraform.io"
  organization = "<YOUR-ORG-NAME>"
  workspaces {
    tags = ["learnterraform"]
  }
}

Your final terraform block will be similar to the following, except your HCP Terraform organization name replaces hashicorp-learn.

terraform {
  cloud {
    hostname = "app.terraform.io"
    organization = "hashicorp-learn"
    workspaces {
      tags = ["learnterraform"]
    }
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.65.0"
    }
  }
}

Since you are migrating multiple Terraform workspaces to HCP Terraform, the workspaces block expects tags. When migrating a single workspace, use the name attribute and a string value instead.

Login to HCP Terraform

Log into your HCP Terraform account in your terminal.

$ terraform login
Terraform will request an API token for app.terraform.io using your browser.

If login is successful, Terraform will store the token in plain text in
the following file for use by subsequent commands:
    /Users/<USER>/.terraform.d/credentials.tfrc.json

Do you want to proceed?
  Only 'yes' will be accepted to confirm.

  Enter a value:

Confirm with a yes and follow the workflow in the browser window that automatically opens. Paste the generated API key into your Terminal when prompted. For more detailed instructions on logging in, review the Authenticate the CLI with HCP Terraform tutorial.

Migrate existing state to HCP Terraform

Run terraform init to migrate your Terraform state file from S3 to HCP Terraform.

$ ​​terraform init

Initializing HCP Terraform...
Migrating from backend "s3" to HCP Terraform.

When prompted, enter main to name your default workspace.

HCP Terraform requires all workspaces to be given an explicit name.
  Please provide a new workspace name (e.g. dev, test) that will be used
  to migrate the existing default workspace.

  Enter a value: main

When prompted, enter 1 to rename your workspaces according to a pattern you will provide on the next prompt.

Would you like to rename your workspaces?
  Unlike typical Terraform workspaces representing an environment associated with a particular
  configuration (e.g. production, staging, development), HCP Terraform workspaces are named uniquely
  across all configurations used within an organization. A typical strategy to start with is
  <COMPONENT>-<ENVIRONMENT>-<REGION> (e.g. networking-prod-us-east, networking-staging-us-east).
  
 ## …

  When migrating existing workspaces from the backend "s3" to HCP Terraform, would you like to
  rename your workspaces? Enter 1 or 2.
  
  1. Yes, I'd like to rename all workspaces according to a pattern I will provide.
  2. No, I would not like to rename my workspaces. Migrate them as currently named.

  Enter a value: 1

When prompted, enter learn-terraform-s3-migrate-tfc-* to specify the pattern for the HCP Terraform workspaces’ names.

How would you like to rename your workspaces?
  Enter a pattern with an asterisk (*) to rename all workspaces based on their
  previous names. The asterisk represents the current workspace name.
  
  For example, if a workspace is currently named 'prod', the pattern 'app-*' would yield
  'app-prod' for a new workspace name; 'app-*-region1' would  yield 'app-prod-region1'.

  Enter a value: learn-terraform-s3-migrate-tfc-*

Acquiring state lock. This may take a few moments...
Acquiring state lock. This may take a few moments...
Migration complete! Your workspaces are as follows:
* learn-terraform-s3-migrate-tfc-dev
  learn-terraform-s3-migrate-tfc-main


Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v3.65.0

HCP Terraform has been successfully initialized!

You may now begin working with HCP Terraform. Try running "terraform plan" to
see any changes that are required for your infrastructure.

If you ever set or change modules or Terraform Settings, run "terraform init"
again to reinitialize your working directory.

Verify HCP Terraform workspaces

In your HCP Terraform dashboard, you will find the two workspaces you migrated from the S3 backend.

Assign variable sets to HCP Terraform workspaces

Go to the Variable sets page and open the AWS Credentials variable set you created earlier.

Under Workspaces, select learn-terraform-s3-migrate-tfc-main and learn-terraform-s3-migrate-tfc-dev to assign the AWS credentials variable set to the workspaces. Click Save variable set.

Go to your learn-terraform-s3-migrate-tfc-main workspace and select the Variables tab. You will find the AWS Credentials variable set.

Start and apply plan in workspaces

Go to the learn-terraform-s3-migrate-tfc-main workspace. Click on Actions from the top navigation bar, then Start new plan to start a new remote HCP Terraform run.

Click Start plan to confirm the plan run.

Once the plan completes, Terraform will propose to change one resource. Click on aws_instance.web to view the changes to it. Terraform updates the instance’s workspace tag from default to learn-terraform-s3-migrate-tfc-main.

This is because the terraform.workspace variable the example configuration uses to define the workspace tag resolves to the HCP Terraform workspace name, instead of the Terraform CLI workspace name.

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t2.micro"

  tags = {
    Name      = "HelloWorld"
    workspace = terraform.workspace
  }
}

Click Confirm & Apply, then click Confirm run to apply the changes.

Start and apply plan in dev workspace

Open the learn-terraform-migrate-s3-backend-dev workspace and follow the same steps to start and apply a plan run.

Similarly to the learn-terraform-migrate-s3-backend-main workspace, HCP Terraform will update the instance’s workspace tag to reflect the HCP Terraform workspace name: learn-terraform-migrate-s3-backend-dev.

Clean up resources

Now that you have migrated your workspaces to HCP Terraform and used HCP Terraform to manage your infrastructure, clean up the resources created in this tutorial to avoid unexpected AWS charges.

First, visit the learn-terraform-migrate-s3-backend-main workspace, and from the Settings menu, select Destruction and Deletion. Ensure that the Allow destroy plans checkbox is checked. Next, click the Queue destroy plan button, and follow the steps to queue and confirm a destroy plan.

Once HCP Terraform has destroyed your infrastructure, delete the workspace by clicking on Delete from HCP Terraform and following the steps in the prompt.

Follow the same steps to destroy the EC2 instance provisioned in the learn-terraform-migrate-s3-backend-dev workspace and the workspace itself.

Optionally, delete your AWS credentials HCP Terraform variable set.

Delete S3 remote state resources

Navigate to the directory containing the configuration for the S3 bucket and DynamoDB table used for the remote S3 backend.

$ cd ../learn-terraform-s3-remote-state

Destroy these resources. Respond yes to the prompt to confirm.

$ terraform destroy

## …

Plan: 0 to add, 0 to change, 2 to destroy.

Changes to Outputs:
  - dynamodb_endpoint = "terraform-state-lock-dynamo" -> null
  - s3_bucket_name    = "learn-s3-remote-backend-20211111183654999200000001" -> null

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

aws_dynamodb_table.terraform_state_lock: Destroying... [id=terraform-state-lock-dynamo]
aws_s3_bucket.terraform_state: Destroying... [id=learn-s3-remote-backend-20211111183654999200000001]
aws_s3_bucket.terraform_state: Destruction complete after 3s
aws_dynamodb_table.terraform_state_lock: Destruction complete after 4s

Destroy complete! Resources: 2 destroyed.

Next steps

Over the course of this tutorial, you migrated Terraform state from a remote S3 backend to HCP Terraform. Then, you created a variable set and assigned it to your migrated workspaces. Finally, you triggered and applied a plan in HCP Terraform.

For more information on topics covered in this tutorial, check out the following resources:

• Complete the Connect Workspaces with Run Triggers tutorial to learn how you can leverage run triggers in your HCP Terraform organization
• Complete the Automate HCP Terraform Workflows tutorial to use the TFE provider to manage your HCP Terraform workspaces
• Read more about the HCP Terraform CLI Integration in the documentation
• The tf-helper provides commands to interact with both HCP Terraform and Terraform Enterprise, including setting workspace variables, view runs and much more.