Manage existing Cloudflare records with Terraform and cf-terraforming

For those who are using Cloudflare DNS to manage the records of your domains and you already have them records created manually and you would like to manage them using Terraform, then this is the guide that will shed some light in that direction. Cloudflare created a tool known as cf-terraforming that generates the records as Terraform files as well as import them into your tfstate files so that they look like they were created by Terraform in the beginning. With those files generated as Terraform files, they will serve as backup in case anything changes and will also serve as documentation of the various records you have in Cloudflare. Another advantage that will accrue is that the records can now be managed via your CI/CD tools and reviewers can be invited before records are created or altered.

As the it has been described by the development team, “the goal with cf-terraforming is to help existing Cloudflare customers get started with Terraform. Currently, cf-terraforming helps to generate the terraform config state by fetching all the resources of a specified type from the account and/or domain of your choosing.”

Step 1: Prepare your Terraform modules

Since you may be having multiple Cloudfare zones, it makes sense to organise them into separate modules so that it becomes easier and clearer to add new records in future as well as provide a good representation of what lies in Cloudflare. We will be consolidating our Cloudflare Terraform files into the following folders

$ cd ~
$ mkdir -p cloudflare/entries/modules
$ cd cloudflare/entries/modules
$ mkdir {neveropen-com,computinggeeks-com}
$ tree -d
├── cloudflare
│   └── entries
│       └── modules
│           ├── neveropen-com
│           ├── computinggeeks-com

In the root directory, that is inside entries, create a ”main.tf” file and populate it as follows

$ vim main.ft
provider "cloudflare" {
  alias = "dns_records"
  email   = var.cloudflare_email
  api_key = var.cloudflare_api_key
}

terraform {
  required_providers {
    cloudflare = {
      source = "cloudflare/cloudflare"
      version = "~> 3.0"
    }
  }
}

#1. geeksforgeeks.org

module "neveropen-com" {
  source                     = "./modules/neveropen-com"
  providers = {
    cloudflare = cloudflare.dns_records
  }
}

#2. computinggeeks.com

module "computinggeeks-com" {
  source                     = "./modules/computinggeeks-com"
  providers = {
    cloudflare = cloudflare.dns_records
  }
}

terraform {
  backend "gcs" {
    bucket      = "geeks-terraform-state-bucket"
    prefix  = "terraform/cloudflare_state"
  }
}

Note that we are using GCP Bucket backend to store our state. Ensure that a bucket with he name given already exists or you can create it if it is not there yet. Otherwise, you can remove that code block
and use your local state if it works for you or you can use s3 or other implementations that suit your use case.

Create a ”vars.tf” file in the same directory and fill ip us as follows:

$ vim vars.tf

variable "cloudflare_api_key" {
  type        = string
  sensitive   = true
  default     = "e5de3ef7282fde61ed8cfeca65788db4d31aa"
}

variable "cloudflare_email" {
  type        = string
  default     = "[email protected]"
}

The variables declared in vars.tf are the credentials that terraform will use to connect to Cloudflare Api and perform its magics. You can follow Step 3 below on how to obtain the Cloudflare Api Key.

Once that is done, in each root module, create a ”providers.tf” file that have this configuration

$ vim ~/cloudflare/entries/modules/neveropen-com/provider.tf 
terraform {
  required_providers {
    cloudflare = {
      source = "cloudflare/cloudflare"
      version = "~> 3.0"
    }
  }
}

#Do the same for the other modules
$ vim ~/cloudflare/entries/modules/computinggeeks-com/provider.tf 
terraform {
  required_providers {
    cloudflare = {
      source = "cloudflare/cloudflare"
      version = "~> 3.0"
    }
  }
}

You can then copy this same file to the rest of the modules. Variables specific to the module can be created in files at the same location where ”provider.tf” file will reside.

Once this is done, you can now initialize terraform so that it can fetch requisite binaries for the various providers declared.

$ terraform init

Step 2: Install cf-terraforming

With that succinct introduction, lets us now begin by first setting up cf-terraforming then we shall proceed to harness the powers that it provides for us to explore generating DNS Resources already configured in Cloudflare

If you use Homebrew on MacOS, you can run the following:

$ brew tap cloudflare/cloudflare
$ brew install --cask cloudflare/cloudflare/cf-terraforming

For Linux users, installing cf-terraforming is pretty easy, install wget and then proceed as follows. The version of cf-terraforming at the time of writing was v 0.6.2. You can check its GitHub Releases Page for the most recent one.

cd ~
CFTERRAFORM_VERSION=$(curl -s https://api.github.com/repos/cloudflare/cf-terraforming/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')

wget https://github.com/cloudflare/cf-terraforming/releases/download/v${CFTERRAFORM_VERSION}/cf-terraforming_${CFTERRAFORM_VERSION}_linux_amd64.tar.gz

tar -xvzf cf-terraforming_${CFTERRAFORM_VERSION}_linux_amd64.tar.gz
sudo cp cf-terraforming /usr/local/bin 
sudo chmod a+x /usr/local/bin/cf-terraforming

Confirm installation by checking its version

$ cf-terraforming version
cf-terraforming v0.6.2

Step 3: Generate Cloudflare DNS Records as Terraform Files

Now our tool is installed and ready for action. First, we shall endeavour to generate all of the existing DNS entries as Terraform files. Cf-terraforming is pretty straightforward about this as you will see shortly. But before we can generate the records, we need a few things including: Cloudflare Api Key, Email, and the Zone ID of your particular zone/domain. If you have several zones, then you will need the zone id’s of all them because they are all unique. After you have these details in your hands, we recommend keeping them as environment variables in case the environment you are working in demands some security.

To have our Cloudflare Api Key, login to Cloudflare then proceed as shown in the screenshots below

To retrieve your API key:

1. Log in to the Cloudflare dashboard.
2. Under the My Profile dropdown, click My Profile.
3. Click the API tokens tab.
4. In the API keys section, choose one of two options: Global API Key or Origin CA Key. Choose the API Key that you would like to view.

Login

Click your Profile

Click API Tokens Tab on the left pane then click new as shown below.

To have our Zone ID, while still logged in, click on each of the zones/domains in your account

And then check the right pane for the details of the particular zone/domain. You will see your Zone ID as illustrated below.

Once we have these credentials, our toolbox is finally ready and we should get to work. To generate the records, get cf-terraforming out of the box and prepare it as follows.

First, conceal the credentials by generating their equivalent environment variables

export CLOUDFLARE_EMAIL = <your email>
export CLOUDFLARE_API_KEY = <your Api Key>
export CLOUDFLARE_ZONE_ID = <your zone id>

Then let us run the cf-terraforming command to generate the records for the particular Zone

$ cf-terraforming generate \
  --resource-type "cloudflare_record" \
  --email $CLOUDFLARE_EMAIL \
  --key $CLOUDFLARE_API_KEY \
  --zone $CLOUDFLARE_ZONE_ID

It should be noted that cf-terraforming outputs to the STDOUT. What we did is that we redirected the output to a file that you can consequently store in one of your modules if you like. For example, to store geeksforgeeks.org zone records, we redirect it to its module directory as follows. Repeat this for the other modules/zones that you have.

$ cf-terraforming generate \
  --resource-type "cloudflare_record" \
  --email $CLOUDFLARE_EMAIL \
  --key $CLOUDFLARE_API_KEY \
  --zone $CLOUDFLARE_ZONE_ID >> ~/cloudflare/entries/modules/neveropen-com/cloudflare_records.tf

That should populate the cloudflare_records.tf file with your Cloudflare DNS records as terraform resources. Example of output include

$ more cloudflare_records.tf
resource "cloudflare_record" "terraform_managed_resource_5cf9db2eb05350af09044a2e5e386265” {
  name    = “testing"
  proxied = true
  ttl     = 1
  type    = "A"
  value   = "192.168.20.21”
  zone_id = "ec14361bc18033b45e8683425e826b4862”
}

In our setup, we arranged all zones as modules. So each zone’s output was re-directed to a file inside the modules. Something like below

.
├── README.md
├── bitbucket-pipelines.yml
└── entries
    ├── main.tf
    ├── modules
    │   ├── neveropen-com
    │   │   ├── cloudflare_records.tf
    │   │   ├── provider.tf
    │   │   └── vars.tf
    │   ├── computinggeeks-com
    │   │   ├── cloudflare_records.tf
    │   │   ├── provider.tf
    │   │   └── vars.tf

Step 4: Importing the Records into Terraform State

This step is pretty manual because cf-terraforming currently at the time of writing this has no way of importing the generated terraform resources into terraform state automatically. This is where the bulk of work will happen in case you have a lot of zones and multiple records inside each zone. This command assumes you have already ran cf-terraforming generate to output your resources as done in the preceding step.

We are going to use “terraform import” command to import all of our resources into tfstate step by step. This is because we re-arranged our resources into modules and cf-terraforming will have challenges importing them. At least we now have our resources as terraform files.

Now, Cloudflare expects the resources to be in this format

terraform import module.moduleY.cloudflare_record.terraform_managed_resource ZoneID/ResourceID

So we should know the resource ids of each resource. Here, your creativity will come to your rescue. You can query Cloudflare’s Api to get the data generated as json and then you can extract the resource id’s from there.

curl -X GET "https://api.cloudflare.com/client/v4/zones/$CLOUDFLARE_ZONE_ID/dns_records?page=1&per_page=3&order=type&direction=asc" \
     -H "Content-Type:application/json" \
     -H "X-Auth-Key:$CLOUDFLARE_API_KEY" \
     -H "X-Auth-Email:$CLOUDFLARE_EMAIL"

Once you have the resource id’s for each resource in one zone, we can create a simple for loop bash script that will import each resource on that zone one by one. You can do a better job than this, so please explore your options here.

$ vim import_resources.py
#!/bin/bash
zoneid = ec14361bc18033b45e8683425e826b4862
echo
for recordid in 3e1cc426bae6733fe7a21a7986a952d9 \
4e1cc426bae6733fe7a21a7986a952d9 \
5e1cc426bae6733fe7a21a7986a952d9 \
6e1cc426bae6733fe7a21a7986a952d9 \
7077466543e1508f9ba38a7986a952d9 

do

  echo "We are importing $recordid"
  terraform import module.neveropen-tech.cloudflare_record.terraform_managed_resource_$recordid $zoneid/$recordid

done

So each record id in this one zone will be imported one by one. You should do this for all of the zones you have. If you can write one script that covers all of the zones then well and good. I ended up creating different scripts for each zone. And please let us know how you did it in the comments below to help all of us.

Once this crucial Step is successfully done, we should be ready to start adding new resources using Terraform by adding them on the particular zone’s module files. We can also leverage CI/CD tools to do the work for us after we commit our code to any of the Git platforms of your choice.
The next guide will cover using BitBucket pipelines CI/CD to update our Cloudflare DNS entries.

Step 5: Running Terraform with protected Cloudflare credentials

The way we have arranged our terraform files will give us the opportunity of hiding the cloudflare_api_key and email inside environment variables. We should be able to successfully do the following

$ export TF_VAR_cloudflare_api_key="our_cloudflare_api_key"
$ export TF_VAR_cloudflare_email="the_email"

##Then Run the following
$ terraform plan -var cloudflare_email=$TF_VAR_cloudflare_email -var cloudflare_api_key=$TF_VAR_cloudflare_api_key

You should now see the following at the end of the output meaning that all records are now in terraform state and behave as though they were created by Terraform from the beginning.

Plan: 0 to add, 0 to change, 0 to destroy

End of the Ballad

The terraform files we managed to generate will serve as documentation, provide means to automate our DNS entries as well as enforce checks and alerts in case DNS records change. We can now focus on code while Terraform does the rest of work. We hope the playbook was informative and improvements on the same are highly invited.

Other related guides

• How To Install CloudFlare CFSSL on Linux | macOS
• Build Private PKI/TLS CA for Certificates Management With CloudFlare CFSSL