401 lines
15 KiB
Markdown
401 lines
15 KiB
Markdown
---
|
||
slug: /1008/aws-cloudformation/
|
||
---
|
||
|
||
# Provision infrastructure with Dagger and AWS CloudFormation
|
||
|
||
In this guide, you will learn how to automatically [provision infrastructure](https://dzone.com/articles/infrastructure-provisioning-–) on AWS by integrating [Amazon Cloudformation](https://aws.amazon.com/cloudformation/) in your Dagger environment.
|
||
|
||
We will start with something simple: provisioning a new bucket on [Amazon S3](https://en.wikipedia.org/wiki/Amazon_S3). But Cloudformation can provision almost any AWS resource, and Dagger can integrate with the full Cloudformation API.
|
||
|
||
## Prerequisites
|
||
|
||
### Reminder
|
||
|
||
#### Guidelines
|
||
|
||
The provisioning strategy detailed below follows S3 best practices. However, to remain agnostic of your current AWS level, it profoundly relies on S3 and Cloudformation documentation.
|
||
|
||
#### Relays
|
||
|
||
The first thing to consider when developing a plan based on relays is to read their universe reference: it summarizes the expected inputs and their corresponding formats.
|
||
|
||
## Initialize a Dagger Project and Environment
|
||
|
||
### (optional) Setup example app
|
||
|
||
You will need the local copy of the [Dagger examples repository](https://github.com/dagger/examples) used in previous guides
|
||
|
||
```shell
|
||
git clone https://github.com/dagger/examples
|
||
```
|
||
|
||
Make sure to run all commands from the todoapp directory:
|
||
|
||
```shell
|
||
cd examples/todoapp
|
||
```
|
||
|
||
### Organize your package
|
||
|
||
Let's create a new directory for our Cue package:
|
||
|
||
```shell
|
||
mkdir cloudformation
|
||
```
|
||
|
||
## Create a basic plan
|
||
|
||
Let's implement the Cloudformation template and convert it to a Cue definition for further flexibility.
|
||
|
||
### Setup the template and the environment
|
||
|
||
#### Setup the template
|
||
|
||
The idea here is to follow best practices in [S3 buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/HostingWebsiteOnS3Setup.html) provisioning. Thankfully, the AWS documentation contains a working [Cloudformation template](https://docs.aws.amazon.com/fr_fr/AWSCloudFormation/latest/UserGuide/quickref-s3.html#scenario-s3-bucket-website) that fits 95% of our needs.
|
||
|
||
##### 1. Tweaking the template: output bucket name only
|
||
|
||
Create a file named `template.cue` and add the following configuration to it.
|
||
|
||
```cue file=./tests/cloudformation/template.cue title="todoapp/cloudformation/template.cue"
|
||
|
||
```
|
||
|
||
##### 2. Cloudformation relay
|
||
|
||
As our plan relies on Cloudformation's relay, let's dissect the expected inputs by gradually incorporating them into our plan.
|
||
|
||
```shell
|
||
dagger doc alpha.dagger.io/aws/cloudformation
|
||
# Inputs:
|
||
# config.region string AWS region
|
||
# config.accessKey dagger.#Secret AWS access key
|
||
# config.secretKey dagger.#Secret AWS secret key
|
||
# source string Source is the Cloudformation template (JSON/YAML…
|
||
# stackName string Stackname is the cloudformation stack
|
||
# parameters struct Stack parameters
|
||
# onFailure *"DO_NOTHING" | "ROLLBACK" | "DELETE" Behavior when failure to create/update the Stack
|
||
# timeout *10 | >=0 & int Maximum waiting time until stack creation/update…
|
||
# neverUpdate *false | true Never update the stack if already exists
|
||
```
|
||
|
||
###### 1. General insights
|
||
|
||
As seen above in the documentation, values starting with `*` are default values. However, as a plan developer, we may need to add default values to inputs from relays without one: Cue gives you this flexibility.
|
||
|
||
###### 2. The config value
|
||
|
||
The config values are all part of the `aws` relay. Regarding this package, as you can see above, five of the required inputs miss default options (`parameters` field is optional):
|
||
|
||
> - _config.region_
|
||
> - _config.accessKey_
|
||
> - _config.secretKey_
|
||
> - _source_
|
||
> - _stackName_
|
||
|
||
Let's implement the first step, use the `aws.#Config` relay, and request its first inputs: the region to deploy and the AWS credentials.
|
||
|
||
```cue file=./tests/cloudformation/source-begin.cue title="todoapp/cloudformation/source.cue"
|
||
|
||
```
|
||
|
||
This defines:
|
||
|
||
- `awsConfig`: AWS CLI Configuration step using the package `alpha.dagger.io/aws`. It takes three user inputs: a `region`, an `accessKey`, and a `secretKey`
|
||
|
||
#### Setup the environment
|
||
|
||
##### 1. Create a new environment
|
||
|
||
Let's create a project:
|
||
|
||
```shell
|
||
dagger init
|
||
```
|
||
|
||
Let's create an environment to run it:
|
||
|
||
```shell
|
||
dagger new 'cloudformation' -p ./cloudformation
|
||
```
|
||
|
||
##### 2. Check plan
|
||
|
||
_Pro tips_: To check whether it worked or not, these three commands might help
|
||
|
||
```shell
|
||
dagger input list -e cloudformation # List our personal plan's inputs
|
||
# Input Value Set by user Description
|
||
# awsConfig.region string false AWS region
|
||
# awsConfig.accessKey dagger.#Secret false AWS access key
|
||
# awsConfig.secretKey dagger.#Secret false AWS secret key
|
||
|
||
dagger query -e cloudformation # Query values / inspect default values (Instrumental in case of conflict)
|
||
# {}
|
||
|
||
dagger up -e cloudformation # Try to run the plan. As expected, we encounter a failure because some user inputs haven't been set
|
||
# 4:11PM ERR system | required input is missing input=awsConfig.region
|
||
# 4:11PM ERR system | required input is missing input=awsConfig.accessKey
|
||
# 4:11PM ERR system | required input is missing input=awsConfig.secretKey
|
||
# 4:11PM FTL system | some required inputs are not set, please re-run with `--force` if you think it's a mistake missing=0s
|
||
```
|
||
|
||
#### Finish template setup
|
||
|
||
Now that we have the `config` definition properly configured, let's modify the Cloudformation one:
|
||
|
||
```cue file=./tests/cloudformation/source-end.cue title="todoapp/cloudformation/source.cue"
|
||
|
||
```
|
||
|
||
This defines:
|
||
|
||
- `suffix`: random suffix leveraging the `random` relay. It doesn't have a seed because we don't care about predictability
|
||
- `cfnStackName`: Name of the stack, either a default value `stack-suffix` or user input
|
||
- `cfnStack`: Cloudformation relay with `AWS config`, `stackName` and `JSON template` as inputs
|
||
|
||
### Configure the environment
|
||
|
||
Before bringing up the deployment, we need to provide the `cfnStack` inputs declared in the configuration. Otherwise, Dagger will complain about missing inputs.
|
||
|
||
```shell
|
||
dagger up -e cloudformation
|
||
# 3:34PM ERR system | required input is missing input=awsConfig.region
|
||
# 3:34PM ERR system | required input is missing input=awsConfig.accessKey
|
||
# 3:34PM ERR system | required input is missing input=awsConfig.secretKey
|
||
# 3:34PM FTL system | some required inputs are not set, please re-run with `--force` if you think it's a mistake missing=0s
|
||
```
|
||
|
||
You can inspect the list of inputs (both required and optional) using dagger input list:
|
||
|
||
```shell
|
||
dagger input list -e cloudformation
|
||
# Input Value Set by user Description
|
||
# awsConfig.region string false AWS region
|
||
# awsConfig.accessKey dagger.#Secret false AWS access key
|
||
# awsConfig.secretKey dagger.#Secret false AWS secret key
|
||
# suffix.length *12 | number false length of the string
|
||
# cfnStack.onFailure *"DO_NOTHING" | "ROLLBACK" | "DELETE" false Behavior when failure to create/update the Stack
|
||
# cfnStack.timeout *10 | >=0 & int false Maximum waiting time until stack creation/update (in minutes)
|
||
# cfnStack.neverUpdate *false | true false Never update the stack if already exists
|
||
```
|
||
|
||
Let's provide the missing inputs:
|
||
|
||
```shell
|
||
dagger input text awsConfig.region us-east-2 -e cloudformation
|
||
dagger input secret awsConfig.accessKey yourAccessKey -e cloudformation
|
||
dagger input secret awsConfig.secretKey yourSecretKey -e cloudformation
|
||
```
|
||
|
||
### Deploying
|
||
|
||
Finally ! We now have a working template ready to be used to provision S3 infrastructures. Let's deploy it:
|
||
<Tabs
|
||
defaultValue="nd"
|
||
values={[
|
||
{ label: 'Normal deploy', value: 'nd', },
|
||
{ label: 'Debug deploy', value: 'dd', },
|
||
]
|
||
}>
|
||
<TabItem value="nd">
|
||
|
||
```shell
|
||
dagger up -e cloudformation
|
||
#2:22PM INF suffix.out | computing
|
||
#2:22PM INF suffix.out | completed duration=200ms
|
||
#2:22PM INF cfnStack.outputs | computing
|
||
#2:22PM INF cfnStack.outputs | #15 1.304 {
|
||
#2:22PM INF cfnStack.outputs | #15 1.304 "Parameters": []
|
||
#2:22PM INF cfnStack.outputs | #15 1.304 }
|
||
#2:22PM INF cfnStack.outputs | #15 2.948 {
|
||
#2:22PM INF cfnStack.outputs | #15 2.948 "StackId": "arn:aws:cloudformation:us-east-2:817126022176:stack/stack-emktqcfwksng/207d29a0-cd0b-11eb-aafd-0a6bae5481b4"
|
||
#2:22PM INF cfnStack.outputs | #15 2.948 }
|
||
#2:22PM INF cfnStack.outputs | completed duration=35s
|
||
|
||
dagger output list -e cloudformation
|
||
# Output Value Description
|
||
# suffix.out "emktqcfwksng" generated random string
|
||
# cfnStack.outputs.Name "arn:aws:s3:::stack-emktqcfwksng-s3bucket-9eiowjs1jab4" -
|
||
```
|
||
|
||
</TabItem>
|
||
<TabItem value="dd">
|
||
|
||
```shell
|
||
dagger up -l debug -e cloudformation
|
||
#Output:
|
||
# 3:50PM DBG system | detected buildkit version version=v0.8.3
|
||
# 3:50PM DBG system | spawning buildkit job localdirs={
|
||
# "/tmp/infra-provisioning/.dagger/env/infra/plan": "/tmp/infra-provisioning/.dagger/env/infra/plan"
|
||
# } attrs=null
|
||
# 3:50PM DBG system | loading configuration
|
||
# ... Lots of logs ... :-D
|
||
# Output Value Description
|
||
# suffix.out "abnyiemsoqbm" generated random string
|
||
# cfnStack.outputs.Name "arn:aws:s3:::stack-abnyiemsoqbm-s3bucket-9eiowjs1jab4" -
|
||
|
||
dagger output list -e cloudformation
|
||
# Output Value Description
|
||
# suffix.out "abnyiemsoqbm" generated random string
|
||
# cfnStack.outputs.Name "arn:aws:s3:::stack-abnyiemsoqbm-s3bucket-9eiowjs1jab4" -
|
||
```
|
||
|
||
</TabItem>
|
||
</Tabs>
|
||
|
||
The deployment went well!
|
||
|
||
In case of a failure, the `Debug deploy` tab shows the command to get more information.
|
||
The name of the provisioned S3 instance lies in the `cfnStack.outputs.Name` output key, without `arn:aws:s3:::`
|
||
|
||
> With this provisioning infrastructure, your dev team will easily be able to instantiate aws infrastructures: all they need to know is `dagger input list -e cloudformation` and `dagger up -e cloudformation` isn't that awesome? :-D
|
||
|
||
## Cue Cloudformation template
|
||
|
||
This section will convert the inlined JSON template to CUE to take advantage of the language features.
|
||
|
||
To do so quickly, we will first transform the template from JSON format to Cue format, then optimize it to leverage Cue's forces.
|
||
|
||
### 1. Create convert.cue
|
||
|
||
We will create a new `convert.cue` file to process the conversion
|
||
|
||
import Tabs from "@theme/Tabs";
|
||
import TabItem from "@theme/TabItem";
|
||
|
||
```cue file=./tests/cloudformation/template/convert.cue title="todoapp/cloudformation/convert.cue"
|
||
|
||
```
|
||
|
||
This defines:
|
||
|
||
- `s3Template`: contains the unmarshalled template.
|
||
|
||
You need to empty the plan and copy the `convert.cue` file to the plan for Dagger to reference it
|
||
|
||
```shell
|
||
mv cloudformation/source.cue ~/tmp/
|
||
```
|
||
|
||
### 2. Retrieve the Unmarshalled JSON
|
||
|
||
Then, still in the same folder, query the `s3Template` value to retrieve the Unmarshalled result of `s3`:
|
||
|
||
```shell
|
||
dagger query s3Template -e cloudformation
|
||
# {
|
||
# "AWSTemplateFormatVersion": "2010-09-09",
|
||
# "Outputs": {
|
||
# "Name": {
|
||
# "Description": "Name S3 Bucket",
|
||
# "Value": {
|
||
# "Fn::GetAtt": [
|
||
# "S3Bucket",
|
||
# "Arn"
|
||
# ...
|
||
```
|
||
|
||
The commented output above is the cue version of the JSON Template, copy it
|
||
|
||
### 3. Remove convert.cue
|
||
|
||
```shell
|
||
rm cloudformation/convert.cue
|
||
```
|
||
|
||
### 4. Store the output
|
||
|
||
Open `cloudformation/template.cue` and append below elements with copied Cue definition of the JSON:
|
||
|
||
```cue file=./tests/cloudformation/template/template-begin.cue title="todoapp/cloudformation/template.cue"
|
||
|
||
```
|
||
|
||
We're using the built-in `json.Marshal` function to convert CUE back to JSON, so Cloudformation still receives the same template.
|
||
|
||
You can inspect the configuration using `dagger query -e cloudformation` to verify it produces the same manifest:
|
||
|
||
```shell
|
||
dagger query template -f text -e cloudformation
|
||
```
|
||
|
||
Now that the template is defined in CUE, we can use the language to add more flexibility to our template.
|
||
|
||
Let's define a re-usable `#Deployment` definition in `todoapp/cloudformation/deployment.cue`:
|
||
|
||
```cue file=./tests/cloudformation/template/deployment.cue title="todoapp/cloudformation/deployment.cue"
|
||
|
||
```
|
||
|
||
`template.cue` can be rewritten as follows:
|
||
|
||
```cue file=./tests/cloudformation/template/template-end.cue title="todoapp/cloudformation/template.cue"
|
||
|
||
```
|
||
|
||
Verify template
|
||
|
||
Double-checks at the template level can be done with manual uploads on Cloudformation's web interface or by executing the below command locally:
|
||
|
||
```shell
|
||
tmpfile=$(mktemp ./tmp.XXXXXX) && dagger query template -f text -e cloudformation > "$tmpfile" && aws cloudformation validate-template --template-body file://"$tmpfile" ; rm "$tmpfile"
|
||
```
|
||
|
||
Let's make sure it yields the same result:
|
||
|
||
```shell
|
||
dagger query template -f text -e cloudformation
|
||
# {
|
||
# "description": "Name S3 Bucket",
|
||
# "indexDocument": "index.html",
|
||
# "errorDocument": "error.html",
|
||
# "version": "2012-10-17",
|
||
# "deletionPolicy": "Retain",
|
||
# "accessControl": "PublicRead",
|
||
# "template": {
|
||
# "AWSTemplateFormatVersion": "2010-09-09",
|
||
# "Outputs": {
|
||
# "Name": {
|
||
# "Description": "Name S3 Bucket",
|
||
# "Value": {
|
||
```
|
||
|
||
Reimplement `source.cue`:
|
||
|
||
```cue file=./tests/cloudformation/source-end.cue title="todoapp/cloudformation/source.cue"
|
||
|
||
```
|
||
|
||
And we can now deploy it:
|
||
|
||
```shell
|
||
dagger up -e cloudformation
|
||
#2:22PM INF suffix.out | computing
|
||
#2:22PM INF suffix.out | completed duration=200ms
|
||
#2:22PM INF cfnStack.outputs | computing
|
||
#2:22PM INF cfnStack.outputs | #15 1.304 {
|
||
#2:22PM INF cfnStack.outputs | #15 1.304 "Parameters": []
|
||
#2:22PM INF cfnStack.outputs | #15 1.304 }
|
||
#2:22PM INF cfnStack.outputs | #15 2.948 {
|
||
#2:22PM INF cfnStack.outputs | #15 2.948 "StackId": "arn:aws:cloudformation:us-east-2:817126022176:stack/stack-emktqcfwksng/207d29a0-cd0b-11eb-aafd-0a6bae5481b4"
|
||
#2:22PM INF cfnStack.outputs | #15 2.948 }
|
||
#2:22PM INF cfnStack.outputs | completed duration=35s
|
||
```
|
||
|
||
Name of the deployed bucket:
|
||
|
||
```shell
|
||
dagger output list -e cloudformation
|
||
# Output Value Description
|
||
# suffix.out "ucwcecwwshdl" generated random string
|
||
# cfnStack.outputs.Name "arn:aws:s3:::stack-ucwcecwwshdl-s3bucket-gaqmj8rzsl08" -
|
||
```
|
||
|
||
The name of the provisioned S3 instance lies in the `cfnStack.outputs.Name` output key, without `arn:aws:s3:::`
|
||
|
||
PS: This plan could be further extended with the AWS S3 example. It could provide infrastructure and quickly deploy it.
|
||
|
||
PS1: As it could be an excellent first exercise for you, this won't be detailed here. However, we're interested in your imagination: let us know your implementations :-)
|