From 4221011064abf020999b2465c870cd12d0038993 Mon Sep 17 00:00:00 2001 From: Andrea Luzzardi Date: Thu, 3 Jun 2021 18:18:32 -0700 Subject: [PATCH 1/5] docs: kubernetes tutorial Signed-off-by: Andrea Luzzardi --- docs/tutorials/_category_.json | 4 + docs/tutorials/kubernetes.md | 423 +++++++++++++++++++++++++++++++++ 2 files changed, 427 insertions(+) create mode 100644 docs/tutorials/_category_.json create mode 100644 docs/tutorials/kubernetes.md diff --git a/docs/tutorials/_category_.json b/docs/tutorials/_category_.json new file mode 100644 index 00000000..ae26fa51 --- /dev/null +++ b/docs/tutorials/_category_.json @@ -0,0 +1,4 @@ +{ + "label": "Tutorials", + "position": 5 +} diff --git a/docs/tutorials/kubernetes.md b/docs/tutorials/kubernetes.md new file mode 100644 index 00000000..e71c2238 --- /dev/null +++ b/docs/tutorials/kubernetes.md @@ -0,0 +1,423 @@ +# Kubernetes + +This example illustrates how to use `dagger` to build, push and deploy Docker +images to Kubernetes. + +## Prerequisites + +### Setup a local Kubernetes cluster + +While dagger supports GKE and EKS, for the purpose of this example, we'll be +using [kind](https://kind.sigs.k8s.io/) to install a local Kubernetes cluster +in addition to a local container registry, no cloud account required. + +1\. Install kind + +Follow [these instructions](https://kind.sigs.k8s.io/docs/user/quick-start) to +install kind. + +Alternatively, on macOS using [homebrew](https://brew.sh/): + +```shell +brew install kind +``` + +2\. Start a local registry + +```shell +docker run -d -p 5000:5000 --name registry registry:2 +``` + +3\. Create a cluster with the local registry enabled in containerd + +```bash +cat < + +Create a file named `.dagger/env/default/plan/manifest.cue` and add the +following configuration to it. + +```cue title=".dagger/env/default/plan/manifest.cue" +package main + +// inlined kubernetes manifest as a string +manifest: """ + apiVersion: apps/v1 + kind: Deployment + metadata: + name: nginx + labels: + app: nginx + spec: + replicas: 1 + selector: + matchLabels: + app: nginx + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:1.14.2 + ports: + - containerPort: 80 + """ +``` + +This will define a `manifest` variable containing the inlined Kubernetes YAML +used to create a _nginx_ deployment. + +Next, create `.dagger/env/default/plan/main.cue`. + +```cue title=".dagger/env/default/plan/main.cue" +package main + +import ( + "dagger.io/kubernetes" +) + +// input: ~/.kube/config file used for deployment +// set with `dagger input text kubeconfig -f ~/.kube/config` +kubeconfig: string @dagger(input) + +// deploy uses the `dagger.io/kubernetes` package to apply a manifest to a +// Kubernetes cluster. +deploy: kubernetes.#Apply & { + // reference the `kubeconfig` input above + "kubeconfig": kubeconfig + + // reference to the manifest defined in `manifest.cue` + "manifest": manifest +} +``` + +This defines: + +- `kubeconfig` a _string_ **input**: kubernetes configuration (`~/.kube/config`) + used for `kubectl` +- `deploy`: Deployment step using the package `dagger.io/kubernetes`. It takes + the `manifest` defined earlier and deploys it to the Kubernetes cluster specified in `kubeconfig`. + +### Configure the environment + +Before we can bring up the deployment, we need to provide the `kubeconfig` input +declared in the configuration. Otherwise, dagger will complain about a missing input: + +```shell +$ dagger up +6:53PM ERR system | required input is missing input=kubeconfig +``` + +You can inspect the list of inputs (both required and optional) using `dagger input list`: + +```shell +$ dagger input list +Input Type Description +kubeconfig string ~/.kube/config file used for deployment +deploy.namespace string Kubernetes Namespace to deploy to +``` + +Let's provide the missing input: + +```shell +# we'll use the ~/.kube/config created by `kind` +dagger input text kubeconfig -f ~/.kube/config +``` + +### Deploying + +Now is time to deploy to kubernetes. + +```shell +$ dagger up +deploy | computing +deploy | #26 0.700 deployment.apps/nginx created +deploy | completed duration=900ms +``` + +Let's verify the deployment worked: + +```shell +$ kubectl get deployments +NAME READY UP-TO-DATE AVAILABLE AGE +nginx 1/1 1 1 1m +``` + +## CUE Kubernetes manifests + +In this section we will convert the inlined YAML manifest to CUE to take advantage of the language features. + +For a more advanced example, see the +[official CUE Kubernetes tutorial](https://github.com/cuelang/cue/blob/v0.4.0/doc/tutorial/kubernetes/README.md) + +First, let's replace `manifest.cue` with the following configuration. This is a +straightforward one-to-one conversion from YAML to CUE, only the syntax has changed. + +```cue title=".dagger/env/default/plan/manifest.cue" +package main + +import ( + "encoding/yaml" +) + +nginx: { + apiVersion: "apps/v1" + kind: "Deployment" + metadata: { + "name": "nginx" + labels: app: "nginx" + } + spec: { + replicas: 1 + selector: matchLabels: app: "nginx" + template: { + metadata: labels: app: "nginx" + spec: containers: [{ + "name": "nginx" + "image": image + ports: [{ + containerPort: port + }] + }] + } + } +} + +manifest: yaml.Marshal(nginx) +``` + +We're using the built-in `yaml.Marshal` function to convert CUE back to YAML so +Kubernetes still receives the same manifest. + +You can inspect the configuration using `dagger query` to verify it produces the +same manifest: + +```shell +$ dagger query manifest -f text +apiVersion: apps/v1 +kind: Deployment +... +``` + +Now that the manifest is defined in CUE, we can take advantage of the language +to remove a lot of boilerplate and repetition. + +Let's define a re-usable `#Deployment` definition in `.dagger/env/default/plan/deployment.cue"`: + +```cue title=".dagger/env/default/plan/deployment.cue" +package main + +// Deployment template containing all the common boilerplate shared by +// deployments of this application. +#Deployment: { + // name of the deployment. This will be used to automatically label resouces + // and generate selectors. + name: string + + // container image + image: string + + // 80 is the default port + port: *80 | int + + // 1 is the default, but we allow any number + replicas: *1 | int + + // Deployment manifest. Uses the name, image, port and replicas above to + // generate the resource manifest. + manifest: { + apiVersion: "apps/v1" + kind: "Deployment" + metadata: { + "name": name + labels: app: name + } + spec: { + "replicas": replicas + selector: matchLabels: app: name + template: { + metadata: labels: app: name + spec: containers: [{ + "name": name + "image": image + ports: [{ + containerPort: port + }] + }] + } + } + } +} +``` + +`manifest.cue` can be rewritten as follows: + +```cue title=".dagger/env/default/plan/manifest.cue" +import ( + "encoding/yaml" +) + +nginx: #Deployment & { + name: "nginx" + image: "nginx:1.14.2" +} + +manifest: yaml.Marshal(nginx.manifest) +``` + +Let's make sure it yields the same result: + +```shell +$ dagger query deploy.manifest -f text +apiVersion: apps/v1 +kind: Deployment +... +``` + +And we can now deploy it: + +```shell +$ dagger up +deploy | computing +deploy | #26 0.700 deployment.apps/nginx unchanged +deploy | completed duration=900ms +``` + +## Building, pushing and deploying Docker images + +Rather than deploying an existing (`nginx`) image, we're going to build a Docker +image from source, push it to a registry and update the kubernetes configuration. + +### Update the plan + +The following configuration will: + +- Declare a `repository` input as a `dagger.#Artifact`. This will be mapped to + the source code directory. +- Declare a `registry` input. This is the address used for docker push +- Use `dagger.io/docker` to build and push the image +- Use the registry image reference (`push.ref`) as the image for the deployment. + +```cue title=".dagger/env/default/plan/manifest.cue" +package main + +import ( + "encoding/yaml" + + "dagger.io/docker" +) + +// input: source code repository, must contain a Dockerfile +// set with `dagger input dir repository ./app` +repository: dagger.#Artifact @dagger(input) + +// registry to push images to +registry: string @dagger(input) + +// docker build the `repository` directory +image: docker.#Build & { + source: repository +} + +// push the `image` to the `registry` +push: docker.#Push & { + source: image + ref: registry +} + +// use the `#Deployment` template to generate the kubernetes manifest +app: #Deployment & { + name: "test" + + // use the reference of the image we just pushed + // this creates a dependency: `app` will only be deployed after the image is + // built and pushed. + "image": push.ref +} + +manifest: yaml.Marshal(app.manifest) +``` + +### Connect the Inputs + +Next, we'll provide the two new inputs, `repository` and `registry`. + +For the purpose of this tutorial we'll be using +[hello-go](https://github.com/aluzzardi/hello-go) as example source code. + +```shell +$ git clone https://github.com/aluzzardi/hello-go.git +dagger input dir repository ./hello-go +dagger input text registry "localhost:5000/image" +``` + +### Bring up the changes + +```shell +$ dagger up +repository | computing +repository | completed duration=0s +image | computing +image | completed duration=1s +deploy | computing +deploy | #26 0.709 deployment.apps/hello created +deploy | completed duration=900ms +``` + +Let's verify the deployment worked: + +```shell +$ kubectl get deployments +NAME READY UP-TO-DATE AVAILABLE AGE +nginx 1/1 1 1 1m +hello 1/1 1 1 1m +``` + +## Next Steps + +Deploy on a hosted Kubernetes cluster: + +- [GKE](https://github.com/dagger/dagger/tree/main/stdlib/gcp/gke) +- [EKS](https://github.com/dagger/dagger/tree/main/stdlib/aws/eks) + +Authenticate to a remote registry: + +- [ECR](https://github.com/dagger/dagger/tree/main/stdlib/aws/ecr) +- [GCR](https://github.com/dagger/dagger/tree/main/stdlib/gcp/gcr) + +Integrate kubernetes tools with Dagger: + +- [Helm](https://github.com/dagger/dagger/tree/main/stdlib/kubernetes/helm) +- [Kustomize](https://github.com/dagger/dagger/tree/main/stdlib/kubernetes/kustomize) From 4554a216f1473b584742d1cb490d923592bd902a Mon Sep 17 00:00:00 2001 From: Andrea Luzzardi Date: Thu, 3 Jun 2021 20:34:18 -0700 Subject: [PATCH 2/5] docs: kubernetes: support kind/gke/eks Signed-off-by: Andrea Luzzardi --- .markdownlint.yaml | 3 + docs/tutorials/kubernetes.md | 219 ++++++++++++++++++++++++++++++++--- 2 files changed, 208 insertions(+), 14 deletions(-) diff --git a/.markdownlint.yaml b/.markdownlint.yaml index bedc3c97..5323121b 100644 --- a/.markdownlint.yaml +++ b/.markdownlint.yaml @@ -3,3 +3,6 @@ default: true # MD013/line-length - Line length MD013: false + +# MD033 - Inline HTML. Needed for tabs in docusaurus +MD033: false diff --git a/docs/tutorials/kubernetes.md b/docs/tutorials/kubernetes.md index e71c2238..2c0e83e6 100644 --- a/docs/tutorials/kubernetes.md +++ b/docs/tutorials/kubernetes.md @@ -1,15 +1,27 @@ # Kubernetes -This example illustrates how to use `dagger` to build, push and deploy Docker +This tutorial illustrates how to use `dagger` to build, push and deploy Docker images to Kubernetes. +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + ## Prerequisites -### Setup a local Kubernetes cluster +For this tutorial, you will need a Kubernetes cluster. -While dagger supports GKE and EKS, for the purpose of this example, we'll be -using [kind](https://kind.sigs.k8s.io/) to install a local Kubernetes cluster -in addition to a local container registry, no cloud account required. + + + + +[Kind](https://kind.sigs.k8s.io/docs/user/quick-start) is a tool for running local Kubernetes clusters using Docker. 1\. Install kind @@ -47,6 +59,23 @@ EOF docker network connect kind registry ``` + + + + +This tutorial can be run against a [GCP GKE](https://cloud.google.com/kubernetes-engine) cluster and [GCR](https://cloud.google.com/container-registry +) + + + + + +This tutorial can be run against a [AWS EKS](https://aws.amazon.com/eks/ +) cluster and [ECR](https://aws.amazon.com/ecr/) + + + + ## Initialize a Dagger Workspace and Environment ```shell @@ -57,13 +86,6 @@ dagger new default ## Create a basic plan - - Create a file named `.dagger/env/default/plan/manifest.cue` and add the following configuration to it. @@ -101,6 +123,17 @@ used to create a _nginx_ deployment. Next, create `.dagger/env/default/plan/main.cue`. + + + + ```cue title=".dagger/env/default/plan/main.cue" package main @@ -114,7 +147,7 @@ kubeconfig: string @dagger(input) // deploy uses the `dagger.io/kubernetes` package to apply a manifest to a // Kubernetes cluster. -deploy: kubernetes.#Apply & { +deploy: kubernetes.#Resources & { // reference the `kubeconfig` input above "kubeconfig": kubeconfig @@ -123,6 +156,65 @@ deploy: kubernetes.#Apply & { } ``` + + + + +```cue title=".dagger/env/default/plan/main.cue" +package main + +import ( + "dagger.io/kubernetes" + "dagger.io/gcp/gke" +) + +// gkeConfig used for deployment +gkeConfig: gke.#KubeConfig @dagger(input) + +kubeconfig: gkeConfig.kubeconfig + +// deploy uses the `dagger.io/kubernetes` package to apply a manifest to a +// Kubernetes cluster. +deploy: kubernetes.#Resources & { + // reference the `kubeconfig` input above + "kubeconfig": kubeconfig + + // reference to the manifest defined in `manifest.cue` + "manifest": manifest +} +``` + + + + + +```cue title=".dagger/env/default/plan/main.cue" +package main + +import ( + "dagger.io/kubernetes" + "dagger.io/aws/eks" +) + +// eksConfig used for deployment +eksConfig: eks.#KubeConfig @dagger(input) + +kubeconfig: eksConfig.kubeconfig + +// deploy uses the `dagger.io/kubernetes` package to apply a manifest to a +// Kubernetes cluster. +deploy: kubernetes.#Resources & { + // reference the `kubeconfig` input above + "kubeconfig": kubeconfig + + // reference to the manifest defined in `manifest.cue` + "manifest": manifest +} +``` + + + + This defines: - `kubeconfig` a _string_ **input**: kubernetes configuration (`~/.kube/config`) @@ -142,6 +234,38 @@ $ dagger up You can inspect the list of inputs (both required and optional) using `dagger input list`: + + + + + + ```shell $ dagger input list Input Type Description @@ -149,13 +273,80 @@ kubeconfig string ~/.kube/config file used for deployment deploy.namespace string Kubernetes Namespace to deploy to ``` -Let's provide the missing input: + + + + +```shell +$ dagger input list +Input Type Description +deploy.namespace string Kubernetes Namespace to deploy to +gkeConfig.config.region string GCP region +gkeConfig.config.project string GCP project +gkeConfig.config.serviceKey dagger.#Secret GCP service key +gkeConfig.clusterName string GKE cluster name +``` + + + + + +```shell +$ dagger input list +Input Type Description +deploy.namespace string Kubernetes Namespace to deploy to +eksConfig.config.region string AWS region +eksConfig.config.accessKey dagger.#Secret AWS access key +eksConfig.config.secretKey dagger.#Secret AWS secret key +eksConfig.clusterName string EKS cluster name +``` + + + + +Let's provide the missing inputs: + + + + ```shell # we'll use the ~/.kube/config created by `kind` dagger input text kubeconfig -f ~/.kube/config ``` + + + + +```shell +dagger input text gkeConfig.config.project +dagger input text gkeConfig.config.region +dagger input text gkeConfig.clusterName +dagger input secret gkeConfig.config.serviceKey -f +``` + + + + + +```shell +dagger input text eksConfig.config.region +dagger input text eksConfig.clusterName +dagger input secret eksConfig.config.accessKey +dagger input secret eksConfig.config.secretKey +``` + + + + ### Deploying Now is time to deploy to kubernetes. From b231c3cb1b9b3ff13b25686b929d150b057898b7 Mon Sep 17 00:00:00 2001 From: Andrea Luzzardi Date: Wed, 9 Jun 2021 16:57:35 -0700 Subject: [PATCH 3/5] docs: kubernetes: move into programming guides Signed-off-by: Andrea Luzzardi --- docs/{tutorials => programming/guides}/kubernetes.md | 0 docs/tutorials/_category_.json | 4 ---- 2 files changed, 4 deletions(-) rename docs/{tutorials => programming/guides}/kubernetes.md (100%) delete mode 100644 docs/tutorials/_category_.json diff --git a/docs/tutorials/kubernetes.md b/docs/programming/guides/kubernetes.md similarity index 100% rename from docs/tutorials/kubernetes.md rename to docs/programming/guides/kubernetes.md diff --git a/docs/tutorials/_category_.json b/docs/tutorials/_category_.json deleted file mode 100644 index ae26fa51..00000000 --- a/docs/tutorials/_category_.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "label": "Tutorials", - "position": 5 -} From 2a36771be035449693f4258521f0d37e6bcdfd12 Mon Sep 17 00:00:00 2001 From: Sam Alba Date: Mon, 14 Jun 2021 18:29:10 +0200 Subject: [PATCH 4/5] docs: updated code in kubernetes guide Signed-off-by: Sam Alba --- docs/programming/guides/kubernetes.md | 70 +++++++++++++-------------- 1 file changed, 35 insertions(+), 35 deletions(-) diff --git a/docs/programming/guides/kubernetes.md b/docs/programming/guides/kubernetes.md index 2c0e83e6..04b3463f 100644 --- a/docs/programming/guides/kubernetes.md +++ b/docs/programming/guides/kubernetes.md @@ -1,6 +1,6 @@ # Kubernetes -This tutorial illustrates how to use `dagger` to build, push and deploy Docker +This tutorial illustrates how to use dagger to build, push and deploy Docker images to Kubernetes. import Tabs from '@theme/Tabs'; @@ -11,13 +11,13 @@ import TabItem from '@theme/TabItem'; For this tutorial, you will need a Kubernetes cluster. +defaultValue="kind" +groupId="provider" +values={[ +{label: 'kind', value: 'kind'}, +{label: 'GKE', value: 'gke'}, +{label: 'EKS', value: 'eks'}, +]}> @@ -63,15 +63,13 @@ docker network connect kind registry -This tutorial can be run against a [GCP GKE](https://cloud.google.com/kubernetes-engine) cluster and [GCR](https://cloud.google.com/container-registry -) +This tutorial can be run against a [GCP GKE](https://cloud.google.com/kubernetes-engine) cluster and [GCR](https://cloud.google.com/container-registry) -This tutorial can be run against a [AWS EKS](https://aws.amazon.com/eks/ -) cluster and [ECR](https://aws.amazon.com/ecr/) +This tutorial can be run against a [AWS EKS](https://aws.amazon.com/eks/) cluster and [ECR](https://aws.amazon.com/ecr/) @@ -124,13 +122,13 @@ used to create a _nginx_ deployment. Next, create `.dagger/env/default/plan/main.cue`. +defaultValue="kind" +groupId="provider" +values={[ +{label: 'kind', value: 'kind'}, +{label: 'GKE', value: 'gke'}, +{label: 'EKS', value: 'eks'}, +]}> @@ -138,12 +136,13 @@ Next, create `.dagger/env/default/plan/main.cue`. package main import ( + "dagger.io/dagger" "dagger.io/kubernetes" ) // input: ~/.kube/config file used for deployment -// set with `dagger input text kubeconfig -f ~/.kube/config` -kubeconfig: string @dagger(input) +// set with `dagger input secret kubeconfig -f ~/.kube/config` +kubeconfig: dagger.#Secret @dagger(input) // deploy uses the `dagger.io/kubernetes` package to apply a manifest to a // Kubernetes cluster. @@ -256,13 +255,13 @@ You can inspect the list of inputs (both required and optional) using `dagger in --> +defaultValue="kind" +groupId="provider" +values={[ +{label: 'kind', value: 'kind'}, +{label: 'GKE', value: 'gke'}, +{label: 'EKS', value: 'eks'}, +]}> @@ -307,13 +306,13 @@ eksConfig.clusterName string EKS cluster name Let's provide the missing inputs: +defaultValue="kind" +groupId="provider" +values={[ +{label: 'kind', value: 'kind'}, +{label: 'GKE', value: 'gke'}, +{label: 'EKS', value: 'eks'}, +]}> @@ -527,6 +526,7 @@ package main import ( "encoding/yaml" + "dagger.io/dagger" "dagger.io/docker" ) From 4882c406769d661ea8d40dfe70491c2b7af29419 Mon Sep 17 00:00:00 2001 From: Sam Alba Date: Mon, 14 Jun 2021 18:31:37 +0200 Subject: [PATCH 5/5] docs: refreshed stdlib Signed-off-by: Sam Alba --- docs/reference/universe/aws/cloudformation.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/reference/universe/aws/cloudformation.md b/docs/reference/universe/aws/cloudformation.md index 79f80d3c..c448310e 100644 --- a/docs/reference/universe/aws/cloudformation.md +++ b/docs/reference/universe/aws/cloudformation.md @@ -10,16 +10,16 @@ AWS CloudFormation Stack ### #Stack Inputs -| Name | Type | Description | -| ------------- |:-------------: |:-------------: | -|*config.region* | `string` |AWS region | -|*config.accessKey* | `dagger.#Secret` |AWS access key | -|*config.secretKey* | `dagger.#Secret` |AWS secret key | -|*source* | `string` |Source is the Cloudformation template (JSON/YAML string) | -|*stackName* | `string` |Stackname is the cloudformation stack | -|*onFailure* | `*"DO_NOTHING" \| "ROLLBACK" \| "DELETE"` |Behavior when failure to create/update the Stack | -|*timeout* | `*10 \| \>=0 & int` |Timeout for waiting for the stack to be created/updated (in minutes) | -|*neverUpdate* | `*false \| bool` |Never update the stack if already exists | +| Name | Type | Description | +| ------------- |:-------------: |:-------------: | +|*config.region* | `string` |AWS region | +|*config.accessKey* | `dagger.#Secret` |AWS access key | +|*config.secretKey* | `dagger.#Secret` |AWS secret key | +|*source* | `string` |Source is the Cloudformation template (JSON/YAML string) | +|*stackName* | `string` |Stackname is the cloudformation stack | +|*onFailure* | `*"DO_NOTHING" \| "ROLLBACK" \| "DELETE"` |Behavior when failure to create/update the Stack | +|*timeout* | `*10 \| \>=0 & int` |Maximum waiting time until stack creation/update (in minutes) | +|*neverUpdate* | `*false \| bool` |Never update the stack if already exists | ### #Stack Outputs