From 5fe04d22e987331be859a50dcdd289fa326433b0 Mon Sep 17 00:00:00 2001 From: Gerhard Lazu Date: Fri, 25 Mar 2022 14:25:01 +0000 Subject: [PATCH 1/5] Revise Europa Docs - Core Concepts - It all starts with a plan Thank you Tanguy for the "fresh pair of eyes" perspective. Thank you Tom for the link suggestions. Supersedes https://github.com/dagger/dagger/pull/1847 Signed-off-by: Gerhard Lazu --- docs/core-concepts/1202-plan.md | 112 ++++++++++++------ .../core-concepts/plan/structure.cue.fragment | 8 ++ .../todoapp/{netlify.cue => todoapp.cue} | 10 +- 3 files changed, 87 insertions(+), 43 deletions(-) rename pkg/universe.dagger.io/examples/todoapp/{netlify.cue => todoapp.cue} (90%) diff --git a/docs/core-concepts/1202-plan.md b/docs/core-concepts/1202-plan.md index 47831c5d..a660060d 100644 --- a/docs/core-concepts/1202-plan.md +++ b/docs/core-concepts/1202-plan.md @@ -5,15 +5,19 @@ displayed_sidebar: europa # It all starts with a plan -A CI/CD pipeline declared in Dagger starts with a plan, specifically `dagger.#Plan` +## Plan structure -This plan is the entrypoint for everything that runs within a pipeline. The simplest plan will usually: +A config declared in Dagger starts with a plan, specifically `dagger.#Plan` -- interact with the client filesystem to read (e.g. source code) or write files (e.g. build output) -- read environment variables -- declare a few actions, e.g. deps, test & build +Within this plan we can: -This is our **Getting Started** example app plan structure: +- interact with the `client` filesystem + - read files, usually the current directory as `.` + - write files, usually the build output as `_build` +- read `env` variables, such as `NETLIFY_TEAM` in our example +- declare a few `actions`, e.g. `deps`, `test` & `build` + +This is our **Getting Started** todoapp plan structure: ```cue file=../tests/core-concepts/plan/structure.cue.fragment ``` @@ -21,53 +25,87 @@ This is our **Getting Started** example app plan structure: When the above plan gets executed via `dagger do build`, it produces the following output: ```shell -[✔] client.filesystem.".".read 0.0s -[✔] actions.deps 1.1s -[✔] actions.test.script 0.0s -[✔] actions.test 0.0s -[✔] actions.build.run.script 0.0s -[✔] actions.build.run 0.0s -[✔] actions.build.contents 0.0s -[✔] client.filesystem.build.write 0.1s +[✔] client.filesystem.".".read 0.0s +[✔] actions.deps 1.1s +[✔] actions.test.script 0.0s +[✔] actions.test 0.0s +[✔] actions.build.run.script 0.0s +[✔] actions.build.run 0.0s +[✔] actions.build.contents 0.0s +[✔] client.filesystem."./_build".write 0.1s ``` -Since these actions have run before, they are cached and take less than 1 second to complete. +Since these actions have run before, they are cached and take less than 2 seconds to complete. -While the names used for the actions above - `deps`, `test` & `build` - are short & descriptive, +While the names used for the actions above - `deps`, `test` & `build` - are short and descriptive, any other names would have worked. Put differently, action naming does not affect plan execution. -In the example above, the `deps` action is an instance of the docker package build definition. +Lastly, notice that even if the `deploy` action is defined, we did not run it. +Similar to Makefile targets, we have the option of running specific actions. -This is written as `deps: docker.#Build` +We ran the `dagger do build` command, which only runs the `build` action (and all its dependent actions). +This Dagger property enables us to keep the entire CI/CD config in a single file, while keeping the integration execution separate from the deployment one. +Separating CI & CD concerns becomes essential as our pipelines grow in complexity, and we learn about operational and security constraints specific to our systems. -Default definition configuration - `docker.#Build` in this case - can be modified via curly brackets, e.g. +## Packages & imports + +In order to understand the correlation between actions, definitions and packages, let us focus on the following fragment from our **Getting Started** todoapp config: ```cue - deps: docker.#Build & { +package todoapp + +import ( + "dagger.io/dagger" + "universe.dagger.io/netlify" +) + +dagger.#Plan & { + // ... + actions: { + // ... + deploy: netlify.#Deploy & { // ... } + // ... + } +} +``` + +We start by declaring the package name, `package todoapp` above. + +Next, we import the packages that we use in our plan. + +The first import is needed for the `dagger.#Plan` definition to be available. + +The second import is for `netlify.#Deploy` to work. + +:::info +Which other imports we are missing? +Look at all the actions in the plan structure at the top of this page. +Now check all the available packages in [universe.dagger.io](https://github.com/dagger/dagger/tree/v0.2.0/pkg/universe.dagger.io). +::: + +We now understand that the `deploy` action is the deploy definition from the netlify package, written as `deploy: netlify.#Deploy` + +Each definition has default values that can be modified via curly brackets. This is what that looks like in practice for our deploy action: + +```cue +// ... +deploy: netlify.#Deploy & { + contents: build.contents.output + site: client.env.APP_NAME + token: client.env.NETLIFY_TOKEN + team: client.env.NETLIFY_TEAM +} +// ... ``` We can build complex pipelines efficiently by referencing any definition, from any package in our actions. -This is one of the fundamental concepts that makes Dagger a powerful language for CI/CD. -Before we can use a package in a plan, we need to declare it at the top of the pipeline configuration, like this: +This is one of the fundamental concepts that makes Dagger a powerful language for building CI/CD pipelines. -```cue -import ( - "universe.dagger.io/docker" -) -``` - -Since we are using the plan definition from the dagger package - `dagger.#Plan` - we also need to declare it at the top of the pipeline configuration: - -```cue -import ( - "dagger.io/dagger" - "universe.dagger.io/docker" -) -``` +If you want to learn more packages in the context of CUE, the config language used by Dagger configs, check out the [Packages](1215-what-is-cue.md#packages) section on the **What is CUE?** page. :::tip Now that we understand the basics of a Dagger plan, we are ready to learn more about how to interact with the client environment. -This will enable us to configure plans just-in-time, save build artefacts, and perform other interactions with the environment within which Dagger runs. +We can read the env (including secrets), run commands, use local sockets, etc. ::: diff --git a/docs/tests/core-concepts/plan/structure.cue.fragment b/docs/tests/core-concepts/plan/structure.cue.fragment index df0cceaa..2e840774 100644 --- a/docs/tests/core-concepts/plan/structure.cue.fragment +++ b/docs/tests/core-concepts/plan/structure.cue.fragment @@ -1,3 +1,8 @@ +// ... +// A plan has pre-requisited that we cover below. +// For now we focus on the dagger.#Plan structure. +// ... + dagger.#Plan & { client: { filesystem: { @@ -22,5 +27,8 @@ dagger.#Plan & { // ... } } + deploy: netlify.#Deploy & { + // ... + } } } diff --git a/pkg/universe.dagger.io/examples/todoapp/netlify.cue b/pkg/universe.dagger.io/examples/todoapp/todoapp.cue similarity index 90% rename from pkg/universe.dagger.io/examples/todoapp/netlify.cue rename to pkg/universe.dagger.io/examples/todoapp/todoapp.cue index 0c5cda96..1722f2d3 100644 --- a/pkg/universe.dagger.io/examples/todoapp/netlify.cue +++ b/pkg/universe.dagger.io/examples/todoapp/todoapp.cue @@ -1,4 +1,4 @@ -package netlify +package todoapp import ( "dagger.io/dagger" @@ -23,12 +23,12 @@ dagger.#Plan & { contents: dagger.#FS exclude: [ "README.md", - "build", - "netlify.cue", + "_build", + "todoapp.cue", "node_modules", ] } - build: write: contents: actions.build.contents.output + "./_build": write: contents: actions.build.contents.output } env: { APP_NAME: string @@ -50,8 +50,6 @@ dagger.#Plan & { contents: client.filesystem.".".read.contents dest: "/src" }, - // bash.#Run is a superset of docker.#Run - // install yarn dependencies bash.#Run & { workdir: "/src" mounts: { From dc5d3340bbb0c30d461a34d08f5b7c88bb25e189 Mon Sep 17 00:00:00 2001 From: Gerhard Lazu Date: Fri, 25 Mar 2022 16:34:37 +0000 Subject: [PATCH 2/5] Capture cyclic task dependency error for #1857 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit universe.dagger.io/examples/todoapp on  europa-docs-plan-tanguy-fresh-pair-of-eyes [$!?] via  v17.4.0 ❯ bin/dagger do build 4:32PM FTL failed to execute plan: cyclic task dependency: task client.filesystem."./_build".write refers to task actions.build.contents._copy refers to task actions.build.run._exec refers to task actions.test._exec refers to task actions.deps._dag."2"._exec refers to task actions.deps._dag."1"._copy refers to task client.filesystem."./_build".write: /Users/gerhard/github.com/gerhard/dagger/pkg/universe.dagger.io/cue.mod/pkg/dagger.io/dagger/fs.cue:114:2 /Users/gerhard/github.com/gerhard/dagger/pkg/universe.dagger.io/docker/build.cue:49:2 /Users/gerhard/github.com/gerhard/dagger/pkg/universe.dagger.io/docker/run.cue:162:2 Signed-off-by: Gerhard Lazu --- pkg/universe.dagger.io/examples/todoapp/todoapp.cue | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/universe.dagger.io/examples/todoapp/todoapp.cue b/pkg/universe.dagger.io/examples/todoapp/todoapp.cue index 1722f2d3..c2143c3e 100644 --- a/pkg/universe.dagger.io/examples/todoapp/todoapp.cue +++ b/pkg/universe.dagger.io/examples/todoapp/todoapp.cue @@ -19,7 +19,7 @@ dagger.#Plan & { } client: { filesystem: { - ".": read: { + "./": read: { contents: dagger.#FS exclude: [ "README.md", From a3cd60d14012f198d6f05e71f6de16f94305bb00 Mon Sep 17 00:00:00 2001 From: Gerhard Lazu Date: Fri, 25 Mar 2022 16:35:29 +0000 Subject: [PATCH 3/5] Cyclic task dependency error fix (workaround?) for #1857 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit universe.dagger.io/examples/todoapp on  europa-docs-plan-tanguy-fresh-pair-of-eyes [$!] via  v17.4.0 took 3s ❯ bin/dagger do build [✔] actions.build.run.script 0.0s [✔] actions.deps 4.2s [✔] client.filesystem."./".read 3.0s [✔] actions.test.script 0.0s [✔] actions.test 0.8s [✔] actions.build.run 3.1s [✔] actions.build.contents 0.0s [✔] client.filesystem."./_build".write 0.1s Signed-off-by: Gerhard Lazu --- pkg/universe.dagger.io/examples/todoapp/todoapp.cue | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/universe.dagger.io/examples/todoapp/todoapp.cue b/pkg/universe.dagger.io/examples/todoapp/todoapp.cue index c2143c3e..9dcd1b02 100644 --- a/pkg/universe.dagger.io/examples/todoapp/todoapp.cue +++ b/pkg/universe.dagger.io/examples/todoapp/todoapp.cue @@ -47,7 +47,7 @@ dagger.#Plan & { } }, docker.#Copy & { - contents: client.filesystem.".".read.contents + contents: client.filesystem."./".read.contents dest: "/src" }, bash.#Run & { From f322327945e80e2ad21fe54bd0efa1dc5f0de65c Mon Sep 17 00:00:00 2001 From: Gerhard Lazu Date: Fri, 25 Mar 2022 16:37:30 +0000 Subject: [PATCH 4/5] =?UTF-8?q?Switch=20build=20=E2=86=92=20=5Fbuild=20in?= =?UTF-8?q?=20.gitignore?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It is much easier to spot this build dir when it gets prefixed with _ Inspired by Elixir: https://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html Signed-off-by: Gerhard Lazu --- pkg/universe.dagger.io/examples/todoapp/.gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/universe.dagger.io/examples/todoapp/.gitignore b/pkg/universe.dagger.io/examples/todoapp/.gitignore index 378eac25..e35d8850 100644 --- a/pkg/universe.dagger.io/examples/todoapp/.gitignore +++ b/pkg/universe.dagger.io/examples/todoapp/.gitignore @@ -1 +1 @@ -build +_build From 3a6d922a4e5ef8b4e3d38aa32f8ac31100165e2b Mon Sep 17 00:00:00 2001 From: Gerhard Lazu Date: Fri, 25 Mar 2022 16:55:23 +0000 Subject: [PATCH 5/5] =?UTF-8?q?Follow-through=20the=20other=20"."=20?= =?UTF-8?q?=E2=86=92=20"./"=20&=20build=20=E2=86=92=20"./=5Fbuild"=20renam?= =?UTF-8?q?es?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Gerhard Lazu --- docs/getting-started/1200-local-dev.md | 37 ++++++++++++-------------- 1 file changed, 17 insertions(+), 20 deletions(-) diff --git a/docs/getting-started/1200-local-dev.md b/docs/getting-started/1200-local-dev.md index 695948cd..644a49a4 100644 --- a/docs/getting-started/1200-local-dev.md +++ b/docs/getting-started/1200-local-dev.md @@ -67,20 +67,20 @@ dagger do build With an empty cache, installing all dependencies, then testing & generating a build for this example app completes in just under 3 minutes: ```shell -[✔] client.filesystem.".".read 0.1s +[✔] client.filesystem."./".read 0.1s [✔] actions.deps 118.8s [✔] actions.test.script 0.1s [✔] actions.test 6.3s [✔] actions.build.run.script 0.0s [✔] actions.build.run 43.7s [✔] actions.build.contents 0.4s -[✔] client.filesystem.build.write 0.1s +[✔] client.filesystem."./_build".write 0.1s ``` Since this is a static application, we can open the files which are generated in `actions.build.contents` in a browser. -The last step - `client.filesystem.build.write` - copies the build result into the `build` directory on the host. +The last step - `client.filesystem.build.write` - copies the build result into the `_build` directory on the host. -On macOS, we run `open build/index.html` in our terminal and see the following app preview: +On macOS, we run `open _build/index.html` in our terminal and see the following app preview: ![todoapp preview](/img/getting-started/todoapp.macos.png) @@ -99,14 +99,14 @@ I change this line to `What must be done today?` and run the build locally again ```shell dagger do build -[✔] client.filesystem.".".read 0.0s +[✔] client.filesystem."./".read 0.0s [✔] actions.deps 7.5s [✔] actions.test.script 0.0s [✔] actions.test 6.0s [✔] actions.build.run.script 0.0s [✔] actions.build.run 29.2s [✔] actions.build.contents 0.0s -[✔] client.filesystem.build.write 0.1s +[✔] client.filesystem."./_build".write 0.1s ``` The total `42.8` time is macOS specific, since the Linux alternative is more than 8x quicker. @@ -160,20 +160,20 @@ dagger do build With an empty cache, installing all dependencies, then testing & generating a build for this example app completes in just under 1 minute: ```shell -[✔] client.filesystem.".".read 0.3s +[✔] client.filesystem."./".read 0.3s [✔] actions.deps 39.7s [✔] actions.test.script 0.2s [✔] actions.test 1.9s [✔] actions.build.run.script 0.1s [✔] actions.build.run 10.0s [✔] actions.build.contents 0.6s -[✔] client.filesystem.build.write 0.1s +[✔] client.filesystem."./_build".write 0.1s ``` Since this is a static application, we can open the files which are generated in `actions.build.contents` in a browser. -The last step - `client.filesystem.build.write` - copies the build result into the `build` directory on the host. +The last step - `client.filesystem.build.write` - copies the build result into the `_build` directory on the host. -On Linux, we run `xdg-open build/index.html` in our terminal and see the following app preview: +On Linux, we run `xdg-open _build/index.html` in our terminal and see the following app preview: ![todoapp preview](/img/getting-started/todoapp.linux.png) @@ -192,14 +192,14 @@ I change this line to `What must be done today?` and run the build locally again ```shell dagger do build -[✔] client.filesystem.".".read 0.0s +[✔] client.filesystem."./".read 0.0s [✔] actions.deps 1.1s [✔] actions.test.script 0.0s [✔] actions.test 0.0s [✔] actions.build.run.script 0.8s [✔] actions.build.run 2.9s [✔] actions.build.contents 0.0s -[✔] client.filesystem.build.write 0.0s +[✔] client.filesystem."./_build".write 0.0s ``` Being able to re-run the test & build loop locally in `4.8s`, at the same speed as running `yarn` scripts locally and without adding any extra dependencies to our host, is likely to change our approach to iterating on changes. @@ -249,16 +249,15 @@ With an empty cache, installing all dependencies, then testing & generating a bu [✔] actions.deps 62.1s [✔] actions.build.run.script 0.4s [✔] actions.test.script 0.5s -[✔] client.filesystem.".".read 0.6s +[✔] client.filesystem."./".read 0.6s [✔] actions.test 2.0s [✔] actions.build.run 12.4s [✔] actions.build.contents 0.1s -[✔] client.filesystem.build.write 0.2s -[✔] client.filesystem.".".read 0.2s +[✔] client.filesystem."./_build".write 0.2s ``` Since this is a static application, we can open the files which are generated in `actions.build.contents` in a browser. -The last step - `client.filesystem.build.write` - copies the build result into the `build` directory on the host. +The last step - `client.filesystem.build.write` - copies the build result into the `_build` directory on the host. On Windows, we run `start build/index.html` in our `Command Prompt` terminal and see the following app preview: @@ -278,16 +277,14 @@ I change this line to `What must be done today?` and run the build locally again ```shell dagger do build -INF upgrading buildkit have host network=true version=v0.10.0 [✔] actions.build.run.script 0.0s [✔] actions.deps 3.4s -[✔] client.filesystem.".".read 0.2s +[✔] client.filesystem."./".read 0.1s [✔] actions.test.script 0.0s [✔] actions.test 1.8s [✔] actions.build.run 7.7s [✔] actions.build.contents 0.2s -[✔] client.filesystem.build.write 0.2s -[✔] client.filesystem.".".read 0.1s +[✔] client.filesystem."./_build".write 0.2s ``` Being able to re-run the test & build loop locally in `13.6s`, without adding any extra dependencies to our host, is likely to change our approach to iterating on changes.