dagger/docs/use-cases/1012-ci.md

slug
/1012/ci

Continuous Integration

Dagger is the perfect tool for CI workflows.

Benefits

• Develop and run your CI pipeline locally. No need to create a Pull Request to trigger CI, you can run the pipeline locally. Since dagger workflows are containerized, you can expect the same results no matter where the pipeline is executed.
• Write once, Run anywhere. The same pipeline can run in any CI, goodbye vendor lock in.
• Blazing Fast: Dagger will automatically build an optimized execution graph, completing the job as fast as possible. Spend less time staring at CI to complete and reduce costs.
• Effortless Cache Optimizations. Dagger will automatically re-use cached execution results if no changes are detected. Made a change to the frontend? The backend won't be built again.
• Composable & Reusable. Define re-usable steps and share them across all your projects. Or with the world. Dagger ships with dozens of re-usable components

Example

This example illustrates how to use Dagger to test and lint a Go project. The use of Go is irrelevant and the example can be easily adapted to other languages.

From the project repository, create a file named ci/main.cue and add the following configuration to it.

package main

import (
    "alpha.dagger.io/dagger"
    "alpha.dagger.io/os"
    "alpha.dagger.io/docker"
)

// Source directory of the repository. We'll connect this from the CLI using `dagger input`
source: dagger.#Artifact

// Here we define a test phase.
// We're using `os.#Container`, a built-in package that will spawn a container
// We're also using `docker.#Pull` to execute the container from a Docker image
// comifrom a registry.
test: os.#Container & {
    image: docker.#Pull & {
        from: "golang:1.16-alpine"
    }
    mount: "/app": from: source
    command: "go test -v ./..."
    dir:     "/app"
}

// Likewise, here we define a lint phase.
lint: os.#Container & {
    image: docker.#Pull & {
        from: "golangci/golangci-lint:v1.39.0"
    }
    mount: "/app": from: source
    command: "golangci-lint run -v"
    dir:     "/app"
}

The configuration above defines:

• source code of the project. More on this later.
• test task which executes go test inside the source artifact using the golang Docker image
• lint task which executes golangci-lint inside the source artifact using the golangci-lint Docker image.

Before we can execute the configuration, we need to set up the Dagger workspace and environment.

# Initialize a dagger workspace at the root of your project
dagger init

# Create a CI environment using the CUE files in the `./ci` directory
dagger new ci -p ./ci

# Link the `source` artifact defined in the configuration with the project
# source code.
dagger input dir source .

Next, bring up the CI environment (e.g. execute the CI configuration):

$ dagger up
# ...
7:15PM INF test | computing    environment=ci
7:15PM INF lint | computing    environment=ci
# ...

Since test and lint do not depend on each other, they are executed in parallel.

Running dagger up a second time will return almost immediately: since no changes were made to source, Dagger will re-use the cached results for both test nor lint.

Integrating with CI

All it takes to execute a Dagger workflow in CI is to run dagger up.

We provide a GitHub Actions to make integration with GitHub easier.

Monorepos

Dagger workflows scale really well with CI complexity.

The following example illustrates how to test two different projects in the same repository: a Node frontend (located in ./frontend) along with a Go backend (located in ./backend).

From the project repository, update the file named ci/main.cue with the following configuration.

package main

import (
    "alpha.dagger.io/dagger"
    "alpha.dagger.io/os"
    "alpha.dagger.io/docker"
)

// Source directory of the repository. We'll connect this from the CLI using `dagger input`
source: dagger.#Artifact

backend: {
    // We use `os.#Dir` to grab a sub-directory of `source`
    code: os.#Dir & {
        from: source
        path: "."
    }

    test: os.#Container & {
        image: docker.#Pull & {
            from: "golang:1.16-alpine"
        }
        mount: "/app": from: code
        command: "go test -v ./..."
        dir:     "/app"
    }
}

frontend: {
    // We use `os.#Dir` to grab a sub-directory of `source`
    code: os.#Dir & {
        from: source
        path: "./frontend"
    }

    test: os.#Container & {
        image: docker.#Pull & {
            from: "node:16-alpine"
        }
        mount: "/app": from: code
        command: """
            # Install Dependencies
            yarn

            # Run the test script
            yarn test
            """
        dir: "/app"
    }
}

:::tip Larger configurations can be split into multiple files, for example backend.cue and frontend.cue :::

The configuration above defines a frontend and backend structure, each containing:

• A code directory, defined as a subdirectory of source
• A language specific test task using either yarn or go

$ dagger up
7:15PM INF frontend.test | computing    environment=ci
7:15PM INF backend.test | computing    environment=ci
7:15PM INF frontend.test | #8 0.370 yarn install v1.22.5    environment=ci
7:15PM INF frontend.test | #8 0.689 [1/4] Resolving packages...    environment=ci
7:15PM INF frontend.test | #8 1.626 [2/4] Fetching packages...    environment=ci
...

frontend.test and backend.test are running in parallel since there are no dependencies between each other.

If you were to make changes to the ./frontend directory, only frontend.test will be executed.