Split CI workflow into individual workflows

I had to re-run all jobs in the CI workflow at least 10 times in the
past 2 days. The problem is that when one jobs fails, all jobs need to
re-run, which sometimes results in different jobs failing. It would be
great if we could only re-run the jobs that failed, rather than all the
jobs in the CI workflow. Going forward, we should focus on improving
flaky tests, and speed the jobs which take the longest, but for now this
is a good start.

Before this change, we were wasting a lot of dev time - 2h in total for
my last PR #1476 - but also wasting CI minutes. Some of us were even
tempted to ignore CI 😱. This is a very slipper slope, and while it may
feel liberating in the short-term, there are many "windmill monsters"
down this path - don't do it.

Have a look at the CI workflow before this change to see how many
failures we had:
https://github.com/dagger/dagger/actions/workflows/ci.yml

Without looking at the jobs that failed, can you guess which areas are
the flakiest and need our attention the most? Integration & Universe are
good guesses, and I wish we could see this without digging into the CI
workflow - this change does that.

There is a lot more that can be improved here, but I didn't want to get
too carried away. The biggest improvement that we can make is switch
this to Dagger, which has some challenges, but I definitely intend to
tackle them because it feels worth it. This is good enough for now.

This is a ship & show PR. If all tests pass, this is a straight merge. I
am keeping it atomic so that we can revert it if we don't like it.

cc @aluzzardi @talentedmrjones @jlongtine @samalba @shykes @grouville

Signed-off-by: Gerhard Lazu <gerhard@lazu.co.uk>
This commit is contained in:
Gerhard Lazu 2022-01-25 13:25:04 +00:00
parent 8697f90285
commit fb09e2a1c6
No known key found for this signature in database
GPG Key ID: A28DE70C9444D7A6
6 changed files with 308 additions and 283 deletions

View File

@ -1,283 +0,0 @@
name: CI
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- name: Check out
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v1
with:
go-version: 1.16
- name: Install Dependencies
run: |
# Cue
export CUE_VERSION="$(grep cue ./go.mod | cut -d' ' -f2 | head -1 | sed -E 's/\.[[:digit:]]\.[[:alnum:]]+-[[:alnum:]]+$//')"
export CUE_TARBALL="cue_${CUE_VERSION}_linux_amd64.tar.gz"
echo "Installing cue version $CUE_VERSION"
curl -L https://github.com/cue-lang/cue/releases/download/${CUE_VERSION}/${CUE_TARBALL} | sudo tar zxf - -C /usr/local/bin
- name: Go Lint
uses: golangci/golangci-lint-action@v2
with:
version: v1.39
skip-go-installation: true
args: --timeout=3m
- name: Lint
run: |
make shellcheck cuelint docslint
- name: Markdown Lint
uses: avto-dev/markdown-lint@v1
with:
config: ".markdownlint.yaml"
args: ./docs README.md
test:
name: Test
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Check out
uses: actions/checkout@v2
- name: Set up Go
uses: actions/setup-go@v1
with:
go-version: 1.16
- name: Test
run: |
make test
integration:
name: Integration
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Check out
uses: actions/checkout@v2
- name: Set up Go
uses: actions/setup-go@v1
with:
go-version: 1.16
- name: Install Dependencies
run: |
# Cue
export CUE_VERSION="$(grep cue ./go.mod | cut -d' ' -f2 | head -1 | sed -E 's/\.[[:digit:]]\.[[:alnum:]]+-[[:alnum:]]+$//')"
export CUE_TARBALL="cue_${CUE_VERSION}_linux_amd64.tar.gz"
echo "Installing cue version $CUE_VERSION"
curl -L https://github.com/cue-lang/cue/releases/download/${CUE_VERSION}/${CUE_TARBALL} | sudo tar zxf - -C /usr/local/bin
- name: Install Dependencies
run: |
# SOPS
sudo curl -L -o /usr/local/bin/sops https://github.com/mozilla/sops/releases/download/v3.7.1/sops-v3.7.1.linux
sudo chmod +x /usr/local/bin/sops
- name: Import Dagger private key
env:
DAGGER_AGE_KEY: ${{ secrets.DAGGER_AGE_KEY }}
run: |
mkdir -p ~/.config/dagger
echo "$DAGGER_AGE_KEY" > ~/.config/dagger/keys.txt
- name: Expose GitHub Runtime
uses: crazy-max/ghaction-github-runtime@v1
- name: Integration test
env:
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
# DAGGER_CACHE_TO: "type=gha,mode=max,scope=test-integration"
# DAGGER_CACHE_FROM: "type=gha,mode=max,scope=test-integration"
run: |
env
make core-integration
universe:
name: Universe
runs-on: ubuntu-latest
timeout-minutes: 20
services:
localstack:
image: localstack/localstack:0.12.16
env:
SERVICES: s3, ecr
LOCALSTACK_API_KEY: ${{ secrets.LOCALSTACK_API_KEY }}
ports:
- 4566:4566
- 4571:4571
- 4510:4510
options: >-
--health-cmd "curl -f http://localhost:4566/health"
--health-start-period 5s
--health-timeout 5s
--health-interval 5s
--health-retries 10
steps:
- name: Check out
uses: actions/checkout@v2
- name: Set up Go
uses: actions/setup-go@v1
with:
go-version: 1.16
- name: Setup Kind Kubernetes Cluster
uses: helm/kind-action@v1.2.0
- name: Import Dagger private key
env:
DAGGER_AGE_KEY: ${{ secrets.DAGGER_AGE_KEY }}
run: |
mkdir -p ~/.config/dagger
echo "$DAGGER_AGE_KEY" > ~/.config/dagger/keys.txt
- name: Provision Localstack AWS resources
env:
AWS_ACCESS_KEY_ID: test
AWS_SECRET_ACCESS_KEY: test
AWS_DEFAULT_REGION: us-east-2
run: |
aws --endpoint-url=http://localhost:4566 s3 mb s3://dagger-ci
aws --endpoint-url=http://localhost:4566 ecr create-repository --repository-name dagger-ci
- name: Expose GitHub Runtime
uses: crazy-max/ghaction-github-runtime@v1
- name: Universe Test
# env:
# DAGGER_CACHE_TO: "type=gha,mode=max,scope=test-universe"
# DAGGER_CACHE_FROM: "type=gha,mode=max,scope=test-universe"
run: |
make universe-test
europa:
name: Universe (Europa)
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- name: Check out
uses: actions/checkout@v2
- name: Set up Go
uses: actions/setup-go@v1
with:
go-version: 1.16
- name: Install Dependencies
run: |
# SOPS
sudo curl -L -o /usr/local/bin/sops https://github.com/mozilla/sops/releases/download/v3.7.1/sops-v3.7.1.linux
sudo chmod +x /usr/local/bin/sops
- name: Import AGE Key
env:
DAGGER_AGE_KEY: ${{ secrets.DAGGER_AGE_KEY }}
run: |
mkdir -p ~/.config/sops/age
echo "$DAGGER_AGE_KEY" > ~/.config/sops/age/keys.txt
- name: Expose GitHub Runtime
uses: crazy-max/ghaction-github-runtime@v1
- name: Universe Test
env:
DAGGER_CACHE_TO: "type=gha,mode=max,scope=test-universe"
DAGGER_CACHE_FROM: "type=gha,mode=max,scope=test-universe"
run: |
make europa-universe-test
doc:
name: Documentation
runs-on: ubuntu-latest
timeout-minutes: 15
# services:
# localstack:
# image: localstack/localstack:0.13.1
# env:
# SERVICES: s3, cloudformation
# LOCALSTACK_API_KEY: ${{ secrets.LOCALSTACK_API_KEY }}
# ports:
# - 4566:4566
# - 4571:4571
# options: >-
# --health-cmd "curl -f http://localhost:4566/health"
# --health-start-period 5s
# --health-timeout 5s
# --health-interval 5s
# --health-retries 10
steps:
- name: Check out
uses: actions/checkout@v2
- name: Set up Go
uses: actions/setup-go@v1
with:
go-version: 1.16
- name: Install Dependencies
run: |
# Cue
export CUE_VERSION="$(grep cue ./go.mod | cut -d' ' -f2 | head -1 | sed -E 's/\.[[:digit:]]\.[[:alnum:]]+-[[:alnum:]]+$//')"
export CUE_TARBALL="cue_${CUE_VERSION}_linux_amd64.tar.gz"
echo "Installing cue version $CUE_VERSION"
curl -L https://github.com/cue-lang/cue/releases/download/${CUE_VERSION}/${CUE_TARBALL} | sudo tar zxf - -C /usr/local/bin
- name: Run local registry
run: |
docker run -d -p 5000:5000 --name registry registry:2
# # DISABLED for CI deadlock debugging
# - name: Write kind echo
# run: |
# echo 'kind: Cluster
# apiVersion: kind.x-k8s.io/v1alpha4
# containerdConfigPatches:
# - |-
# [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5000"]
# endpoint = ["http://registry:5000"]' > ./kind-config.yaml
# cat ./kind-config.yaml
# - name: Setup Kind Kubernetes Cluster
# uses: helm/kind-action@v1.2.0
# with:
# config: "./kind-config.yaml"
# - name: Connect registry to cluster
# run: |
# docker network connect kind registry
- name: Import Dagger private key
env:
DAGGER_AGE_KEY: ${{ secrets.DAGGER_AGE_KEY }}
run: |
mkdir -p ~/.config/dagger
echo "$DAGGER_AGE_KEY" > ~/.config/dagger/keys.txt
- name: Expose GitHub Runtime
uses: crazy-max/ghaction-github-runtime@v1
- name: Documentation Test
# env:
# DAGGER_CACHE_TO: "type=gha,mode=max,scope=test-docs"
# DAGGER_CACHE_FROM: "type=gha,mode=max,scope=test-docs"
run: |
make doc-test

55
.github/workflows/lint.yml vendored Normal file
View File

@ -0,0 +1,55 @@
name: Lint
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
everything:
name: Everything
runs-on: ubuntu-latest
steps:
- name: "Check out"
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: "Set up go"
uses: actions/setup-go@v1
with:
go-version: 1.16
- name: "Install deps"
run: |
# Cue
export CUE_VERSION="$(grep cue ./go.mod | cut -d' ' -f2 | head -1 | sed -E 's/\.[[:digit:]]\.[[:alnum:]]+-[[:alnum:]]+$//')"
export CUE_TARBALL="cue_${CUE_VERSION}_linux_amd64.tar.gz"
echo "Installing cue version $CUE_VERSION"
curl -L https://github.com/cue-lang/cue/releases/download/${CUE_VERSION}/${CUE_TARBALL} | sudo tar zxf - -C /usr/local/bin
- name: "Lint go"
uses: golangci/golangci-lint-action@v2
with:
version: v1.39
skip-go-installation: true
args: --timeout=3m
- name: "Lint shell"
run: |
make shellcheck
- name: "Lint CUE"
run: |
make cuelint
- name: "Lint docs"
run: |
make docslint
- name: "Lint markdown"
uses: avto-dev/markdown-lint@v1
with:
config: ".markdownlint.yaml"
args: ./docs README.md

71
.github/workflows/test-docs.yml vendored Normal file
View File

@ -0,0 +1,71 @@
name: "Test Docs"
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
docs:
name: Docs
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: "Check out"
uses: actions/checkout@v2
- name: "Setup Go"
uses: actions/setup-go@v1
with:
go-version: 1.16
- name: "Install CUE"
run: |
# Cue
export CUE_VERSION="$(grep cue ./go.mod | cut -d' ' -f2 | head -1 | sed -E 's/\.[[:digit:]]\.[[:alnum:]]+-[[:alnum:]]+$//')"
export CUE_TARBALL="cue_${CUE_VERSION}_linux_amd64.tar.gz"
echo "Installing cue version $CUE_VERSION"
curl -L https://github.com/cue-lang/cue/releases/download/${CUE_VERSION}/${CUE_TARBALL} | sudo tar zxf - -C /usr/local/bin
- name: "Run local registry"
run: |
docker run -d -p 5000:5000 --name registry registry:2
# TODO: DISABLED for CI deadlock debugging
# - name: Generate KiND config
# run: |
# echo 'kind: Cluster
# apiVersion: kind.x-k8s.io/v1alpha4
# containerdConfigPatches:
# - |-
# [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5000"]
# endpoint = ["http://registry:5000"]' > ./kind-config.yaml
# cat ./kind-config.yaml
# - name: "Setup KiND"
# uses: helm/kind-action@v1.2.0
# with:
# config: "./kind-config.yaml"
# - name: Connect local registry to KiND
# run: |
# docker network connect kind registry
- name: "Import Dagger private key"
env:
DAGGER_AGE_KEY: ${{ secrets.DAGGER_AGE_KEY }}
run: |
mkdir -p ~/.config/dagger
echo "$DAGGER_AGE_KEY" > ~/.config/dagger/keys.txt
- name: "Expose GitHub Runtime"
uses: crazy-max/ghaction-github-runtime@v1
- name: Test
# TODO: https://github.com/dagger/dagger/pull/1341
# env:
# DAGGER_CACHE_TO: "type=gha,mode=max,scope=test-docs"
# DAGGER_CACHE_FROM: "type=gha,mode=max,scope=test-docs"
run: |
make doc-test

54
.github/workflows/test-integration.yml vendored Normal file
View File

@ -0,0 +1,54 @@
name: "Test Integration"
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
integration:
name: Integration
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: "Check out"
uses: actions/checkout@v2
- name: "Setup Go"
uses: actions/setup-go@v1
with:
go-version: 1.16
- name: "Install CUE"
run: |
export CUE_VERSION="$(grep cue ./go.mod | cut -d' ' -f2 | head -1 | sed -E 's/\.[[:digit:]]\.[[:alnum:]]+-[[:alnum:]]+$//')"
export CUE_TARBALL="cue_${CUE_VERSION}_linux_amd64.tar.gz"
echo "Installing cue version $CUE_VERSION"
curl -L https://github.com/cue-lang/cue/releases/download/${CUE_VERSION}/${CUE_TARBALL} | sudo tar zxf - -C /usr/local/bin
- name: "Install SOPS"
run: |
# SOPS
sudo curl -L -o /usr/local/bin/sops https://github.com/mozilla/sops/releases/download/v3.7.1/sops-v3.7.1.linux
sudo chmod +x /usr/local/bin/sops
- name: "Import Dagger private key"
env:
DAGGER_AGE_KEY: ${{ secrets.DAGGER_AGE_KEY }}
run: |
mkdir -p ~/.config/dagger
echo "$DAGGER_AGE_KEY" > ~/.config/dagger/keys.txt
- name: "Expose GitHub Runtime"
uses: crazy-max/ghaction-github-runtime@v1
- name: Test
env:
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
# TODO: https://github.com/dagger/dagger/pull/1341
# DAGGER_CACHE_TO: "type=gha,mode=max,scope=test-integration"
# DAGGER_CACHE_FROM: "type=gha,mode=max,scope=test-integration"
run: |
env
make core-integration

25
.github/workflows/test-unit.yml vendored Normal file
View File

@ -0,0 +1,25 @@
name: "Test Unit"
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
unit:
name: Unit
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: "Check out"
uses: actions/checkout@v2
- name: "Setup Go"
uses: actions/setup-go@v1
with:
go-version: 1.16
- name: Test
run: |
make test

103
.github/workflows/test-universe.yml vendored Normal file
View File

@ -0,0 +1,103 @@
name: "Test Universe"
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
universe:
name: Universe
runs-on: ubuntu-latest
timeout-minutes: 30
services:
localstack:
image: localstack/localstack:0.12.16
env:
SERVICES: s3, ecr
LOCALSTACK_API_KEY: ${{ secrets.LOCALSTACK_API_KEY }}
ports:
- 4566:4566
- 4571:4571
- 4510:4510
options: >-
--health-cmd "curl -f http://localhost:4566/health"
--health-start-period 5s
--health-timeout 5s
--health-interval 5s
--health-retries 10
steps:
- name: "Check out"
uses: actions/checkout@v2
- name: "Setup Go"
uses: actions/setup-go@v1
with:
go-version: 1.16
- name: "Setup KiND"
uses: helm/kind-action@v1.2.0
- name: "Import Dagger private key"
env:
DAGGER_AGE_KEY: ${{ secrets.DAGGER_AGE_KEY }}
run: |
mkdir -p ~/.config/dagger
echo "$DAGGER_AGE_KEY" > ~/.config/dagger/keys.txt
- name: "Provision Localstack AWS resources"
env:
AWS_ACCESS_KEY_ID: test
AWS_SECRET_ACCESS_KEY: test
AWS_DEFAULT_REGION: us-east-2
run: |
aws --endpoint-url=http://localhost:4566 s3 mb s3://dagger-ci
aws --endpoint-url=http://localhost:4566 ecr create-repository --repository-name dagger-ci
- name: "Expose GitHub Runtime"
uses: crazy-max/ghaction-github-runtime@v1
- name: Test
# TODO: https://github.com/dagger/dagger/pull/1341
# env:
# DAGGER_CACHE_TO: "type=gha,mode=max,scope=test-universe"
# DAGGER_CACHE_FROM: "type=gha,mode=max,scope=test-universe"
run: |
make universe-test
universe-europa:
name: "Universe - Europa"
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: "Check out"
uses: actions/checkout@v2
- name: "Set up Go"
uses: actions/setup-go@v1
with:
go-version: 1.16
- name: "Install SOPS"
run: |
sudo curl -L -o /usr/local/bin/sops https://github.com/mozilla/sops/releases/download/v3.7.1/sops-v3.7.1.linux
sudo chmod +x /usr/local/bin/sops
- name: "Import Dagger private key"
env:
DAGGER_AGE_KEY: ${{ secrets.DAGGER_AGE_KEY }}
run: |
mkdir -p ~/.config/sops/age
echo "$DAGGER_AGE_KEY" > ~/.config/sops/age/keys.txt
- name: "Expose GitHub Runtime"
uses: crazy-max/ghaction-github-runtime@v1
- name: Test
env:
DAGGER_CACHE_TO: "type=gha,mode=max,scope=test-universe"
DAGGER_CACHE_FROM: "type=gha,mode=max,scope=test-universe"
run: |
make europa-universe-test