Europa: integrate core packages, separate universe

Signed-off-by: Solomon Hykes <solomon@dagger.io>
This commit is contained in:
Solomon Hykes 2021-12-14 00:16:12 +00:00
parent dd4c360a7b
commit c1c585bcd5
57 changed files with 612 additions and 142 deletions

View File

@ -16,10 +16,13 @@
- [azure/storage](./azure/storage.md) - -
- [bats](./bats.md) - -
- [dagger](./dagger/README.md) - Dagger core types
- [dagger/engine](./dagger/engine.md) - -
- [dagger/op](./dagger/op.md) - op: low-level operations for Dagger processing pipelines
- [docker](./docker/README.md) - Docker container operations
- [docker/compose](./docker/compose.md) - Docker-compose operations
- [europa/dagger](./europa/dagger/README.md) - The Dagger API.
- [europa/dagger/engine](./europa/dagger/engine/README.md) - -
- [europa/dagger/engine/spec](./europa/dagger/engine/spec/README.md) - Placeholder package, to keep docs generating tool happy.
- [europa/dagger/engine/spec/engine](./europa/dagger/engine/spec/engine.md) - -
- [gcp](./gcp/README.md) - Google Cloud Platform
- [gcp/cloudrun](./gcp/cloudrun.md) - -
- [gcp/gcr](./gcp/gcr.md) - Google Container Registry

View File

@ -0,0 +1,95 @@
---
sidebar_label: dagger
---
# alpha.dagger.io/europa/dagger
The Dagger API.
```cue
import "alpha.dagger.io/europa/dagger"
```
## dagger.#DAG
A special kind of program which `dagger` can execute.
### dagger.#DAG Inputs
_No input._
### dagger.#DAG Outputs
_No output._
## dagger.#FS
A reference to a filesystem tree. For example: - The root filesystem of a container - A source code repository - A directory containing binary artifacts Rule of thumb: if it fits in a tar archive, it fits in a #FS.
### dagger.#FS Inputs
_No input._
### dagger.#FS Outputs
_No output._
## dagger.#Plan
A deployment plan executed by `dagger up`
### dagger.#Plan Inputs
_No input._
### dagger.#Plan Outputs
_No output._
## dagger.#Secret
A reference to an external secret, for example: - A password - A SSH private key - An API token Secrets are never merged in the Cue tree. They can only be used by a special filesystem mount designed to minimize leak risk.
### dagger.#Secret Inputs
_No input._
### dagger.#Secret Outputs
_No output._
## dagger.#Service
A reference to a network service endpoint, for example: - A TCP or UDP port - A unix socket - An HTTPS endpoint
### dagger.#Service Inputs
_No input._
### dagger.#Service Outputs
_No output._
## dagger.#Stream
A reference to a stream of bytes, for example: - The standard output or error stream of a command - The standard input stream of a command - The contents of a file or named pipe
### dagger.#Stream Inputs
_No input._
### dagger.#Stream Outputs
_No output._
## dagger.#Subdir
Select a subdirectory from a filesystem tree
### dagger.#Subdir Inputs
_No input._
### dagger.#Subdir Outputs
_No output._

View File

@ -0,0 +1,91 @@
---
sidebar_label: engine
---
# alpha.dagger.io/europa/dagger/engine
```cue
import "alpha.dagger.io/europa/dagger/engine"
```
## engine.#Context
### engine.#Context Inputs
_No input._
### engine.#Context Outputs
_No output._
## engine.#FS
A reference to a filesystem tree. For example: - The root filesystem of a container - A source code repository - A directory containing binary artifacts Rule of thumb: if it fits in a tar archive, it fits in a #FS.
### engine.#FS Inputs
_No input._
### engine.#FS Outputs
_No output._
## engine.#ImageConfig
Container image config. See [OCI](https://www.opencontainers.org/). Spec left open on purpose to account for additional fields. [Image Spec](https://github.com/opencontainers/image-spec/blob/main/specs-go/v1/config.go) [Docker Superset](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/dockerfile2llb/image.go)
### engine.#ImageConfig Inputs
_No input._
### engine.#ImageConfig Outputs
_No output._
## engine.#Plan
A deployment plan executed by `dagger up`
### engine.#Plan Inputs
_No input._
### engine.#Plan Outputs
_No output._
## engine.#Pull
Download a container image from a remote repository
### engine.#Pull Inputs
_No input._
### engine.#Pull Outputs
_No output._
## engine.#Secret
A reference to an external secret, for example: - A password - A SSH private key - An API token Secrets are never merged in the Cue tree. They can only be used by a special filesystem mount designed to minimize leak risk.
### engine.#Secret Inputs
_No input._
### engine.#Secret Outputs
_No output._
## engine.#Service
A reference to a network service endpoint, for example: - A TCP or UDP port - A unix or npipe socket - An HTTPS endpoint
### engine.#Service Inputs
_No input._
### engine.#Service Outputs
_No output._

View File

@ -0,0 +1,11 @@
---
sidebar_label: spec
---
# alpha.dagger.io/europa/dagger/engine/spec
Placeholder package, to keep docs generating tool happy.
```cue
import "alpha.dagger.io/europa/dagger/engine/spec"
```

View File

@ -0,0 +1,261 @@
---
sidebar_label: engine
---
# alpha.dagger.io/europa/dagger/engine/spec/engine
```cue
import "alpha.dagger.io/europa/dagger/engine/spec/engine"
```
## engine.#Build
Build a container image using buildkit
### engine.#Build Inputs
_No input._
### engine.#Build Outputs
_No output._
## engine.#CacheDir
A (best effort) persistent cache dir
### engine.#CacheDir Inputs
_No input._
### engine.#CacheDir Outputs
_No output._
## engine.#Context
### engine.#Context Inputs
_No input._
### engine.#Context Outputs
_No output._
## engine.#Copy
### engine.#Copy Inputs
_No input._
### engine.#Copy Outputs
_No output._
## engine.#CopyInfo
### engine.#CopyInfo Inputs
_No input._
### engine.#CopyInfo Outputs
_No output._
## engine.#Exec
Execute a command in a container
### engine.#Exec Inputs
_No input._
### engine.#Exec Outputs
_No output._
## engine.#FS
A reference to a filesystem tree. For example: - The root filesystem of a container - A source code repository - A directory containing binary artifacts Rule of thumb: if it fits in a tar archive, it fits in a #FS. A filesystem state
### engine.#FS Inputs
_No input._
### engine.#FS Outputs
_No output._
## engine.#GitPull
Pull a directory from a git remote
### engine.#GitPull Inputs
_No input._
### engine.#GitPull Outputs
_No output._
## engine.#GitPush
Push a directory to a git remote
### engine.#GitPush Inputs
_No input._
### engine.#GitPush Outputs
_No output._
## engine.#ImageConfig
Container image config. See [OCI](https://www.opencontainers.org/). Spec left open on purpose to account for additional fields. [Image Spec](https://github.com/opencontainers/image-spec/blob/main/specs-go/v1/config.go) [Docker Superset](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/dockerfile2llb/image.go) Container image config See [OCI](https://www.opencontainers.org)
### engine.#ImageConfig Inputs
_No input._
### engine.#ImageConfig Outputs
_No output._
## engine.#Merge
### engine.#Merge Inputs
_No input._
### engine.#Merge Outputs
_No output._
## engine.#Mount
A transient filesystem mount.
### engine.#Mount Inputs
_No input._
### engine.#Mount Outputs
_No output._
## engine.#Plan
A deployment plan executed by `dagger up`
### engine.#Plan Inputs
_No input._
### engine.#Plan Outputs
_No output._
## engine.#Pull
Download a container image from a remote repository
### engine.#Pull Inputs
_No input._
### engine.#Pull Outputs
_No output._
## engine.#Push
Upload a container image to a remote repository
### engine.#Push Inputs
_No input._
### engine.#Push Outputs
_No output._
## engine.#ReadFile
### engine.#ReadFile Inputs
_No input._
### engine.#ReadFile Outputs
_No output._
## engine.#Scratch
Produce an empty directory
### engine.#Scratch Inputs
_No input._
### engine.#Scratch Outputs
_No output._
## engine.#Secret
A reference to an external secret, for example: - A password - A SSH private key - An API token Secrets are never merged in the Cue tree. They can only be used by a special filesystem mount designed to minimize leak risk. An external secret
### engine.#Secret Inputs
_No input._
### engine.#Secret Outputs
_No output._
## engine.#Service
A reference to a network service endpoint, for example: - A TCP or UDP port - A unix or npipe socket - An HTTPS endpoint An external network service
### engine.#Service Inputs
_No input._
### engine.#Service Outputs
_No output._
## engine.#Stream
A stream of bytes
### engine.#Stream Inputs
_No input._
### engine.#Stream Outputs
_No output._
## engine.#TempDir
A temporary directory for command execution
### engine.#TempDir Inputs
_No input._
### engine.#TempDir Outputs
_No output._
## engine.#WriteFile
### engine.#WriteFile Inputs
_No input._
### engine.#WriteFile Outputs
_No output._

76
europa-universe/README.md Normal file
View File

@ -0,0 +1,76 @@
# Europa Universe
## About this directory
`europa-universe/` is a staging area for the upcoming `universe.dagger.io` package namespace,
which will be shipped as part of the [Europa release](https://github.com/dagger/dagger/issues/1088).
## What is Universe?
The Dagger Universe is a catalog of reusable Cue packages, curated by Dagger but possibly authored by third parties. Most packages in Universe contain reusable actions; some may also contain entire configuration templates.
The import domain for Universe will be `universe.dagger.io`. It will deprecate the current domain `alpha.dagger.io`.
## Where is the `dagger` package?
Europa will also introduce a new package for the Dagger Core API: `dagger.io/dagger`.
This is a core package, and is *not* part of Universe (note the import domain).
The development version of the Europa core API can be imported as [alpha.dagger.io/europa/dagger](../stdlib/europa/dagger).
## Where is the `dagger/engine` package?
Europa will also introduce a new package for the Low-Level Dagger Engine API : `dagger.io/dagger/engine`.
This is a core package, and is *not* part of Universe (note the import domain).
The development version of the Europa Low-Level Engine API can be imported as either:
* [alpha.dagger.io/europa/dagger/engine/spec/engine](../stdlib/europa/dagger/engine/spec/engine) for the full spec
* [alpha.dagger.io/dagger/europa/engine](../stdlib/europa/dagger/engine) for the implemented subset of the spec
## Universe vs other packages
This table compares Dagger core packages, Dagger Universe packages, and the overall CUE package ecosystem.
| | *Dagger core* | *Dagger Universe* | *CUE ecosystem* |
|----------|--------------|------|
| Import path | `dagger.io` | `universe.dagger.io` | Everything else |
| Purpose | Access core Dagger features | Safely reuse code from the Dagger community | Reuse any CUE code from anyone |
| Author | Dagger team | Dagger community, curated by Dagger | Anyone |
| Release cycle | Released with Dagger engine | Released continuously | No release cycle |
| Size | Small | Large | Very large |
| Growth rate | Grows slowly, with engine features | Grows fast, with Dagger community | Grows even faster, with CUE ecosystem |
## Notable packages
### Docker API
*Import path: [`universe.dagger.io/docker`](./universe/docker)*
The `docker` package is a native Cue API for Docker. You can use it to build, run, push and pull Docker containers directly from Cue.
The Dagger container API defines the following types:
* `#Image`: a container image
* `#Run`: run a comand in a container
* `#Push`: upload an image to a repository
* `#Pull`: download an image from a repository
* `#Build`: build an image
### Examples
*Import path: [`universe.dagger.io/examples`](./examples)*
This package contains examples of complete Dagger configurations, including the result of following tutorials in the documentations.
For example, [the todoapp example](./examples/todoapp) corresponds to the [Getting Started tutorial](https://docs.dagger.io/1003/get-started/)
## TODO LIST
* Support native language dev in `docker.#Run` with good DX (Python, Go, Typescript etc.)
* Coding style. When to use verbs vs. nouns?
* Easy file injection API (`container.#Image.files` ?)
* Use file injection instead of inline for `#Command.script` (to avoid hitting arg character limits)
* Organize universe packages in sub-categories?

View File

@ -0,0 +1 @@
../../../stdlib/europa

View File

@ -2,7 +2,7 @@ package docker
import (
"dagger.io/dagger"
"dagger.io/dagger/engine"
"dagger.io/dagger/engine/spec/engine"
)
// Modular build API for Docker containers

View File

@ -4,7 +4,7 @@ package docker
import (
"list"
"dagger.io/dagger/engine"
"dagger.io/dagger/engine/spec/engine"
"dagger.io/dagger"
)

View File

@ -2,9 +2,6 @@
set -e
# Run all tests from universe/
cd universe
targets=(
dagger.io/dagger
dagger.io/dagger/engine

View File

@ -5,7 +5,7 @@ import (
"strings"
"dagger.io/dagger"
"dagger.io/dagger/engine" // FIXME: should not be needed for common cases
"dagger.io/dagger/engine/spec/engine"
"universe.dagger.io/alpine"
"universe.dagger.io/bash"

View File

@ -1,120 +0,0 @@
# Europa release staging
## About the europa/ directory
This directory is a staging area for the upcoming Europa release.
It is intended for experimentation and review without requiring long-lived development branches.
Its contents MUST NOT BE USED by `dagger` or its build and release tooling.
As part of the Europa release, this directory will be removed.
## About the Europa release
Europa is the codename of the final major release of Dagger before launch.
For more details on the Europa release, see the [Europa epic](https://github.com/dagger/dagger/issues/1088).
## New CUE packages
Europa introduces a new set of CUE packages for developers to use. These new packages are a complete, incompatible replacement for the pre-Europa packages.
* The bad news is that pre-Europa configurations will need to be manually adapted
* The good news is that Europa APIs are much better. So once ported to Europa, configurations will be shorter, easier to maintain, faster and more reliable (at least that's the goal!)
We intend for Europa to be the last breaking update. Going forward, we will aim for 100% compatibility
whenever possible, and when that is not possible, a migration path that is as automated and painless
as possible.
Starting with Europa, Dagger separates its Cue packages in two distinct namespaces: *stdlib* and *universe*.
* The Dagger Universe is a catalog of reusable Cue packages, curated by Dagger but possibly authored by third parties. Most packages in Universe contain reusable actions; some may also contain entire configuration templates.
* The Dagger Stdlib are core packages shipped with the Dagger engine.
| | *Stdlib* | *Universe* |
|----------|--------------|------|
| Import path | `dagger.io` | `universe.dagger.io` |
| Purpose | Access core Dagger features | Safely reuse code from the community |
| Author | Dagger team | Dagger community, curated by Dagger |
| Release cycle | Released with Dagger engine | Released continuously |
| Size | Small | Large |
| Growth rate | Grows slowly, with engine features | Grows fast, with community |
### Dagger Core API
*Import path: [`dagger.io/dagger`](./stdlib/dagger)*
The Dagger Core API defines core types and utilities for programming Dagger:
* `#Plan`: a complete configuration executable by `dagger up`
* `#FS` to reference filesystem state
* `#Secret` to (securely) reference external secrets
* `#Service` to reference network service endpoints
* `#Stream` to reference byte streams
### Low-level engine API
*Import path: [`dagger.io/dagger/engine`](./stdlib/dagger/engine)*
`engine` is a low-level API for accessing the raw capabilities of the Dagger Engine. Most developers should use the Dagger Core API instead (`dagger.io/dagger`), but experts and framework developers can target the engine API directly for maximum control.
This API prioritizes robustness, consistency, and feature completeness. It does NOT prioritize developer convenience or leveraging Cue for composition.
In Europa, `engine` will deprecate the following implicit API:
* Low-level operations defined in `alpha.dagger.io/dagger/op`
* Imperative DSL to assemble Dockerfile-like arrays as Cue arrays
* Convention to embed pipelines in the Cue lattice with the special nested definition `#up`
* Convention to reference filesystem state from the Cue lattice with `@dagger(artifact)`
* Convention to reference external secrets from the Cue lattice with `@dagger(secret)`
* Convention to reference external network endpoints from the Cue lattive with `@dagger(stream)`
* Convention that some operations (specifically `op.#Local`) are meant to be generated at runtime rather than authored manually.
### Docker API
*Import path: [`universe.dagger.io/docker`](./universe/docker)*
The `docker` package is a native Cue API for Docker. You can use it to build, run, push and pull Docker containers directly from Cue.
The Dagger container API defines the following types:
* `#Image`: a container image
* `#Run`: run a comand in a container
* `#Push`: upload an image to a repository
* `#Pull`: download an image from a repository
* `#Build`: build an image
### Examples
*Import path: [`universe.dagger.io/examples`](https://github.com/shykes/dagger/tree/llb2/europa/universe/examples)*
This package contains examples of complete Dagger configurations, including the result of following tutorials in the documentations.
For example, [the todoapp example](https://github.com/shykes/dagger/tree/llb2/europa/universe/examples/todoapp/deploy) corresponds to the [Getting Started tutorial](https://docs.dagger.io/1003/get-started/)
### More packages
More packages are being developed under [universe.dagger.io](./universe)
## TODO LIST
* Support native language dev in `docker.#Run` with good DX (Python, Go, Typescript etc.)
* #Scratch: replace with null #FS?
* Coding style. When to use verbs vs. nouns?
* Resolve registry auth special case (buildkit does not support scoping registry auth)
* Easy file injection API (`container.#Image.files` ?)
* Use file injection instead of inline for `#Command.script` (to avoid hitting arg character limits)
* Organize universe packages in sub-categories?
* Are there runtime limitations in….
* using hidden fields `_foo` as part of the DAG?
* using `if` statements as part of the DAG?
* using inlined Cue expressions as part of the DAG?
* Do we really need CUE definitions? cue/cmd doesnt need them… This one is 💣🔥, pure speculation. We must pursue the best DX wherever that may lead us!
* Readability of error messages
* At a minimum dont make it worse!
* Small improvements are good (eg.
* Make sure we dont make error messages LESS readable
* Add input.params as proposed by Richard
* Combining all container operations under an opinionated universe.dagger.io/docker package: [good or bad idea](https://github.com/dagger/dagger/pull/1117#discussion_r765178454)?
* [Outstanding questions on proxy features](https://github.com/dagger/dagger/pull/1117#discussion_r765211280)
* [Outstanding questions on #Stream and emulating unix pipes with them](https://github.com/dagger/dagger/pull/1117#discussion_r766145864)
* [Outstanding questions on engine.#Pull and information loss](https://github.com/dagger/dagger/pull/1117#discussion_r765219049)
* [Outstanding questions on global registry auth scope in buildkit](https://github.com/dagger/dagger/pull/1117#discussion_r765963051)
* [Outstanding questions on platform key](https://github.com/dagger/dagger/pull/1117#discussion_r766085610)

View File

@ -1,2 +0,0 @@
node_modules
report.xml

View File

@ -1 +0,0 @@
module: "dagger.io"

View File

@ -1,2 +0,0 @@
# dagger universe
alpha.dagger.io

View File

@ -3,7 +3,7 @@ package dagger
import (
"alpha.dagger.io/dagger/op"
"alpha.dagger.io/dagger/engine"
"alpha.dagger.io/europa/dagger/engine"
)
// An artifact such as source code checkout, container image, binary archive...

59
stdlib/europa/README.md Normal file
View File

@ -0,0 +1,59 @@
# Europa Core packages
## About this directory
`stdlib/europa/` holds the development version of the Core packages for the upcoming [Europa release](https://github.com/dagger/dagger/issues/1088).
Once Europa is released, `stdlib/europa` will become the new `stdlib/`
## What are Dagger core packages?
Dagger core packages are CUE packages released alongside the Dagger engine, to allow developers to access its features.
### Dagger Core API: `dagger.io/dagger`
*Development import path: `alpha.dagger.io/europa/dagger`*
The Dagger Core API defines core types and utilities for programming Dagger:
* `#Plan`: a complete configuration executable by `dagger up`
* `#FS` to reference filesystem state
* `#Secret` to (securely) reference external secrets
* `#Service` to reference network service endpoints
* `#Stream` to reference byte streams
### Low-level Engine API: `dagger.io/dagger/engine`
* *Development import path (implemented subset): `alpha.dagger.io/europa/dagger/engine`*
* *Development importa pth (full spec): `alpha.dagger.io/dagger/europa/dagger/engine/spec/engine`*
`engine` is a low-level API for accessing the raw capabilities of the Dagger Engine. Most developers should use the Dagger Core API instead (`dagger.io/dagger`), but experts and framework developers can target the engine API directly for maximum control.
This API prioritizes robustness, consistency, and feature completeness. It does NOT prioritize developer convenience or leveraging Cue for composition.
In Europa, `engine` will deprecate the following implicit API:
* Low-level operations defined in `alpha.dagger.io/dagger/op`
* Imperative DSL to assemble Dockerfile-like arrays as Cue arrays
* Convention to embed pipelines in the Cue lattice with the special nested definition `#up`
* Convention to reference filesystem state from the Cue lattice with `@dagger(artifact)`
* Convention to reference external secrets from the Cue lattice with `@dagger(secret)`
* Convention to reference external network endpoints from the Cue lattive with `@dagger(stream)`
* Convention that some operations (specifically `op.#Local`) are meant to be generated at runtime rather than authored manually.
## TODO LIST
* #Scratch: replace with null #FS?
* Resolve registry auth special case (buildkit does not support scoping registry auth)
* Are there runtime limitations in….
* using hidden fields `_foo` as part of the DAG?
* using `if` statements as part of the DAG?
* using inlined Cue expressions as part of the DAG?
* Readability of error messages
* At a minimum dont make it worse!
* Small improvements are good (eg.
* Make sure we dont make error messages LESS readable
* [Outstanding questions on proxy features](https://github.com/dagger/dagger/pull/1117#discussion_r765211280)
* [Outstanding questions on #Stream and emulating unix pipes with them](https://github.com/dagger/dagger/pull/1117#discussion_r766145864)
* [Outstanding questions on engine.#Pull and information loss](https://github.com/dagger/dagger/pull/1117#discussion_r765219049)
* [Outstanding questions on global registry auth scope in buildkit](https://github.com/dagger/dagger/pull/1117#discussion_r765963051)
* [Outstanding questions on platform key](https://github.com/dagger/dagger/pull/1117#discussion_r766085610)

View File

@ -9,7 +9,7 @@ package engine
// - "index.docker.io/dagger:latest@sha256:a89cb097693dd354de598d279c304a1c73ee550fbfff6d9ee515568e0c749cfe"
#Ref: string
// Container image config. See [OCI](https://opencontainers.org/).
// Container image config. See [OCI](https://www.opencontainers.org/).
// Spec left open on purpose to account for additional fields.
// [Image Spec](https://github.com/opencontainers/image-spec/blob/main/specs-go/v1/config.go)
// [Docker Superset](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/dockerfile2llb/image.go)

View File

@ -1,7 +1,7 @@
package engine
// Container image config
// See https://opencontainers.org
// See [OCI](https://www.opencontainers.org)
#ImageConfig: {
env?: [...string]
user?: string

View File

@ -0,0 +1,2 @@
// Placeholder package, to keep docs generating tool happy.
package spec

View File

@ -1,7 +1,7 @@
package dagger
import (
"dagger.io/dagger/engine"
"alpha.dagger.io/europa/dagger/engine/spec/engine"
)
// A reference to a filesystem tree.

View File

@ -1,7 +1,7 @@
package dagger
import (
"dagger.io/dagger/engine"
"alpha.dagger.io/europa/dagger/engine/spec/engine"
)
// Select a subdirectory from a filesystem tree

View File

@ -14,12 +14,11 @@ import (
var (
// FS contains the filesystem of the stdlib.
//go:embed **/*.cue **/*/*.cue
//go:embed **/*.cue **/*/*.cue europa/dagger/*.cue europa/dagger/engine/*.cue europa/dagger/engine/spec/*.cue europa/dagger/engine/spec/engine/*.cue
FS embed.FS
ModuleName = "alpha.dagger.io"
PackageName = fmt.Sprintf("%s/dagger", ModuleName)
EnginePackage = fmt.Sprintf("%s/engine", PackageName)
EnginePackage = fmt.Sprintf("%s/europa/dagger/engine", ModuleName)
Path = path.Join("cue.mod", "pkg", ModuleName)
lockFilePath = path.Join("cue.mod", "dagger.lock")
)

View File

@ -1,7 +1,7 @@
package main
import (
"alpha.dagger.io/dagger/engine"
"alpha.dagger.io/europa/dagger/engine"
"alpha.dagger.io/os"
)

View File

@ -1,7 +1,7 @@
package main
import (
"alpha.dagger.io/dagger/engine"
"alpha.dagger.io/europa/dagger/engine"
)
engine.#Plan & {