Commit Graph

47 Commits

Author SHA1 Message Date
Andrea Luzzardi
0aea10d23e dagger.#FS support
- Implement dagger.#FS support
- Migrate `context.imports` to dagger.#FS
- Backward compat: dagger.#FS can be passed in lieu of a
  dagger.#Artifact
- For instance, an import (`dagger.#FS`) can be passed to the current
  `yarn.#Package` implementation

Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-11-30 12:58:50 -08:00
Andrea Luzzardi
81ec746d2c core: component -> task
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-11-24 16:58:24 -08:00
Andrea Luzzardi
608f254449 runtime: support legacy Pipelines in new execution engine
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-11-24 16:51:52 -08:00
Andrea Luzzardi
b5e2ba63c6 runner: fix empty CUE value check
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-11-24 16:08:40 -08:00
Andrea Luzzardi
a61e8dcb62 prepare the transition to #Plan.context
This change helps the transition between `dagger input` and `#Plan.context`.

In summary, the codebase now relies on a *context* for execution with mapping to *IDs*.
In the future, *context* will come from a `#Plan.context`.
In the meantime, a bridge converts `dagger input` to a plan context. This allows both *old* and *new* style configurations to co-exist with the same underlying engine.

- Implement `plancontext`. Context holds the execution context for a plan. Currently this includes the platform, local directories, secrets and services (e.g. unix/npipe).
- Contextual data can be registered at any point. In the future, this will be done by `#Plan.context`
- Migrated the `dagger input` codebase to register inputs in a `plancontext`
- Migrated low-level types/operations to the *Context ID* pattern.
  - `dagger.#Stream` now only includes an `id` (instead of `unix` path)
  - `dagger.#Secret` still includes only an ID, but now it's based off `plancontext`
  - `op.#Local` now only includes an `id` (instead of `path`, `include`, `exclude`.

Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-11-19 11:29:38 -08:00
Andrea Luzzardi
53b588029b remove forgotten debug printf
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-11-12 17:27:26 -08:00
Sam Alba
e6de59a340 added support for stream type npipe (Windows named pipe)
Signed-off-by: Sam Alba <samalba@users.noreply.github.com>
2021-11-08 17:16:38 -08:00
Tom Chauveau
a63c4e989a
Change "arch" into "platform" because it's more accurate.
Signed-off-by: Tom Chauveau <tom.chauveau@epitech.eu>
2021-11-05 16:21:21 +01:00
Tom Chauveau
59f67f9cb5
Fix export architecture configuration to resolves #1087
Signed-off-by: Tom Chauveau <tom.chauveau@epitech.eu>
2021-11-02 12:59:51 +01:00
Tom Chauveau
4a461a0021
Fix engine to don't write architecture in values.yaml if no one provided.
Signed-off-by: Tom Chauveau <tom.chauveau@epitech.eu>
2021-10-30 12:07:33 +02:00
Tom Chauveau
caa03a37cc
Update dagger engine to uses a given architecture instead of default one.
Check #1071 for more details

Signed-off-by: Tom Chauveau <tom.chauveau@epitech.eu>
2021-10-22 21:02:12 +02:00
Andrea Luzzardi
e37f8c5e53 pipeline: analysis: ignore CUE errors
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-10-20 14:37:31 -07:00
Andrea Luzzardi
6c48f8b7a9 pipeline: more explicit non-executable errors
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-10-20 12:27:58 -07:00
Andrea Luzzardi
c08f619b02 Fix local directory error handling
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-10-20 11:38:42 -07:00
Andrea Luzzardi
e5c571b8a9 op.#SaveImage: fix data race
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-10-12 16:05:43 -07:00
Andrea Luzzardi
50d55a8885 op.#SaveImage: export the image ID
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-10-12 15:39:04 -07:00
Andrea Luzzardi
5a1d4bff62 Support loading artifacts into a Docker Engine
This adds support to loading artifacts (e.g. docker.#Build,
os.#Container, ...) into any arbitrary docker engine (through a
dagger.#Stream for UNIX sockets or SSH for a remote engine)

Implementation:
- Add op.#SaveImage which serializes an artifact into an arbitrary path
  (docker tarball format)
- Add docker.#Load which uses op.#SaveImage to serialize to disk and
  executes `docker load` to load it back

Caveats: Because we're doing this in userspace rather than letting
dagger itself load the image, the performance is pretty bad.

The buildkit API is meant for streaming (get a stream of a docker image
pipe it into docker load). Because of userspace, we have to load the
entire docker image into memory, then serialize it in a single WriteFile
LLB operation.

Example:

```cue
package main

import (
	"alpha.dagger.io/dagger"
	"alpha.dagger.io/docker"
)

source: dagger.#Input & dagger.#Artifact

dockersocket: dagger.#Input & dagger.#Stream

build: docker.#Build & {
	"source": source
}

load: docker.#Load & {
	source: build
	tag:    "testimage"
	socket: dockersocket
}
```

Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-10-12 14:16:01 -07:00
Andrea Luzzardi
dd1bf18ec2 logger: TTY logs support (live update)
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-10-05 15:38:34 -07:00
Kevin Poirot
51f96172bd Add #Secret type to Docker #Build args
Signed-off-by: Kevin Poirot <kevin@vazee.fr>
2021-09-23 11:13:59 +02:00
Andrea Luzzardi
0f4a55a482 socket -> stream
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-09-17 15:19:15 -07:00
Andrea Luzzardi
5480fb991d dagger.#Socket support
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-09-17 11:53:09 -07:00
Tom Chauveau
cf13257b10
Improve SecretStore integration with new method
Signed-off-by: Tom Chauveau <tom.chauveau@epitech.eu>
2021-09-01 21:46:56 +02:00
Tom Chauveau
eabf1b52e5
Update docker push tests
Signed-off-by: Tom Chauveau <tom.chauveau@epitech.eu>
2021-08-31 14:44:06 +02:00
Tom Chauveau
a9fd97d7fe
Handle secrets in DockerLogin operation
Before, secret was a plain text string, but it could lead to security issue
so we are now handling secrets as `dagger.#Secret` or string.
I've add a new struct SecretStore that expose the inputStore to easily
retrieve secret value.

Signed-off-by: Tom Chauveau <tom.chauveau@epitech.eu>
2021-08-31 13:04:16 +02:00
Guillaume de Rouville
08f395b70d Add auth to Git.#Repository
Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
2021-08-27 15:58:55 +02:00
Tom Chauveau
d7194f161d Add secret management to op.#FetchGit operation to fetch private repository
- Update `op.cue` to support secrets
- Update `pipeline.go` to use authTokenSecret & authHeaderSecret

Signed-off-by: Tom Chauveau <tom.chauveau@epitech.eu>
2021-08-27 14:41:31 +02:00
Andrea Luzzardi
b8dcc02bb8 performance: compile CUE client side
Restructured the compile logic to happen on the CLI instead of the
BuildKit frontend.

- Avoid uploading the entire workspace to BuildKit on every compilation
- Let the CUE loader scan the files instead of going through the
  BuildKit filesystem gRPC APIs.

Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-08-20 16:34:49 +02:00
Guillaume de Rouville
9f90aca2b7 op.#Exec hack: improve performance by 100x on big configs
Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
2021-07-29 16:37:47 +02:00
Alvise
99d2514aaa Move to OpenTelemetry
Signed-off-by: Alvise <vitalvise@gmail.com>
2021-07-09 09:47:36 +02:00
Andrea Luzzardi
f58ee5811b universe vendoring
Rather than injecting universe at runtime, this change will vendor
alpha.dagger.io in `cue.mod` directly.

Fixes #700

Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-07-08 17:52:21 +02:00
Andrea Luzzardi
f8531fdb0b
Merge pull request #772 from aluzzardi/default-plan-dir
plan: default cue module to .
2021-07-07 17:34:23 +02:00
Andrea Luzzardi
6e215b194e use the workspace as the plan module
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-07-07 13:12:10 +02:00
Edgar Lee
0501ae91a5 Adds support for op.#FetchHTTP
Signed-off-by: Edgar Lee <edgarhinshunlee@gmail.com>
2021-07-06 14:37:52 -07:00
Sam Alba
efed4f73ea input/dir: ignore .dagger/ directory by default
Signed-off-by: Sam Alba <sam.alba@gmail.com>
2021-06-21 11:37:48 +02:00
Sam Alba
7d9ff46ebd revert input dir management to llb.Copy to fix cache for yarn source pkg
Signed-off-by: Sam Alba <sam.alba@gmail.com>
2021-06-20 08:32:37 +02:00
Andrea Luzzardi
f39a88e644 cue native: environments can reference a module instead of embedding
one.

Fixes #631

Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-06-16 18:58:56 +02:00
Andrea Luzzardi
5f849bb13e cmd: dagger input edit -> dagger edit
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-06-16 18:58:56 +02:00
Sam Alba
09617b89da core: replaced persistent cache mounts with directory
Signed-off-by: Sam Alba <sam.alba@gmail.com>
2021-06-09 15:22:06 +02:00
Sam Alba
7b4421b9a0 cmd/doc: boiler plate and for inputs / outputs scanning
Signed-off-by: Sam Alba <sam.alba@gmail.com>
2021-06-04 09:01:32 +02:00
Andrea Luzzardi
c95b5d2bae docs: deploy using dagger
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-06-02 15:45:51 -07:00
Sam Alba
06a515f496 environments only ignore the compute layer if it is empty instead of logging a fatal error
Signed-off-by: Sam Alba <sam.alba@gmail.com>
2021-06-01 14:14:45 +02:00
Sam Alba
efb4ee209e cmd/up: disable output list when not a tty + better error handling
Signed-off-by: Sam Alba <sam.alba@gmail.com>
2021-05-31 18:25:58 +02:00
Sam Alba
6e3ec02ceb added support for output scanning
Signed-off-by: Sam Alba <sam.alba@gmail.com>
2021-05-31 15:24:42 +02:00
Andrea Luzzardi
b627b4bc88 add support for excludes in input dir
This adds support for `--include` and `--exclude` for directory inputs.

For instance, this is what you would want to use when passing dagger
repository as an input:

```
inputs:
    repository:
        dir:
            path: .
            exclude:
                - '**/node_modules'
                - cmd/dagger/dagger
                - cmd/dagger/dagger-debug
```

Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-05-28 16:22:32 -07:00
Andrea Luzzardi
9c0e2d1d95 buildkit secrets support
- Secrets are never exposed in plaintext in the Cue tree. `dagger query`
  won't dump secrets anymore, Cue errors won't contain them either.
- BuildKit-native secrets support through a new `mount` type. This
  ensures secrets will never be part of containerd layers, buildkit
  cache and generally speaking will never be saved to disk in plaintext.
- Updated netlify as an example
- Added tests
- Changed the Cue definition of a secret to:

```
	@dagger(secret)

	id: string
}
```

This is to ensure both that setting the wrong input type on a secret
(e.g. `dagger input text`) will fail, and attempting to misuse the
secret (e.g. interpolating, passing as an env variable, etc) will also
fail properly.

Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-05-27 19:14:08 -07:00
Sam Alba
eba35e285b environment inputs scanning can now scan including the user inputs with the plan
Signed-off-by: Sam Alba <sam.alba@gmail.com>
2021-05-27 18:45:24 +02:00
Andrea Luzzardi
af776b8abe cleanup: move packages to top level, change vanity URL
Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>
2021-05-25 16:54:00 -07:00