64 lines
3.4 KiB
Markdown
64 lines
3.4 KiB
Markdown
|
# Prototype 69-dagger-archon
|
||
|
|
|
||
|
|
This is an improved version of 64-blagger with the cuellb pattern, using
|
||
|
|
everything we learned in 64,64,66.
|
||
|
|
|
||
|
|
This prototype marks a strong convergence between SH and AL on the high-level DX and architecture
|
||
|
|
of the future OSS project. The next step is to get to a working POC implementation together,
|
||
|
|
that can run a configuration written by SA, end-to-end, and make SA happy with the DX :)
|
||
|
|
|
||
|
|
## Points of convergence:
|
||
|
|
|
||
|
|
### 1. Cue DX: the cuellb pattern
|
||
|
|
|
||
|
|
The "cuellb" pattern is a promising DX for writing delivery workflows in Cue.
|
||
|
|
On the one hand, the raw arrays are verbose, but they are easy to understand. And they
|
||
|
|
can be abstracted away by the community as they wish, using the native features of Cue.
|
||
|
|
TO quote AL: "It's easier to take something verbose and simple, and make it less verbose,
|
||
|
|
than to take something concise and complex, and make it simple."
|
||
|
|
|
||
|
|
### 2. Integration of SAAS features: runtime+engine
|
||
|
|
|
||
|
|
The OSS project must combine features from 3 codebases: `bl-runtime`, `bl-api`, and `bl`.
|
||
|
|
And it must combine them in a way that allows maximum code sharing between SAAS and OSS,
|
||
|
|
so we don't have to do everything twice.
|
||
|
|
|
||
|
|
So in addition to defining a Cue DX and runtime, we also need to define how `dagger` will
|
||
|
|
implement (or avoid implementing) saas features like settings, secrets, connectors, stacks,
|
||
|
|
envs, slugs, job history, provider catalog, etc.
|
||
|
|
|
||
|
|
A promising approach to do this is to split the OSS project in 2 components:
|
||
|
|
|
||
|
|
- 1) the runtime which loads and executes cue confiurations, and
|
||
|
|
- 2) the engine which orchestrates the runtime, and all its inputs and outputs: config repositories,
|
||
|
|
settings, dependencies, previous state, execution history, etc.
|
||
|
|
The engine is also responsible for end-user interaction.
|
||
|
|
(see 64-blagger/README.md for more details on this architecture).
|
||
|
|
|
||
|
|
### 3. Possible performance gains
|
||
|
|
|
||
|
|
One consequence of the cuellb model + engine/runtime split, is that it becomes possible (we think)
|
||
|
|
to compile a dagger job to a single LLB slug, and run it once.
|
||
|
|
This has several benefits, including making the runtime simpler, and removing the hard dependency
|
||
|
|
on a registry (which opens the door to new use cases in local development).
|
||
|
|
|
||
|
|
One possible benefit is performance. In theory, single-pass llb compilation is automatically faster
|
||
|
|
because it removes the multiple round-trips between 1) cue eval 2) buildkit run 3) registry push 4) registry pull.
|
||
|
|
The larger the configuration, the more round-trips, and the bigger the potential performance gain in dagger.
|
||
|
|
|
||
|
|
*[SH]* I have done lots of research work on this (prototypes 64-68) and the results are very encouraiging.
|
||
|
|
I have strong conviction that the performance benefits are huge. But I only have bits and pieces of POC implementations,
|
||
|
|
each focusing on a small aspect of the problem. The next step is to implement an end-to-end POC, and run a crude benchmark.
|
||
|
|
|
||
|
|
|
||
|
|
### 4. Next step: working end-to-end prototype
|
||
|
|
|
||
|
|
We agree on a possible high-level architecture, DX and UX, built on several hypothesis:
|
||
|
|
is cuellb really a good DX in practice? can dir refs really be eliminated?
|
||
|
|
is single-pass llb jit really feasible, and if so does it solve our performance issues?
|
||
|
|
how easy will it be to port Blocklayer to dagger? How much work to migrate beta users?
|
||
|
|
|
||
|
|
Now is the time to test these hypothesis with a working end-to-end implementation.
|
||
|
|
There are many remaining questions but we have enough alignment to find the answers
|
||
|
|
together.
|