First pass at a README
This commit is contained in:
Solomon Hykes
parent
30f75da114
commit
22500ad9db
106
README.md
106
README.md
@ -1,63 +1,63 @@
|
|||||||
# Prototype 69-dagger-archon
|
# dagger: the devops superglu, by the creators of docker
|
||||||
|
|
||||||
This is an improved version of 64-blagger with the cuellb pattern, using
|
Dagger is an automation platform that lets software teams bind all their tools and infrastructure together into a unified supply chain.
|
||||||
everything we learned in 64,64,66.
|
|
||||||
|
|
||||||
This prototype marks a strong convergence between SH and AL on the high-level DX and architecture
|
Thanks to its vast catalog of adapters, developed in the open and curated for safety and quality,
|
||||||
of the future OSS project. The next step is to get to a working POC implementation together,
|
you can drop Dagger into an existing stack *without changing it*, and immediately start automating the most repetitive and complex tasks.
|
||||||
that can run a configuration written by SA, end-to-end, and make SA happy with the DX :)
|
|
||||||
|
|
||||||
## Points of convergence:
|
And most importantly, Dagger is *programmable*. Thanks to a powerful scripting environment,
|
||||||
|
anyone with basic programming knowledge can extend Dagger with their own custom adapters and workflows.
|
||||||
|
Whether you're a seasoned SRE building a custom PAAS for your organization, a hobbyist on a fun
|
||||||
|
over-engineered week-end project, or a developer trying to setup CICD because, well, someone has to do it..
|
||||||
|
There's a whole community of fellow automation enthusiasts ready to help you write your first Dagger script.
|
||||||
|
|
||||||
### 1. Cue DX: the cuellb pattern
|
## Usage examples
|
||||||
|
|
||||||
The "cuellb" pattern is a promising DX for writing delivery workflows in Cue.
|
A few examples of how Dagger is used in the wild:
|
||||||
On the one hand, the raw arrays are verbose, but they are easy to understand. And they
|
|
||||||
can be abstracted away by the community as they wish, using the native features of Cue.
|
|
||||||
TO quote AL: "It's easier to take something verbose and simple, and make it less verbose,
|
|
||||||
than to take something concise and complex, and make it simple."
|
|
||||||
|
|
||||||
### 2. Integration of SAAS features: runtime+engine
|
- Deploy a new API to AWS Elastic Container Service while continuing to deploy the main app on Heroku
|
||||||
|
- Deploy lightweight staging environments on-demand for QA, integration testing or product demos.
|
||||||
The OSS project must combine features from 3 codebases: `bl-runtime`, `bl-api`, and `bl`.
|
- Run integration tests on a live production-like deployment, automatically, for each pull request.
|
||||||
And it must combine them in a way that allows maximum code sharing between SAAS and OSS,
|
- Deploy the same app on Netlify for testing, and on Kubernetes for production
|
||||||
so we don't have to do everything twice.
|
- Replace a 500-line deploy.sh with a 10-line configuration file
|
||||||
|
- Control sprawl of serverless functions on AWS, Google Cloud, Cloudflare, Netlify etc. by gradually
|
||||||
So in addition to defining a Cue DX and runtime, we also need to define how `dagger` will
|
moving them to a generic interface, and switching backend at will.
|
||||||
implement (or avoid implementing) saas features like settings, secrets, connectors, stacks,
|
- When the ML team uploads a new model to their S3 bucket, automatically incorporate it into staging
|
||||||
envs, slugs, job history, provider catalog, etc.
|
deployments, but not into production until manual confirmation!
|
||||||
|
- Rotate database credentials, and automatically re-deploy all staging environments with the new credentials.
|
||||||
A promising approach to do this is to split the OSS project in 2 components:
|
- Allocate cool auto-generated URLs to development instances, and automatically configure your DNS,
|
||||||
|
load-balancer and SSL certificate manager to route traffic to them.
|
||||||
- 1) the runtime which loads and executes cue confiurations, and
|
- Orchestrate application deployment across 2 infrastructure siloes, one managed with CloudFormation, the other with Terraform.
|
||||||
- 2) the engine which orchestrates the runtime, and all its inputs and outputs: config repositories,
|
- Migrate from Helm to Kustomize, without disrupting next week's big release.
|
||||||
settings, dependencies, previous state, execution history, etc.
|
|
||||||
The engine is also responsible for end-user interaction.
|
|
||||||
(see 64-blagger/README.md for more details on this architecture).
|
|
||||||
|
|
||||||
### 3. Possible performance gains
|
|
||||||
|
|
||||||
One consequence of the cuellb model + engine/runtime split, is that it becomes possible (we think)
|
|
||||||
to compile a dagger job to a single LLB slug, and run it once.
|
|
||||||
This has several benefits, including making the runtime simpler, and removing the hard dependency
|
|
||||||
on a registry (which opens the door to new use cases in local development).
|
|
||||||
|
|
||||||
One possible benefit is performance. In theory, single-pass llb compilation is automatically faster
|
|
||||||
because it removes the multiple round-trips between 1) cue eval 2) buildkit run 3) registry push 4) registry pull.
|
|
||||||
The larger the configuration, the more round-trips, and the bigger the potential performance gain in dagger.
|
|
||||||
|
|
||||||
*[SH]* I have done lots of research work on this (prototypes 64-68) and the results are very encouraiging.
|
|
||||||
I have strong conviction that the performance benefits are huge. But I only have bits and pieces of POC implementations,
|
|
||||||
each focusing on a small aspect of the problem. The next step is to implement an end-to-end POC, and run a crude benchmark.
|
|
||||||
|
|
||||||
|
|
||||||
### 4. Next step: working end-to-end prototype
|
## Comparison to other automation platforms
|
||||||
|
|
||||||
We agree on a possible high-level architecture, DX and UX, built on several hypothesis:
|
|
||||||
is cuellb really a good DX in practice? can dir refs really be eliminated?
|
|
||||||
is single-pass llb jit really feasible, and if so does it solve our performance issues?
|
|
||||||
how easy will it be to port Blocklayer to dagger? How much work to migrate beta users?
|
|
||||||
|
|
||||||
Now is the time to test these hypothesis with a working end-to-end implementation.
|
### CICD
|
||||||
There are many remaining questions but we have enough alignment to find the answers
|
|
||||||
together.
|
Github, Gitlab, Jenkins, Spinnaker, Tekton
|
||||||
|
|
||||||
|
### Build systems
|
||||||
|
|
||||||
|
Bazel, Nix, Skopeo
|
||||||
|
|
||||||
|
### Infrastructure automation
|
||||||
|
|
||||||
|
Terraform, Pulumi, Ansible
|
||||||
|
|
||||||
|
### Traditional scripting
|
||||||
|
|
||||||
|
Bash, Make, Python
|
||||||
|
|
||||||
|
### PaaS
|
||||||
|
|
||||||
|
Heroku, Elastic Beanstalk, Cloud Foundry, Openshift
|
||||||
|
|
||||||
|
### Kubernetes management
|
||||||
|
|
||||||
|
Kustomize, Helm, jsonnet
|
||||||
|
|
||||||
|
### Gitops
|
||||||
|
|
||||||
|
Flux, ...
|
||||||
|
|||||||
Reference in New Issue
Block a user