diff --git a/README.md b/README.md index 4d3c96ae..ee6694ae 100644 --- a/README.md +++ b/README.md @@ -1,63 +1,63 @@ -# Prototype 69-dagger-archon +# dagger: the devops superglu, by the creators of docker -This is an improved version of 64-blagger with the cuellb pattern, using -everything we learned in 64,64,66. +Dagger is an automation platform that lets software teams bind all their tools and infrastructure together into a unified supply chain. -This prototype marks a strong convergence between SH and AL on the high-level DX and architecture -of the future OSS project. The next step is to get to a working POC implementation together, -that can run a configuration written by SA, end-to-end, and make SA happy with the DX :) +Thanks to its vast catalog of adapters, developed in the open and curated for safety and quality, +you can drop Dagger into an existing stack *without changing it*, and immediately start automating the most repetitive and complex tasks. -## Points of convergence: +And most importantly, Dagger is *programmable*. Thanks to a powerful scripting environment, +anyone with basic programming knowledge can extend Dagger with their own custom adapters and workflows. +Whether you're a seasoned SRE building a custom PAAS for your organization, a hobbyist on a fun +over-engineered week-end project, or a developer trying to setup CICD because, well, someone has to do it.. +There's a whole community of fellow automation enthusiasts ready to help you write your first Dagger script. -### 1. Cue DX: the cuellb pattern +## Usage examples -The "cuellb" pattern is a promising DX for writing delivery workflows in Cue. -On the one hand, the raw arrays are verbose, but they are easy to understand. And they -can be abstracted away by the community as they wish, using the native features of Cue. -TO quote AL: "It's easier to take something verbose and simple, and make it less verbose, -than to take something concise and complex, and make it simple." +A few examples of how Dagger is used in the wild: -### 2. Integration of SAAS features: runtime+engine - -The OSS project must combine features from 3 codebases: `bl-runtime`, `bl-api`, and `bl`. -And it must combine them in a way that allows maximum code sharing between SAAS and OSS, -so we don't have to do everything twice. - -So in addition to defining a Cue DX and runtime, we also need to define how `dagger` will -implement (or avoid implementing) saas features like settings, secrets, connectors, stacks, -envs, slugs, job history, provider catalog, etc. - -A promising approach to do this is to split the OSS project in 2 components: - -- 1) the runtime which loads and executes cue confiurations, and -- 2) the engine which orchestrates the runtime, and all its inputs and outputs: config repositories, -settings, dependencies, previous state, execution history, etc. -The engine is also responsible for end-user interaction. -(see 64-blagger/README.md for more details on this architecture). - -### 3. Possible performance gains - -One consequence of the cuellb model + engine/runtime split, is that it becomes possible (we think) -to compile a dagger job to a single LLB slug, and run it once. -This has several benefits, including making the runtime simpler, and removing the hard dependency -on a registry (which opens the door to new use cases in local development). - -One possible benefit is performance. In theory, single-pass llb compilation is automatically faster -because it removes the multiple round-trips between 1) cue eval 2) buildkit run 3) registry push 4) registry pull. -The larger the configuration, the more round-trips, and the bigger the potential performance gain in dagger. - -*[SH]* I have done lots of research work on this (prototypes 64-68) and the results are very encouraiging. -I have strong conviction that the performance benefits are huge. But I only have bits and pieces of POC implementations, -each focusing on a small aspect of the problem. The next step is to implement an end-to-end POC, and run a crude benchmark. +- Deploy a new API to AWS Elastic Container Service while continuing to deploy the main app on Heroku +- Deploy lightweight staging environments on-demand for QA, integration testing or product demos. +- Run integration tests on a live production-like deployment, automatically, for each pull request. +- Deploy the same app on Netlify for testing, and on Kubernetes for production +- Replace a 500-line deploy.sh with a 10-line configuration file +- Control sprawl of serverless functions on AWS, Google Cloud, Cloudflare, Netlify etc. by gradually + moving them to a generic interface, and switching backend at will. +- When the ML team uploads a new model to their S3 bucket, automatically incorporate it into staging + deployments, but not into production until manual confirmation! +- Rotate database credentials, and automatically re-deploy all staging environments with the new credentials. +- Allocate cool auto-generated URLs to development instances, and automatically configure your DNS, + load-balancer and SSL certificate manager to route traffic to them. +- Orchestrate application deployment across 2 infrastructure siloes, one managed with CloudFormation, the other with Terraform. +- Migrate from Helm to Kustomize, without disrupting next week's big release. -### 4. Next step: working end-to-end prototype +## Comparison to other automation platforms -We agree on a possible high-level architecture, DX and UX, built on several hypothesis: -is cuellb really a good DX in practice? can dir refs really be eliminated? -is single-pass llb jit really feasible, and if so does it solve our performance issues? -how easy will it be to port Blocklayer to dagger? How much work to migrate beta users? -Now is the time to test these hypothesis with a working end-to-end implementation. -There are many remaining questions but we have enough alignment to find the answers -together. +### CICD + +Github, Gitlab, Jenkins, Spinnaker, Tekton + +### Build systems + +Bazel, Nix, Skopeo + +### Infrastructure automation + +Terraform, Pulumi, Ansible + +### Traditional scripting + +Bash, Make, Python + +### PaaS + +Heroku, Elastic Beanstalk, Cloud Foundry, Openshift + +### Kubernetes management + +Kustomize, Helm, jsonnet + +### Gitops + +Flux, ...