nodata/README.md

# nodata

Nodata is a simple binary that consists of two parts:

1. Data ingest
2. Data storage
3. Data aggregation
4. Data API / egress

## Data ingest

Nodata presents a simple protobuf grpc api for ingesting either single events or batch

## Data storage

Nodata stores data locally in a parquet partitioned scheme

## Data aggregation

Nodata accepts wasm routines for running aggregations over data to be processed

## Data Egress

Nodata exposes aggregations as apis, or events to be sent as grpc streamed apis to a service.

# Architecture

## Data flow

Data enteres nodata

1. Application uses SDK to publish data
2. Data is sent over grpc using, a topic, id and data
3. Data is sent to a topic
4. A broadcast is sent that said topic was updated with a given offset
5. A client can consume from said topic, given a topic and id
// 6. We need a partition in here to separate handling between partitions and consumer groups
6. A queue is running consuming each broadcast message, assigning jobs for each consumer group to delegate messages

## Components

A component is a consumer on a set topic, it will either act as a source, sink or a tranformation between topics. It can declare topics, use topics, transform data and much more.

A topic at its most basic is a computational unit implementing a certain interface, source, sink, transformation. 

The most simple is a source and sink, where we respectively push or pull data from the topics.

A component implements either or all of 3 sdk interfaces

1. Create a new sample rust application
2. Add dependency nodata-component
3. In the main functin use: nodata_component::component
4. Implement the interfaces you want to use
5. Build the application into a dockerfile, optionally use the nodata cli to build the app
6. Register the application as a component
7. nodata client add-component --image docker.io/kjuulh/nodata-example-transform:latest --tranform `<input-topic>:<output-topic>`