2024-08-10 00:09:11 +02:00
|
|
|
# nodata
|
2024-08-10 00:17:12 +02:00
|
|
|
|
|
|
|
|
Nodata is a simple binary that consists of two parts:
|
|
|
|
|
|
|
|
|
|
1. Data ingest
|
|
|
|
|
2. Data storage
|
|
|
|
|
3. Data aggregation
|
|
|
|
|
4. Data API / egress
|
|
|
|
|
|
|
|
|
|
## Data ingest
|
|
|
|
|
|
|
|
|
|
Nodata presents a simple protobuf grpc api for ingesting either single events or batch
|
|
|
|
|
|
|
|
|
|
## Data storage
|
|
|
|
|
|
|
|
|
|
Nodata stores data locally in a parquet partitioned scheme
|
|
|
|
|
|
|
|
|
|
## Data aggregation
|
|
|
|
|
|
|
|
|
|
Nodata accepts wasm routines for running aggregations over data to be processed
|
|
|
|
|
|
|
|
|
|
## Data Egress
|
|
|
|
|
|
|
|
|
|
Nodata exposes aggregations as apis, or events to be sent as grpc streamed apis to a service.
|
2024-08-16 00:25:43 +02:00
|
|
|
|
|
|
|
|
# Architecture
|
|
|
|
|
|
|
|
|
|
## Data flow
|
|
|
|
|
|
|
|
|
|
Data enteres nodata
|
|
|
|
|
|
|
|
|
|
1. Application uses SDK to publish data
|
|
|
|
|
2. Data is sent over grpc using, a topic, id and data
|
|
|
|
|
3. Data is sent to a topic
|
|
|
|
|
4. A broadcast is sent that said topic was updated with a given offset
|
|
|
|
|
5. A client can consume from said topic, given a topic and id
|
|
|
|
|
6. A queue is running consuming each broadcast message, assigning jobs for each consumer group to delegate messages
|
