How Streamdal Works

We're updating our documentation, so the presented info might not be the most recent.

Streamdal operates by integrating one of our open-source SDKs (or, importing a Streamdal-enhanced library) into your code. The SDK then communicates with it’s server and can be managed through it’s Console UI.

Typical Workflow

To start using Streamdal, you’ll typically do the following:

Streamdal workflow

Server and SDK: A Closer Look

You don’t need to go really deep to use Streamdal. But, understanding the interplay between the server and SDK can be very helpful:

  1. Registration: Post integration, the SDK registers with the Streamdal server, affirming its status and its readiness to exchange data.
  2. Communication via gRPC: The SDK and server converse over gRPC, ensuring efficient bidirectional streaming and reliable data transmission.
  3. Metrics Reporting: After registration, the SDK sends metrics to the server. These insights reveal data flow patterns, potential bottlenecks, and other key metrics.
  4. Fetching Rules: The SDK retrieves the latest Wasm-compiled rules from the server. These rules, vital for data processing, ensure immediate reflection of any changes in data logic within the SDK.
SDK flow

Access the Console UI

Once the SDK and Server are connected you will manage your data via the Console UI

  1. Manage Pipelines: Create, edit, and delete pipelines.
    1. You can attach/remove as many pipelines as needed
  2. View Metrics: View metrics and data flow patterns.
  3. Tail: View Data flow in real-time.
  4. Discover Schemas: View and edit schemas.
Console UI
Console UI
Open in new tab ⇱

How It All Works Together

This is a high-level overview of how Streamdal’s components all work together.

How It all Works Together Diagram