Glossary

We're updating our documentation, so the presented info might not be the most recent.

Streamdal uses a variety of terms that can have various meanings across industries and use cases. This section exists to help clear up any ambiguity around Streamdal core-components or functionality.

Audience

A term we have come up with that identifies a specific service + operation type (consumer or producer), name for the operation type and component. The audience can be referenced using the following slug:

  • $service_name:$operation_type:$operation_name:$component_name

The only exposure to the audience concept that an SDK has is via the NewAudience() internal gRPC method which is used by the SDK to announce the existence of a new audience. The audience contains the elements needed to be able to draw the most basic Data Graph entry, eg: service ↔ producer ↔ component.

Condition

A condition is associated with a step. It defines how the SDK should react to the result of a step execution. The condition might be configured to ABORT the pipeline if a step results in a failure or an error.

Consumer

A consumer generally refers to any function, method, or component that takes input (or data) and does something with it. A consumer consumes data, but may not necessarily produce output. Notice how the arrow is going up on the Data Graph.

For example, a function that logs data to the console, writes data to a file, or updates a UI element can be considered a consumer because it acts on the data but doesn’t give you something to work with afterward.

Sometimes a consumer will take in data, process it, and produce new or transformed data - this hybrid of “consumer/producer” functionality is sometimes referred to as a “transformer.”

Destination

A destination refers to any location, system, or facility where data is sent, stored, or displayed after it has been processed or generated by an application.

While sources provide or feed data into systems (input), destinations are where data ends up (output). Some examples of destinations would be:

  • Databases
  • Files
  • API Endpoints
  • Message Queues
  • User Interfaces
  • Data Warehouses & Data Lakes
  • Cloud-based Storage Solutions
  • Email and Notification Systems
  • IoT or External Devices
  • Logging Systems

Data Graph

The data graph is a graphic layout showcasing the relationship between consumers, producers, and topics in each pipeline.

It’s the pretty figma-like portion of the Console UI:

Streamdal Data Graph
Streamdal Data Graph
Open in new tab ⇱

It will show the direction of the flow of data (whether it is a read or a write), and will always have:

Firewall for Data

Streamdal does not configure/reconfigure or affect your network firewall settings. When we say “firewall for data,” it is a figure of speech to convey the power that creating rules and attaching pipelines to the flow of your data establishes.

Our data governance sections somewhat cover these beta functionalities, and will be much more verbose once these features are production.

Observability

Observability can be a really loaded term, which is why we also cover this in our FAQ and have a whole section dedicated to it. Depending on what you’re working with or who you’re speaking to, it can mean many different things.

When we say observability, we mean:

The ability to simply see the actual real-time data, its schema, and the services generating data or interacting with it.

DORA (DevOps Research and Assessment) also has a good definition on observability within this context, which defines it as:

…tooling or a technical solution that allows teams to actively debug their system. Observability is based on exploring properties and patterns not defined in advance.

We see Tail as a crucial part of this, and the primary function enabling data observability with Streamdal. Here are the following features we consider as “observability” features:

Pipeline or Ruleset

A pipeline is a set of steps and conditions. A user would configure one or more steps for a specific producer or consumer - the result would be a pipeline (that is sent to the SDK client via gRPC).

Check out the Pipeline guide for more information.

Producer

A producer typically refers to a function, method, or component that generates or produces data. It can be thought of as a source of data in a data flow pipeline. Notice how the arrow is going down on the Data Graph.

Sometimes a producer will take in data, process it, and produce new or transformed data - this hybrid of “producer/consumer” functionality is sometimes referred to as a “transformer.”

Service

A service can have various meanings depending on the context, but it generally refers to a distinct functional unit that provides specific functionalities or operations that are well-defined and independent.

Here is some common contexts in which the term “service” might be used:

  • Web Services
  • Microservices
  • Windows Services (or Daemon in UNIX)
  • Cloud Services (Like IaaS, PaaS, or SaaS services)

Specifically to Streamdal, when instrumenting the SDK, ServiceName is a required field to be defined, which gives your “service” a name on the Data Graph. Our Golang example shows how this works.

Shim

A wrapper library for popular libraries that incorporate the usage of our SDK to simplify integration and adoption of Streamdal.

  • Example: we would develop a shim for Python’s HTTP request library called requests. This shim would be a lightweight fork of requests that would call on streamdal_sdk.Process(..) whenever the user would POST, PUT, DELETE, or GET data. The shim simplifies and speeds up initial onboarding and integration.

Source

A source refers to any location, system, or facility where data is generated, stored, or maintained and from which data can be retrieved for further use or processing.

Data sources can range from databases to external APIs, and their nature can be both static or dynamic. Some examples of a data source would be:

  • Databases
  • Files
  • APIs (Application Programming Interfaces)
  • Message Queues (Such as RabbitMQ, SQS, Kafka, etc)
  • Streaming Data
  • Data Warehouses & Data Lakes
  • Cloud-based Storage Solutions
  • ERP (Enterprise Resource Planning) Systems
  • Content Management Systems (CMS)

Rule or Step

A rule (or a step) is a single unit of “work” that the SDK will run against a payload.

  • Example: The user might create a step that ensures that the payload always contains a field called id which MUST be of the uuid format.

Wasm

Wasm is WebAssembly. We use this technology in steps, and it’s a principal element of Streamdal functionality.

A step consists of a configuration (ie. what is the name of this step, what Wasm function should it run with what parameters, etc.) - AND the actual Wasm bytes that will be executed by the SDK.