The Streamdal Server

We're updating our documentation, so the presented info might not be the most recent.


The Server ensures that rules defined in the Console UI are fluently transformed into real-time operations. Boasting high availability, it doesn’t merely store rules. Instead, it transmutes them into WebAssembly (WASM), a format impeccably deciphered and executed by the SDK.

For most up-to-date information, please refer to the Streamdal Server GitHub repository.

Core Features

  • Bridge Between UI and Execution: Seamlessly channels intentions from rule creation into actionable processes.
  • Wasm Integration: Efficiently transcribes rules into Wasm, optimizing real-time operations for speed and accuracy.
  • High Availability: Engineered for reliability, ensuring continuous and uninterrupted data operations.

Server’s Role & Functionality

The server plays a pivotal role in the Streamdal ecosystem - it is the bridge between the Console and the SDKs.

More specifically, the server is responsible for:

  1. Translating pipelines into WebAssembly rules for the SDKs
  2. Distributing configuration updates to SDKs
  3. Keeping track of “active” (currently running) clients (SDKs)
  4. Storing pipeline configuration
  5. Providing an API for the Console to allow configuring pipelines, steps, SDKs
  6. Aggregating metrics from SDKs
  7. .. and many other things

For a more thorough understanding of the server’s role, refer to the server repo.

Multi-Component Communication

The server is pivotal in liaising between various components, ensuring streamlined communication and data operations.

The server talks gRPC with both the Console and SDKs. The server does not perform any outbound connections, instead, the Console and SDKs establish a connection with the server.

Authentication

In order for the Console and SDKs to be able to communicate with the server, they all must share the same authentication token.

To set the authentication token, set the STREAMDAL_SERVER_AUTH_TOKEN environment variable to a unique value.

Similarly, all SDKs and console must use this token when talking to the server.

Configuration

The server is configured using environment variables that use a STREAMDAL_SERVER prefix.

At minimum, the following environment variables MUST be set:

  1. STREAMDAL_SERVER_AUTH_TOKEN - the authentication token
  2. STREAMDAL_SERVER_NODE_NAME - the unique name for this service instance
  3. STREAMDAL_SERVER_REDIS_URL - the URL of the Redis instance or cluster
Note

Please refer to the server README for the most up-to-date and complete list of configuration options.

Redis

The server uses Redis for the following purposes:

  1. To track actively connected and disconnected clients (SDKs)
  2. To store pipeline configurations
  3. To store the WebAssembly rules
  4. To store SDK metrics

All Redis keys used by the server will be prefixed with streamdal_.

If Redis goes away during the server’s operation, the server will continue to operate normally and will automatically reconnect to Redis once it becomes available again.

Distributed by Default

The Streamdal server is “distributed by default” - it is highly scalable, fault-tolerant, and designed to handle a large number of clients and pipelines.

You can run as many server instances as you like as the server is stateless which means that there is no need for leader election or requiring an odd number of nodes.

All you have to do to launch additional server instances is:

  1. Specify a unique node name
  2. Point the server instance to the same Redis cluster/instance

Server High Level Architecture

Below is a visual representation of the server architecture:

  • Three Streamdal Servers in HA, each equipped with an Envoy Sidecar port 8083 to handle gRPC web traffic.
  • Redis operates in a failover mode, comprising one master and three replicas. Streamdal servers communicate with the Redis master for task distribution.
  • The Streamdal Console illustrates how the Streamdal servers connect and serve gRPC web traffic.
  • The Client App with SDK is an external entity, representing a typical client application that integrates with the Streamdal servers via API/SDK calls directly over grpc on port 8082. It’s important to note that this Client App is not deployed by the Helm chart but is illustrated to demonstrate potential end-user application integration.
Streamdal Server Arch

The Streamdal Server, integral to Streamdal’s narrative, is the linchpin between rule creation and real-time execution. Ensuring data compliance, transformation, and security isn’t merely theoretical but is actualized with every operational tick.