Instrumentation Guide

We're updating our documentation, so the presented info might not be the most recent.

The SDK should initiliazed in your application where data consumption or production takes place. The SDK is responsible for sending data to the Streamdal server, which then processes it and returns a response.

SDK flow

  1. Initializing Streamdal Client: The SDK’s streamdal.New() method is invoked with a configuration object, initializing a new Streamdal client.
  2. Processing a Request: The sc.Process() method processes a request using the Streamdal client, detailing operation specifics and data to be handled.
  3. Error Handling (Again): Errors from the processing method are checked similarly.

Streamdal Client Configuration

Note

This example uses the Golang SDK.

The configuration parameters are similar across all SDKs, but the syntax may vary slightly.

When initializing the Streamdal client with streamdal.New(), you pass a configuration object to customize its behavior. Here’s a breakdown of the parameters in that configuration:

ParameterDescriptionExampleRequired?
StreamdalURLThe URL of the Streamdal server instance."localhost:8082"
StreamdalTokenThe authentication token for the Streamdal server."streamdal"
ServiceNameA name that represents the specific service in your infrastructure using this client."billing-svc"
StepTimeoutSets the maximum duration for Wasm execution in each step.time.Millisecond * 10
PipelineTimeoutDefines the total allowable execution time for the entire pipeline.time.Millisecond * 100
DryRunA switch to toggle between actual execution and a logging-only mode. If set to true, it won’t actually execute but will only log.false
ShutdownCtxContext used to manage the lifecycle and potential shutdown of the Streamdal client.context.Background()

Alternatively, you can also configure many of these settings via environment variables. Refer to the Go SDK README for more details.

Authentication

All of the SDKs will require you to provide a “Streamdal (Auth) Token” as one of the parameters in the configuration object. This token is used to authenticate the SDK in your app with the Streamdal server.

This is the same token as the STREAMDAL_SERVER_AUTH_TOKEN environment variable that you set when running the Streamdal server.

Process() Parameters

Let’s delve into the specifics of the Process() method:

AttributeDescriptionTypeRequired?
NameA descriptive name related to the producer or consumer in focus. For instance: billing-processor, db-flusher, or doc-handler.String✅ Yes
OperationSpecifies the nature of the operation, whether it’s a consumer or producer.Enum✅ Yes
ComponentRepresents the data source or destination. Examples include main-kafka, slow-db, or salesforce-api.String✅ Yes
DataThe actual data payload that’s being consumed or produced.Byte array✅ Yes

How the SDK parameters defines the Data Graph

Audience

Golang SDK Example

Using the SDK in your Golang applications is straightforward. Here’s a simple example to get you started:

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/streamdal-go-client"
)

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	sc, err := streamdal.New(&streamdal.Config{
		StreamdalURL:     "localhost:8082",
		StreamdalToken:   "streamdal",
		ServiceName:     "billing-svc",
		ShutdownCtx:     ctx,
	})
	if err != nil {
		panic(err)
	}

	resp, err := sc.Process(ctx, &streamdal.ProcessRequest{
		OperationType: streamdal.OperationTypeConsumer,
		OperationName: "new-order-topic",
		ComponentName: "kafka",
		Data:          []byte(`{"object": {"field": true}}`),
	})
	if err != nil {
		panic(err)
	}

	fmt.Printf("%#v\n", resp)
}

Available SDKs

The SDK is currently available in the following languages:


With this guide at your side, integrating Streamdal into your operations becomes a piece of cake. Remember, the true magic unfolds when Streamdal’s capabilities meld seamlessly with your application’s data flows. So, jump in, integrate, and watch your data operations transform!