Relaying data using plumber is the most reliable and performant way to get data into Streamdal.
You can launch
plumber relays in multiple ways:
plumberin single-relay mode via CLI
- Best for quick, one-offs
plumberas a docker container
- Best for ephemeral workloads
plumberin server mode
- Best for production
The following examples show how to run
plumber in single relay mode.
For production deployments, we suggest to deploy
plumber running in server mode.
plumber relay kafka \ --address "your-kafka-address.com:9092" \ --token YOUR-COLLECTION-TOKEN-HERE \ --topics orders \ --tls-skip-verify
In this example, all messages from kafka topic new_orders will be automatically sent to the collection with the specified relay token.
docker run --name plumber-rabbit -p 8080:8080 \ -e PLUMBER_RELAY_TYPE=rabbit \ -e PLUMBER_RELAY_TOKEN=$YOUR-BATCHSH-TOKEN-HERE \ -e PLUMBER_RELAY_RABBIT_EXCHANGE=my_exchange \ -e PLUMBER_RELAY_RABBIT_QUEUE=my_queue \ -e PLUMBER_RELAY_RABBIT_ROUTING_KEY=some.routing.key \ -e PLUMBER_RELAY_RABBIT_QUEUE_EXCLUSIVE=false \ -e PLUMBER_RELAY_RABBIT_QUEUE_DURABLE=true \ streamdal/plumber \ rabbit
In this example, all messages sent to
my_exchange that match the routing key
some.routing.key will be sent to
At that point, plumber will pick up the messages and send them to Streamdal using the specified token.
A full suite of environment variables are provided in ENV.md for configuring plumber’s relay mode.
Example of running plumber via kubernetes
apiVersion: apps/v1 kind: Deployment metadata: name: plumber-deployment spec: selector: matchLabels: app: plumber replicas: 1 template: metadata: labels: app: plumber spec: containers: - name: plumber image: streamdal/plumber:latest command: ["/plumber-linux", "relay", "kafka"] args: ["--stats-enable"] ports: - containerPort: 9191 env: - name: PLUMBER_RELAY_TOKEN value: "--- COLLECTION TOKEN HERE ---" - name: PLUMBER_RELAY_KAFKA_ADDRESS value: "kafka.server.com:9092" - name: PLUMBER_RELAY_KAFKA_TOPIC value: "new-orders" - name: PLUMBER_RELAY_KAFKA_GROUP_ID value: "plumber" resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" cpu: "500m"
More examples of relaying from various systems can be found in EXAMPLES.md.
When should you use this API?
Plumber is the easiest way to relay throughput heavy workloads and should be used by anyone wanting to get up and running quickly.
plumber uses gRPC under the hood to communicate with Streamdal’s collectors.
You should be able to comfortably reach 25K-50K messages/sec on a single
plumber instance. To reach higher levels, you should run
plumber in cluster server mode and launch 2+ replicas of
Make sure to use the same consumer group if relaying for backends such as Kafka or NATS.