Streamdal Open Source

We're updating our documentation, so the presented info might not be the most recent.

We make significant use of open source across many of the Streamdal components. Here is a non-exhaustive list of some of the stuff we use in our projects:

CLI

• Kong: A nice CLI argument and flag library in Golang which leverages struct tags for declaration
• Log: A fun and colorful logger
• Tview: An excellent terminal UI library, used by k9s and many other well-known projects
• Tcell: Used for working with the terminal - used by tview under the hood
• Godotenv: A staple in many of our Go-based projects - loads .env files (if they’re there)

Console UI

• Deno-Fresh — Typescript runtime and full stack web framework
• Preact — Fast JSX for rendering and templating
• Protobuf-ts — Typescript gRPC and protocol buffers for Deno
• Zod — Schema declaration and validation
• React-flow — Interactive node-based UI
• Flowbite — Tailwind compatible UI widgets
• Tailwindcss — CSS utility framework

Server

• Go: This is what the server is written in

• gRPC: Nearly all Streamdal components use gRPC for communication. We use both unary and streaming functionality for communicating with SDKs, communicating with other cluster nodes, and talking with the Console UI

• Protocol Buffers: Our protos are open source and you can find them on GitHub. As with gRPC, Protobuf is a crucial part of the Streamdal ecosystem

  It’s used for:

  • Defining internal and external APIs
  • Establishing a stable API spec used by SDKs
  • Facilitating Wasm functionality
  • And a many other uses

• UUID: This library generates and inspects UUIDs based on RFC 4122 and DCE 1.1: Authentication and Security Services.

• Redis: Used for internal communications between cluster nodes and as the primary storage mechanism. We specifically use redis for Pubsub and its traditional key/value functionality.

  • Go-Redis: The Redis lib we use

Streamdal Trivia:

We used etcd and NATS in the beginning, but they both proved to be too much for what we needed. Also, most folks recognize redis and won’t raise eyebrows when they see it.

• Kong: A really awesome CLI arg and flag library that uses struct tags for declaration (More trivia: alecthomas also wrote the excellent kingpin library)
• Ginkgo & Gomega: Our choice of BDD test framework + matcher. The server is complex, highly concurrent, and has many moving parts. These libs have helped us write more thorough tests that are still easy to grok (even if you haven’t looked at them in a while)
• Logrus: A popular, structured logging library
• Httprouter: A fast, lightweight muxer we use for our REST API. While most internal communication in Streamdal happens with gRPC, the pieces that are not are powered by this library
• Ratecounter: We use this library to calculate rates (such as throughput over time)
• Go-director: This library enables us to declare for loops as injectable dependencies, which greatly simplifies testing

SDKs

Currently we support Go, Node and Python. We have plans to add more SDKs in the future. Check the roadmap for more information.

Wasm

The rule functionality of Streamdal uses Wasm extensively under the hood. Most of our Wasm modules are open source.

• Rust — The language most of the modules are written in
• Serde — For working with JSON
• Protobuf - Used for facilitating communication between the executor (SDKs) and the Wasm modules themselves. In other words, we encode requests in Protobuf, pass them to Wasm, and the modules return Protobuf-encoded responses

Website & Other Tools

• Astro - Used for the website and documentation
• Algolia - Used for indexing and search for the docs
• Excalidraw - For technical diagrams and whiteboarding
• kube-prometheus-stack - A collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator