Frequently Asked Questions

We're updating our documentation, so the presented info might not be the most recent.

Get clarification or answers for the most common questions we receive about Streamdal. Can’t find what you’re looking for, or need more details? Get in touch with us.



    1. What is Streamdal?

  1. Streamdal is an open source observability tool.


    It’s comprised of a server, a UI, and a bunch of Wasm-powered SDKs. By using our SDK and deploying it’s server, you’ll get a dynamic graph of your producers and consumers as they scale up and down, with the ability to tap in and view all the real-time data flowing through them with Tail.


    You can use Tail from the Console UI. Or, for those who prefer a terminal, you can use the CLI.


    The stuff you can use to modify your “in-flight” data such as stripping PII, masking, obfuscating, or validating data is of a certain format or structure, etc is done with rules. Currently, rules and the preventive/data governance features are in beta.

    2. 2. How do I use Streamdal, and where is it deployed?

    1. Import our SDK for your language.
    2. Weave the Process() method into the fabric of your code where data production or consumption takes place.
    3. Deploy the Streamdal server where the services using the Streamdal SDK are running.


    Check out the quickstart guide.


    You can find more details in our guides on instrumentation and deployment.

    3. 3. What language SDKs do you support?

  3. Currently we support:



    We plan on adding more language support in the future, along with libraries/shims for easier instrumentation. Check out the roadmap for more details.

    4. 4. What data sources and destinations are supported?

  4. Any source, and any destination.


    The way Streamdal is instrumented, there isn’t a need for connectors or interfacing with various API’s. The SDK uses a Process() method that is wrapped around the parts of your code where data production and consumption take place.

    5. 5. Will this cause HTTP bottlenecks, or impact my app performance in any way?

  5. The work all happens locally in the SDK, and on the client. The server component is just there to configure things on the client by shipping down rules created in the Console UI.


    This happens with only a negligible resource hit to the client (usally <0.1ms, thanks to Wasm). There is no sending data, or shipping logs, metrics, or traces anywhere.


    Check out our benchmark tests on GitHub.

    6. 6. Why aren’t my services, producers, or consumers showing up on the data graph?

  6. In order for the data graph to visually populate services:


    1. The SDK should be imported and properly instrumented for your language. See the Golang example on importing the SDK.
    2. The server needs to be accessible by the SDK. If you have confirmed the containers are up, try running curl <streamdal-server>:<endpoints>. Check out the various endpoints exposed by the server components for further testing.
      • The STREAMDAL_URL default environment variable value is localhost:9090. If you are running the server somewhere other than where your instrumented services live (which is generally not recommended), you will need to update this value to the correct address.

    7. 7. I see Rules in the Console UI, but they are listed as Beta. Can I use this feature?

  7. Rules are currently in beta along with the other data governance features of Streamdal.


    They are being tested by a select group of contributors, companies, and design partners.


    These features aren’t production-ready! But, if you would like to get test access or become a design partner, contact us or hit us up on
    discord.

    8. 8. What is open source, what is free, and what is paid? Will this change in the future?

  8. The observability aspects and features of Streamdal, including Tail, will always be free and open source using the Apache License v2.0.


    Currently, the data governance features are in beta, but once they are fully fleshed out and production-ready, they may be a paid feature. We haven’t decided which parts of the governance features might need to be paid.


    Regardless of how that decision pans out, we will not be rug-pulling anything we put out as open source.


    Check out our launch manifesto for details on this design decision.

    9. 9. When will you support <this-feature-language-or-library>?

  9. Our roadmap is publicly available on GitHub.


    Don’t see your library, or think a feature timeline should be shifted? Join our community and let us know!

    10. 10. How do I maintain the Data Graph?

  10. The Data Graph is real-time and dynamic.


    What that means is so long as the server component is running, and new services are instrumented with the SDK, the Data Graph will not require maintenance.


    The view you get will always be an accurate representation of data flowing through your systems regardless of scale.

    11. 11. How do I manage schemas?

  11. Currently, there is no need to manage schemas. Once you have instrumented the SDK, you will be able to view the active schemas of data throughout your systems.


    When you weave the Process() method throughout your code, behind the scenes the SDK will infer and decode schemas, and present them in a human-readable format via the Console UI.


    We have big plans to expand schemas in the future. This could include tools, management capabilities, or functionality such as history and diff views.


    Reference the roadmap for more information.

    12. 12. What is Tail?

  12. Tail gets its name from the Unix command and argument tail -f.


    It is a feature that can be used via the Console UI (or CLI) which will allow you to tap into any of your producers or consumers and view the real-time data flowing through them.


    Because schemas are inferred and decoded, you’ll be able to observe your data in a human-readable format.

    13. 13. What are Shims?

  13. A Shim is a ‘wrapper library’ for popular libraries that incorporate the usage of our SDK to simplify the integration and adoption of Streamdal.


    Our primary focus has been on the SDKs, and shims will land sometime in the future. Check the roadmap to see when they could be added.

    14. 14. What is Observability?

  14. Observability can be a really loaded term, which is why we also define it in our glossary and have a whole section dedicated to it. Depending on what you’re working with or who you’re speaking to, it can mean many different things.


    When we say observability, we mean:

    The ability to simply see the actual real-time data, its schema, and the services generating data or interacting with it.


    DORA (DevOps Research and Assessment) also has a good definition on observability within this context, which defines it as:

    …tooling or a technical solution that allows teams to actively debug their system. Observability is based on exploring properties and patterns not defined in advance.


    We see Tail as a crucial part of this, and the primary function enabling data observability with Streamdal. Here are the following features we consider as “observability” features:



    👆That list is subject to additions over time, and not subtractions. Check out our manifesto for more information on this design choice.

    15. 15. What is Wasm?

  15. Wasm is WebAssembly.

    16. 16. What role does Wasm play in Streamdal?

  16. We make significant use of Wasm.


    Rules are supported by Wasm 100%. Schema inference, rules, and any sort of business logic is going to be executed by Wasm in the SDK.


    The reason we’re able to have such a low impact on any given client’s resources, usually less than 0.1ms, is because it’s being executed via Wasm. It’s open source.


    Get more information on how we use it in the Wasm section of the monorepo, or read about this design choice in our manifesto.

    17. 17. How do I contribute or become a design partner?

  17. Streamdal is open-source using the Apache License v2.0.


    Review our contributing guidelines for more information.

    18. 18. Does Streamdal support Protobuf or other binary encoding formats?

  18. We are unabashed fans of and experts with Protobuf. It’s used in the primary API - we use it everywhere. We have even released previous open source tools like plumber which has extensive protobuf support.


    Right now Streamdal will only work with JSON or non-binary data, but we have plans to add support in the near future.


    Check out our roadmap for updates.

    19. 19. What are the numbers on the lines between my services and backends?

  19. Those numbers are throughput details for the edges in those particular groups of components.


    In the screenshot below, the arrows are pointing to the producer throughput details of the welcome-service service:


    Streamdal producer edge details

    20. 20. What is live/real-time in the app?

  20. Nearly everything.


    The app is always connected to the backend and the server. It will receive and react to every update that happens on the server:


    • Throughput information
    • the Data Graph (i.e. when a new SDK is instrumented and the Process() is called, it will start showing up on the data graph)
    • Tail
    • Schemas

    21. 21. I see direction indicators on the lines. What are those for?

  21. These are very important. They show the direction of the flow of data.


    • Going up means it’s a read, and is a consumer
    • Going down means it’s a write, and is a producer

    22. 22. I saw that I must pass a ServiceName, ComponentName, OperationType, and OperationName in the SDK - what are those?

  22. We refer to these as the audience, which identifies a specific service + operation type (consume or producer), and name for the operation type and component.


    You will always see these. There is an envar for some of them, but these are declared when the SDK is instrumented.