A message envelope is a common “schema” that you use for ALL events in your system. Think “namespace”.
By having a common envelope, you are less likely to run into bugs and make mistakes as you construct events that are to be processed by other systems.
Just like with messaging systems, it is up to you to choose what message envelope you wish to use - all of them have their own PROS and CONS.
By far, the most common message envelope utilizes
And for good reason:
- It is simple
- It is well supported in virtually all programming languages
- And is mostly human readable
JSON for your message envelope is (probably) a good idea if:
- You are adding asynchronous elements to an existing system
- You are interfacing with many different programming languages
- Establishing new standards in your org is difficult
JSONcomes with its fair share of CONS though:
No type safety
Schema-enforcement is not easy
- This can be alleviated by using something like JSON Schema but that will require org-wide adoption for it to be effective
Without schema-enforcement, you are more likely to end up with events that are missing critical event data (ie. team B forgot to fill out the “payment_source” field)
No built-in compression; this will influence:
- message transfer speed
- bandwidth usage
- storage cost
No automatic client or server code generation
JSON and does automatic schema inference.
Protobuf is fairly complex but offers a high-degree of confidence in your event quality due to built-in schema enforcement, type safety and excellent cross-language support.
- You are tasked with establishing a long-lasting, sophisticated and reliable event-driven architecture at your org
- You MUST have schema-enforcement and type safety
- You need rich type support
- You are (or are planning on) using gRPC
- You want to generate client/server code from schemas
- You want to be able to point folks at good documentation
- You will have to create your own schema repository and setup a build process to generate your compiled protobufs
- The CLI tools are complex
- Deprecation is less-than-ideal
- Protobuf messages are semi-human-readable - as in, you will not be able to view all values in a message via
cat- you will need to properly decode the message
Streamdal has full support for
protobuf - all fields in a protobuf message are indexed and available for search and replay.
WARNING: If you use
google.protobuf/any.proto, please view the following article: Risks with
Avro is the default message serialization format used by Kafka. While it is language-neutral, it is best paired with Java and Kafka.
While it is very similar to
protobuf and offers many of the same advantages, it also has some unique properties:
- Reads and writes are tightly coupled with schemas which enables you to have granular control over how message envelopes evolve
- Along with static types, it is also possible to include untyped data
- You are a Java and/or Kafka shop
- You are using Confluent’s platform
- You need both static and dynamic types
- Avro is not as well supported as Protobuf
- Need schema to read/write data (might be a PRO in some cases)
Streamdal has full support for Avro - all fields in an Avro message are indexed and available for search and replay.
There are many other message serialization formats.
Streamdal also supports:
In most cases, we advise to use
JSON if you are new to messaging patterns and are not sure about all of your requirements.
If you are familiar with asynchronous patterns and are comfortable with distributed systems, choosing
protobuf is a good call.