In most cases a static approach fits the needs quite well, in that case Thrift lets you benefit from the better performance of generated code. That’s why I have chosen Protocol Buffer vs Avro (from Hadoop) for the final comparison. This is independent of Kafka Streams. AVRO might generally be de-/serialized faster than JSON. Compared to AVRO, JSON might be slower in general, because JSON is a text-based format whereas AVRO is a binary format. Avro is being driven largely by Hadoop, afaict. The size of data encoded in JSON is generally larger, which impacts network transmission throughput. Support and tools for Java and Scala are on a very good level. ^ The "classic" format is plain text, and an XML format is also supported. Avro versus Protobuf There is an interesting comparison in this post that compares Avro, Protobuf and Thrift of binary messages sizes and how well the protocol supports schema evolution. Kafka with AVRO vs., Kafka with Protobuf vs., Kafka with JSON Schema. The libraries also provide compatibility checks between the writer and reader schema. Apache Avro – Avro is a newer project designed to accomplish many of the same goals of Protobuf or Thrift but without the static compilation step and greater interop with dynamic languages. I wrote a JMH benchmark to compare the serialization performance of Avro (1.8.2) & Protobuf (3.5.0) in java 1.8. According to JMH, Protobuf can serialize some data 4.7 million times in a second where as Avro can only do 800k per second. Thrift — from Facebook, almost the same when it comes to functionalities as Google’s Protocol Buffers, but subjectively Protobuf is easier to use. For C++ I used Visual Studio 2017 (not an update version) with Cereal 1.2.2 (which uses rapidjson and rapidxml), protobuf 3.2.0 (static library can be found on the repository) Methodology. e. ^ Means that generic tools/libraries know how to encode, decode, and dereference a reference to another piece of … Also it is worth mentioning that besides Thrift, Protobuf and Avro there are some more solutions on the market, such as Capt'n'proto or BOLT. c. ^ Theoretically possible due to abstraction, but no implementation is included. The test data that was serialized is around 200 bytes and I generated schema for both Avro and Protobuf. Protocol buffers, also known as Protobuf, is a protocol that Google developed internally to enable serialization and deserialization of structured data between different services.Google’s design goal was to create a better method than XML to make systems communicate with each other over a wire or for the storage of data. d. ^ The primary format is binary, but a text format is available. What most other benchmarks do, is create a couple of objects to serialize and deserialize, run those a number of times in a row and calculate the average. The key difference is that protobuf has a language-agnostic schema definition but Avro uses compact JSON. The producer and consumer use those classes and libraries to serialize and deserialize the payload.  a. Messages are defined in JSON (truly more painful than Protobuf or Thrift). Protocol Buffers offer several compelling advantages over JSON for sending data over the wire between internal services. Documentation is very detailed and extensive. If that is not the case, Avro might be more suitable. All serializers will implement the following simple interface in the sample project b. JSON; Avro; Protobuf; I am going to use simple project which wraps and abstracts different serialization formats through simple interface and we'll use Unit test project to check the speed of the process and size of serialized data. Another interesting data transfer protocol is Parquet , which is optimized for column- oriented data. ^ The current default format is binary. Both protobuf and Apache Avro follow that approach. Apache Avro was has been the defacto Kafka serialization mechanism for a long time. Confluent just updated their Kafka streaming platform with additioinal support for serializing data with Protocol buffers (or protobuf) and JSON Schema serialization. With Protobuf and JSON both being sequential, it is very hard to achieve a 5x performance boost running in the same CPU and the same core.