Implementing Data Serialization
| Format | Type | Schema | Size | Use Case |
|---|
| JSON | Text | None | Large | Web APIs, debugging |
| XML | Text | XSD optional | Largest | Legacy, SOAP |
| YAML | Text | None | Large | Config files |
| Protobuf | Binary | Required (.proto) | Compact | gRPC, microservices |
| Avro | Binary | Required (JSON) | Compact | Kafka, Hadoop |
| Thrift | Binary | Required (.thrift) | Compact | Cross-language RPC |
| MessagePack | Binary | Schemaless | Compact | JSON drop-in |
| CBOR | Binary | Schemaless | Compact | IoT, COSE |
| FlatBuffers | Binary | Required | Zero-copy | Games, latency-critical |
2. Working with Protocol Buffers (protobuf)
Example: protobuf schema and Java usage
syntax = "proto3";
package orders;
message Order {
string id = 1;
string customer_id = 2;
repeated LineItem items = 3;
int64 created_at_ms = 4;
}
message LineItem {
string sku = 1;
int32 quantity = 2;
double price = 3;
}
Order order = Order.newBuilder()
.setId("o-42")
.setCustomerId("c-7")
.addItems(LineItem.newBuilder().setSku("A1").setQuantity(2).setPrice(9.99))
.setCreatedAtMs(System.currentTimeMillis())
.build();
byte[] bytes = order.toByteArray();
Order parsed = Order.parseFrom(bytes);
| Feature | Detail |
|---|
| Wire format | Tag-length-value, varint encoding |
| Field tags | 1-15 = 1 byte; reserve low tags for hot fields |
| Forward compat | Unknown fields preserved (proto3.5+) |
| Languages | Java, Go, Python, C++, Rust, JS, Kotlin, Swift, etc. |
3. Working with Apache Avro
| Feature | Detail |
|---|
| Schema | JSON, embedded in file or referenced from registry |
| Reader/Writer schemas | Resolution rules enable evolution |
| Encoding | Binary or JSON |
| Best for | Long-term data lake storage, Kafka payloads |
Example: Avro schema
{
"type": "record",
"name": "Order",
"namespace": "com.shop",
"fields": [
{ "name": "id", "type": "string" },
{ "name": "amount", "type": "double" },
{ "name": "currency", "type": "string", "default": "USD" }
]
}
4. Working with Apache Thrift
| Feature | Detail |
|---|
| IDL | .thrift files define structs & services |
| Transports | TSocket, TFramedTransport, THttpTransport |
| Protocols | TBinary, TCompact, TJSON |
| Origin | Facebook, used by Cassandra, HBase |
5. Implementing Binary Serialization (MessagePack)
| Feature | Detail |
|---|
| Schema | Schemaless (like JSON) |
| Size | ~30-50% smaller than JSON |
| Speed | Faster parse/serialize than JSON |
| Use Case | Redis (RESP3), Fluentd, mobile APIs |
6. Implementing Schema Evolution
| Change | Protobuf | Avro |
|---|
| Add optional field | Safe | Safe (with default) |
| Remove field | Reserve tag | Reader uses default |
| Rename field | Tag preserves identity | Use aliases |
| Change type | Limited (int↔long) | Promotion rules |
| Required → optional | N/A in proto3 | Add default |
7. Implementing Forward and Backward Compatibility
| Direction | Meaning | Strategy |
|---|
| Backward | New code reads old data | Defaults for new fields |
| Forward | Old code reads new data | Ignore unknown fields |
| Full | Both directions | Required for rolling upgrades |
8. Working with Schema Registry
| Feature | Detail |
|---|
| Purpose | Centralized versioned schemas referenced by ID |
| Wire format | [magic byte][schema id][payload] |
| Compatibility modes | BACKWARD, FORWARD, FULL, NONE |
| Tools | Confluent Schema Registry, AWS Glue, Apicurio |
| Format | Encode (MB/s) | Decode (MB/s) | Size |
|---|
| JSON | ~200 | ~150 | 1.0× (baseline) |
| Protobuf | ~700 | ~900 | 0.3-0.5× |
| Avro | ~500 | ~600 | 0.3-0.5× |
| FlatBuffers | ~1500 | ~5000 (zero-copy) | 0.5-0.7× |
| MessagePack | ~400 | ~500 | 0.5-0.7× |
Note: Numbers are order-of-magnitude only; benchmark with your payload shape.
10. Handling Schema Versioning
| Strategy | Mechanism |
|---|
| Embedded version | Field schema_version in payload |
| Registry ID | Schema ID in wire format header |
| Topic-per-version | Kafka orders-v1, orders-v2 |
| Dual write | Producer writes both during migration |