Designing Message Formats
1. Designing Message Structure
| Field | Purpose |
|---|---|
v | Schema version |
type | Dispatch key |
id | Correlation ID |
ts | Timestamp (ms epoch) |
seq | Per-channel sequence |
payload | Type-specific body |
meta | Trace IDs, source, etc. |
2. Using JSON Format
| Aspect | Detail |
|---|---|
| Pros | Human-readable, debuggable, native parsers |
| Cons | Verbose, no binary, slow parse for large msgs |
| Tools | JSON Schema, zod, ajv |
Example: JSON envelope
{ "v": 1, "type": "trade", "id": "abc", "ts": 1715900000000, "payload": { "sym": "BTC", "px": 65000.5 } }
3. Using MessagePack
| Aspect | Detail |
|---|---|
| Encoding | Schema-less binary; JSON-like model |
| Size | ~30-50% smaller than JSON |
| JS library | @msgpack/msgpack |
| Types | int, float, bool, str, bin, array, map, ext |
Example: MessagePack
import { encode, decode } from "@msgpack/msgpack";
ws.binaryType = "arraybuffer";
ws.send(encode({ "type": "tick", "px": 65000.5 }));
ws.addEventListener("message", (e) => console.log(decode(new Uint8Array(e.data))));
4. Using Protocol Buffers
| Aspect | Detail |
|---|---|
| Schema | Required (.proto file) |
| Library | protobufjs, google-protobuf |
| Size | Smallest of common formats |
| Evolution | Backward/forward compatible if rules followed |
Example: protobuf.js
const root = await protobuf.load("chat.proto");
const Msg = root.lookupType("chat.Message");
const buf = Msg.encode(Msg.create({ "text": "hi" })).finish();
ws.send(buf);
5. Using CBOR
| Aspect | Detail |
|---|---|
| Spec | RFC 8949 |
| Library | cbor-x (fast) |
| Strengths | Self-describing, IoT-friendly, supports tags |
| vs MessagePack | Similar; CBOR is IETF-standard |
6. Implementing Custom Binary Format
Example: Fixed-layout header
// [u8 type][u32 length][bytes payload]
function encode(type, payload) {
const buf = new ArrayBuffer(5 + payload.byteLength);
const v = new DataView(buf);
v.setUint8(0, type);
v.setUint32(1, payload.byteLength, false);
new Uint8Array(buf, 5).set(new Uint8Array(payload));
return buf;
}
| Field | Use |
|---|---|
| Magic | Validate protocol |
| Version | Schema rev |
| Type | Routing |
| Length | Payload bound |
| CRC | Integrity (optional) |
7. Adding Message Versioning
| Strategy | Detail |
|---|---|
Envelope field v | Per-message version |
| Subprotocol | Negotiate at handshake (chat.v2) |
| URL path | /v2/ws |
| Feature flags | Server announces capabilities |
8. Including Message Metadata
| Metadata | Use |
|---|---|
traceId | Distributed tracing |
correlationId | Match req/resp |
userId / sessionId | Auth context |
source | Origin service |
ttl | Drop after expiry |
9. Implementing Message Acknowledgment
Example: Ack flow
// sender
ws.send(JSON.stringify({ "type": "evt", "id": id, "payload": data }));
// receiver
ws.send(JSON.stringify({ "type": "ack", "id": id }));
| Ack Type | Use |
|---|---|
| Application ack | Confirm processing |
| Negative ack (nack) | Request resend |
| Cumulative ack | "All up to seq N" |
10. Validating Message Schema
| Tool | Approach |
|---|---|
| JSON Schema + ajv | Declarative, language-agnostic |
| zod / valibot | TS-native, runtime + types |
| Protobuf | Schema is compiled in |
| Avro | Schema evolution heavy |