Managing Schema Evolution

1. Understanding Schema Compatibility

Aspect Description Detail
Goal Evolve without breaking Decoupled
Reader/Writer Different schema versions Resolution
Defaults Enable safe changes Field-level

Example: Field with default

{"name":"discount","type":["null","double"],"default":null}

2. Implementing Backward Compatibility

Rule Allowed Detail
Delete Field Yes New reads old
Add Optional Yes (with default) Backward safe
Consumer First Upgrade consumers first Order

Example: Backward-safe add

{"name":"currency","type":"string","default":"USD"}

3. Implementing Forward Compatibility

Rule Allowed Detail
Add Field Yes Old ignores it
Delete Optional Yes Forward safe
Producer First Upgrade producers first Order

Example: Forward-safe add

// Old consumers simply skip the unknown "region" field
{"name":"region","type":"string","default":"NA"}

4. Using Full Compatibility

Aspect Description Detail
Both Ways Backward + forward Strictest
Only Allows Add/remove optional w/ default Limited
Any Order Producers/consumers independent Flexible deploy

Example: Full mode

curl -X PUT --data '{"compatibility":"FULL"}' \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  http://localhost:8081/config/orders-value

5. Using Transitive Compatibility

Mode Description Detail
BACKWARD_TRANSITIVE Compatible with all prior Full history
Non-Transitive Only previous version Adjacent
Use Long-lived data Replay old

Example: Transitive mode

curl -X PUT --data '{"compatibility":"BACKWARD_TRANSITIVE"}' \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  http://localhost:8081/config/orders-value

6. Configuring Compatibility Level

Scope Description Detail
Global /config default All subjects
Subject /config/{subject} Override
Precedence Subject over global Specific wins

Example: Global default

curl -X PUT --data '{"compatibility":"BACKWARD"}' \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  http://localhost:8081/config

7. Handling Schema Evolution Errors

Error Cause Fix
409 Conflict Incompatible schema Add defaults
SerializationException Schema mismatch Check registry
Missing Default Required field added Make optional

Example: Pre-deploy check

# 409 means incompatible — fix before registering
curl -s -o /dev/null -w "%{http_code}" -X POST --data @schema.json \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  http://localhost:8081/compatibility/subjects/orders-value/versions/latest

8. Versioning Schemas

Aspect Description Detail
Version Per-subject sequence 1, 2, 3...
Global ID Unique across registry In message
Immutable Registered = frozen No edits

Example: List versions

curl http://localhost:8081/subjects/orders-value/versions

9. Using Schema References

Aspect Description Detail
Reference Reuse schema in another Modular
name + subject Points to registered schema + version
Use Shared types/records DRY

Example: Schema with reference

{"schema":"...","references":[
  {"name":"com.acme.Address","subject":"address","version":1}]}

10. Migrating Schema Formats

Aspect Description Detail
Formats Avro, Protobuf, JSON Schema Supported
Dual Topics New topic for new format Safe transition
Converter Bridge old → new Streams app

Example: Set schema type

curl -X POST --data '{"schemaType":"PROTOBUF","schema":"..."}' \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  http://localhost:8081/subjects/orders-value/versions

Related Cheat Sheets

Kafka

Using KRaft Consensus Protocol

Kafka

Monitoring Broker Metrics

Kafka

Optimizing Broker Performance

Kafka

Implementing Custom Partitioners