Implementing Custom Partitioners
1. Understanding Partitioner Concept
| Aspect | Description | Detail |
|---|---|---|
| Partitioner | Maps record → partition | Producer-side |
| Default | Hash key / sticky null | Built-in |
| Custom | Implement Partitioner | Override |
2. Creating Custom Partitioner
| Method | Description | Detail |
|---|---|---|
| configure() | Read settings | Init |
| partition() | Return partition | Core logic |
| close() | Cleanup | Optional |
Example: Class skeleton
public class TenantPartitioner implements Partitioner {
public void configure(Map<String,?> c) {}
public void close() {}
// partition() below
}3. Implementing partition() Method
| Param | Description | Detail |
|---|---|---|
| key/keyBytes | Record key | May be null |
| cluster | partitionsForTopic | Count source |
| Return | int partition | 0..n-1 |
Example: partition()
public int partition(String topic, Object key, byte[] kb,
Object val, byte[] vb, Cluster c) {
int n = c.partitionsForTopic(topic).size();
return Math.abs(key.hashCode()) % n;
}4. Configuring Partitioner Class
| Config | Description | Detail |
|---|---|---|
| partitioner.class | FQN of partitioner | Producer prop |
| Default | Built-in if unset | DefaultPartitioner |
| Per-Producer | Set on producer config | Instance-level |
Example: Register partitioner
props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG,
"com.acme.TenantPartitioner");5. Using Hash-Based Partitioning
| Aspect | Description | Detail |
|---|---|---|
| Murmur2 | Default key hash | Even spread |
| Deterministic | Same key → same partition | Ordering |
| Skew Risk | Hot keys overload | Monitor |
6. Implementing Round-Robin Partitioning
| Aspect | Description | Detail |
|---|---|---|
| Cycle | Next partition each call | Even load |
| No Key Order | Ignores key | Spread |
| Counter | AtomicInteger | Thread-safe |
7. Creating Sticky Partitioner
| Aspect | Description | Detail |
|---|---|---|
| Default (null key) | Sticks to one partition per batch | Better batching |
| Switch | On batch full | New partition |
| Benefit | Lower latency, fewer requests | Throughput |
Example: Built-in sticky behavior
// Null-key records use sticky partitioning automatically since 2.4
producer.send(new ProducerRecord<>("events", null, payload));8. Implementing Range Partitioner
| Aspect | Description | Detail |
|---|---|---|
| Ranges | Map key range → partition | Ordered keys |
| Use | Time/ID buckets | Locality |
| Care | Avoid skew | Balance ranges |
Example: Range mapping
long id = Long.parseLong(key);
return (int) (id / BUCKET_SIZE) % numPartitions;9. Using Composite Key Partitioning
| Aspect | Description | Detail |
|---|---|---|
| Composite | Part of key for routing | e.g. tenant |
| Co-Location | Related records together | Join-friendly |
| Extract | Parse prefix from key | Custom |
Example: Tenant prefix routing
String tenant = key.split(":")[0];
return Math.abs(tenant.hashCode()) % numPartitions;10. Handling Null Keys
| Aspect | Description | Detail |
|---|---|---|
| Default | Sticky partition | Batched |
| Custom | Fallback logic needed | Avoid NPE |
| Option | Random or round-robin | Even spread |