Producing Messages
1. Creating Producer Instance
| Element | Description | Detail |
|---|---|---|
| KafkaProducer | Thread-safe client for sending records | Reuse one instance |
| Generics | Key and value types | <String, String> |
| close() | Flushes and releases resources | Always close |
Example: Construct a producer
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
2. Setting Producer Properties
| Property | Purpose | Common Value |
|---|---|---|
| acks | Durability level | all |
| retries | Retry attempts | Integer.MAX_VALUE |
| enable.idempotence | Dedup on retry | true |
| compression.type | Payload compression | lz4 |
Example: Durable producer config
props.put("acks", "all");
props.put("enable.idempotence", "true");
props.put("compression.type", "lz4");
3. Configuring Bootstrap Servers
| Aspect | Description | Detail |
|---|---|---|
| bootstrap.servers | Initial brokers for discovery | host:port list |
| Failover | List multiple for resilience | 2–3 entries |
| Discovery | Full cluster learned at connect | Auto-refresh |
Example: Multiple bootstrap brokers
props.put("bootstrap.servers", "kafka1:9092,kafka2:9092,kafka3:9092");
4. Configuring Serializers
| Serializer | Type | Class |
|---|---|---|
| String | UTF-8 text | StringSerializer |
| ByteArray | Raw bytes | ByteArraySerializer |
| Long/Integer | Numeric | LongSerializer |
| Avro | Schema-based | KafkaAvroSerializer |
Example: Avro value serializer
props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("schema.registry.url", "http://localhost:8081");
5. Sending Messages
| Method | Behavior | Returns |
|---|---|---|
| send(record) | Async, fire-and-forget | Future |
| send(record, cb) | Async with callback | Future |
| ProducerRecord | Wraps topic, key, value | — |
Example: Send a record
ProducerRecord<String, String> record =
new ProducerRecord<>("orders", "order-99", "{\"amount\":42}");
producer.send(record);
6. Using Synchronous Send
| Aspect | Description | Tradeoff |
|---|---|---|
| get() | Blocks until ack received | Lowest throughput |
| Ordering | Strict per record | Guaranteed |
| Errors | Thrown immediately | Easy handling |
Example: Synchronous send
try {
RecordMetadata md = producer.send(record).get();
System.out.printf("offset=%d%n", md.offset());
} catch (ExecutionException e) {
log.error("send failed", e.getCause());
}
7. Using Asynchronous Send
| Aspect | Description | Benefit |
|---|---|---|
| Callback | Invoked on ack or error | Non-blocking |
| Throughput | Batches in background | Highest |
| Exception | Passed to callback param | Check for null |
Example: Async with callback
producer.send(record, (md, ex) -> {
if (ex != null) log.error("failed", ex);
else log.info("sent to {}-{}", md.topic(), md.partition());
});
8. Setting Message Key
| Aspect | Effect | Detail |
|---|---|---|
| Key | Determines partition & ordering | Hashed |
| Compaction | Key identifies record for dedup | Required for compact |
| null Key | No co-location | Sticky assignment |
9. Specifying Target Partition
| Aspect | Description | Detail |
|---|---|---|
| Partition Arg | Explicit partition index | Bypasses partitioner |
| Validation | Must be < partition count | Else error |
| Use Case | Custom routing logic | Advanced |
Example: Send to specific partition
// topic, partition, key, value
producer.send(new ProducerRecord<>("orders", 3, "k", "v"));
10. Adding Message Headers
| Aspect | Description | Detail |
|---|---|---|
| Header | Key/byte-value metadata pair | Not in payload |
| Use Case | Tracing, routing, schema id | Cross-cutting |
| Multiple | Same key allowed repeatedly | Iterable |
Example: Add headers
ProducerRecord<String, String> r = new ProducerRecord<>("orders", "k", "v");
r.headers().add("trace-id", traceId.getBytes(StandardCharsets.UTF_8));
r.headers().add("source", "web".getBytes(StandardCharsets.UTF_8));
producer.send(r);
11. Configuring Acknowledgments
| acks | Guarantee | Tradeoff |
|---|---|---|
| 0 | No wait, may lose data | Fastest |
| 1 | Leader-only ack | Loss if leader fails |
| all / -1 | All ISR acked | Strongest durability |
Example: Strongest durability
props.put("acks", "all");
props.put("min.insync.replicas", "2"); // enforced at broker/topic
12. Flushing Producer Buffer
| Method | Behavior | Detail |
|---|---|---|
| flush() | Blocks until all buffered sent | Drains accumulator |
| close() | Flushes then shuts down | Final flush |
| Use Case | Checkpoint before commit | Ensures delivery |