Producing Messages

1. Creating Producer Instance

Element Description Detail
KafkaProducer Thread-safe client for sending records Reuse one instance
Generics Key and value types <String, String>
close() Flushes and releases resources Always close

Example: Construct a producer

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);

2. Setting Producer Properties

Property Purpose Common Value
acks Durability level all
retries Retry attempts Integer.MAX_VALUE
enable.idempotence Dedup on retry true
compression.type Payload compression lz4

Example: Durable producer config

props.put("acks", "all");
props.put("enable.idempotence", "true");
props.put("compression.type", "lz4");

3. Configuring Bootstrap Servers

Aspect Description Detail
bootstrap.servers Initial brokers for discovery host:port list
Failover List multiple for resilience 2–3 entries
Discovery Full cluster learned at connect Auto-refresh

Example: Multiple bootstrap brokers

props.put("bootstrap.servers", "kafka1:9092,kafka2:9092,kafka3:9092");

4. Configuring Serializers

Serializer Type Class
String UTF-8 text StringSerializer
ByteArray Raw bytes ByteArraySerializer
Long/Integer Numeric LongSerializer
Avro Schema-based KafkaAvroSerializer

Example: Avro value serializer

props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("schema.registry.url", "http://localhost:8081");

5. Sending Messages

Method Behavior Returns
send(record) Async, fire-and-forget Future
send(record, cb) Async with callback Future
ProducerRecord Wraps topic, key, value

Example: Send a record

ProducerRecord<String, String> record =
    new ProducerRecord<>("orders", "order-99", "{\"amount\":42}");
producer.send(record);

6. Using Synchronous Send

Aspect Description Tradeoff
get() Blocks until ack received Lowest throughput
Ordering Strict per record Guaranteed
Errors Thrown immediately Easy handling

Example: Synchronous send

try {
    RecordMetadata md = producer.send(record).get();
    System.out.printf("offset=%d%n", md.offset());
} catch (ExecutionException e) {
    log.error("send failed", e.getCause());
}

7. Using Asynchronous Send

Aspect Description Benefit
Callback Invoked on ack or error Non-blocking
Throughput Batches in background Highest
Exception Passed to callback param Check for null

Example: Async with callback

producer.send(record, (md, ex) -> {
    if (ex != null) log.error("failed", ex);
    else log.info("sent to {}-{}", md.topic(), md.partition());
});

8. Setting Message Key

Aspect Effect Detail
Key Determines partition & ordering Hashed
Compaction Key identifies record for dedup Required for compact
null Key No co-location Sticky assignment

Example: Keyed record

producer.send(new ProducerRecord<>("accounts", accountId, balanceJson));

9. Specifying Target Partition

Aspect Description Detail
Partition Arg Explicit partition index Bypasses partitioner
Validation Must be < partition count Else error
Use Case Custom routing logic Advanced

Example: Send to specific partition

// topic, partition, key, value
producer.send(new ProducerRecord<>("orders", 3, "k", "v"));

10. Adding Message Headers

Aspect Description Detail
Header Key/byte-value metadata pair Not in payload
Use Case Tracing, routing, schema id Cross-cutting
Multiple Same key allowed repeatedly Iterable

Example: Add headers

ProducerRecord<String, String> r = new ProducerRecord<>("orders", "k", "v");
r.headers().add("trace-id", traceId.getBytes(StandardCharsets.UTF_8));
r.headers().add("source", "web".getBytes(StandardCharsets.UTF_8));
producer.send(r);

11. Configuring Acknowledgments

acks Guarantee Tradeoff
0 No wait, may lose data Fastest
1 Leader-only ack Loss if leader fails
all / -1 All ISR acked Strongest durability

Example: Strongest durability

props.put("acks", "all");
props.put("min.insync.replicas", "2"); // enforced at broker/topic

12. Flushing Producer Buffer

Method Behavior Detail
flush() Blocks until all buffered sent Drains accumulator
close() Flushes then shuts down Final flush
Use Case Checkpoint before commit Ensures delivery

Example: Flush before exit

for (Order o : batch) producer.send(toRecord(o));
producer.flush();   // wait for all acks
producer.close();