Using Kafka Streams

1. Understanding Streams Concept

Concept Description Detail
KStream Record stream (events) Append
KTable Changelog (latest per key) Upsert
Library Embedded, no cluster Just a JAR

Example: Streams dependency

// org.apache.kafka:kafka-streams:3.9.0
import org.apache.kafka.streams.*;

2. Creating Streams Application

Step Description Detail
Properties application.id + bootstrap Required
StreamsBuilder Define topology DSL
KafkaStreams Runtime instance start()

Example: App skeleton

StreamsBuilder b = new StreamsBuilder();
b.stream("input").to("output");
KafkaStreams ks = new KafkaStreams(b.build(), props);
ks.start();

3. Building Stream Topology

Element Description Detail
Source Input topic node stream/table
Processor Transformation node map/filter
Sink Output topic node to()

Example: Describe topology

Topology t = builder.build();
System.out.println(t.describe());

4. Transforming Streams

Operator Description Type
mapValues Transform value Stateless
filter Keep matching Stateless
flatMap One-to-many Stateless

Example: Map and filter

stream.filter((k, v) -> v.getAmount() > 100)
      .mapValues(v -> v.getAmount() * 1.1);

5. Aggregating Streams

Operator Description Output
groupByKey Prepare aggregation KGroupedStream
count Count per key KTable
aggregate Custom accumulator KTable

Example: Count per key

KTable<String, Long> counts = stream
    .groupByKey()
    .count();

6. Joining Streams

Join Description Detail
join Inner join Both present
leftJoin Left preserved Null right
outerJoin Either side Stream-stream

Example: Stream-table join

orders.join(customers,
    (order, cust) -> enrich(order, cust));

7. Windowing Operations

Window Description Detail
Tumbling Fixed, non-overlapping Size only
Hopping Fixed, overlapping Size + advance
Session Activity-based Inactivity gap

Example: Windowed count

stream.groupByKey()
  .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
  .count();

8. Using State Stores

Aspect Description Detail
RocksDB Default local store Disk-backed
Changelog Backup topic for store Fault-tolerant
Interactive Query Read store directly store()

Example: Query a store

ReadOnlyKeyValueStore<String, Long> store =
  ks.store(StoreQueryParameters.fromNameAndType(
    "counts", QueryableStoreTypes.keyValueStore()));
Long n = store.get("key1");

9. Processing Records

Operator Description Detail
peek Side-effect, no change Logging
foreach Terminal side-effect No return
process Custom Processor State access

Example: Peek for logging

stream.peek((k, v) -> log.info("{}={}", k, v))
      .to("output");

10. Using Processors API

Element Description Detail
Processor Low-level record handler process()
ProcessorContext Forward, schedule, store Runtime hook
punctuate Time-driven callback schedule()

Example: Processor API

builder.stream("in").process(() -> new Processor<>() {
  public void process(Record<String,String> r) {
    context.forward(r.withValue(r.value().toUpperCase()));
  }
});

11. Starting Streams Application

Step Description Detail
start() Begin processing Async
Shutdown Hook Graceful close close()
State Listener Track RUNNING/ERROR Monitoring

Example: Start with hook

ks.start();
Runtime.getRuntime().addShutdownHook(new Thread(ks::close));

12. Handling Streams Exceptions

Handler Description Detail
DeserializationExceptionHandler Bad input records Skip/fail
ProductionExceptionHandler Output send errors Continue/fail
UncaughtExceptionHandler Thread crash policy REPLACE_THREAD

Example: Replace failed thread

ks.setUncaughtExceptionHandler(e ->
  StreamThreadExceptionResponse.REPLACE_THREAD);