Using Kafka Streams
1. Understanding Streams Concept
| Concept | Description | Detail |
|---|---|---|
| KStream | Record stream (events) | Append |
| KTable | Changelog (latest per key) | Upsert |
| Library | Embedded, no cluster | Just a JAR |
Example: Streams dependency
// org.apache.kafka:kafka-streams:3.9.0
import org.apache.kafka.streams.*;
2. Creating Streams Application
| Step | Description | Detail |
|---|---|---|
| Properties | application.id + bootstrap | Required |
| StreamsBuilder | Define topology | DSL |
| KafkaStreams | Runtime instance | start() |
Example: App skeleton
StreamsBuilder b = new StreamsBuilder();
b.stream("input").to("output");
KafkaStreams ks = new KafkaStreams(b.build(), props);
ks.start();
3. Building Stream Topology
| Element | Description | Detail |
|---|---|---|
| Source | Input topic node | stream/table |
| Processor | Transformation node | map/filter |
| Sink | Output topic node | to() |
4. Transforming Streams
| Operator | Description | Type |
|---|---|---|
| mapValues | Transform value | Stateless |
| filter | Keep matching | Stateless |
| flatMap | One-to-many | Stateless |
Example: Map and filter
stream.filter((k, v) -> v.getAmount() > 100)
.mapValues(v -> v.getAmount() * 1.1);
5. Aggregating Streams
| Operator | Description | Output |
|---|---|---|
| groupByKey | Prepare aggregation | KGroupedStream |
| count | Count per key | KTable |
| aggregate | Custom accumulator | KTable |
6. Joining Streams
| Join | Description | Detail |
|---|---|---|
| join | Inner join | Both present |
| leftJoin | Left preserved | Null right |
| outerJoin | Either side | Stream-stream |
7. Windowing Operations
| Window | Description | Detail |
|---|---|---|
| Tumbling | Fixed, non-overlapping | Size only |
| Hopping | Fixed, overlapping | Size + advance |
| Session | Activity-based | Inactivity gap |
Example: Windowed count
stream.groupByKey()
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
.count();
8. Using State Stores
| Aspect | Description | Detail |
|---|---|---|
| RocksDB | Default local store | Disk-backed |
| Changelog | Backup topic for store | Fault-tolerant |
| Interactive Query | Read store directly | store() |
Example: Query a store
ReadOnlyKeyValueStore<String, Long> store =
ks.store(StoreQueryParameters.fromNameAndType(
"counts", QueryableStoreTypes.keyValueStore()));
Long n = store.get("key1");
9. Processing Records
| Operator | Description | Detail |
|---|---|---|
| peek | Side-effect, no change | Logging |
| foreach | Terminal side-effect | No return |
| process | Custom Processor | State access |
10. Using Processors API
| Element | Description | Detail |
|---|---|---|
| Processor | Low-level record handler | process() |
| ProcessorContext | Forward, schedule, store | Runtime hook |
| punctuate | Time-driven callback | schedule() |
Example: Processor API
builder.stream("in").process(() -> new Processor<>() {
public void process(Record<String,String> r) {
context.forward(r.withValue(r.value().toUpperCase()));
}
});
11. Starting Streams Application
| Step | Description | Detail |
|---|---|---|
| start() | Begin processing | Async |
| Shutdown Hook | Graceful close | close() |
| State Listener | Track RUNNING/ERROR | Monitoring |
12. Handling Streams Exceptions
| Handler | Description | Detail |
|---|---|---|
| DeserializationExceptionHandler | Bad input records | Skip/fail |
| ProductionExceptionHandler | Output send errors | Continue/fail |
| UncaughtExceptionHandler | Thread crash policy | REPLACE_THREAD |