Configuring Kafka Connect

1. Setting Bootstrap Servers

Config Description Detail
bootstrap.servers Worker → cluster Required
Shared All workers same cluster Consistency
Security SSL/SASL prefixes producer./consumer.

Example: Bootstrap

bootstrap.servers=b1:9092,b2:9092

2. Configuring Key and Value Converters

Converter Description Detail
JsonConverter JSON with/without schema Common
AvroConverter Schema Registry Compact
StringConverter Plain strings Simple

Example: Converters

key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false

3. Setting Offset Storage

Config Description Detail
offset.storage.topic Source offsets topic Distributed
offset.storage.
partitions		 Partition count 25 default
offset.flush.
interval.ms		 Commit frequency 60000

Example: Offset topic

offset.storage.topic=connect-offsets
offset.storage.replication.factor=3

4. Configuring Config Storage

Config Description Detail
config.storage.topic Connector configs Compacted
Partitions Must be 1 Ordering
RF 3 in prod Durability

Example: Config topic

config.storage.topic=connect-configs
config.storage.replication.factor=3

5. Setting Status Storage

Config Description Detail
status.storage.topic Connector/task state Compacted
Partitions Multiple OK 5 default
RF 3 in prod Durability

Example: Status topic

status.storage.topic=connect-status
status.storage.replication.factor=3

6. Configuring Replication Factor

Topic Config Prod
Offsets offset.storage.replication.factor 3
Configs config.storage.replication.factor 3
Status status.storage.replication.factor 3

Example: RF for all internal topics

offset.storage.replication.factor=3
config.storage.replication.factor=3
status.storage.replication.factor=3

7. Setting Group ID

Config Description Detail
group.id Connect cluster identity Shared by workers
Unique Differ from consumer groups No clash
Cluster Boundary Same id = one cluster Coordination

Example: Group id

group.id=connect-cluster

8. Configuring Plugin Path

Config Description Detail
plugin.path Connector JAR dirs Isolated loaders
Isolation Avoids classpath clashes Per-plugin
Multiple Comma-separated List

Example: Plugin path

plugin.path=/usr/share/kafka/plugins,/opt/connectors

9. Setting REST Port

Config Description Default
rest.port REST API listen port 8083
listeners Bind address/protocol http://
rest.advertised.
host.name		 Inter-worker comms Required cluster

Example: REST listener

listeners=http://0.0.0.0:8083
rest.advertised.host.name=worker1.example.com

10. Configuring Internal Topics

Aspect Description Detail
Auto-Create Workers create if missing On startup
cleanup.policy compact for config/status Latest state
Pre-Create Manual in locked clusters No auto-topic

Example: Pre-create config topic

kafka-topics.sh --create --topic connect-configs \
  --partitions 1 --replication-factor 3 \
  --config cleanup.policy=compact --bootstrap-server localhost:9092

Related Cheat Sheets

Kafka

Managing Schema Evolution

Kafka

Monitoring Producer Metrics

Kafka

Implementing Backup and Recovery

Kafka

Working with Message Serialization