Deploying Kafka on Cloud Platforms

1. Using Amazon MSK

Aspect Description Detail
Managed AWS runs brokers No ops
IAM Auth SASL/OAUTHBEARER Native
Serverless Auto-scaling option Pay per use

Example: MSK IAM client

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;

2. Deploying on Confluent Cloud

Aspect Description Detail
Fully Managed SaaS Kafka Multi-cloud
API Keys SASL/PLAIN auth Per cluster
ksqlDB + Connect Managed add-ons Ecosystem

Example: Confluent Cloud client

bootstrap.servers=pkc-xxxx.region.aws.confluent.cloud:9092
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule \
  required username="KEY" password="SECRET";

3. Using Azure Event Hubs

Aspect Description Detail
Kafka Endpoint Protocol-compatible Port 9093
Connection String SASL/PLAIN $ConnectionString
Namespace = cluster Hub = topic

Example: Event Hubs client

bootstrap.servers=NS.servicebus.windows.net:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule \
  required username="$ConnectionString" password="Endpoint=sb://...";

4. Deploying on Google Cloud

Option Description Detail
Managed Kafka GCP Apache Kafka service Native
GKE Strimzi operator Self-managed
Pub/Sub Bridge Kafka Connect Integration

Example: Strimzi Kafka CR

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    replicas: 3

5. Configuring Cloud-Specific Settings

Aspect Description Detail
Private Link VPC-private access No public IP
Multi-AZ Spread brokers HA
Storage Tier EBS/PD type Throughput

Example: Private bootstrap

bootstrap.servers=b-1.privatelink.region.amazonaws.com:9098

6. Managing Cloud Kafka Security

Aspect Description Detail
Encryption TLS in transit, KMS at rest Default
RBAC Cloud IAM roles Fine-grained
Audit Logs Access tracking Compliance

Example: Managed Service for Kafka auth (GCP)

security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER

7. Scaling Cloud Kafka Clusters

Aspect Description Detail
Add Brokers Console/API call Auto-rebalance
Storage Expansion Grow volumes No downtime
Serverless Elastic capacity Hands-off

Example: MSK scale CLI

aws kafka update-broker-count --cluster-arn $ARN \
  --target-number-of-broker-nodes 6 --current-version $VER

8. Monitoring Cloud Deployments

Aspect Description Detail
CloudWatch AWS metrics MSK
Cloud Monitoring GCP metrics Stackdriver
Open Monitoring Prometheus/JMX export Portable

Example: MSK open monitoring scrape

scrape_configs:
  - job_name: msk
    static_configs:
      - targets: ['b-1.cluster:11001']

9. Optimizing Cloud Costs

Lever Description Detail
Tiered Storage Offload to object store Cheaper
Right-Size Match broker type Avoid waste
Compression Cut egress/storage zstd

Example: Enable tiered storage

remote.storage.enable=true
local.retention.ms=3600000

10. Implementing Hybrid Cloud Setup

Aspect Description Detail
On-Prem + Cloud MM2/Cluster Link Bridge
VPN/Interconnect Private connectivity Secure
Migration Path Gradual cutover Mirror first

Example: On-prem to cloud mirror

clusters=onprem,cloud
onprem->cloud.enabled=true
cloud.security.protocol=SASL_SSL

Related Cheat Sheets

Kafka

Working with Message Serialization

Kafka

Using Kafka CLI Tools

Kafka

Installing and Configuring Kafka

Kafka

Optimizing Producer Performance