Spring Kafka with Docker-Compose: Local Kafka Setup for Development

Kafka spring boot example, Spring Boot Kafka producer consumer example, Kafka spring boot tutorial, Apache Kafka Spring Boot microservices example, Kafka Spring Boot interview questions, Kafka spring boot example github, Spring Kafka documentation, spring-kafka maven

Apache Kafka is an essential component of modern distributed systems, enabling robust event streaming and data processing. However, when setting up Kafka for local development, configuring it directly on your machine can be challenging and time-consuming. Docker Compose simplifies this process, providing an isolated environment for Kafka and its dependencies without modifying your primary system configuration.

This article provides a step-by-step guide to setting up Kafka locally using Docker Compose. By the end, you’ll have a fully operational Kafka environment tailored for Spring Boot development, along with tools to test and troubleshoot your integration.

Why Use Docker for Local Kafka?
docker-compose.yml for Kafka and Zookeeper
Kafka UI Tool Integration
Connecting Spring Boot to Local Kafka
Testing Producer with Kafka CLI
Consuming from Spring @KafkaListener
Resetting Topics and Partitions
Troubleshooting Container Network Issues
Persisting Kafka Logs in Docker
Clean-Up and Repeatable Local Setup

Why Use Docker for Local Kafka?

Running Kafka manually requires installing dependencies like Zookeeper, configuring network ports, and managing the lifecycle of multiple services. Docker Compose eliminates these challenges by encapsulating Kafka and its dependencies in lightweight containers.

Key Benefits of Docker:

Simplicity: Minimal configuration is needed to start Kafka.
Reproducibility: Ensures consistency across development environments.
Isolation: Prevents conflicts with existing system services.
Scalability: Enables quick scaling by adding brokers for testing partitioned setups.

Using Docker for Kafka provides a hassle-free way to experiment, test, and debug applications.

docker-compose.yml for Kafka and Zookeeper

Docker Compose simplifies running Kafka and Zookeeper in tandem. Below is an example docker-compose.yml file that spins up a single Kafka broker with its required Zookeeper instance.

Example docker-compose.yml:

version: '3.8'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

  kafka:
    image: confluentinc/cp-kafka
    hostname: kafka
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

How It Works:

The zookeeper service manages metadata for the Kafka cluster.
The kafka service represents the broker that handles producer and consumer requests.
Ports 2181 and 9092 expose Zookeeper and Kafka for local usage.

Run docker-compose up to bring the services online.

Kafka UI Tool Integration

Testing Kafka interactions often requires monitoring broker activity, such as messages being published or topic configurations. Kafka UI tools like Kafdrop make this monitoring intuitive.

Kafdrop Setup:

Add the following service to your docker-compose.yml:

  kafdrop:
    image: obsidiandynamics/kafdrop
    container_name: kafdrop
    ports:
      - "9000:9000"
    environment:
      KAFKA_BROKER_CONNECT: kafka:9092

Access the UI at http://localhost:9000 to monitor topics, partitions, and messages.

Connecting Spring Boot to Local Kafka

Spring Boot simplifies Kafka integration with the spring-kafka dependency.

Maven Dependency:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

application.yml Configuration:

spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: example-group
      auto-offset-reset: earliest
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer

Spring Boot will now connect automatically to the local Kafka broker, ready to produce and consume messages.

Testing Producer with Kafka CLI

Kafka’s CLI tools allow you to manually produce and consume messages for testing.

Step 1. Create a Topic:

Run the following command to create a topic named test-topic:

docker exec kafka kafka-topics --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Step 2. Produce a Message:

docker exec -it kafka kafka-console-producer --topic test-topic --bootstrap-server localhost:9092

Input data (e.g., {"id":1,"name":"test"}) to send messages to the topic.

Consuming from Spring @KafkaListener

After publishing test messages, set up a consumer microservice in Spring Boot to read from the topic.

Kafka Listener Example:

@Component
public class Consumer {

    @KafkaListener(topics = "test-topic", groupId = "example-group")
    public void consume(String message) {
        System.out.println("Consumed message: " + message);
    }
}

Check the application logs to verify the messages are consumed successfully.

Resetting Topics and Partitions

During development, you may need to reset or reconfigure topics frequently.

Delete a Topic:

docker exec kafka kafka-topics --delete --topic test-topic --bootstrap-server localhost:9092

Modify Partitions:

Kafka does not allow reducing partitions. To reconfigure partitions, recreate the topic with the desired settings.

Troubleshooting Container Network Issues

Networking misconfigurations can interrupt Kafka communication.

Common Solutions:

Ensure Correct Listeners: Use KAFKA_ADVERTISED_LISTENERS to expose the broker to clients.
DNS Resolution: Use service names (kafka, zookeeper) in the docker-compose.yml file for inter-container communication.
Check Network Configurations: Verify containers are on the same Docker network.

Run docker network ls to list networks and docker network inspect <network-name> to review configurations.

Persisting Kafka Logs in Docker

Kafka logs enable debugging and replaying messages but are lost when containers restart. Prevent this by persisting logs.

Persistent Storage:

Add a volume mapping to your docker-compose.yml:

  kafka:
    volumes:
      - ./kafka-logs:/var/lib/kafka/data

Logs will now persist in the kafka-logs directory.

Clean-Up and Repeatable Local Setup

After testing, it’s crucial to clean up resources to avoid system clutter.

Stop and Remove Containers:

docker-compose down

Delete Persistent Data:

rm -rf kafka-logs

By maintaining a clean environment and reusing the docker-compose.yml file, you ensure consistency across setups.

Summary

Setting up Apache Kafka locally using Docker Compose simplifies the development process, providing a reliable and isolated environment to test real-time event-driven applications. Kafka’s integration with Spring Boot further enhances development efficiency, making it a powerful tool for microservices and event streaming. From connecting Kafka to monitoring topics via Kafdrop, this guide equips you with all the tools and techniques needed for local Kafka development.

FAQs

Q1. Can I add multiple Kafka brokers in Docker Compose?

Yes, simply replicate the kafka service with unique IDs and ports, and update the KAFKA_ZOOKEEPER_CONNECT to include all brokers.

Q2. Why is my Spring Boot application failing to connect to Kafka?

Ensure the bootstrap-servers in your application.yml matches the KAFKA_ADVERTISED_LISTENERS in the docker-compose.yml.

Q3. Is local Kafka setup sufficient for production testing?

No, local setups are ideal for development. For production, consider a multi-node Kafka cluster with proper monitoring and security configurations.

Mastering local Kafka setups ensures smooth and productive development workflows. Start building your event-driven applications today!

Table of Contents