When integrating Apache Kafka with Spring Boot, there are several default configuration settings provided by Spring Kafka and Kafka itself. These settings are defined in the application.properties or application.yml file of your Spring Boot project and are used to configure Kafka producers, consumers, and topics.
Here’s a breakdown of default and commonly used configuration settings:
1. Basic Kafka Configuration
Kafka Broker Settings
spring.kafka.bootstrap-servers=localhost:9092
- Specifies the Kafka broker addresses (can be a comma-separated list of broker URLs).
- Default:
localhost:9092
2. Producer Configuration
Default producer settings:
spring.kafka.producer.bootstrap-servers=localhost:9092
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.acks=1
spring.kafka.producer.linger-ms=0
spring.kafka.producer.batch-size=16384
spring.kafka.producer.buffer-memory=33554432
| Setting | Default | Description |
|---|---|---|
bootstrap-servers | localhost:9092 | The Kafka broker address. |
key-serializer | StringSerializer | Serializer class for the key of the message. |
value-serializer | StringSerializer | Serializer class for the value of the message. |
acks | 1 | Specifies the acknowledgment level (0, 1, or all). |
linger-ms | 0 | Time the producer waits before sending a batch. |
batch-size | 16384 (16 KB) | Maximum size of a batch of messages before sending to Kafka. |
buffer-memory | 33554432 (32 MB) | Memory available to the producer for buffering messages. |
3. Consumer Configuration
Default consumer settings:
spring.kafka.consumer.bootstrap-servers=localhost:9092
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.group-id=default-group
spring.kafka.consumer.auto-offset-reset=latest
spring.kafka.consumer.enable-auto-commit=true
spring.kafka.consumer.max-poll-records=500
| Setting | Default | Description |
|---|---|---|
bootstrap-servers | localhost:9092 | The Kafka broker address. |
key-deserializer | StringDeserializer | Deserializer class for the key of the message. |
value-deserializer | StringDeserializer | Deserializer class for the value of the message. |
group-id | default-group | Consumer group ID to which this consumer belongs. |
auto-offset-reset | latest | Specifies the offset reset policy (latest, earliest, or none). |
enable-auto-commit | true | If true, Kafka automatically commits offsets. |
max-poll-records | 500 | Maximum number of records returned in a single poll. |
4. Common Kafka Settings
Topic Configuration
spring.kafka.topic.default-partitions=1
spring.kafka.topic.default-replication-factor=1
- These settings are used when creating topics programmatically through Spring Boot:
default-partitions: Number of partitions for the topic.default-replication-factor: Number of replicas for each partition.
SSL and Security
spring.kafka.security.protocol=PLAINTEXT
spring.kafka.ssl.key-password=
spring.kafka.ssl.keystore-location=
spring.kafka.ssl.keystore-password=
spring.kafka.ssl.truststore-location=
spring.kafka.ssl.truststore-password=
- For secured Kafka clusters, you need to configure SSL/TLS settings.
5. Spring Kafka Listener Settings
Default listener container settings:
spring.kafka.listener.concurrency=3
spring.kafka.listener.poll-timeout=3000
| Setting | Default | Description |
|---|---|---|
listener.concurrency | 1 | Number of concurrent threads for the Kafka listener. |
listener.poll-timeout | 3000 ms | Maximum time (in ms) a consumer blocks while waiting for records. |
6. Advanced Settings
Retry and Error Handling
spring.kafka.producer.retries=3
spring.kafka.consumer.max-poll-interval-ms=300000
spring.kafka.consumer.session-timeout-ms=10000
spring.kafka.consumer.heartbeat-interval-ms=3000
| Setting | Default | Description |
|---|---|---|
producer.retries | 0 | Number of retries the producer will attempt on send failures. |
consumer.max-poll-interval-ms | 300000 (5 min) | Maximum time between polls before a consumer is considered unresponsive. |
consumer.session-timeout-ms | 10000 (10 sec) | Timeout for consumer heartbeat with Kafka. |
consumer.heartbeat-interval-ms | 3000 (3 sec) | Frequency at which the consumer sends heartbeats to the broker. |
7. Example application.properties
# Kafka Broker
spring.kafka.bootstrap-servers=localhost:9092
# Producer Configuration
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.acks=all
spring.kafka.producer.retries=3
# Consumer Configuration
spring.kafka.consumer.group-id=my-group
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
# Listener Settings
spring.kafka.listener.concurrency=2
spring.kafka.listener.poll-timeout=3000
# Topic Configuration (Optional)
spring.kafka.topic.default-partitions=3
spring.kafka.topic.default-replication-factor=2
8. Customizing Kafka with a Configuration Class
You can override and customize Kafka settings by defining a @Configuration class:
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.core.ProducerFactory;
import java.util.HashMap;
import java.util.Map;
@Configuration
public class KafkaConfig {
@Bean
public ProducerFactory<String, String> producerFactory() {
Map<String, Object> config = new HashMap<>();
config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
config.put(ProducerConfig.ACKS_CONFIG, "all");
return new DefaultKafkaProducerFactory<>(config);
}
@Bean
public KafkaTemplate<String, String> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
}
These settings cover the most common default configurations and can be customized to meet your application requirements. Always monitor and fine-tune for production environments!