First steps

After going through the Installation section and having installed all the operators, you now deploy a Kafka cluster and the required dependencies. Afterward you can verify that it works by producing test data into a topic and consuming it.

Setup

Two things need to be installed to create a Kafka cluster:

  • A ZooKeeper instance for internal use by Kafka

  • The Kafka cluster itself

Create them in this order by applying the corresponding manifest files. The operators you just installed then create the resources according to the manifest.

ZooKeeper

Create a file named zookeeper.yaml with the following content:

---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperCluster
metadata:
  name: simple-zk
spec:
  image:
    productVersion: 3.9.4
  servers:
    roleGroups:
      default:
        replicas: 1

and apply it:

kubectl apply -f zookeeper.yaml

Create a file kafka-znode.yaml with the following content:

---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: simple-kafka-znode
spec:
  clusterRef:
    name: simple-zk

and apply it:

kubectl apply -f kafka-znode.yaml

Kafka

Create a file named kafka.yaml with the following contents:

---
apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
metadata:
  name: simple-kafka
spec:
  image:
    productVersion: 3.9.1
  clusterConfig:
    tls:
      serverSecretClass: null
    zookeeperConfigMapName: simple-kafka-znode
  brokers:
    config:
      bootstrapListenerClass: external-unstable # This exposes your Stacklet outside of Kubernetes. Remove this property if this is not desired
      brokerListenerClass: external-unstable # This exposes your Stacklet outside of Kubernetes. Remove this property if this is not desired
    roleGroups:
      default:
        replicas: 3

and apply it:

kubectl apply --server-side -f kafka.yaml

This creates the actual Kafka instance.

Verify that it works

Next, use the Kafka client scripts to create a topic and publish and consume data.

The Kafka operator has created a service called simple-kafka-broker-default-bootstrap. This service represents the endpoint clients should initially connect to in order to publish and consume data. First, make sure that the service exists and it is healthy:

kubectl describe svc simple-kafka-broker-default-bootstrap

The output should look somewhat like this:

Name:                     simple-kafka-broker-default-bootstrap
Namespace:                default
Labels:                   app.kubernetes.io/component=broker
                          app.kubernetes.io/instance=simple-kafka-broker-default-bootstrap
                          app.kubernetes.io/managed-by=listeners.stackable.tech_listener
                          app.kubernetes.io/name=listener
                          app.kubernetes.io/role-group=default
                          app.kubernetes.io/version=3.9.1-stackable0.0.0-dev
                          stackable.tech/vendor=Stackable
Annotations:              <none>
Selector:                 listener.stackable.tech/mnt.9555cbb6f38d4b0ca1771e6d83d28e27=simple-kafka-broker-default-bootstrap
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.105.88.52
IPs:                      10.105.88.52
Port:                     kafka  9092/TCP
TargetPort:               9092/TCP
NodePort:                 kafka  32608/TCP
Endpoints:                10.244.4.22:9092,10.244.4.24:9092,10.244.4.23:9092
Session Affinity:         None
External Traffic Policy:  Local
Internal Traffic Policy:  Cluster
Events:                   <none>

The output shows that there are three endpoints serviced here. They correspond to the three broker pods belonging to the Kafka cluster.

Then, create a port-forward on this service:

kubectl port-forward svc/simple-kafka-broker-default-bootstrap 9092 2>&1 >/dev/null &

Now, create a new topic called test-data-topic:

kubectl exec -n default simple-kafka-broker-default-0 -c kafka -t -- /stackable/kafka/bin/kafka-topics.sh \
--create \
--topic test-data-topic \
--partitions 1 \
--bootstrap-server localhost:9092

Use the Kafka performance producer script to send a couple of messages to the topic previously created:

kubectl exec -n default simple-kafka-broker-default-0 -c kafka -t -- /stackable/kafka/bin/kafka-producer-perf-test.sh \
--producer-props bootstrap.servers=localhost:9092 \
--topic test-data-topic \
--payload-monotonic \
--throughput 1 \
--num-records 5

The output should contain the following line:

5 records sent, 1.138434 records/sec (0.00 MB/sec), 83.40 ms avg latency, 395.00 ms max latency, 3 ms 50th, 395 ms 95th, 395 ms 99th, 395 ms 99.9th.

This confirms that there were five messages sent to the topic and it also displays performance timers. We are not interested in any performance indicators but appreciate the fact that there were five unique messages that we consume later.

Now let’s consume the messages from above:

kubectl exec -n default simple-kafka-broker-default-0 -c kafka -t -- /stackable/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic test-data-topic \
--offset earliest \
--partition 0 \
--timeout-ms 1000

The consumer should print the messages in between logging statements

0
1
2
3
4

You successfully created a Kafka cluster and produced and consumed data.

What’s next

Have a look at the Usage guide page to find out more about the features of the Kafka Operator.