How to install kafkacat on kafka container resulted from docker-compose

Question

My context is I am trying to create a docker-compose wich will start few containers for running ELK+FileBeats and 3 Kafka containers. The whole docker compose and its yml configuration files can be found in github docker-compose

To sumarize, this is my docker-compose part downloading and starting one of my kafka containers:

  kafka1:
    image: wurstmeister/kafka
    command: [start-kafka.sh]
    depends_on:
      - zoo1
      - zoo2
      - zoo3
    links:
      - zoo1
      - zoo2
      - zoo3
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_LOG_RETENTION_HOURS: "168"
      KAFKA_LOG_RETENTION_BYTES: "100000000"
      KAFKA_ZOOKEEPER_CONNECT:  zoo1:2181,zoo2:2181,zoo3:2181
      KAFKA_CREATE_TOPICS: "log:3:3"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'

I want investigate kafka using kafkat and I got

C:\Dockers\megalog-try-1>docker exec -it megalog-try-1_kafka1_1 bash
bash-4.4# kafka-topics.sh --list --bootstrap-server localhost:9092
__consumer_offsets
log
bash-4.4# kafkacat -b megalog-try-1_kafka1_1:9092 L
bash: kafkacat: command not found
bash-4.4# apt-get install kafkacat
bash: apt-get: command not found
bash-4.4# exit
exit

edited

Logs after tentative to install kafkacat

/ # kafkacat -C -b localhost:9092 -t log -o beginning -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\t
Offset: %o\n--\n'
/bin/sh: kafkacat: not found
/ # apk --no-cache add ca-certificates wget
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/1) Installing wget (1.20.3-r0)
Executing busybox-1.29.3-r10.trigger
OK: 359 MiB in 76 packages
/ # wget --quiet --output-document=/etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub
/ # wget https://github.com/sgerrand/alpine-pkg-kafkacat/releases/download/1.5.0-r0/kafkacat-1.5.0-r0.apk
--2020-02-05 22:56:46--  https://github.com/sgerrand/alpine-pkg-kafkacat/releases/download/1.5.0-r0/kafkacat-1.5.0-r0.apk
Resolving github.com... 18.228.52.138
Connecting to github.com|18.228.52.138|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/120518176/95d10080-eee3-11e9-8e83-c0ba853d6714?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200206T131952Z&X-Amz-Expires=300&X-Amz-Signature=19d4778753313a7a535c3f8c10645e13851d482f49035903de83ba93c087a247&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dkafkacat-1.5.0-r0.apk&response-content-type=application%2Fvnd.android.package-archive [following]
--2020-02-05 22:56:46--  https://github-production-release-asset-2e65be.s3.amazonaws.com/120518176/95d10080-eee3-11e9-8e83-c0ba853d6714?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200206T131952Z&X-Amz-Expires=300&X-Amz-Signature=19d4778753313a7a535c3f8c10645e13851d482f49035903de83ba93c087a247&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dkafkacat-1.5.0-r0.apk&response-content-type=application%2Fvnd.android.package-archive
Resolving github-production-release-asset-2e65be.s3.amazonaws.com... 52.216.236.203
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com|52.216.236.203|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22602 (22K) [application/vnd.android.package-archive]
Saving to: 'kafkacat-1.5.0-r0.apk'

kafkacat-1.5.0-r0.apk         100%[=================================================>]  22.07K  --.-KB/s    in 0.1s

2020-02-05 22:56:48 (167 KB/s) - 'kafkacat-1.5.0-r0.apk' saved [22602/22602]

/ # apk add --no-cache kafkacat-1.5.0-r0.apk
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/6) Installing lz4-libs (1.8.3-r2)
(2/6) Installing db (5.3.28-r1)
(3/6) Installing libsasl (2.1.27-r2)
(4/6) Installing librdkafka (0.11.6-r1)
(5/6) Installing yajl (2.1.0-r0)
(6/6) Installing kafkacat (1.5.0-r0)
Executing busybox-1.29.3-r10.trigger
OK: 361 MiB in 82 packages
/ # kafkacat -C -b localhost:9092 -t log -o beginning -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\t
Offset: %o\n--\n'
Error relocating /usr/bin/kafkacat: rd_kafka_fatal_error: symbol not found
/ # kafkacat -b megalog-try-1_kafka1_1:9092 L
Error relocating /usr/bin/kafkacat: rd_kafka_fatal_error: symbol not found
/ # kafka-topics.sh --list --bootstrap-server localhost:9092
__consumer_offsets
log
/ #

edited after Cricket_007's suggestion

docker-compose.yml (only lines edited)

...
image: confluentinc/cp-kafka:latest
...
image: confluentinc/cp-zookeeper:latest
...

relevant logs

kafka:

===> Configuring ...
advertised.port is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead.

zookeeper:

ZOOKEEPER_CLIENT_PORT is required.

Command [/usr/local/bin/dub ensure ZOOKEEPER_CLIENT_PORT] FAILED !

I guess you mean replace wurstmeister/kafka by confluent/kafka, right? In other words, switch the images, right? If so, well, I can give a try. — Jim C, Feb 06 '20 at 13:29
@cricket_007 now I ca't start zookeeper and kafka containers via docker-compose. It seems confluent and wurstmeister has different parameter names. I will add above what I have changed in docker-compose.yml and the relevant logs for kafka and zookeeper — Jim C, Feb 06 '20 at 13:57
All the variables values are the same, but the keys might be different... Not Confluent/kafka. Those are deprecated. confluentinc/cp-kafkacat, I think. And you can run it separately from the brokers... So you can keep wurstmeister and add the other — OneCricketeer, Feb 06 '20 at 14:12
`rd_kafka_fatal_error: symbol not found` indicates that your librdkafka version is too old. You will need librdkafka v1.0.0 or later, preferably v1.3.0. librdkafka rpms and debs are available from confluent, see https://docs.confluent.io/current/installation/installing_cp/index.html#installation — Edenhill, Feb 10 '20 at 08:15

Munish · Answer 1 · 2020-02-05T21:05:05.230

3

You have to use apk for installing any package on this kafka image container.

The current installation method for this package is to pull it in using wget or curl and install the local file with apk:

apk --no-cache add ca-certificates wget
wget --quiet --output-document=/etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub
wget https://github.com/sgerrand/alpine-pkg-kafkacat/releases/download/1.5.0-r0/kafkacat-1.5.0-r0.apk
apk add --no-cache kafkacat-1.5.0-r0.apk

edited Feb 05 '20 at 21:05

answered Feb 05 '20 at 20:30

Munish

673
3
11

1

Thanks. I am getting "Error relocating /usr/bin/kafkacat: rd_kafka_fatal_error: symbol not found". I will paste above all logs. PS.: do you know how set in my docker-compose.yml commands for installing kafkacat? Otherwise I will have to install manually in all containers. – Jim C Feb 06 '20 at 13:24
1

you can build a new docker image using kafka as a base image and then run above commands using Dockerfile, use that newly build image to run your containers – Munish Feb 06 '20 at 16:17

score 1 · Accepted Answer · answered Feb 06 '20 at 14:15

1

Please use KAFKA_ADVERTISED_LISTENERS instead

Just remove the port variable. You already have this KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092

ZOOKEEPER_CLIENT_PORT is required

Add it

ZOOKEEPER_CLIENT_PORT: 2181

You can mix and match containers... I was only suggesting that you use the existing kafkacat container with your existing setup, which could connect to any remote broker.

Or just install kafkacat on your local system

answered Feb 06 '20 at 14:15

OneCricketeer

126,858
14
92
185

Thanks, now I can start again kafka using confluent image. By the way, I still can't use kafkacat. Here is the log: # kafkacat -C -b localhost:9092 -t log -o beginning -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\tOffset: %o\n--\n' /bin/sh: 1: kafkacat: not found – Jim C Feb 06 '20 at 15:34
Well, after I switched from wurstmeister/kafka to confluentinc/cp-kafka:latest I have now two issues: I can't use kafkacat (the most import tool for me now) and I can't use apk to install kafkacat: # apk --no-cache add ca-certificates wget /bin/sh: 2: apk: not found. I just want to see if there is some content in my Kafka topic. That is the reason I am trying to use kafkacat. How would you approach this challenge? Imagine you want to analize certain kafka topic and you can't use kafkacat neither install it successfully. – Jim C Feb 06 '20 at 15:39
1

The Confluent images are built on Debian, not Alpine. And kafkacat might already be installed... But if not, you'd use apt, not apk. But like I said, you can use an entirely separate, **additional** container... Also, what's wrong with using kafka-console-consumer?? – OneCricketeer Feb 06 '20 at 21:37

How to install kafkacat on kafka container resulted from docker-compose

2 Answers2