How to access the MSK cluster with UI

How to access the MSK cluster with UI? This post will show you the easy way to do it.

This way may use for the other Kafka cluster, something like your on-premise system.

Check your IAM permissions

Yup, this guide will show the way to work with the AWS MSK cluster, so you have to check your IAM account has enough the permission.

You can give your IAM with the policies like: AmazonMSKFullAccess or AdministratorAccess.

This is depend on you, normally this permission only granted to the DevOps/System team. The Dev team will doesn’t have it.

I think for guys are working with AWS, this step is simply. If you need any details, please refer to the official docs.

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_change-permissions.html

Install and configure AWS CLI

And the next step, you will need to install the AWS CLI and configure the IAM key to your local machine.

Because of the kafka-ui will access the MSK cluster through the IAM key.

Please follow to the official docs, I will not mention its details here.

https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html

Clone the repository kafka-ui and configure it

Thanks for the guys who created this repository, by using it, you don’t have to work with Kafka/MSK cluster cli anymore.

Clone the repository to your local machine.

git clone https://github.com/provectus/kafka-ui

Then move to the folder kafka-ui and create a docker compose file. We will use this file to run the container.

cd kafka-ui && nano docker-compose.yaml

Copy the below content into the docker compose file.

version: '3.4'
services:
  kafka-ui:
    container_name: kafka-ui
    image: provectuslabs/kafka-ui:latest
    ports:
      - 8888:8080
    environment:
      KAFKA_CLUSTERS_0_NAME: YOUR_CLUSTER_NAME
      KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SSL
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "YOUR_BOOTSTRAP_PRIVATE_ENDPOINT"
      KAFKA_CLUSTERS_0_ZOOKEEPER: "YOUR_ZOOKEEPER_TLS"
      KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: "SSL"

Now, you will need to update some parameters:

• KAFKA_CLUSTERS_0_NAME : you need to change YOUR_CLUSTER_NAME to the name you want, something like: production, dev-cluster, etc…
• KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS : need to change to the value of the cluster’s private endpoint.
• KAFKA_CLUSTERS_0_ZOOKEEPER : you will need to change the value to the Zookeeper TLS connection string.

These 2 parameters KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS and KAFKA_CLUSTERS_0_ZOOKEEPER you will get the values from AWS Console. Go to the MSK service, choose your cluster and click to the button View client information .

Now, you will the see the dashboard look like below, get the values of the red circles.

Run the kafka-ui container

And this next step, after you’re configured the docker compose file with the correct cluster’s information.

Time to run it up. But the first of this step, you have to make sure your local machine already installed Docker and Docker Compose packages.

You can use this commands to verify it.

docker --version
docker compose verion

If you need more details about 2 packages above, refer to this docs.

https://docs.docker.com/engine/install/ubuntu/

https://docs.docker.com/compose/install/linux/#install-using-the-repository

Then move to the kafka-ui folder again, run below command.

# Switch to sudo or root user
sudo su

# Then run the compose
docker compose up

If you see the console log look like this, it’s running fine.

[+] Running 1/0
 ⠿ Container kafka-ui  Recreated                                                                                                                                                                                                         0.1s
Attaching to kafka-ui
kafka-ui  |  _   _ ___    __             _                _          _  __      __ _
kafka-ui  | | | | |_ _|  / _|___ _ _    /_\  _ __ __ _ __| |_  ___  | |/ /__ _ / _| |_____
kafka-ui  | | |_| || |  |  _/ _ | '_|  / _ \| '_ / _` / _| ' \/ -_) | ' </ _` |  _| / / _`|
kafka-ui  |  \___/|___| |_| \___|_|   /_/ \_| .__\__,_\__|_||_\___| |_|\_\__,_|_| |_\_\__,|
kafka-ui  |                                  |_|                                             
kafka-ui  | 
kafka-ui  | 2023-03-06 18:14:28,664 INFO  [background-preinit] o.h.v.i.u.Version: HV000001: Hibernate Validator 6.2.5.Final
kafka-ui  | 2023-03-06 18:14:28,714 INFO  [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 17.0.5 on 27862292b2df with PID 1 (/kafka-ui-api.jar started by kafkaui in /)
kafka-ui  | 2023-03-06 18:14:28,714 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v2.7.5, Spring v5.3.23
kafka-ui  | 2023-03-06 18:14:28,715 INFO  [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to 1 default profile: "default"
kafka-ui  | 2023-03-06 18:14:31,377 DEBUG [main] c.p.k.u.s.SerdesInitializer: Configuring serdes for cluster prod
kafka-ui  | 2023-03-06 18:14:32,011 INFO  [main] o.s.b.a.e.w.EndpointLinksResolver: Exposing 2 endpoint(s) beneath base path '/actuator'
kafka-ui  | 2023-03-06 18:14:32,169 INFO  [main] o.s.b.a.s.r.ReactiveUserDetailsServiceAutoConfiguration: 
kafka-ui  | 
...

Access the MSK cluster UI with kafka-ui dashboard

After above step, maybe a min, you can open your browser and type this one:

https://localhost:8888

If you can see the dashboard look like exactly above, congratulation! You’re almost done.

And now, you can click to Brokers, Topics, Consumers menu in the left and work with MSK cluster through them.

It’s cool right. I got this tool from my bro, he recommend it to me and now, I shared it with you guys.

Conclusion

Well, this post already showed you how to use the kafka-ui tool to connect to the MSK cluster. By using this tool, you can interact with the cluster via a dashboard instead of CLI. This is really cool and I hope it’s will useful for you guys. Cheers!