Restarting Rabbitmq systematically in vRealize Automation 8.x

Distributed Architecture in vRealize Automation 8.x has 3 nodes. Each node is part of the RabbitMQ cluster.

It is important RabbitMQ POD's are restarted sequentially.

The order does not matter, just kill one POD, wait for it to come (be up and running) do the same for the second one, and finally the last one.

To restart a pod just delete it by typing

kubectl -n prelude delete pod rabbitmq-ha-0

Wait until the pod is up and running: you may check it by calling

kubectl -n prelude get pods -l app=rabbitmq-ha

Result shall be something like
NAME READY STATUS RESTARTS AGE
 rabbitmq-ha-0 1/1 Running 0 1m
 rabbitmq-ha-1 1/1 Running 0 1d
 rabbitmq-ha-2 1/1 Running 0 1d

In the end, all 3 POD shall be restarted

Verify RabbitMQ cluster status is OK by executing the below command. Remember, this should be in a single line

for n in {0,1,2}; do echo node-$n; kubectl -n prelude exec -ti rabbitmq-ha-$n -- rabbitmqctl cluster_status; done

Each of the RMQ cluster nodes shall see all 3 members Then, restart EBS POD. The order doesn't matter, but it is important to restart all the 3 of them

kubectl -n prelude delete pod ebs-app-<cid>-<rid>

The POD name could be queried by typing

kubectl -n prelude get pods -l app=ebs-app