Search Results

I was working on a Migration Project where we did encounter a NULL POINTER EXCEPTION during migration Below Snippet is from migration-service-app.log 2021-07-22T05:37:12.928Z ERROR migration-service [host='migration-service-app-76b6d4c445-glgs6' thread='tasks-5' user='configadmin' org='1d22a685-bd12-4f22-89b1-cf0b5213a16f' trace='4cdb68a9-c202-4b27-851b-9c1ff4a6ace9'] c.v.a.m.s.AbstractMigrationService - null java.lang.NullPointerException: null at com.vmware.automation.migration.service.converter.VsphereReservationDeserializer.deserialize(VsphereReservationDeserializer.java:156) at com.vmware.automation.migration.service.converter.VsphereReservationDeserializer.deserialize(VsphereReservationDeserializer.java:29) at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4526) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3468) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3436) at com.vmware.automation.migration.service.converter.VsphereReservationConverterService.convert(VsphereReservationConverterService.java:105) at com.vmware.automation.migration.service.ReservationMigrationService.migrateReservation(ReservationMigrationService.java:152) at com.vmware.automation.migration.service.ReservationMigrationService.lambda$migrateReservations$0(ReservationMigrationService.java:110) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at com.vmware.automation.migration.service.ReservationMigrationService.migrateReservations(ReservationMigrationService.java:106) at com.vmware.automation.migration.service.ContentMigrationService.migrateContent(ContentMigrationService.java:124) at com.vmware.automation.migration.service.ContentMigrationService.lambda$migrateContent$6(ContentMigrationService.java:92) at com.vmware.automation.migration.service.ContentMigrationService.lambda$handleBusinessGroups$9(ContentMigrationService.java:161) at java.base/java.util.Optional.ifPresent(Optional.java:183) at com.vmware.automation.migration.service.ContentMigrationService.lambda$handleBusinessGroups$10(ContentMigrationService.java:161) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1603) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at com.vmware.automation.migration.service.ContentMigrationService.handleBusinessGroups(ContentMigrationService.java:158) at com.vmware.automation.migration.service.ContentMigrationService.migrateContent(ContentMigrationService.java:89) at com.vmware.automation.migration.service.ContentMigrationService$$FastClassBySpringCGLIB$$6f9a601d.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at io.opentracing.contrib.spring.cloud.async.TraceAsyncAspect.traceBackgroundThread(TraceAsyncAspect.java:56) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at io.opentracing.contrib.concurrent.TracedRunnable.run(TracedRunnable.java:30) at org.springframework.security.concurrent.DelegatingSecurityContextRunnable.run(DelegatingSecurityContextRunnable.java:82) at com.vmware.symphony.task.MDCTaskDecorator.lambda$decorate$0(MDCTaskDecorator.java:41) at com.vmware.symphony.task.RequestAttributesTaskDecorator.lambda$decorate$0(RequestAttributesTaskDecorator.java:27) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) What happens when you start a migration in vRA 8.x ? 1. Migration Triggers 2. Requests Business Groups assessment report 3. Marks Business Group from Assessing to Migrating 4. Get's the content: Source vRA Server ,Business Group ID , Content Type and Content ID 5. Converts vRA endpoints to Cloud Account ( that's the vSphere endpoint ) 6. Creates vSphere Cloud Account associations 7. Starts Migrating Business Group 8. Converts vSphere Reservation 9. Get's Endpoint Link 10. Get's Resource Pool Link 11. Get's Provisioning Region Link 12. Get's Storage Descriptions 13. Get's Subnet Name 14. Updates Subnet Network ID 15. Marks reservation tagged with the Business Group as converted 16. Assigns Tags & Creates Subnet Range 17. Assigns Tags & Creates Network Profile 18. Assigns Tags & Creates Storage Profile 19. Updates Property Configuration 20. Migrates Property Group for business group 21. Blueprint Migration starts 22. Parses Custom form if one exists, If there are any XaaS stuff , it get's it here 23. Converts blueprint 24. Versions and Releases Blueprint to Service Broker 25. Creates Policies 26. Migrates IP Allocations from Source 27. Creates IP Address State 28. Marks completion of Successful Business Group Migration The exception mentioned above happened during vSphere Reservation conversion We looked into the JSON response which we get during vSphere reservation conversion, as we can see below we are missing a whole computeResource section as compared to my lab This was the reason for the NULL POINTER EXCEPTION. But why is this not picking up the data was the question The answer was lying inside assessment logs 2021-07-21T01:24:22.418Z WARN assessment-service [host='assessment-service-app-6dfbfd896c-ljctk' thread='ForkJoinPool-2-worker-3' user='' org='' trace=''] c.v.a.capture.BusinessGroupServiceImpl - Error occurred while capturing machine prefix id 4e13bfce-dd56-4ddb-8c65-1d8ea5b9233a with error: I/O error on GET request for "https://<>/repository/data/ManagementModelEntities.svc/HostNamePrefixes": Connect to <>:443 [iaaswebfqdn/10.xx.xx.xx] failed: Connection timed out (Connection timed out); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to iaaswebfqdn:443 [iaaswebfqdn/10.xx.xx.xx] failed: Connection timed out (Connection timed out) Due to these timeouts during assessment of data, We are getting partial information and hence seems to be causing Null Pointer Exception So basically it turned out to be firewall port between the IAAS Web node and the vRA 8 Appliance is not open. Which led to failures. So ensure all appropriate ports are open before Migration.

After External vRealize Orchestrator 8.x upgrade to 8.4.2 , Users were unable to launch vRO UI , it fails with HTTP 400 response *** vco-server-app.log *** 2021-08-04T08:06:34.280Z WARN vco [host='vco-app-6df77fc467-tfzvt' thread='http-nio-8280-exec-2' user='-' org='-' trace='-'] {} com.vmware.identity.websso.client.SiteAffinity - Failed to init CdcSession. likely due to missing vmafd jar. Message: com/vmware/identity/cdc/CdcFactory 2021-08-04T08:06:34.280Z INFO vco [host='vco-app-6df77fc467-tfzvt' thread='http-nio-8280-exec-2' user='-' org='-' trace='-'] {} com.vmware.identity.websso.client.endpoint.SsoRequestSender - Added Renewable condition 2021-08-04T08:06:34.280Z INFO vco [host='vco-app-6df77fc467-tfzvt' thread='http-nio-8280-exec-2' user='-' org='-' trace='-'] {} com.vmware.identity.websso.client.endpoint.SsoRequestSender - Added Delegable condition 2021-08-04T08:06:34.280Z INFO vco [host='vco-app-6df77fc467-tfzvt' thread='http-nio-8280-exec-2' user='-' org='-' trace='-'] {} com.vmware.identity.websso.client.endpoint.SsoRequestSender - Destination URL: https://<>/websso/SAML2/SSO/vsphere.local *** vCenter Logs *** [2021-08-04T08:01:54.451Z tomcat-http--4 vsphere.local 7c66102f-55ad-44c6-b46f-235b056d20d0 ERROR com.vmware.identity.BaseSsoController] Could not parse tenant request java.lang.IllegalStateException: Issuer not recognized [2021-08-04T08:01:54.451Z tomcat-http--4 vsphere.local 7c66102f-55ad-44c6-b46f-235b056d20d0 INFO com.vmware.identity.samlservice.impl.SAMLAuthnResponseSender] Responded with ERROR 400 message Issuer not recognized [2021-08-04T08:01:54.451Z tomcat-http--4 vsphere.local 7c66102f-55ad-44c6-b46f-235b056d20d0 INFO com.vmware.identity.BaseSsoController] End processing SP-Initiated SSO response. Session was created. [2021-08-04T08:06:34.335Z tomcat-http--5 vsphere.local e1c95121-03fc-4d95-afc8-2a82e4e46499 INFO com.vmware.identity.SsoController] Welcome to SP-initiated AuthnRequest handler! The client locale is en_US, tenant is vshere.local [2021-08-04T08:06:34.335Z tomcat-http--5 vsphere.local e1c95121-03fc-4d95-afc8-2a82e4e46499 INFO com.vmware.identity. For remediation , go ahead and perform re-authentication. Post that it would restart the services of vco-server. Wait till its complete and then check the URL, it should be opening now and working

While installing kubernetes on my on-prem lab , i did face strange issues on making them working. Thought of putting all of the steps together that made this work Deployed 3 Ubuntu VM's Ensured all three virtual machines are communicating between each other Note : Below steps are to be performed on all of the three nodes As a first step , Login as ‘sudo’ user because the following set of commands need to be executed with ‘sudo’ permissions. Then, update your ‘apt-get’ repository. Do this on all of the three nodes We have to turn off the swap space because Kubernetes will start throwing random errors otherwise. After that you need to open the ‘fstab’ file and comment out the line which has mention of swap partition. $ sudo su # apt-get update #swapoff -a Second step , we need to install docker. Run the following commands # sudo su # apt-get update # apt-get install -y docker.io Third Step , we have to install these 3 essential components for setting up Kubernetes environment: kubeadm, kubectl, and kubelet Run below commands before installing kubernetes environment # apt-get update && apt-get install -y apt-transport-https curl # curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - # cat </etc/apt/sources.list.d/kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main EOF # apt-get update As Fourth Step, now its time to install the 3 essential components. kubelet is the lowest level component in Kubernetes. It’s responsible for what’s running on an individual machine. Kuebadm is used for administrating the Kubernetes cluster. Kubectl is used for controlling the configurations on various nodes inside the cluster. apt-get install -y kubelet kubeadm kubectl As Fifth step , we will change the configuration file of Kubernetes. Run the following command: # nano /etc/systemd/system/kubelet.service.d/10-kubeadm.conf This will open a text editor, enter the following line after the last “Environment Variable”: Environment="cgroup-driver=systemd/cgroup-driver=cgroupfs" As Sixth Step , create a file daemon.json under /etc/docker touch /etc/docker/daemon.json vi /etc/docker/daemon.json then paste this content in the json file { "exec-opts": ["native.cgroupdriver=systemd"] } Restart docker services sudo systemctl enable docker && sudo systemctl daemon-reload && sudo systemctl restart docker Seventh Step would be to initialize Kubernetes # kubeadm init --apiserver-advertise-address= --pod-network-cidr=10.244.0.0/16 Once initialized output would look like below Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 10.109.xx.xx:6443 --token jgzjjl.yyllcpqw5u2c9u9y \ --discovery-token-ca-cert-hash sha256:67feac133c1b3bda335a6179b99d5f88bf9cb3701ca7978fbc574d91d06abc92 Execute below commands as a non-root user mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Execute below command as root user export KUBECONFIG=/etc/kubernetes/admin.conf Unless Pod Networking or CNI is installed , the core-dns pods will be in pending state Eighth Step , Installing CNI. I've chosen flannel Execute below command to install it kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml root@master:/home/osadmin# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created Ninth Step , Join nodes to cluster root@workerone:/home/osadmin# kubeadm join 10.109.xx.xx:6443 --token jgzjjl.yyllcpqw5u2c9u9y --discovery-token-ca-cert-hash sha256:67feac133c1b3bda335a6179b99d5f88bf9cb3701ca7978fbc574d91d06abc92 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. root@workertwo:/home/osadmin# kubeadm join 10.109.46.53:6443 --token jgzjjl.yyllcpqw5u2c9u9y --discovery-token-ca-cert-hash sha256:67feac133c1b3bda335a6179b99d5f88bf9cb3701ca7978fbc574d91d06abc92 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. Last and Tenth Step is to list all the nodes and pods and see if they are ready This concludes the kubernetes cluster installation and configuration

Created a small mind map which will help in understand what goes on when we upgrade vRealize Automation 8.x. Below map also contains various logs one can monitor when need arises Created a runbook out of an upgrade performed in my lab recently. This should be helpful while performing an upgrade in your environments. This is no where a replacement of official VMware documentation or your own environment's runbook but this would act as a good reference point. Download the runbook I created by clicking on the attachment below !! Good Luck !!

While performing an Inventory Sync for vRealize Automation environment we've seen an error stating ERROR [scheduling-1] c.v.v.l.r.c.p.ProductInventorySyncPlanner - -- Error queuing the engine request for performing product sync java.util.NoSuchElementException: No value present So how do you debug and resolve this problem ? First of all let's understand what happens in the background when you initiate an inventory sync when it's working and compare it with this failed scenario This will give us a picture on where the problem is and how to resolve it Reference Log Snippet from my lab , taken from /var/log/vrlcm/vmware_vrlcm.log **** Inventory Sync starts here **** 2021-10-04 04:38:38.168 INFO [http-nio-8080-exec-4] c.v.v.l.l.u.RequestSubmissionUtil - -- ++++++++++++++++++ Creating request to Request_Service :::>>> { "vmid" : "4026aa6b-4157-4da5-917b-f8b7952eadd5", "transactionId" : null, "requestName" : "productinventorysync", "requestReason" : "VRA in Environment ARUN - Product Inventory Sync", "requestType" : "PRODUCT_INVENTORY_SYNC", "requestSource" : "3eb03039-1ea0-4924-b251-8fb14c63315c", "requestSourceType" : "user", "inputMap" : { "environmentId" : "3eb03039-1ea0-4924-b251-8fb14c63315c", "productId" : "vra", "tenantId" : "" }, "outputMap" : { }, "state" : "CREATED", "executionId" : null, "executionPath" : null, "executionStatus" : null, "errorCause" : null, "resultSet" : null, "isCancelEnabled" : null, "lastUpdatedOn" : 1633322318167, "createdBy" : null } **** Generates the request id **** 2021-10-04 04:38:38.172 INFO [http-nio-8080-exec-4] c.v.v.l.l.u.RequestSubmissionUtil - -- Generic Request Response : { "requestId" : "4026aa6b-4157-4da5-917b-f8b7952eadd5" } 2021-10-04 04:38:38.707 INFO [scheduling-1] c.v.v.l.r.c.RequestProcessor - -- Number of request to be processed : 1 **** Creates Spec **** 2021-10-04 04:38:38.725 INFO [scheduling-1] c.v.v.l.r.c.p.ProductInventorySyncPlanner - -- Creating spec for inventory sync for product : vra in environment : ARUN 2021-10-04 04:38:38.725 INFO [scheduling-1] c.v.v.l.r.u.InfrastructurePropertiesHelper - -- VCF properties: { "vcfEnabled" : false, "sddcManagerDetails" : [ ] } **** Finds the product id with vIDM **** **** Since it's not a clustered environment it goes ahead and moves on with the Environment Planner **** 2021-10-04 04:38:38.727 INFO [scheduling-1] c.v.v.l.r.c.p.ProductInventorySyncPlanner - -- Found product with id vidm 2021-10-04 04:38:38.728 INFO [scheduling-1] c.v.v.l.r.c.p.ProductInventorySyncPlanner - -- CreateEnvironmentPlanner.addVsscImportStateMachine environmentId : 3eb03039-1ea0-4924-b251-8fb14c63315c 2021-10-04 04:38:38.728 INFO [scheduling-1] c.v.v.l.c.u.StringUtil - -- QUERYING CONTENT :: ProductInventory::resources::dataCenters->8efd48d7-d813-408f-aa55-be997e4463f7 2021-10-04 04:38:38.728 INFO [scheduling-1] c.v.v.l.d.i.u.InventoryWriteUtil - -- QUERY MAP LENGTH :: 3 2021-10-04 04:38:38.728 INFO [scheduling-1] c.v.v.l.d.i.u.InventorySchemaQueryUtil - -- GETTING ROOT NODE FOR :: ProductInventory 2021-10-04 04:38:38.831 INFO [scheduling-1] c.v.v.l.c.u.StringUtil - -- QUERYING CONTENT :: ProductInventory::resources::dataCenters->8efd48d7-d813-408f-aa55-be997e4463f7 2021-10-04 04:38:38.831 INFO [scheduling-1] c.v.v.l.d.i.u.InventoryWriteUtil - -- QUERY MAP LENGTH :: 3 2021-10-04 04:38:38.831 INFO [scheduling-1] c.v.v.l.d.i.u.InventorySchemaQueryUtil - -- GETTING ROOT NODE FOR :: ProductInventory 2021-10-04 04:38:38.947 INFO [scheduling-1] c.v.v.l.c.u.StringUtil - -- QUERYING CONTENT :: ProductInventory::resources::dataCenters->8efd48d7-d813-408f-aa55-be997e4463f7::regions->default::zones->default::vCenters 2021-10-04 04:38:38.947 INFO [scheduling-1] c.v.v.l.d.i.u.InventoryWriteUtil - -- QUERY MAP LENGTH :: 6 2021-10-04 04:38:38.947 INFO [scheduling-1] c.v.v.l.d.i.u.InventorySchemaQueryUtil - -- GETTING ROOT NODE FOR :: ProductInventory 2021-10-04 04:38:39.041 INFO [scheduling-1] c.v.v.l.l.s.DataCenterServiceImpl - -- VCENTER RESOURCE RETRIEVED IS { * * * aged\":true},{\"storageName\":\"ISOs\",\"totalSize\":6.333186977792E12,\"availableSize\":2.783035342848E12,\"isManaged\":true},{\"storageName\":\"local_13\",\"totalSize\":4.61977419776E11,\"availableSize\":4.6046117888E11,\"isManaged\":true},{\"storageName\":\"local_11\",\"totalSize\":4.61977419776E11,\"availableSize\":4.6046117888E11,\"isManaged\":true},{\"storageName\":\"local_14\",\"totalSize\":4.61977419776E11,\"availableSize\":4.6046117888E11,\"isManaged\":true}],\"networks\":[{\"network\":\"mgmtnetwork\",\"isManaged\":null},{\"network\":\"ncdvswitch-DVUplinks-1050\",\"isManaged\":null},{\"network\":\"vmnetwork\",\"isManaged\":null},{\"network\":\"storage\",\"isManaged\":null},{\"network\":\"vmdvswitch-DVUplinks-1055\",\"isManaged\":null}],\"virtualMachineTemplates\":[],\"isManaged\":null,\"clusterName\":\"nccluster\",\"resourcePools\":[{\"name\":\"tkgresourcegroup\",\"children\":null,\"morId\":\"resgroup-1046\"}]}],\"folders\":[{\"name\":\"harpreet\",\"morId\":\"group-v1039\",\"children\":null},{\"name\":\"vramanaged\",\"morId\":\"group-v1028\",\"children\":null},{\"name\":\"templates\",\"morId\":\"group-v42\",\"children\":null},{\"name\":\"infra\",\"morId\":\"group-v38\",\"children\":null},{\"name\":\"tkgmvms\",\"morId\":\"group-v1047\",\"children\":null},{\"name\":\"kubernetes\",\"morId\":\"group-v46\",\"children\":null}]}],\"templateCustomSpecs\":[],\"contentLibraries\":[],\"name\":\"nc\"}", "vcUsername" : "arun@nc.com", "vCenterHost" : "vc.nc.com", "vCenterName" : "nc", "vcPassword" : "JXJXJXJX", "vcUsedAs" : "MANAGEMENT_AND_WORKLOAD" } In the environment where the failure was seen **** Inventory Sync starts here **** 2021-09-28 01:48:02.378 INFO [http-nio-8080-exec-6] c.v.v.l.l.u.RequestSubmissionUtil - -- ++++++++++++++++++ Creating request to Request_Service :::>>> { "vmid" : "3a1ce775-f36f-4753-9c6a-83f09ef52647", "transactionId" : null, "requestName" : "productinventorysync", "requestReason" : "VRA in Environment TESTENV - Product Inventory Sync", "requestType" : "PRODUCT_INVENTORY_SYNC", "requestSource" : "c150a269-6278-4f1e-b16a-fdc203756e11", "requestSourceType" : "user", "inputMap" : { "environmentId" : "c1dd269-6278-4f1e-b16a-fdc2ddd756e11", "productId" : "vra" }, "outputMap" : { }, "state" : "CREATED", "executionId" : null, "executionPath" : null, "executionStatus" : null, "errorCause" : null, "resultSet" : null, "isCancelEnabled" : null, "lastUpdatedOn" : 1632793682377, "createdBy" : null } **** Generates the request id **** 2021-09-28 01:48:02.384 INFO [http-nio-8080-exec-6] c.v.v.l.l.u.RequestSubmissionUtil - -- Generic Request Response : { "requestId" : "3a1ce775-f36f-4753-9c6a-83f09ef52647" } 2021-09-28 01:48:02.853 INFO [scheduling-1] c.v.v.l.r.c.RequestProcessor - -- Number of request to be processed : 1 **** Creates Spec **** **** Because this is a vCF Managed environment , we do see the properties of vCF listed below else , these would be blank **** 2021-09-28 01:48:02.863 INFO [scheduling-1] c.v.v.l.r.c.p.ProductInventorySyncPlanner - -- Creating spec for inventory sync for product : vra in environment : TESTENV 2021-09-28 01:48:02.863 INFO [scheduling-1] c.v.v.l.r.u.UserLoginUtil - -- Is local admin false 2021-09-28 01:48:02.863 INFO [scheduling-1] c.v.v.l.r.u.UserLoginUtil - -- Is service admin false 2021-09-28 01:48:02.863 INFO [scheduling-1] c.v.v.l.r.u.UserLoginUtil - -- Is VCF admin true 2021-09-28 01:48:02.864 INFO [scheduling-1] c.v.v.l.r.u.InfrastructurePropertiesHelper - -- VCF properties: { "vcfEnabled" : true, "sddcManagerDetails" : [ { "sddcManagerName" : "default", "sddcManagerVmid" : "default", "sddcManagerHostName" : "sddcmanager.nc.com", "properties" : null } ] } **** Finds the product id with vIDM **** 2021-09-28 01:48:02.866 INFO [scheduling-1] c.v.v.l.r.c.p.ProductInventorySyncPlanner - -- Found product with id vidm **** Identifies vIDm clustered lb hostname **** **** Only occurs if vIDM is in clustered mode **** 2021-09-28 01:48:02.866 INFO [scheduling-1] c.v.v.l.r.c.p.CreateEnvironmentPlanner - -- vIDM clustered LB hostname idm.nc.com **** Exception occurs here **** 2021-09-28 01:48:02.866 ERROR [scheduling-1] c.v.v.l.r.c.p.ProductInventorySyncPlanner - -- Error queuing the engine request for performing product sync java.util.NoSuchElementException: No value present at java.util.Optional.get(Unknown Source) ~[?:?] at com.vmware.vrealize.lcm.requestservice.core.planner.ProductInventorySyncPlanner.processRequest(ProductInventorySyncPlanner.java:115) ~[vmlcm-requestservice-core-8.2.0-SNAPSHOT.jar!/:?] at com.vmware.vrealize.lcm.requestservice.core.RequestProcessor.invokeExecutionPlanner(RequestProcessor.java:296) ~[vmlcm-requestservice-core-8.2.0-SNAPSHOT.jar!/:?] at com.vmware.vrealize.lcm.requestservice.core.RequestProcessor$$FastClassBySpringCGLIB$$58375e43.invoke() ~[vmlcm-requestservice-core-8.2.0-SNAPSHOT.jar!/:?] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.17.RELEASE.jar!/:5.1.17.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:752) ~[spring-aop-5.1.17.RELEASE.jar!/:5.1.17.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.17.RELEASE.jar!/:5.1.17.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) [spring-tx-5.1.17.RELEASE.jar!/:5.1.17.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) [spring-tx-5.1.17.RELEASE.jar!/:5.1.17.RELEASE] * * * at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?] 2021-09-28 01:48:02.878 INFO [scheduling-1] c.v.v.l.r.c.RequestProcessor - -- Engine Spec generated is null for Request ID : 3a1ce775-f36f-4753-9c6a-83f09ef52647, So marking the request as FAILED What does the "no value present mean and what is it referring to ? It's telling you that when we select the product in an environment it's does not see the product information and it's missing or null Ideally it should be in this way , one should be able to see the nodes which constitute a product and it's properties To resolve , all we had to do is to remove the product from inventory and then re-import it back to vRSLCM. Remember when your deleting the product from vRSLCM never check "Delete VM's from vCenter" option. This will delete the whole VM's from vCenter. So be very careful and extra cautious.

This blog complies with the new version released on 21st December 2021 Note All instructions and procedures are taken from VMware KB: https://kb.vmware.com/s/article/87120 , all i am trying to do is to add some screenshots and outputs by implementing this workaround in my lab Download this Note which has screenshots and detailed snippets VMware updated KB article 87120 with new commands on 21st December 2021. This blog article complies with it Both the PDF document attached and the blog screenshots are taken after the new commands are tested Symptoms CVE-2021-44228 has been determined to impact vRA and vRO from 8.0 to 8.6.1 via the Apache Log4j open source component it ships. This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing: CVE-2021-44228 - VMSA-2021-0028 (link: https://www.vmware.com/security/advisories/VMSA-2021-0028.html) Notice: On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believe the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors.  We expect to fully address both CVE-2021-44228 and CVE-2021-45046 by updating log4j to version 2.16 in forthcoming releases of 8.6.2, as outlined by our software support policies. VMSA-2021-0028 will be updated when these releases are available. In the interim, we will be updating this Knowledge Base article with revised guidance to remove all JndiLookup classes per Apache Software Foundation guidance. Please subscribe to this article to be informed when updates are published. The workarounds described in this document are meant to be a temporary solution only. Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 when available. Long-term resolution will be available in vRA and vRO versions 8.6.2 or later. Purpose CVE-2021-44228 and CVE-2021-45046 have been determined to impact vRA and vRO from 8.0 to 8.6.1 via the Apache Log4j open source component it ships. This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing: CVE-2021-44228, CVE-2021-45046 - VMSA-2021-0028 Impact and Risks Please note the following prior to executing the workaround procedure: vRA and vRO versions 8.0, 8.0.1 and 8.1 are no longer supported at the time of this article's publication. Re-apply this KB, If you have previously applied the workaround released prior to 12/20/2021. If not re-applied, upgrading to a future release will not be possible. This change is persistent over upgrades and the KB should not be re-applied. This change applies to vRA and vRO (both standalone and embedded). Can be applied to all vRA and vRO deployments versions 8.1 through 8.6.1. For clustered setups, in vRO Control Center/Cluster management page, a warning message "Local changes detected" may appear after applying this KB. This message should be disregarded. Note: Automated vulnerability scanners may report that vRA/vRO products are still vulnerable to CVE-2021-44228 and CVE-2021-45046 after this KB article has been applied. These findings can be safely ignored. Resolution The workarounds described in this document can be considered as permanent solution as they update log4j libraries in the VA to 2.17.0. Future releases will include log4j 2.17.0 or later Workaround For each vRA and vRO deployments with versions from 8.1 to 8.6.1, execute the following procedure Pre-requisites Take simultaneous VM snapshots without memory of all nodes in the cluster For this task i would leverage vRSLCM as shown below in the screenshots Select the product and click on the day-2 actions pane on the product and choose Create Snapshot I will choose an option which would create snapshot after shutting down the appliances. This is the best option and recommended option too. If you cannot shutdown the production then that is fine , but take snapshots through LCM Run the Precheck Ensure precheck is successful Snapshot task does not take a long time but shutting down the application and bringing it back on takes a little bit of time but it's worth it Procedure Note: This workaround applies to vRA and vRO (both standalone and embedded). Note: To be applied to all vRA and vRO deployments versions 8.2 through 8.6.1. SSH login or virtual machine console into one of the nodes in the vRA / vRO cluster. Ensure all the pods are in running state Step:1 Upload <87120-kb-v2.tar.gz> and <87120-kb-v2-validate.tar.gz> under /root on all nodes. Step:2 Validate whether the system is vulnerable by running the command below on all nodes. Error reports related to log4j will show up for the affected artifacts that are vulnerable. cd /root; base64 -d <<< "W1sgIiQoc2hhMjU2c3VtIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiB8IGN1dCAtZiAxIC1kICcgJykiID09ICJmY2IxZjQ0YWRkMmNjZTg3MzNjNzAzOTYzNjg4Njk3NjU4ZjA3NTkzMzYyZTRhMzU0MjZlYjc5OGFjMTM5YzRlIiBdXSAmJiAocm0gLXJmIC90bXAvODcxMjAta2I7IG1rZGlyIC1wIC90bXAvODcxMjAta2IvOyB0YXIgLXh2ZiA4NzEyMC1rYi12Mi12YWxpZGF0ZS50YXIuZ3ogLUMgL3RtcC84NzEyMC1rYjsgY2htb2QgK3ggL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgcm0gLXJmIC90bXAvODcxMjAta2IpIHx8IGVjaG8gIkZpbGUgbm90IGZvdW5kIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiBvciBjaGVja3N1bSBtaXNtYXRjaCIK" | bash - Complete Output looks like this ... root@vra [ ~ ]# cd /root; base64 -d <<< "W1sgIiQoc2hhMjU2c3VtIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiB8IGN1dCAtZiAxIC1kICcgJykiID09ICJmY2IxZjQ0YWRkMmNjZTg3MzNjNzAzOTYzNjg4Njk3NjU4ZjA3NTkzMzYyZTRhMzU0MjZlYjc5OGFjMTM5YzRlIiBdXSAmJiAocm0gLXJmIC90bXAvODcxMjAta2I7IG1rZGlyIC1wIC90bXAvODcxMjAta2IvOyB0YXIgLXh2ZiA4NzEyMC1rYi12Mi12YWxpZGF0ZS50YXIuZ3ogLUMgL3RtcC84NzEyMC1rYjsgY2htb2QgK3ggL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgcm0gLXJmIC90bXAvODcxMjAta2IpIHx8IGVjaG8gIkZpbGUgbm90IGZvdW5kIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiBvciBjaGVja3N1bSBtaXNtYXRjaCIK" | bash - 87120-kb-v2-validate.sh Scanning blueprint-webapp_private:latest at: > Scanning /tmp/patc-verify-container.rvwtc ERROR: found ./opt/vmware/lib/log4j-core-2.14.1.jar Scanning catalog-service_private:latest at: > Scanning /tmp/patc-verify-container.R7Doo >> Scanning ./opt/service/cs-host.jar ERROR: found ./BOOT-INF/lib/log4j-core-2.14.1.jar Scanning codestream_private:latest at: > Scanning /tmp/patc-verify-container.6WwhN ERROR: found ./opt/codestream/lib/log4j-core-2.13.3.jar Scanning content-service_private:latest at: > Scanning /tmp/patc-verify-container.GovpH ERROR: found ./opt/vmware/lib/log4j-core-2.14.1.jar Scanning identity-service_private:latest at: > Scanning /tmp/patc-verify-container.P5Zyo >> Scanning ./opt/bc-fips/bctls-fips-1.0.12.1.jar >> Scanning ./opt/bc-fips/bcpkix-fips-1.0.5.jar >> Scanning ./opt/bc-fips/bcmail-fips-1.0.3.jar >> Scanning ./opt/bc-fips/bc-fips-1.0.2.1.jar >> Scanning ./jdk/lib/jrt-fs.jar >> Scanning ./identity-service/lib/identity-service-1.5.4-SNAPSHOT.jar ERROR: found ./BOOT-INF/lib/log4j-core-2.14.1.jar Scanning provisioning-service_private:latest at: > Scanning /tmp/patc-verify-container.mYkqN ERROR: found ./admiral/log4j-core-2.13.3.jar Scanning relocation-service_private:latest at: > Scanning /tmp/patc-verify-container.yesgE >> Scanning ./opt/relocation/relocation-service.jar ERROR: found ./BOOT-INF/lib/log4j-core-2.12.1.jar Scanning vco_private:latest at: > Scanning /tmp/patc-verify-container.YSm3Q >> Scanning ./var/opt/apache-tomcat/lib/websocket-api.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-websocket.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-util.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-util-scan.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-jni.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-jdbc.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-zh-CN.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-ru.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-ko.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-ja.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-fr.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-es.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-de.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-dbcp.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-coyote.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-api.jar >> Scanning ./var/opt/apache-tomcat/lib/servlet-api.jar >> Scanning ./var/opt/apache-tomcat/lib/jsp-api.jar >> Scanning ./var/opt/apache-tomcat/lib/jaspic-api.jar >> Scanning ./var/opt/apache-tomcat/lib/jasper.jar >> Scanning ./var/opt/apache-tomcat/lib/jasper-el.jar >> Scanning ./var/opt/apache-tomcat/lib/el-api.jar >> Scanning ./var/opt/apache-tomcat/lib/ecj-4.6.3.jar >> Scanning ./var/opt/apache-tomcat/lib/catalina.jar >> Scanning ./var/opt/apache-tomcat/lib/catalina-tribes.jar >> Scanning ./var/opt/apache-tomcat/lib/catalina-storeconfig.jar >> Scanning ./var/opt/apache-tomcat/lib/catalina-ha.jar >> Scanning ./var/opt/apache-tomcat/lib/catalina-ant.jar >> Scanning ./var/opt/apache-tomcat/lib/annotations-api.jar >> Scanning ./var/opt/apache-tomcat/bin/tomcat-juli.jar >> Scanning ./var/opt/apache-tomcat/bin/commons-daemon.jar >> Scanning ./var/opt/apache-tomcat/bin/bootstrap.jar >> Scanning ./var/opt/apache-ant/lib/maven-ant-tasks-2.1.3.jar >> Scanning ./var/opt/apache-ant/lib/ant.jar >> Scanning ./var/opt/apache-ant/lib/ant-xz.jar >> Scanning ./var/opt/apache-ant/lib/ant-testutil.jar >> Scanning ./var/opt/apache-ant/lib/ant-swing.jar >> Scanning ./var/opt/apache-ant/lib/ant-netrexx.jar >> Scanning ./var/opt/apache-ant/lib/ant-launcher.jar >> Scanning ./var/opt/apache-ant/lib/ant-junitlauncher.jar >> Scanning ./var/opt/apache-ant/lib/ant-junit4.jar >> Scanning ./var/opt/apache-ant/lib/ant-junit.jar >> Scanning ./var/opt/apache-ant/lib/ant-jsch.jar >> Scanning ./var/opt/apache-ant/lib/ant-jmf.jar >> Scanning ./var/opt/apache-ant/lib/ant-jdepend.jar >> Scanning ./var/opt/apache-ant/lib/ant-javamail.jar >> Scanning ./var/opt/apache-ant/lib/ant-jai.jar >> Scanning ./var/opt/apache-ant/lib/ant-imageio.jar >> Scanning ./var/opt/apache-ant/lib/ant-commons-net.jar >> Scanning ./var/opt/apache-ant/lib/ant-commons-logging.jar >> Scanning ./var/opt/apache-ant/lib/ant-apache-xalan2.jar >> Scanning ./var/opt/apache-ant/lib/ant-apache-resolver.jar >> Scanning ./var/opt/apache-ant/lib/ant-apache-regexp.jar >> Scanning ./var/opt/apache-ant/lib/ant-apache-oro.jar >> Scanning ./var/opt/apache-ant/lib/ant-apache-log4j.jar >> Scanning ./var/opt/apache-ant/lib/ant-apache-bsf.jar >> Scanning ./var/opt/apache-ant/lib/ant-apache-bcel.jar >> Scanning ./var/opt/apache-ant/lib/ant-antlr.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/security/policy/unlimited/local_policy.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/security/policy/unlimited/US_export_policy.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/security/policy/limited/local_policy.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/rt.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/resources.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/management-agent.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/jsse.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/jfr.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/jce.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/zipfs.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/sunpkcs11.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/sunjce_provider.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/sunec.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/nashorn.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/localedata.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/jaccess.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/dnsns.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/ext/cldrdata.jar >> Scanning ./usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/charsets.jar >> Scanning ./tmp/dejavu-fonts-ttf-2.37.zip Scanning /data/vco at: > Scanning /tmp/patc-verify-directory.baK9I/ ERROR: found ./vco/usr/lib/vco/configuration/webapps/vco-controlcenter/WEB-INF/lib/log4j-core-2.13.3.jar ./vco/usr/lib/vco/app-server/temp/dars/o11nplugin-configurator.dar/lib/log4j-core-2.13.3.jar ./vco/usr/lib/vco/app-server/temp/dars/o11nplugin-multi-node.dar/lib/log4j-core-2.13.3.jar ./vco/usr/lib/vco/app-server/temp/dars/o11nplugin-vsphere.dar/lib/log4j-core-2.13.3.jar ./vco/usr/lib/vco/app-server/lib/log4j-core-2.13.3.jar Scanning Java processes Done. Step:3 If the system is vulnerable, install the KB by executing the following command on all nodes: cd /root; base64 -d <<< "W1sgIiQoc2hhMjU2c3VtIDg3MTIwLWtiLXYyLnRhci5neiB8IGN1dCAtZiAxIC1kICcgJykiID09ICJkNDdlYzc4ZWJmZjY1M2JjNTg3Y2I5NTMwOTkwNDhlMTBhMzc5OTUzZGFjZWZlMGY5ODMzODRiOTRiMDAyZTFiIiBdXSAmJiAocm0gLXJmIC90bXAvODcxMjAta2I7IG1rZGlyIC1wIC90bXAvODcxMjAta2IvOyB0YXIgLXh2ZiA4NzEyMC1rYi12Mi50YXIuZ3ogLUMgL3RtcC84NzEyMC1rYjsgY2htb2QgK3ggL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi5zaDsgL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi5zaDsgcm0gLWZyIC90bXAvODcxMjAta2IpIHx8IGVjaG8gIkZpbGUgbm90IGZvdW5kIDg3MTIwLWtiLXYyLnRhci5neiBvciBjaGVja3N1bSBtaXNtYXRjaCIK" | bash - Step:4 Make the installation effective by executing /opt/scripts/deploy.sh from the node that is typically used as primary (e.g. where /var/log/deploy.log file exists from previous runs). This is run only once across the vRA/vRO cluster nodes. Step:5 Verify the KB is active on the system by running the verification command below on all nodes. There should be no error reports related to log4j. cd /root; base64 -d <<< "W1sgIiQoc2hhMjU2c3VtIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiB8IGN1dCAtZiAxIC1kICcgJykiID09ICJmY2IxZjQ0YWRkMmNjZTg3MzNjNzAzOTYzNjg4Njk3NjU4ZjA3NTkzMzYyZTRhMzU0MjZlYjc5OGFjMTM5YzRlIiBdXSAmJiAocm0gLXJmIC90bXAvODcxMjAta2I7IG1rZGlyIC1wIC90bXAvODcxMjAta2IvOyB0YXIgLXh2ZiA4NzEyMC1rYi12Mi12YWxpZGF0ZS50YXIuZ3ogLUMgL3RtcC84NzEyMC1rYjsgY2htb2QgK3ggL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgcm0gLXJmIC90bXAvODcxMjAta2IpIHx8IGVjaG8gIkZpbGUgbm90IGZvdW5kIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiBvciBjaGVja3N1bSBtaXNtYXRjaCIK" | bash - Step:6 Validate Pods and see it's all running vRealize Automation has the workaround implemented now. Go ahead and perform respective tests and it's all BAU now

Recently was working on an vRealize Automation environment 8.x version ( vCF based ) where an authenticated proxy was involved for external connectivity. We were trying to add an VMware Cloud on AWS cloud account and it was failing with the below error Looking into provisioning-service-app.log , we see the actual error 2021-10-05T04:06:51.975Z [priority='INFO' thread='reactor-http-epoll-13' user='' org='' context='' parent='' token=''] com.vmware.xenon.common.SpringHostUtils.responseEntityToOperation:901 - [POST https://console. cloud.vmware.com/csp/gateway/am/api/auth/api-tokens/authorize] Exception received with no ClientResponse: java.util.concurrent.CompletionException: javax.net.ssl.SSLException: failure when writing TLS control frames The whole exceptions is as below 2021-10-05T04:06:51.726Z [priority='INFO' thread='reactor-http-epoll-15' user='arun' org='ce3fdd9b-f3f1-41f1-8622-7cb1e53fae71' context='f9899447-69e3-4843-bae0-303ace93b1e4' parent='' token='c2559493-fae 5-4faf-ba28-7f1fddcb58dc'] com.vmware.xenon.common.SpringHostUtils.sendRequest:233 - Sending POST http://10.244.10.110:8282/provisioning/mgmt/vmc-sddc (referer http://10.244.10.110:8282/provisioning/uerp) as a remote request to change auth context 2021-10-05T04:06:51.975Z [priority='INFO' thread='reactor-http-epoll-13' user='' org='' context='' parent='' token=''] com.vmware.xenon.common.SpringHostUtils.responseEntityToOperation:901 - [POST https://console. cloud.vmware.com/csp/gateway/am/api/auth/api-tokens/authorize] Exception received with no ClientResponse: java.util.concurrent.CompletionException: javax.net.ssl.SSLException: failure when writing TLS control frames at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) at reactor.core.publisher.MonoToCompletableFuture.onError(MonoToCompletableFuture.java:76) at io.opentracing.contrib.reactor.TracedSubscriber.onError(TracedSubscriber.java:79) at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) * * * at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:818) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: javax.net.ssl.SSLException: failure when writing TLS control frames at io.netty.handler.ssl.SslHandler.setHandshakeFailureTransportFailure(SslHandler.java:1855) Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: Error has been observed at the following site(s): |_ checkpoint ⇢ Request to POST https://console.cloud.vmware.com/csp/gateway/am/api/auth/api-tokens/authorize [DefaultWebClient] Stack trace: at io.netty.handler.ssl.SslHandler.setHandshakeFailureTransportFailure(SslHandler.java:1855) at io.netty.handler.ssl.SslHandler.access$600(SslHandler.java:167) at io.netty.handler.ssl.SslHandler$2.operationComplete(SslHandler.java:970) at io.netty.handler.ssl.SslHandler$2.operationComplete(SslHandler.java:965) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)* * * * at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: io.netty.handler.proxy.ProxyConnectException: http, none, proxy-service.prelude.svc.cluster.local/10.244.14.224:3128 => console.cloud.vmware.com:443, disconnected at io.netty.handler.proxy.ProxyHandler.channelInactive(ProxyHandler.java:234) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) * * at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:818) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) When this error occurs we see following statements under proxy-service-xxxxx/squid-proxy.log 1633406811.973 epoch timestamp relates to the timestamp when the error occurred in the UI. 1633406811.973 --> 2021-10-05T04:06:51 1633406811.973 240 10.244.0.110 TCP_TUNNEL/407 293 CONNECT console.cloud.vmware.com:443 - FIRSTUP_PARENT/59.154.134.108 - Inspecting older squid-proxy logs, whenever an attempt to add a new VMC vCenter cloud account was done , we did see this statement proxy-service/squid-proxy.log-202110020000.xz_extracted/squid-proxy.log-202110020000:1633063657.998 108 10.244.10.110 TCP_TUNNEL/407 293 CONNECT console.cloud.vmware.com:443 - FIRSTUP_PARENT/59.154.134.108 - proxy-service/squid-proxy.log:1633406811.973 240 10.244.10.110 TCP_TUNNEL/407 293 CONNECT console.cloud.vmware.com:443 - FIRSTUP_PARENT/59.154.134.108 - proxy-service/squid-proxy.log-202110010000.xz_extracted/squid-proxy.log-202110010000:1633008305.662 980 10.244.10.110 TCP_TUNNEL/407 293 CONNECT console.cloud.vmware.com:443 - FIRSTUP_PARENT/59.154.134.108 - proxy-service/squid-proxy.log-202110010000.xz_extracted/squid-proxy.log-202110010000:1633008582.091 1413 10.244.10.110 TCP_TUNNEL/407 293 CONNECT console.cloud.vmware.com:443 - FIRSTUP_PARENT/59.154.134.108 - proxy-service/squid-proxy.log-202110010000.xz_extracted/squid-proxy.log-202110010000:1633008851.485 2341 10.244.10.110 TCP_TUNNEL/407 287 CONNECT console.cloud.vmware.com:443 - FIRSTUP_PARENT/59.154.134.109 - proxy-service/squid-proxy.log-202110010000.xz_extracted/squid-proxy.log-202110010000:1633008862.656 24 10.244.10.110 TCP_TUNNEL/407 287 CONNECT console.cloud.vmware.com:443 - FIRSTUP_PARENT/59.154.134.109 - Let's understand this squid-proxy statement in depth once As we can see all of the result codes are TCP_TUNNEL/407. What do we understand by this based on the search on google The HTTP 407 Proxy Authentication Required client error status response code indicates that the request has not been applied because it lacks valid authentication credentials for a proxy server that is between the browser and the server that can access the requested resource. The vracli proxy show output on the node was as below root@node02[ ~ ]# vracli proxy show { "enabled": true, "host": "proxy-service.prelude.svc.cluster.local", "java-proxy-exclude": ********", "java-user": "arun@nc.com", "password": null, "port": 3128, "proxy-exclude": "************", "scheme": "http", "upstream_proxy_host": "******.lb.service.dev", "upstream_proxy_password_encoded": "*CENSORED*", "upstream_proxy_port": 80, "upstream_proxy_user_encoded": "arun%40nc.com", "user": null, "internal.proxy.config": "*******", "internal.proxy.config.type": "non-default" } After we reached this point , one of our peer pointed out that he did recently face an issue where if the java_user is set to UPN and not just a USERNAME , then TCP_TUNNEL/407 will occur So based on this we decided to reconfigure the proxy which has been already set root@node02[ ~ ]# vracli proxy show { "enabled": true, "host": "proxy-service.prelude.svc.cluster.local", "java-proxy-exclude": ********", "java-user": "arun", "password": null, "port": 3128, "proxy-exclude": "************", "scheme": "http", "upstream_proxy_host": "******.lb.service.dev", "upstream_proxy_password_encoded": "*CENSORED*", "upstream_proxy_port": 80, "upstream_proxy_user_encoded": "arun", "user": null, "internal.proxy.config": "*******", "internal.proxy.config.type": "non-default" } Once we made the change , we were able to validate the API and there were no exceptions I will try and gather the reason behind why it does not work with UPN and works with USERNAME and update it here. Yea , if you have a proxy configured and adding a VMware Cloud on AWS endpoint keep this in mind

Source of Truth : VMware KB: 87121 ( https://kb.vmware.com/s/article/87121 ) Note: I am trying to share screenshots / outputs from the steps mentioned in the knowledge base : 87121 by implementing them in my lab You may download PDF version here Purpose Notice: On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believe the instructions in this article to be an effective mitigation for CVE-2021-44228 and CVE-2021-45046, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors. Resolution The workarounds described in this document are meant to be a temporary solution only. Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 when available. Workaround Prerequisites Backup the vRA appliance nodes Snapshots ( My vRealize Automation environment is managed by vRSLCM so i'll use it to take snapshots ) Snapshot is now complete Let's now move to the actual procedure of workaround implementation Procedure Note: Steps #2 - #3 apply to the embedded vRealize Orchestrator 7.6 instance. For external instances, see Workaround instructions to address CVE-2021-44228 in vRealize Orchestrator 7. SSH into each vRA appliance node and run the following steps: Step:1 Stop the vco-configurator service on each vRA node Step:2 Run the following command to update vRO configuration on each vRA node: base64 -d <<< "bG9nX21lc3NhZ2VfbG9nNGooKSB7CiAgZWNobyAiWyQoZGF0ZSAtLXV0YyAiKyVGVCVULiUzTloiKV0gJDEiIHwgdGVlIC1hIC92YXIvbG9nL3Ztd2FyZS92Y28vYXBwLXNlcnZlci92Y29fbG9nNGpfY3ZlLmxvZwp9Cgpsb2dfZXJyb3JfbG9nNGooKSB7CiAgbG9nX21lc3NhZ2VfbG9nNGogIkVSUk9SOiAkMSIKICBleGl0IDEKfQoKc2V0X2phdmFfb3B0KCkgewogIGxvY2FsIGZpbGU9IiQxIgogIGxvY2FsIGJha3VwX3N1ZmZpeD0iJChkYXRlIC0tdXRjICsiJVklbSVkJUglTSIpIgogIAoKICBpZiBncmVwIC1xICdEbG9nNGoyLmZvcm1hdE1zZ05vTG9va3Vwcz10cnVlJyAkZmlsZQogIHRoZW4gCiAgICBsb2dfbWVzc2FnZV9sb2c0aiAiVGhlIGphdmEgcHJvcGVydHkgbG9nNGoyLmZvcm1hdE1zZ05vTG9va3Vwcz10cnVlIGlzIGFscmVhZHkgc2V0IGluICRmaWxlLiIKICBlbHNlCiAgICBsb2dfbWVzc2FnZV9sb2c0aiAiQ3JlYXRpbmcgYmFjayB1cCBmb3Igc2V0ZW52IGZpbGUgaW4gJGZpbGUuJGJha3VwX3N1ZmZpeCIKICAgIGNwIC1mICIkZmlsZSIgIiRmaWxlLiRiYWt1cF9zdWZmaXgiCiAgICBsb2dfbWVzc2FnZV9sb2c0aiAiQWRkaW5nIC1EbG9nNGoyLmZvcm1hdE1zZ05vTG9va3Vwcz10cnVlIHRvIEpWTV9PUFRTIGluICRmaWxlIgogICAgcmVzPSQoYXdrICdGTlI9PU5SeyBpZiAoL15KVk1fT1BUUz0vKSBwPU5SOyBuZXh0fSAxOyBGTlI9PXB7IHByaW50ICJKVk1fT1BUUz1cIiRKVk1fT1BUUyAtRGxvZzRqMi5mb3JtYXRNc2dOb0xvb2t1cHM9dHJ1ZVwiIiB9JyAkZmlsZSAkZmlsZSkgfHwgbG9nX2Vycm9yX2xvZzRqICJGYWlsZWQgdG8gZWRpdCAkZmlsZSIKICAgIGVjaG8gIiRyZXMiID4gJGZpbGUKICBmaQp9Cgp1cGRhdGVfdnJvX3RvbWNhdF9zdGFydCgpIHsKICBsb2NhbCBmaWxlPSIkMSIKCiAgaWYgZ3JlcCAtcSAncGF0Y2hfYWxsX3BsdWdpbnNfdjIgPicgJGZpbGUKICB0aGVuIAogICAgbG9nX21lc3NhZ2VfbG9nNGogIk5vdGhpbmcgdG8gZG8uIEtCIGFscmVhZHkgYXBwbGllZCEiCiAgZWxzZQogICAgbG9jYWwgcmVzCiAgICBsb2NhbCBiYWt1cF9zdWZmaXg9IiQoZGF0ZSAtLXV0YyArIiVZJW0lZCVIJU0iKSIKICAgIGxvZ19tZXNzYWdlX2xvZzRqICJDcmVhdGluZyBiYWNrIHVwIGZvciB0b21jYXQgc3RhcnR1cCBjb25maWcgaW4gJGZpbGUuJGJha3VwX3N1ZmZpeCIKICAgIGNwIC1mICIkZmlsZSIgIiRmaWxlLiRiYWt1cF9zdWZmaXgiCgogICAgbG9nX21lc3NhZ2VfbG9nNGogIk1vZGlmeWluZyB2Uk8gdG9tY2F0IHN0YXJ0dXAgY29uZmlnIC0gJGZpbGUiCgogICAgaWYgZ3JlcCAtcSAnI2xvZzRqX2N2ZV93b3JrYXJvdW5kJyAkZmlsZQogICAgdGhlbgogICAgICBzZWQgLWkgJ3MvI2xvZzRqX2N2ZV93b3JrYXJvdW5kL1xuI2xvZzRqX2N2ZV93b3JrYXJvdW5kL2cnICRmaWxlCiAgICAgIHNlZCAtaSAnL2xvZ19tZXNzYWdlX2xvZzRqMl9jdmUgIlBhdGNoaW5nIGRvbmUuIiQve247cy8uKi99XG4jbG9nNGpfY3ZlX3dvcmthcm91bmRfZW5kL30nICRmaWxlCiAgICAgIHNlZCAtaSAnL3ZSTyBzZXJ2ZXIgc2VydmljZSBkaWQgbm90IHN0YXJ0IHdpdGhpbiB0aGUgZXhwZWN0ZWQgcGVyaW9kLiokL3tuO247bjtzLy4qL31cbiNsb2c0al9jdmVfd29ya2Fyb3VuZF9lbmQvfScgJGZpbGUKICAgICAgc2VkIC1pICcvI2xvZzRqX2N2ZV93b3JrYXJvdW5kLywvI2xvZzRqX2N2ZV93b3JrYXJvdW5kX2VuZC9jXGRlbGV0ZV9tYXJrZXJcJyAkZmlsZQogICAgICBwZXJsIC1pIC0wcGUgJ3MvKC4qKVxuLipkZWxldGVfbWFya2VyXG4vXDEvZzsnICRmaWxlCgogICAgICBzZWQgLWkgJy9sb2c0al9jdmVfd29ya2Fyb3VuZCA+L2QnICRmaWxlIAogICAgICBzZWQgLWkgJy9wYXRjaF9hbGxfcGx1Z2lucyA+L2QnICRmaWxlCiAgICBmaQogICAgc2VkIC1pICcvYmFzZTY0IC1kIDw8PC9kJyAkZmlsZSAKCiAgICBpZiAhIGdyZXAgLXFFICdhY3Rpb249InN0YXJ0InxTdGFydGluZyB0Y1NlcnZlcicgIiRmaWxlIgogICAgdGhlbgogICAgICBsb2dfZXJyb3JfbG9nNGogIlVuYWJsZSB0byBhcHBseSBwYXRjaDogVW5leHBlY3RlZCBmb3JtYXQgb2YgdlJPIHRvbWNhdCBzdGFydHVwIGNvbmZpZyAtICRmaWxlLiIKICAgICAgZXhpdCAxCiAgICBmaQoKICAgIGxvY2FsIHV0aWw9IiQoZGlybmFtZSAiJGZpbGUiKS9jdmVfdXRpbC5zaCIKICAgIGJhc2U2NCAtZCA8PDwgIkkyeHZaelJxWDJOMlpWOTNiM0pyWVhKdmRXNWtDbXh2WjE5dFpYTnpZV2RsWDJ4dlp6UnFNbDlqZG1WZmRqSW9LU0I3Q2dsbFkyaHZJQ0piSkNoa1lYUmxJQzB0ZFhSaklDSXJKVVpVSlZRdUpUTk9XaUlwWFNBa01TSUtmUW9LQ25CaGRHTm9YM0JzZFdkcGJsOTJNaWdwSUhzS0NXeHZZMkZzSUhCc2RXZHBibDl3WVhSb1BTSWtNU0lLQ1d4dlkyRnNJSEJzZFdkcGJsOXVZVzFsUFNJa0tHSmhjMlZ1WVcxbElDSWtjR3gxWjJsdVgzQmhkR2dpS1NJS0NXeHZZMkZzSUhSbGJYQmZjR0YwYUQwaUwzUnRjQzhrY0d4MVoybHVYMjVoYldVaUNnbHNiMk5oYkNCaVlXTnJkWEJ6WDNCaGRHZzlJaTkxYzNJdmJHbGlMM1pqYnk5aVlXTnJkWEJ6SWdvS0NXMXJaR2x5SUMxd0lDSWtZbUZqYTNWd2MxOXdZWFJvSWdvS0NYSnRJQzF5WmlBaUpIUmxiWEJmY0dGMGFDSUtDWFZ1ZW1sd0lDMXhJQ0lrY0d4MVoybHVYM0JoZEdnaUlDMWtJQ0lrZEdWdGNGOXdZWFJvSWlCOGZDQmxlR2wwSURFS0Nna2pJRU5vWldOcklHbG1JSFJvWlhKbElHbHpJR0VnYm1WbFpDQjBieUIxY0dSaGRHVWdkR2hsSUhCc2RXZHBiZ29KYVdZZ1ptbHVaQ0FpSkhSbGJYQmZjR0YwYUNJZ0xYaGtaWFlnTFhSNWNHVWdaaUF0Ym1GdFpTQW5iRzluTkdvdFkyOXlaUzB5S21waGNpY2dMV1Y0WldNZ0wzVnpjaTlpYVc0dmVtbHdJQzF6WmlBaWUzMGlJRnc3SUh3Z1ozSmxjQ0J2Y21jdllYQmhZMmhsTDJ4dloyZHBibWN2Ykc5bk5Hb3ZZMjl5WlM5c2IyOXJkWEF2U201a2FVeHZiMnQxY0M1amJHRnpjenNLQ1hSb1pXNEtDUWxzYjJkZmJXVnpjMkZuWlY5c2IyYzBhakpmWTNabFgzWXlJQ0pOYjJScFpubHBibWNnY0d4MVoybHVPaUFrY0d4MVoybHVYMjVoYldVaUNna0piWFlnSWlSd2JIVm5hVzVmY0dGMGFDSWdJaVJpWVdOcmRYQnpYM0JoZEdnaUNna0pabWx1WkNBaUpIUmxiWEJmY0dGMGFDSWdMWGhrWlhZZ0xYUjVjR1VnWmlBdGJtRnRaU0FuYkc5bk5Hb3RZMjl5WlMweUttcGhjaWNnTFdWNFpXTWdjMmdnTFdNZ0p5OTFjM0l2WW1sdUwzcHBjQ0F0Y1NBdFpDQWllMzBpSUc5eVp5OWhjR0ZqYUdVdmJHOW5aMmx1Wnk5c2IyYzBhaTlqYjNKbEwyeHZiMnQxY0M5S2JtUnBURzl2YTNWd0xtTnNZWE56T3lCMGIzVmphQ0F0ZENBeU1ESXhNREV3TVRBd01EQWdJbnQ5SWljZ1hEc0tDUWtvWTJRZ0lpUjBaVzF3WDNCaGRHZ2lJRHNnZW1sd0lDMXhJQzF5SUMxWUlDMUVJQ0lrY0d4MVoybHVYM0JoZEdnaUlDb3BJSHg4SUdWNGFYUWdNUW9LQ1FraklFWnBlQ0J5YVdkb2RITUtDUWxqYUc5M2JpQjJZMjg2ZG1OdklDSWtjR3gxWjJsdVgzQmhkR2dpQ2drSlkyaHRiMlFnTURZME5DQWlKSEJzZFdkcGJsOXdZWFJvSWdvSkNXeHZaMTl0WlhOellXZGxYMnh2WnpScU1sOWpkbVZmZGpJZ0lsTjFZMk5sYzNObWRXeHNlU0J3WVhSamFHVmtJSEJzZFdkcGJqb2dKSEJzZFdkcGJsOXVZVzFsSWdvSlpXeHpaUW9KQ1d4dloxOXRaWE56WVdkbFgyeHZaelJxTWw5amRtVmZkaklnSWs1dmRHaHBibWNnZEc4Z1pHOGdabTl5SUhCc2RXZHBiam9nSkhCc2RXZHBibDl1WVcxbElnb0pabWtLQ1hKdElDMXlaaUFpSkhSbGJYQmZjR0YwYUNJS2ZRb0tjR0YwWTJoZllXeHNYM0JzZFdkcGJuTmZkaklvS1NCN0NnbG1iM0lnWm1sc1pTQnBiaUF2ZFhOeUwyeHBZaTkyWTI4dllYQndMWE5sY25abGNpOXdiSFZuYVc1ekx5b3VaR0Z5Q2dsa2J3b0pDWEJoZEdOb1gzQnNkV2RwYmw5Mk1pQWlKR1pwYkdVaUlIeDhJR3h2WjE5dFpYTnpZV2RsWDJ4dlp6UnFNbDlqZG1WZmRqSWdJa1ZTVWs5U09pQkdZV2xzWldRZ2RHOGdjR0YwWTJnZ2NHeDFaMmx1T2lBa0tHSmhjMlZ1WVcxbElDUm1hV3hsS1NJS0NXUnZibVVLQ2dsc2IyZGZiV1Z6YzJGblpWOXNiMmMwYWpKZlkzWmxYM1l5SUNKUVlYUmphR2x1WnlCa2IyNWxMaUlLZlFvPSIgPiAiJHV0aWwiCgogICAgc2VkIC1pICdzLGV2YWwgZXhlYywnInNvdXJjZSAkdXRpbCInXG4mLCcgIiRmaWxlIgogICAgZ3JlcCAtcSAic291cmNlICR1dGlsIiAiJGZpbGUiIHx8IHsgbG9nX2Vycm9yX2xvZzRqICJGYWlsZWQgdG8gZWRpdCAkZmlsZS4gQmFja3VwIGNhbiBiZSBmb3VuZCBpbiAkZmlsZS4kYmFrdXBfc3VmZml4IjsgZXhpdCAxOyB9CgogICAgc2VkIC1yIC1pICcvYWN0aW9uPSJzdGFydCJ8U3RhcnRpbmcgdGNTZXJ2ZXIvYSBcXHRcdFx0cGF0Y2hfYWxsX3BsdWdpbnNfdjIgPj4gL3Zhci9sb2cvdm13YXJlL3Zjby9hcHAtc2VydmVyL3Zjb19sb2c0al9jdmUubG9nJyAkZmlsZQogICAgZ3JlcCAtcSAncGF0Y2hfYWxsX3BsdWdpbnNfdjIgPicgIiRmaWxlIiB8fCB7IGxvZ19lcnJvcl9sb2c0aiAiRmFpbGVkIHRvIGVkaXQgJGZpbGUuIEJhY2t1cCBjYW4gYmUgZm91bmQgaW4gJGZpbGUuJGJha3VwX3N1ZmZpeCI7IGV4aXQgMTsgfQoKICAgIGxvZ19tZXNzYWdlX2xvZzRqICJTdWNjZXNzZnVsbHkgbW9kaWZpZWQgdGhlIHZSTyB0b21jYXQgc3RhcnR1cCBjb25maWcgLSAkZmlsZSIKICBmaQp9CgoKKHNldF9qYXZhX29wdCAiL3Vzci9saWIvdmNvL2NvbmZpZ3VyYXRpb24vYmluL3NldGVudi5zaCIgJiYgc2V0X2phdmFfb3B0ICIvdXNyL2xpYi92Y28vYXBwLXNlcnZlci9iaW4vc2V0ZW52LnNoIiAmJiB1cGRhdGVfdnJvX3RvbWNhdF9zdGFydCAiL3Zhci9saWIvdmNvL2FwcC1zZXJ2ZXIvYmluL2luaXQuZC5zaCIpIHx8IGxvZ19lcnJvcl9sb2c0aiAiRmFpbGVkIHRvIGFwcGx5IHRoZSBsb2c0aiBDVkUgd29ya2Fyb3VuZCBmb3IgdlJPLiBGb3IgbW9yZSBkZXRhaWxzIHNlZSAvdmFyL2xvZy92bXdhcmUvdmNvL2FwcC1zZXJ2ZXIvdmNvX2xvZzRqX2N2ZS5sb2cuIg==" | sh - Note: My environment has a standalone vRA so i'll be running this command only on one node. If there are more than one node then this has to be run on all of the nodes Output [2021-12-18T01:26:31.436Z] Creating back up for setenv file in /usr/lib/vco/configuration/bin/setenv.sh.202112180126 [2021-12-18T01:26:31.470Z] Adding -Dlog4j2.formatMsgNoLookups=true to JVM_OPTS in /usr/lib/vco/configuration/bin/setenv.sh [2021-12-18T01:26:31.493Z] Creating back up for setenv file in /usr/lib/vco/app-server/bin/setenv.sh.202112180126 [2021-12-18T01:26:31.505Z] Adding -Dlog4j2.formatMsgNoLookups=true to JVM_OPTS in /usr/lib/vco/app-server/bin/setenv.sh [2021-12-18T01:26:31.525Z] Creating back up for tomcat startup config in /var/lib/vco/app-server/bin/init.d.sh.202112180126 [2021-12-18T01:26:31.535Z] Modifying vRO tomcat startup config - /var/lib/vco/app-server/bin/init.d.sh [2021-12-18T01:26:31.577Z] Successfully modified the vRO tomcat startup config - /var/lib/vco/app-server/bin/init.d.sh Step:3 Run the following (on any single vRA Node) to update the vRO Control Center (not applicable to versions 7.2 and 7.3) /usr/lib/vco/tools/configuration-cli/bin/vro-configure-inner.sh controlcenter-update Output Will vary if you have a distributed environment [master] svra:~ # /usr/lib/vco/tools/configuration-cli/bin/vro-configure-inner.sh controlcenter-update Orchestrator's root folder: /var/lib/vco Orchestrator Configuration Tool. Version: 7.6.0.12923317 Build: 12923317 Start 'controlcenter-update' command. Dec 18, 2021 1:29:23 AM org.apache.tomcat.jdbc.pool.ConnectionPool checkPoolConfiguration WARNING: initialSize is larger than maxActive, setting initialSize to: 4 Dec 18, 2021 1:29:23 AM org.apache.tomcat.jdbc.pool.ConnectionPool checkPoolConfiguration WARNING: minIdle is larger than maxActive, setting minIdle to: 4 Dec 18, 2021 1:29:23 AM org.apache.tomcat.jdbc.pool.ConnectionPool checkPoolConfiguration WARNING: maxIdle is smaller than minIdle, setting maxIdle to: 4 'controlcenter-update' command finished successfully. The command does not need database configuration update. Step:4 Run the following command to update vRA and vIDM configuration on each vRA Node base64 -d <<< "bG9nX21lc3NhZ2UoKSB7CiAgZWNobyAiWyQoZGF0ZSAtLXV0YyAiKyVGVCVULiUzTloiKV0gJDEiIHwgdGVlIC1hICAgL3Zhci9sb2cvdm13YXJlL3ZjYWMvdmNhY19sb2c0al9jdmUubG9nCn0KCmxvZ19lcnJvcigpIHsKICBsb2dfbWVzc2FnZSAiRVJST1I6ICQxIgogIGV4aXQgMQp9CgpzZXRfamF2YV9vcHQoKSB7CiAgbG9jYWwgZmlsZT0iJDEiCgogIGlmIGdyZXAgLXEgJ0Rsb2c0ajIuZm9ybWF0TXNnTm9Mb29rdXBzPXRydWUnICRmaWxlCiAgdGhlbiAKICAgIGxvZ19tZXNzYWdlICJUaGUgamF2YSBwcm9wZXJ0eSBsb2c0ajIuZm9ybWF0TXNnTm9Mb29rdXBzPXRydWUgaXMgYWxyZWFkeSBzZXQgaW4gJGZpbGUuIgogIGVsc2UKICAgIGxvZ19tZXNzYWdlICJBZGRpbmcgLURsb2c0ajIuZm9ybWF0TXNnTm9Mb29rdXBzPXRydWUgdG8gVkNBQ19PUFRTIGluICRmaWxlIgogICAgZWNobyAnVkNBQ19PUFRTPSIkVkNBQ19PUFRTIC1EbG9nNGoyLmZvcm1hdE1zZ05vTG9va3Vwcz10cnVlIicgPj4gJGZpbGUgfHwgbG9nX2Vycm9yICJGYWlsZWQgdG8gZWRpdCAkZmlsZSIKICBmaQp9CgpkZWxldGVfam5kaV9jbGFzcygpIHsKICBsb2dfbWVzc2FnZSAiRGVsZXRpbmcgYWxsIEpuZGlMb29rdXAuY2xhc3MgZmlsZXMgZm91bmQgZm9yIGxvZzRqIDIueCB2ZXJzaW9ucyIKICBmaW5kIC8gLXhkZXYgLXR5cGUgZiAtbmFtZSAnbG9nNGotY29yZS0yKmphcicgLWV4ZWMgL3Vzci9iaW4vemlwIC1xIC1kICJ7fSIgb3JnL2FwYWNoZS9sb2dnaW5nL2xvZzRqL2NvcmUvbG9va3VwL0puZGlMb29rdXAuY2xhc3MgXDsgfCB0ZWUgLWEgL3Zhci9sb2cvdm13YXJlL3ZjYWMvdmNhY19sb2c0al9jdmUubG9nCn0KCihzZXRfamF2YV9vcHQgIi9ldGMvdmNhYy9zZXRlbnYtdXNlciIgJiYgZGVsZXRlX2puZGlfY2xhc3MgKSB8fCBsb2dfZXJyb3IgIkZhaWxlZCB0byBhcHBseSB0aGUgbG9nNGogQ1ZFIHdvcmthcm91bmQgZm9yIHZSQS4gRm9yIG1vcmUgZGV0YWlscyBzZWUgL3Zhci9sb2cvdm13YXJlL3ZjYWMvdmNhY19sb2c0al9jdmUubG9nLiI=" | sh - Note: Warnings of the type "zip error: Nothing to do!..." indicate there is no need for patching the specified binary. Output [master] svra:~ # base64 -d <<< "bG9nX21lc3NhZ2UoKSB7CiAgZWNobyAiWyQoZGF0ZSAtLXV0YyAiKyVGVCVULiUzTloiKV0gJDEiIHwgdGVlIC1hICAgL3Zhci9sb2cvdm13YXJlL3ZjYWMvdmNhY19sb2c0al9jdmUubG9nCn0KCmxvZ19lcnJvcigpIHsKICBsb2dfbWVzc2FnZSAiRVJST1I6ICQxIgogIGV4aXQgMQp9CgpzZXRfamF2YV9vcHQoKSB7CiAgbG9jYWwgZmlsZT0iJDEiCgogIGlmIGdyZXAgLXEgJ0Rsb2c0ajIuZm9ybWF0TXNnTm9Mb29rdXBzPXRydWUnICRmaWxlCiAgdGhlbiAKICAgIGxvZ19tZXNzYWdlICJUaGUgamF2YSBwcm9wZXJ0eSBsb2c0ajIuZm9ybWF0TXNnTm9Mb29rdXBzPXRydWUgaXMgYWxyZWFkeSBzZXQgaW4gJGZpbGUuIgogIGVsc2UKICAgIGxvZ19tZXNzYWdlICJBZGRpbmcgLURsb2c0ajIuZm9ybWF0TXNnTm9Mb29rdXBzPXRydWUgdG8gVkNBQ19PUFRTIGluICRmaWxlIgogICAgZWNobyAnVkNBQ19PUFRTPSIkVkNBQ19PUFRTIC1EbG9nNGoyLmZvcm1hdE1zZ05vTG9va3Vwcz10cnVlIicgPj4gJGZpbGUgfHwgbG9nX2Vycm9yICJGYWlsZWQgdG8gZWRpdCAkZmlsZSIKICBmaQp9CgpkZWxldGVfam5kaV9jbGFzcygpIHsKICBsb2dfbWVzc2FnZSAiRGVsZXRpbmcgYWxsIEpuZGlMb29rdXAuY2xhc3MgZmlsZXMgZm91bmQgZm9yIGxvZzRqIDIueCB2ZXJzaW9ucyIKICBmaW5kIC8gLXhkZXYgLXR5cGUgZiAtbmFtZSAnbG9nNGotY29yZS0yKmphcicgLWV4ZWMgL3Vzci9iaW4vemlwIC1xIC1kICJ7fSIgb3JnL2FwYWNoZS9sb2dnaW5nL2xvZzRqL2NvcmUvbG9va3VwL0puZGlMb29rdXAuY2xhc3MgXDsgfCB0ZWUgLWEgL3Zhci9sb2cvdm13YXJlL3ZjYWMvdmNhY19sb2c0al9jdmUubG9nCn0KCihzZXRfamF2YV9vcHQgIi9ldGMvdmNhYy9zZXRlbnYtdXNlciIgJiYgZGVsZXRlX2puZGlfY2xhc3MgKSB8fCBsb2dfZXJyb3IgIkZhaWxlZCB0byBhcHBseSB0aGUgbG9nNGogQ1ZFIHdvcmthcm91bmQgZm9yIHZSQS4gRm9yIG1vcmUgZGV0YWlscyBzZWUgL3Zhci9sb2cvdm13YXJlL3ZjYWMvdmNhY19sb2c0al9jdmUubG9nLiI=" | sh - [2021-12-18T01:31:25.078Z] Adding -Dlog4j2.formatMsgNoLookups=true to VCAC_OPTS in /etc/vcac/setenv-user [2021-12-18T01:31:25.082Z] Deleting all JndiLookup.class files found for log4j 2.x versions zip error: Nothing to do! (/usr/lib/vcac/server/webapps/notification-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/approval-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/network-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/o11n-gateway-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/iaas-proxy-provider/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/console-proxy-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/vcac/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/branding-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/portal-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/workitem-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/healthbroker-proxy-server/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/event-broker-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/placement-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/reservation-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/endpoint-configuration-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/config-management-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/catalog-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/properties-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/forms-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/content-management-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/identity/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/software-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/ipam-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/advanced-designer-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/composition-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/server/webapps/management-service/WEB-INF/lib/log4j-core-2.11.2.jar) zip error: Nothing to do! (/usr/lib/vcac/tools/config/repo/log4j-core-2.11.2.jar) Step:5 Restart the services on each vRA Node service elasticsearch restart service vco-server status | grep PID && service vco-server restart service vco-configurator start service vcac-server restart Note: Below command can be used to stop and start all the services systematically. Not mentioned in the knowledge base article but can be followed vcac-vami service-manage stop vco-configurator vco-server vcac-server horizon-workspace hzn-dots elasticsearch vcac-vami service-manage start elasticsearch hzn-dots horizon-workspace vcac-server vco-server vco-configurator Step:6 ( not present in the kb , an extra step just for validation ) Verify if all vRealize Automation services are registered curl --insecure -f -s -H "Content-Type: application/json" "https:/$HOSTNAME/component-registry/services/status/current?limit=200" | sed "s/}/\n/g" | grep -E -o ".serviceName.*serviceInitializationStatus.[^,]*" | sed "s/\"serviceTypeId.*,//g" | sed -e "s/\"//g" -e "s/:/=/g" -e "s/,/, /" | sed -e "s/serviceName\|serviceInitializationStatus\|=\|,\|null//g" | column -t | sort | cat -n Validation To validate that the workaround has succeeded, take the following steps on all nodes Verify that all vco and vcac processes are running with the java property "log4j2.formatMsgNoLookups=true": Monitor the log /var/log/vco/app-server/vco_log4j_cve.log file, until you see 'Patching done.' Run the command below to verify that the JndiLookup.class is not present in any log4j jar file for 2.x versions find / -xdev -type f -name 'log4j-core-2*jar' -exec sh -c '/usr/bin/unzip -l "{}" | grep org/apache/logging/log4j/core/lookup/JndiLookup.class' \; The command output will be empty if the scripting was successful. This completes implementation of the workaround in vRealize Automation 7.6

There was a question on how do i identify who logged in and logged out of my vRealize Automation 8.x recently The easiest way to identify this is through vRealize Loginsight . Usually customer want's to know about this for audit purposes Login Events Flow 1. Request reaches out to connector for authentication 2. Password based authentication begins with AD 3. Authentication succeeds in connector 4. Horizon states that the login succeeded 5. vRA recieves the authentication information which then authorizes the users and displays respective services connector.log 2022-04-06T06:39:25,393 INFO (Thread-3) [IDM;-;10.104.75.48;] com.vmware.horizon.directory.ldap.LdapDirectoryService - Password-based authentication: arun@cap.org - BEGIN 2022-04-06T06:39:25,432 INFO (Thread-3) [IDM;-;10.104.75.48;] com.vmware.horizon.directory.ldap.dc.service.context.JNDIContextFetcher - LDAP Context env Json Values: { "java.naming.factory.initial" : "com.sun.jndi.ldap.LdapCtxFactory", "javax.security.sasl.server.authentication" : "true", "com.sun.jndi.ldap.connect.timeout" : "5000", "java.naming.ldap.attributes.binary" : "objectGUID pae-IconData objectSid securityIdentifier", "javax.security.sasl.strength" : "high,medium,low", "javax.security.sasl.qop" : "auth-conf,auth-int,auth", "com.sun.jndi.ldap.read.timeout" : "600000", "java.naming.provider.url" : "ldap://ad.cap.org:389", "java.naming.security.authentication" : "GSSAPI" } 2022-04-06T06:39:25,439 INFO (Thread-3) [IDM;-;10.104.75.48;] com.vmware.horizon.directory.ldap.LdapDirectoryService - Password-based authentication: arun@cap.org - SUCCESS 2022-04-06T06:39:25,439 INFO (Thread-3) [IDM;-;10.104.75.48;] com.vmware.horizon.adapters.passwordAdapter.PasswordIdpAdapter - Login: arun - SUCCESS 2022-04-06T06:39:25,440 INFO (Thread-3) [IDM;-;10.104.75.48;] com.vmware.horizon.connector.controller.AdapterLoginController - samlRequestInfo: SamlRequestInfo[acsUrl=https://idm.cap.org/SAAS/auth/saml/response,relayState=df3e88a2-e213-41f0-97cc-f9826732fc73,nameId=,requestId=_cf2c85aa30e8d7be47485fb87f6c9988,authnContextClassRefList=[urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport]] 2022-04-06T06:39:25,440 INFO (Thread-3) [IDM;-;10.104.75.48;] com.vmware.horizon.connector.controller.IdPInitiatedSSOController - samlRequestInfo: SamlRequestInfo[acsUrl=https://idm.cap.org/SAAS/auth/saml/response,relayState=df3e88a2-e213-41f0-97cc-f9826732fc73,nameId=,requestId=_cf2c85aa30e8d7be47485fb87f6c9988,authnContextClassRefList=[urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport]] horizon.log 2022-04-06T06:39:25,861 INFO (Thread-8) [IDM;-;10.104.75.48;] com.vmware.horizon.components.authentication.monitoring.LoginMetricsPublisher - Login succeeded. identity-service-app.log 2022-04-06T06:39:26.008Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-2' user='' org='' trace='dae05107-bd99-4b3e-886f-005dc3c08164'] com.vmware.identity.rest.RestClient.lambda$logRequest$1:74 - POST https://idm.cap.org/SAAS/API/1.0/oauth2/token?grant_type=authorization_code 2022-04-06T06:39:26.088Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='pool-4-thread-3' user='' org='' trace='dae05107-bd99-4b3e-886f-005dc3c08164'] com.vmware.identity.rest.RestClient.lambda$logRequest$1:74 - POST https://idm.cap.org/SAAS/API/1.0/oauth2/token?grant_type=client_credentials 2022-04-06T06:39:26.129Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='' org='' trace='dae05107-bd99-4b3e-886f-005dc3c08164'] com.vmware.identity.rest.RestClient.lambda$logRequest$1:74 - POST https://idm.cap.org/SAAS/jersey/manager/api/scim/Groups/.search 2022-04-06T06:39:27.091Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-4' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='5ba4e9dc-16cf-4760-9436-1ccea66197c3'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/am/api/loggedin/user/profile HTTP/1.1" 200 301 8080 65 ms 2022-04-06T06:39:27.403Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-4' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='5c6f8238-4292-4d05-8b15-c725f529c8a8'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/am/api/loggedin/user?expand HTTP/1.1" 200 295 8080 188 ms 2022-04-06T06:39:27.457Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-2' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='db7f5f63-4426-470b-9646-e401eefe0506'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/configuration/api/public HTTP/1.1" 200 196 8080 183 ms 2022-04-06T06:39:27.462Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-4' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='5fd3557f-5cab-43b6-8e29-a70a25ee758e'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/am/api/loggedin/user/orgs?expand=1 HTTP/1.1" 200 290 8080 207 ms 2022-04-06T06:39:27.468Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-1' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='f83d5aee-adf0-4a83-9508-629968620459'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/am/api/loggedin/user/profile HTTP/1.1" 200 301 8080 194 ms 2022-04-06T06:39:27.473Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='7615046f-ef57-455a-9147-e9676faa874e'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/am/api/loggedin/user HTTP/1.1" 200 295 8080 199 ms 2022-04-06T06:39:27.645Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='193e19bc-38c5-4e54-9f1f-e6d124b24c01'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/am/api/loggedin/user/orgs?expand=1 HTTP/1.1" 200 290 8080 135 ms 2022-04-06T06:39:27.661Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-1' user='' org='' trace='6d6aef4c-b10e-4dc8-a74e-8eeb944a8683'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /cs p/gateway/am/api/auth/discovery?username=service_type&state=aHR0cHM6Ly9zcy5jYXAub3JnL2lkZW50aXR5L2FwaS9jb3JlL2F1dGhuL2NzcA%3D%3D&redirect_uri=https%3A%2F%2Fss.cap.org%2Fidentity%2Fapi%2Fcore%2Fauthn%2Fcsp&client_id=ssc-NTnLPaiCwQ HTTP/1.1" 200 246 8080 2 ms 2022-04-06T06:39:27.717Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-2' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='67d01c4c-3c95-4786-b9a0-aeb5d89512c4'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/slc/api/definitions/?expand=1 HTTP/1.1" 200 9395 8080 191 ms 2022-04-06T06:39:27.757Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-4' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='afcdf509-00ed-4443-855f-f82b661d8e1a'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/am/api/loggedin/user/orgs/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a/roles HTTP/1.1" 200 362 8080 231 ms 2022-04-06T06:39:27.758Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-4' user='' org='' trace='b467fc16-0519-4e64-8a43-fee8e3954e76'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /cs p/gateway/am/api/auth/discovery?username=service_type&state=aHR0cHM6Ly9zcy5jYXAub3JnL2lkZW50aXR5L2FwaS9jb3JlL2F1dGhuL2NzcA%3D%3D&redirect_uri=https%3A%2F%2Fss.cap.org%2Fidentity%2Fapi%2Fcore%2Fauthn%2Fcsp&client_id=ssc-NTnLPaiCwQ HTTP/1.1" 200 246 8080 1 ms 2022-04-06T06:39:27.795Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-1' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='3766d64e-66ac-43f6-8db8-6fc730ec65a0'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/am/api/loggedin/user/orgs/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a/service-roles HTTP/1.1" 200 1471 8080 269 ms 2022-04-06T06:39:27.797Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-4' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='bd4749e0-c0d2-4a29-85e0-e7304471b02c'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/portal/api/orgs/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a/services HTTP/1.1" 200 4175 8080 156 ms 2022-04-06T06:39:27.810Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-1' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='08c3dfa7-1daf-437e-bcca-5980650d0ffc'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/slc/api/principal/org/service-families HTTP/1.1" 200 31 8080 95 ms 2022-04-06T06:39:27.816Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='b2b88069-cf36-4b08-9865-c9065925a53e'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/slc/api/orgs/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a/services HTTP/1.1" 200 8401 8080 101 ms 2022-04-06T06:39:27.865Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-4' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='9f01081b-8c72-4029-ac6f-cd4d02b40f42'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/am/api/loggedin/user/orgs/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a/roles HTTP/1.1" 200 362 8080 61 ms 2022-04-06T06:39:27.910Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='a4f38830-1146-4a7f-b5a7-34c7fd1eb5ec'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/am/api/loggedin/user/orgs/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a/service-roles HTTP/1.1" 200 1471 8080 64 ms 2022-04-06T06:39:27.977Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='0700a87d-6e1f-4a44-b346-d9f9d3eb35d1'] reactor.netty.http.se rver.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:27 +0000] "GET /csp/gateway/am/api/loggedin/user/orgs/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a/roles HTTP/1.1" 200 362 8080 65 ms 2022-04-06T06:39:28.070Z ERROR identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='a746ddea-1a89-4003-8d6b-18883b8e01df'] c.v.i.c.RestResponse EntityExceptionHandler.logBriefError:213 - Handling bad request exception: org.springframework.web.server.ResponseStatusException: 404 NOT_FOUND "No matching handler" thrown at org.springframework.web.reactive.DispatcherHandler.lambda$createNotFoundError$3:159 Picking up certain keywords like the one shown below we can determine the login event Filters would be as follows text "GET /csp/gateway/configuration/api/public HTTP/1.1" 200 namespace prelude app identity-service-app Logout Events Flow 1. Identity Service App revokes the token 2. Horizon records the logout event identity-service-app.log 2022-04-06T06:39:29.460Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-1' user='' org='' trace='5ebd1ebb-a410-49f0-bc1c-a46275cd2588'] com.vmware.identity.rest.RestClient.lambda$logRequest$1:74 - GET https://idm.cap.org/SAAS/jersey/manager/api/scim/Users/92aa3ece-b78c-4a11-9c43-7f2b0ef9c462 2022-04-06T06:39:29.550Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-1' user='' org='' trace='5ebd1ebb-a410-49f0-bc1c-a46275cd2588'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.40 - - [06/Apr/2022:06:39:29 +0000] "POST /csp/gateway/am/api/auth/authorize HTTP/1.1" 200 2225 8080 187 ms 2022-04-06T06:39:31.265Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='' org='' trace='3364bcbe-3f2e-4243-a8ef-878bc79368c4'] reactor.netty.http.server.AccessLog.info:270 - 127.0.0.1 - - [06/Apr/2022:06:39:31 +0000] "GET /identity/status HTTP/1.1" 200 634 8080 2 ms 2022-04-06T06:39:32.524Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='' org='' trace='44c96c60-babb-48da-9029-1678465bef55'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:32 +0000] "GET /csp/gateway/am/api/auth/discovery?username=service_type&state=aHR0cHM6Ly9zcy5jYXAub3JnL2lkZW50aXR5L2FwaS9jb3JlL2F1dGhuL2NzcA%3D%3D&redirect_uri=https%3A%2F%2Fss.cap.org%2Fidentity%2Fapi%2Fcore%2Fauthn%2Fcsp&client_id=ssc-NTnLPaiCwQ HTTP/1.1" 200 246 8080 2 ms 2022-04-06T06:39:34.394Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-4' user='' org='' trace='65d3b1ba-4dde-4399-b11c-37fb95d0be82'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.1 - - [06/Apr/2022:06:39:34 +0000] "GET /identity/healthcheck HTTP/1.1" 200 1362 8080 1 ms 2022-04-06T06:39:37.536Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='' org='' trace='b3b39ce9-34ee-4cda-b3c7-1659155835a7'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:37 +0000] "GET /csp/gateway/am/api/auth/discovery?username=service_type&state=aHR0cHM6Ly9zcy5jYXAub3JnL2lkZW50aXR5L2FwaS9jb3JlL2F1dGhuL2NzcA%3D%3D&redirect_uri=https%3A%2F%2Fss.cap.org%2Fidentity%2Fapi%2Fcore%2Fauthn%2Fcsp&client_id=ssc-NTnLPaiCwQ HTTP/1.1" 200 246 8080 1 ms 2022-04-06T06:39:37.661Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='' org='' trace='140664d5-f71e-40c0-88e1-a7818c49bc57'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:37 +0000] "GET /csp/gateway/am/api/auth/discovery?username=service_type&state=aHR0cHM6Ly9zcy5jYXAub3JnL2lkZW50aXR5L2FwaS9jb3JlL2F1dGhuL2NzcA%3D%3D&redirect_uri=https%3A%2F%2Fss.cap.org%2Fidentity%2Fapi%2Fcore%2Fauthn%2Fcsp&client_id=ssc-NTnLPaiCwQ HTTP/1.1" 200 246 8080 2 ms 2022-04-06T06:39:38.364Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='' org='' trace='aaf4ce29-8707-496f-a5be-088f006abad8'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:38 +0000] "GET /csp/gateway/am/api/auth/discovery?username=service_type&state=aHR0cHM6Ly9zcy5jYXAub3JnL2lkZW50aXR5L2FwaS9jb3JlL2F1dGhuL2NzcA%3D%3D&redirect_uri=https%3A%2F%2Fss.cap.org%2Fidentity%2Fapi%2Fcore%2Fauthn%2Fcsp&client_id=ssc-NTnLPaiCwQ HTTP/1.1" 200 246 8080 3 ms 2022-04-06T06:39:39.394Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-1' user='' org='' trace='7428b6fa-a3c5-42ce-b8db-46d585e79dee'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.1 - - [06/Apr/2022:06:39:39 +0000] "GET /identity/healthcheck HTTP/1.1" 200 1362 8080 1 ms 2022-04-06T06:39:42.543Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='' org='' trace='f7a2d487-b26e-4842-8d32-c8aa188fa0a8'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:42 +0000] "GET /csp/gateway/am/api/auth/discovery?username=service_type&state=aHR0cHM6Ly9zcy5jYXAub3JnL2lkZW50aXR5L2FwaS9jb3JlL2F1dGhuL2NzcA%3D%3D&redirect_uri=https%3A%2F%2Fss.cap.org%2Fidentity%2Fapi%2Fcore%2Fauthn%2Fcsp&client_id=ssc-NTnLPaiCwQ HTTP/1.1" 200 246 8080 3 ms 2022-04-06T06:39:44.398Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='parallel-1' user='' org='' trace='3a88d87d-e7d7-4d4f-9547-8de6f777e759'] com.vmware.identity.rest.RestClient.lambda$logRequest$1:74 - POST https://idm.cap.org/SAAS/API/1.0/oauth2/token?grant_type=client_credentials 2022-04-06T06:39:44.440Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='' org='' trace='3a88d87d-e7d7-4d4f-9547-8de6f777e759'] com.vmware.identity.rest.RestClient.lambda$logRequest$1:74 - GET https://idm.cap.org/SAAS/API/1.0/REST/system/health/heartbeat 2022-04-06T06:39:44.448Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-2' user='' org='' trace='3a88d87d-e7d7-4d4f-9547-8de6f777e759'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.1 - - [06/Apr/2022:06:39:44 +0000] "GET /identity/healthcheck HTTP/1.1" 200 1362 8080 55 ms 2022-04-06T06:39:46.357Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='pool-4-thread-8' user='' org='' trace='ed4efb33-32b0-46af-a5ab-73ab2850e4d6'] com.vmware.identity.common.AuditLogger.audit:28 - Revoking token 34a3793c-2db8-4a1c-b1a6-e96432650507 for user arun (91fb23fd-6ce1-4f69-a556-ce2ceffbef4a) in c2eae67a-ff6d-4dae-9fd3-6594352a1f8a organization. 2022-04-06T06:39:46.362Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-4' user='' org='' trace='ed4efb33-32b0-46af-a5ab-73ab2850e4d6'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.40 - - [06/Apr/2022:06:39:46 +0000] "POST /csp/gateway/am/api/auth/logout HTTP/1.1" 200 58 8080 10 ms 2022-04-06T06:39:46.441Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='' org='' trace='a9376843-e852-4daf-9f86-ccb073310817'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:46 +0000] "GET /csp/gateway/discovery?logout HTTP/1.1" 302 0 8080 1 ms 2022-04-06T06:39:47.550Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='' org='' trace='8fa0f70e-2c73-4323-bc72-d6ba1399ecfa'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:47 +0000] "GET /csp/gateway/am/api/auth/discovery?username=service_type&state=aHR0cHM6Ly9zcy5jYXAub3JnL2lkZW50aXR5L2FwaS9jb3JlL2F1dGhuL2NzcA%3D%3D&redirect_uri=https%3A%2F%2Fss.cap.org%2Fidentity%2Fapi%2Fcore%2Fauthn%2Fcsp&client_id=ssc-NTnLPaiCwQ HTTP/1.1" 200 246 8080 2 ms 2022-04-06T06:39:47.709Z INFO identity-service [host='identity-service-app-6954fb5598-gk6ll' thread='reactor-http-epoll-3' user='' org='' trace='babb7ae3-3380-4196-b513-8116bb71dba6'] reactor.netty.http.server.AccessLog.info:270 - 10.244.0.18 - - [06/Apr/2022:06:39:47 +0000] "GET /csp/gateway/am/api/auth/discovery?username=service_type&state=aHR0cHM6Ly9zcy5jYXAub3JnL2lkZW50aXR5L2FwaS9jb3JlL2F1dGhuL2NzcA%3D%3D&redirect_uri=https%3A%2F%2Fss.cap.org%2Fidentity%2Fapi%2Fcore%2Fauthn%2Fcsp&client_id=ssc-NTnLPaiCwQ HTTP/1.1" 200 246 8080 3 ms horizon.log 2022-04-06T06:39:46,761 INFO (Thread-8) [IDM;344d8070-fe39-4c6e-8eaa-a22640e83f5f;10.104.75.48;] com.vmware.horizon.service.controller.auth.LoginController - Logout request. Invalidating session for request: https://idm.cap.org/SAAS/auth/logout with query string:dest=https%3A%2F%2Fvra.cap.org connector.log *** No Information would be available here ** Filters for logout event would be like text Revoking token namespace prelude app identity-service-app If screenshots are not clear , then download this PDF and use it

As a first step fetch the password using below command cat /usr/local/horizon/conf/db.pwd Example:- labidm:~ # cat /usr/local/horizon/conf/db.pwd D4RSIrlpRNU5btYHYGOAySkFyykamPHa Then execute psql -U horizon saas This will prompt you to enter the password, copy the password extracted using the first command and then paste it and hit enter. You would be logged into vIDM database labidm:~ # psql -U horizon saas Password for user horizon: psql.bin (9.6.16 (VMware Postgres 9.6.16.0-15591078 release)) Type "help" for help. saas=> Enter \dt at the saas=> prompt to list all the tables

After upgrading to vRealize Automation 8.4.1 you might encounter rest host exceptions 2021-06-11T03:17:53.826Z ERROR vco [host='vco-app-b995d8bc6-dc6kh' thread='AsyncActionExecutorPool-Thread-21' user='' org='' trace=''] {} com.vmware.o11n.plugin.rest.Request - Cannot execute the request. java.lang.NullPointerException: null at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011) ~[?:?] at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006) ~[?:?] at org.apache.http.impl.client.BasicCredentialsProvider.setCredentials(BasicCredentialsProvider.java:62) ~[httpclient-4.5.13.jar:4.5.13] at com.vmware.o11n.plugin.rest.HttpContextFactory.create(HttpContextFactory.java:27) ~[o11nplugin-rest-model-2.4.0.jar:?] at com.vmware.o11n.plugin.rest.RequestExecutor.execute(RequestExecutor.java:236) ~[o11nplugin-rest-model-2.4.0.jar:?] at com.vmware.o11n.plugin.rest.Request.doExecute(Request.java:419) [o11nplugin-rest-model-2.4.0.jar:?] at com.vmware.o11n.plugin.rest.Request.execute(Request.java:138) [o11nplugin-rest-model-2.4.0.jar:?] at jdk.internal.reflect.GeneratedMethodAccessor462.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at ch.dunes.vso.sdk.WrappedJavaMethod.invoke(WrappedJavaMethod.java:233) [o11n-sdkcenter-8.4.1.jar:?] at ch.dunes.vso.sdk.WrappedJavaMethod.call(WrappedJavaMethod.java:175) [o11n-sdkcenter-8.4.1.jar:?] at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1473) [rhino-1.7R4.jar:1.7R4] at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) [rhino-1.7R4.jar:1.7R4] at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) [rhino-1.7R4.jar:1.7R4] at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394) [rhino-1.7R4.jar:1.7R4] at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091) [rhino-1.7R4.jar:1.7R4] at org.mozilla.javascript.InterpretedFunction.exec(InterpretedFunction.java:120) [rhino-1.7R4.jar:1.7R4] at ch.dunes.scripting.server.script.MainScriptingObject.executeScript(MainScriptingObject.java:275) [o11n-scriptingengine-8.4.1.jar:?] at ch.dunes.scripting.server.script.MainScriptingObject.executeScript(MainScriptingObject.java:261) [o11n-scriptingengine-8.4.1.jar:?] at ch.dunes.scripting.handler.mbean.ScriptUnit.executeScript(ScriptUnit.java:81) [o11n-scriptingengine-8.4.1.jar:?] at ch.dunes.scripting.handler.mbean.ScriptUnitContext.executeSynchronousUnit(ScriptUnitContext.java:100) [o11n-scriptingengine-8.4.1.jar:?] at ch.dunes.scripting.handler.mbean.ScriptLauncherImpl.executeFunction(ScriptLauncherImpl.java:86) [o11n-scriptingengine-8.4.1.jar:?] at ch.dunes.scripting.server.util.ServerActionInvoker.executeServerAction(ServerActionInvoker.java:115) [o11n-scriptingengine-8.4.1.jar:?] at com.vmware.o11n.service.impl.ScriptModuleRuntimeServiceImpl.executeServerAction(ScriptModuleRuntimeServiceImpl.java:88) [o11n-services-server-8.4.1.jar:?] at com.vmware.o11n.service.impl.ScriptModuleRuntimeServiceImpl.executeServerActionPriviliged(ScriptModuleRuntimeServiceImpl.java:201) [o11n-services-server-8.4.1.jar:?] at com.vmware.o11n.service.impl.ScriptModuleRuntimeServiceImpl$1.delegatedCall(ScriptModuleRuntimeServiceImpl.java:145) [o11n-services-server-8.4.1.jar:?] at ch.dunes.scripting.server.script.AsyncAction.call(AsyncAction.java:27) [o11n-scriptingengine-8.4.1.jar:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] This is a bug which has been fixed in the latest version of HTTP REST plugin. Out of the box version of HTTP REST plugin in vRealize Automation 8.4.1 is 2.4.0.17862790 Fix is available under 2.4.0-18124475 Download the plugin by clicking here Implementation Steps Take snapshot of vRA appliance Go to Controlcenter and then take a backup of existing plugin 3. Upload the new plugin 4. Install the plugin 5. Validate the build Post this test your workflows and they should work as expected.

There was an environment where virtual machine dispose tasks were stuck and they were repeating dispose tasks every 2.5 hours Agents were throwing exception stating Exception occured when retrieving work item from Manager Service: System.ServiceModel.CommunicationException: The maximum message size quota for incoming messages (65536) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element. The calls are failing because agent is unable to read the workitems. There is a default maximum size of 65536 on incoming workitems.If the machines being deployed or destroyed have enough custom properties to swell workitem size beyond the maximum allowed value then these sort of exceptions happen To fix this, need to edit VRMAgent.exe.config and locate this block ( Proxy Agent Location ) and change the line that reads to read Then restart the proxy agent where the issue is seen. This exception should now go away and any virtual machines in stuck states will clear over a period of time during it's retry cycle