Search Results

To export the embedded Postgres database from the VMware vRealize Suite LifeCycle Manager Appliance Log in to the VMware vRealize Automation virtual appliance using SSH. Change directory using this command: cd /tmp Run this command to create a copy of the database in /tmp su -m -c "/opt/vmware/vpostgres/11/bin/pg_dump -Fc vrlcm > /tmp/vrlcm.sql" - postgres Note: -Fc switch already provides a compressed file.  No need to bzip. Use SCP or WinSCP to transfer the vcac.sql file off of the appliance.

List of health URL's which can be used during vRealize Automation troubleshooting Horizon System Health URL: https://</SAAS/API/1.0/REST/system/health RESULT: {"AnalyticsUrl":"http://localhost:8080","EhCacheClusterPeers":"","AuditPollInterval":"1000","EncryptionServiceVersion":"unknown","AnalyticsConnectionOk":"true","EncryptionServiceVerified":"Master Keystore verified","FederationBrokerStatus":"ok","ServiceReadOnlyMode":"false","AuditWorkerThreadAlive":"true","BuildVersion":"3.1.0.0 Build 12694081","AuditQueueSize":"0","DatabaseStatus":"connection successful","HostName":"sevenvra.prem.com","EncryptionStatus":"connected","FederationBrokerOk":"true","EncryptionConnectionOk":"true","EncryptionServiceImpl":"Encryption Service DB","ClusterId":"add760d8-b9cd-453d-a476-abf323758b59","EhCacheClusterDiagnostics":"","DatabaseConnectionOk":"true","StatusDate":"2020-06-11 14:19:14 UTC","ClockSyncOk":"true","MaintenanceMode":"false","MessagingConnectionOk":"true","fipsModeEnabled":"false","ServiceVersion":"3.1.0","IpAddress":"10.109.46.59","AuditDisabled":"false","AllOk":"true"} As shown above, the "AllOk" tag should be true Horizon Cluster Instances URL: https://<>/API/1.0/REST/system/clusterInstances RESULT: [{"version":"3.1.0.0 Build 12694081","uuid":"5d22e506-0529-3111-b8ca-beb20b620da8","status":"Active","lastUpdated":1591885520764,"hostname":"sevenvra.prem.com","datacenterId":0,"ipaddress":"10.109.46.59"}] ElasticSearch Health URL: ssh to vRA appliance and then execute curl -kv http://localhost:9200/_cluster/health?pretty=true RESULT: { "cluster_name" : "horizon", "status" : "green", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 15, "active_shards" : 15, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0 } vRA Service status URL: https://va-fqdn/vcac/services/api/status RESULT: true REGISTERED shell-ui-app cafe-7efxBfzyew 2020-05-28T01:34:51.230Z d823b6f9-688e-4819-acf7-e773060a1e64 true CN=sevenvra.prem.com,OU=PREM,O=PREM,C=SG 2025-04-27T08:16:10Z 2020-04-28T08:16:10Z CN=sevenvra.prem.com,OU=PREM,O=PREM,C=SG 8E:68:E8:F0:EE:BC:12:2B:2D:78:89:C5:F9:37:5E:7C:25:38:C0:32 vRA Component Registry URL: https://vra-fqdn/component-registry/services/status/current?limit=200 RESULT: One would information on all services in the vRA appliance. As shown below notAvailable should always be false and serviceInitializationStatus should be REGISTERED 2020-05-28T01:39:25.430Z https://sevenvra.prem.com/composition-service/api/status true REGISTERED composition-service cafe-7efxBfzyew 2020-05-28T01:39:29.284Z com.vmware.csp.component.cafe.composition.api ca11071c-f0aa-4407-bcc9-980102c2a239 true CN=sevenvra.prem.com,OU=PREM,O=PREM,C=SG 2025-04-27T08:16:10Z 2020-04-28T08:16:10Z CN=sevenvra.prem.com,OU=PREM,O=PREM,C=SG 8E:68:E8:F0:EE:BC:12:2B:2D:78:89:C5:F9:37:5E:7C:25:38:C0:32 Another way to check is from VAMI as shown below Also through ssh by executing the command CURL Command curl --insecure -f -s -H "Content-Type: application/json" "https://$HOSTNAME/component-registry/services/status/current?limit=200" | sed "s/}/\n/g" | grep -E -o ".serviceName.*serviceInitializationStatus.[^,]*" | sed "s/\"serviceTypeId.*,//g" | sed -e "s/\"//g" -e "s/:/=/g" -e "s/,/, /" | sed -e "s/serviceName\|serviceInitializationStatus\|=\|,\|null//g" | column -t | sort | cat -n [master] sevenvra:~ # curl --insecure -f -s -H "Content-Type: application/json" "https://$HOSTNAME/component-registry/services/status/current?limit=200" | sed "s/}/\n/g" | grep -E -o ".serviceName.*serviceInitializationStatus.[^,]*" | sed "s/\"serviceTypeId.*,//g" | sed -e "s/\"//g" -e "s/:/=/g" -e "s/,/, /" | sed -e "s/serviceName\|serviceInitializationStatus\|=\|,\|null//g" | column -t | sort | cat -n 1 advanced-designer-service REGISTERED 2 approval-service REGISTERED 3 authentication REGISTERED 4 authorization REGISTERED 5 branding-service REGISTERED 6 catalog-service REGISTERED 7 component-registry REGISTERED 8 composition-service REGISTERED 9 config-management-service REGISTERED 10 console-proxy-service REGISTERED 11 container-service REGISTERED 12 content-management REGISTERED 13 endpoint-configuration-service REGISTERED 14 event-broker-service REGISTERED 15 eventlog-service REGISTERED 16 fabric-service REGISTERED 17 forms-service REGISTERED 18 healthbroker-proxy-server REGISTERED 19 iaas-proxy-provider REGISTERED 20 iaas-service REGISTERED 21 identity REGISTERED 22 ipam-service REGISTERED 23 licensing-service REGISTERED 24 management-service REGISTERED 25 network-service REGISTERED 26 notification-service REGISTERED 27 o11n-gateway-service REGISTERED 28 placement-service REGISTERED 29 plugin-service REGISTERED 30 portal-service REGISTERED 31 properties-service REGISTERED 32 provisioning-service REGISTERED 33 reservation-service REGISTERED 34 shell-ui-app REGISTERED 35 software-service REGISTERED 36 sts-service REGISTERED 37 vco REGISTERED 38 workitem-service REGISTERED IaaS Web URL: https://iaas-web-fqdn/WAPI/api/status/web RESULT: Repository URL: https://iaas-web-fqdn/Repository/Data/MetaModel.svc RESULT: IaaS Manager URL: https://iaas-manager/VMPSProvision RESULT: IaaS Manager URL: https://iaas-mgr-fqdn/VMPS2 RESULT: DEM Orchestrator Login to Tenant > Infrastructure > Monitoring > DEM Status DEM Worker Login to Tenant > Infrastructure > Monitoring > DEM Status Proxy Agents Login to Tenant > Infrastructure > Compute Resources > Compute Resource > View Proxy Agent vRealize Orchestrator URL: https://vra-va-fqdn:8283/vco-controlcenter Click on validate the configuration RESULT:

Here's a small article to identify the Java version being used inside vRO or vRA 8.x As a first step one has to identify container id Let's first grep for VCO processes docker ps | grep vco The output of above command would be as follows root@premvra [ ~ ]# docker ps | grep vco cb770fa8f97d 07b9846ebc64 "/bin/bash -c './cre…" 2 weeks ago Up 2 weeks k8s_vco-controlcenter-app_vco-app-54cfbdbdc-2k7wc_prelude_473e2e29-9b70-11ea-8c90-005056a77fe1_0 d572660d055c 07b9846ebc64 "/bin/bash -c './cre…" 2 weeks ago Up 2 weeks k8s_vco-server-app_vco-app-54cfbdbdc-2k7wc_prelude_473e2e29-9b70-11ea-8c90-005056a77fe1_0 d59aebe8bf54 0ced08cb52f4 "dockerd-entrypoint.…" 2 weeks ago Up 2 weeks k8s_vco-polyglot-runner_vco-app-54cfbdbdc-2k7wc_prelude_473e2e29-9b70-11ea-8c90-005056a77fe1_0 95fbc06fa017 vmware/pause:3.1 "/pause" 2 weeks ago Up 2 weeks k8s_POD_vco-app-54cfbdbdc-2k7wc_prelude_473e2e29-9b70-11ea-8c90-005056a77fe1_0 We need to fetch container id from the above output one we want to know the java version Since we are interested in vco-server-app and this has an id: d572660d055c from the above Then use the following command to find out the java version root@premvra [ ~ ]# docker exec -it d572660d055c java -version openjdk version "1.8.0-internal" OpenJDK Runtime Environment (build 1.8.0-internal-_2019_10_29_05_18-b00) OpenJDK 64-Bit Server VM (build 25.71-b00, mixed mode) We can also check from vco-app-server logs under /var/log/container/ which will ve logged during service startup. {"log":"2020-05-21 14:37:49.278+0000 [localhost-startStop-1] INFO {} [O11N] Sysprop: java.runtime.name = Java(TM) SE Runtime Environment\n","stream":"stdout","time":"2020-05-21T14:37:49.278367624Z"} {"log":"2020-05-21 14:37:49.278+0000 [localhost-startStop-1] INFO {} [O11N] Sysprop: java.runtime.version = 1.8.0_221-b11\n","stream":"stdout","time":"2020-05-21T14:37:49.278453768Z"}

This morning, I was involved in an escalation where NIC flaps were seen on 6 out of 7 hosts on a brand new vxRail Cluster Looks like it all started once administrator started changing MTU values hosted logs didn't help a great deal as it was reporting that vmnic has gone down. When we did check vmkernel logs, MTU was constantly flipping between 1500 and 9000 grep -i "changing MTU" vmkernel.log 2017-12-18T02:54:18.954Z cpu19:65645)<6>ixgbe 0000:01:00.1: vmnic1: changing MTU from 9000 to 1500 2017-12-18T02:54:19.594Z cpu19:65645)<6>ixgbe 0000:01:00.0: vmnic0: changing MTU from 9000 to 1500 2017-12-18T03:02:28.343Z cpu21:65645)<6>ixgbe 0000:01:00.1: vmnic1: changing MTU from 1500 to 9000 2017-12-18T03:02:28.985Z cpu26:65645)<6>ixgbe 0000:01:00.0: vmnic0: changing MTU from 1500 to 9000 2017-12-18T03:02:59.633Z cpu25:65645)<6>ixgbe 0000:01:00.1: vmnic1: changing MTU from 9000 to 1500 2017-12-18T03:03:00.269Z cpu30:65645)<6>ixgbe 0000:01:00.0: vmnic0: changing MTU from 9000 to 1500 2017-12-18T03:08:48.374Z cpu25:65645)<6>ixgbe 0000:01:00.1: vmnic1: changing MTU from 1500 to 9000 2017-12-18T03:08:49.006Z cpu25:65645)<6>ixgbe 0000:01:00.0: vmnic0: changing MTU from 1500 to 9000 What we see in vmkernel logs is MTU flap. The server was using ixgbe driver. Whenever an MTU change is made, it would cause the driver to bring down the NIC, make necessary changes to hardware, and then bring it back up. The link status will be reported to vmkernel and vobd would take down these changes. vmkernel.log snippet 2017-12-18T03:45:09.454Z cpu23:69310 opID=4758f26a)NetOverlay: 1107: class:vxlan is already instantiated one on depth 0 2017-12-18T03:45:09.470Z cpu20:65645)<6>ixgbe 0000:01:00.1: vmnic1: changing MTU from 1500 to 9000 2017-12-18T03:45:10.101Z cpu23:66107)vxlan: VDL2PortOutputUplinkChangeCB:649: Output Uplink change event with priority :5 was ignored for portID: 400000e. 2017-12-18T03:45:10.101Z cpu23:66107)netschedHClk: NetSchedHClkNotify:2892: vmnic1: link down notification 2017-12-18T03:45:10.101Z cpu22:65646)vdrb: VdrHandleUplinkEvent:1605: SYS:DvsPortset-0: Uplink event 2 for port 0x400000a, linkstate 0 2017-12-18T03:45:10.101Z cpu28:65645)<6>ixgbe 0000:01:00.0: vmnic0: changing MTU from 1500 to 9000 2017-12-18T03:45:10.738Z cpu2:66105)vxlan: VDL2GetlEndpointAndSetUplink:387: Now, no active uplinks in tunnel group:67108878. 2017-12-18T03:45:10.738Z cpu2:66105)netschedHClk: NetSchedHClkNotify:2892: vmnic0: link down notification 2017-12-18T03:45:10.738Z cpu2:66105)netschedHClk: NetSchedHClkDoFlushQueue:3818: vmnic0: dropping 6 packets from queue netsched.pools.persist.default 2017-12-18T03:45:10.738Z cpu2:66105)netschedHClk: NetSchedHClkDoFlushQueue:3818: vmnic0: dropping 3 packets from queue netsched.pools.persist.mgmt 2017-12-18T03:45:10.738Z cpu8:65647)vdrb: VdrHandleUplinkEvent:1605: SYS:DvsPortset-0: Uplink event 2 for port 0x4000008, linkstate 0 2017-12-18T03:45:10.739Z cpu28:69310 opID=4758f26a)VMKAPIMOD: 86: Failed to check if port is Uplink : Failure 2017-12-18T03:45:10.739Z cpu28:69310 opID=4758f26a)Team.etherswitch: TeamESLACPLAGEventCB:6277: Received a LAG DESTROY event version :0, lagId :0, lagLinkStatus :NOT USED,lagName :, uplinkName :, portLinkStatus :NOT USED, portID :0x0 2017-12-18T03:45:10.739Z cpu28:69310 opID=4758f26a)netioc: NetIOCSetRespoolVersion:245: Set netioc version for portset: DvsPortset-0 to 3,old threshold: 3 2017-12-18T03:45:10.739Z cpu28:69310 opID=4758f26a)netioc: NetIOCSetupUplinkReservationThreshold:135: Set threshold for portset: DvsPortset-0 to 75, old threshold: 75 2017-12-18T03:45:10.741Z cpu28:69310 opID=4758f26a)netioc: NetIOCPortsetNetSchedStatusSet:1207: Set sched status for portset: DvsPortset-0 to Active, old:Active 2017-12-18T03:45:10.741Z cpu28:69310 opID=4758f26a)VLANMTUCheck: NMVCDeployClear:871: can't not find psReq for ps DvsPortset-0 2017-12-18T03:45:10.784Z cpu15:69324 opID=1731b730)World: 12230: VC opID de31f2ec maps to vmkernel opID 1731b730 2017-12-18T03:45:10.784Z cpu15:69324 opID=1731b730)Tcpip_Vmk: 263: Lookup route failed 2017-12-18T03:45:13.357Z cpu19:10575834)CMMDS: AgentSendHeartbeatRequest:211: Agent requesting a reliable heartbeat from node 5a1e3e8b-2e6e-82a4-bfac-a0369fdec1c4 2017-12-18T03:45:41.383Z cpu22:66079)netschedHClk: NetSchedHClkNotify:2892: vmnic1: link down notification 2017-12-18T03:45:41.383Z cpu22:66079)netschedHClk: NetSchedHClkDoFlushQueue:3818: vmnic1: dropping 10 packets from queue netsched.pools.persist.mgmt 2017-12-18T03:45:41.383Z cpu1:65647)vdrb: VdrHandleUplinkEvent:1605: SYS:DvsPortset-0: Uplink event 2 for port 0x400000a, linkstate 0 2017-12-18T03:45:41.383Z cpu21:65645)<6>ixgbe 0000:01:00.0: vmnic0: changing MTU from 9000 to 1500 Now, after a detailed investigation of logs noticed that all DVS operations are coming from vCenter Moreover, we do see the following error message stating "The operation reconfigureDistributedVirtualSwitch on the host <> disconnected the host and was rolled back" From the above message we could clearly interpret that there is a connectivity issue between vCenter and the hosts. Since we do not see any driver/hardware related errors in the logs, we wanted to attempt increasing timeout value for network rollback under vpxd advanced settings. Procedure Use the vSphere Web Client to increase the timeout for a rollback on vCenter Server. If you encounter the same problem again, increase the rollback timeout with 60 seconds incrementally until the operation has enough time to succeed. On the Manage tab of a vCenter Server instance, click Settings. Select Advanced Settings and click Edit. If the property is not present, add the config.vpxd.network.rollbackTimeout parameter to the settings. Type a new value, in seconds, for the config.vpxd.network.rollbackTimeout parameter Click OK. Restart the vCenter Server system to apply the changes. The value was changed to 600 seocnds Once done, All hosts in the cluster were in a stable state and ready for NSX configuration. It looks like the changes were not being saved properly by vCenter Server in time. With this new timeout value, it had enough time to commit the transaction eventually stopping nic flaps. #vSphere

Browse to path /var/log/vmware/vco/app-server/ and then execute below command grep heap.usage metrics.log* | grep -v non | sed 's/.*value=//g' | perl -e 'use List::Util qw(max min sum); @a=();while(<>){$sqsum+=$_*$_; push(@a,$_)};$n=@a;$s=sum(@a);$a=$s/@a;$m=max(@a);$mm=min(@a);$std=sqrt($sqsum/$n-($s/$n)*($s/$n));$mid=int @a/2;@srtd=sort @a;if(@a%2){$med=$srtd[$mid];}else{$med=($srtd[$mid-1]+$srtd[$mid])/2;};print "records:$n\nsum:$s\navg:$a\nstd:$std\nmed:$med\max:$m\min:$mm";' this will give values for current heap usage inside vRO appliance ( example below )

Request filters restrict the types of HTTP requests that IIS processes. By blocking specific HTTP requests, request filters help prevent potentially harmful requests from reaching the server. The request filter module scans incoming requests and rejects request that is unwanted based upon the rules that you set up For example, if you set the allowUnlisted attribute to false, all requests for files with extensions that are not contained in the list of allowed extensions will be denied. When request filtering blocks an HTTP request because of a denied file name extension, IIS will return an HTTP 404 error to the client and log the following HTTP status with a unique sub status that identifies the reason that the request was denied When request filtering is enabled Infrastructure as a Service component of vRealize Automation 7.x needs this option to be checked or enabled. IaaS uses .jar , .dll , .aspx .config .workflow and many more file extensions which ensures it's IIS functionality is intact and it serves it's application pools as expected. By no means, this setting has to be disabled. The moment you disallow unlisted file name extensions your Manager Service would go down as the extensions needed to run your application are not whitelisted and would be blocked. [UTC:2020-05-21 10:22:41 Local:2020-05-21 10:22:41] [Error]: [sub-thread-Id="100" context="" token=""] falseCollectedDataImportService: Ignoring exception: System.Data.Services.Client.DataServiceQueryException: An error occurred while processing this request. ---> System.Data.Services.Client.DataServiceClientException: * * * HTTP Error 404.7 - Not Found The request filtering module is configured to deny the file extension. Most likely causes: Request filtering is configured for the Web server and the file extension for this request is explicitly denied. Things you can try: Verify the configuration/system.webServer/security/requestFiltering/fileExtensions settings in applicationhost.config and web.config. The moment you re-enable "Allow Unlisted File Name Extensions" your Manager Service would automatically start functioning [UTC:2020-05-21 10:22:41 Local:2020-05-21 10:22:41] [Error]: [sub-thread-Id="100" context="" token=""] Error occurred writing to the repository tracking logSystem.Net.WebException: The remote server returned an error: (404) Not Found. at System.Data.Services.Client.BatchSaveResult.BatchRequest()at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)at DynamicOps.Repository.RepositoryServiceContext.SaveChanges(SaveChangesOptions options) [UTC:2020-05-21 10:22:42 Local:2020-05-21 10:22:42] [Info]: [sub-thread-Id="7" context="" token=""] Processing ping report, report queue depth is 0 [UTC:2020-05-21 10:23:12 Local:2020-05-21 10:23:12] [Info]: [sub-thread-Id="7" context="" token=""] Processing ping report, report queue depth is 0 [UTC:2020-05-21 10:23:18 Local:2020-05-21 10:23:18] [Debug]: [sub-thread-Id="49" context="" token=""] DC: Created data collection item, WorkitemID 89c645a6-21c1-4086-857b-466d74fc32af, Task state, Agent premvc.prem.com, Entity primary, StatusID = f95f216b-ca19-41ef-9565-60326cdc94cd VMware's IIS hardening recommendation states that one has to go contact Microsoft for vendor's hardening guidelines VMware does not provide a list of extensions that have to be whitelisted. This is how it is from vCAC 4.x days. So if your hardening your IAAS system ensure you do not deselect "Allow unlisted file name extensions" and get into a problem !!! I hope this helps !!!

Here's the guide on how you can upgrade your vRA 7.x environment using vRSLCM 2.x For this example, I am performing an upgrade from version 7.5 HF16 to 7.6 GA I have a simple environment wpacvra.prem.com being my vRealize Automation appliance wpaciaas.prem.com being my Infrastructure as a Service node IaaS database is installed on the same node where IaaS services are installed Before we start to upgrade we have to take a snapshot on all the components. This is a MANDATORY step Enter Snapshot Prefix and ensure do not check snapshot with memory box. We should not take snapshots on vRA components with memory Once the snapshots are completed, let's create a request to upgrade Click on the three dots on the right side of the environment and then click on upgrade You will be redirected to this below pane. Ensure we click on the checkbox for IaaS Snapshot. So that if IaaS upgrade fails so whatsoever reason, you can revert to Post-VA upgrade state and then retry upgrade again In this example, as shown below I would be using vRealize Suite Lifecycle Manager Repository which contains upgrade binaries Once done, when we click NEXT, we are now into Precheck phase Here we need to click on RUN-PRECHECK so that the request for validation is submitted successfully The first phase of precheck would be your data validation The second phase of precheck would be vRealize Automation validations During this phase, if there are any exceptions it would let you know as shown in the screenshot below The exception thrown above indicates that there is a reboot pending on my IAAS node. So let's reboot IAAS node and then rerun precheck As you can see below after we performed a reboot on our IAAS node the PreCheck phase did complete successfully I have attached PRECHECK report from my environment here for reference Since PRECHECK has now been successfully completed, click on NEXT to move on to review SUMMARY of the request. On the top right corner of this page, you have an option to run precheck on submit. This is optional. If you would like to run it again that's absolutely fine. It would just take a few more minutes extra. Once we click on SUBMIT, request of upgrading vRealize Automation is created If we have selected to run pre-check on submit, then there would be 4 steps, else 3 steps for a successful upgrade in a simple environment Step 2/4 performs vRA appliance upgrade This has various phases upgradevrava upgradevrava-com.vmware.vrealize.lcm.plugin.core.vra70.task.StartGenericVraTask FROM STATE com.vmware.vrealize.lcm.plugin.core.vra70.task.StartGenericVraTask TO STATE com.vmware.vrealize.lcm.core.vra70.task.upgrade.VraUpgradeTask upgradevrava-com.vmware.vrealize.lcm.core.vra70.task.upgrade.VraUpgradeTask This is the phase where your appliance is being upgraded. There are three phases in a vRA appliance upgrade Pre Update Upgrade Post Update Here's the list of logs you need to monitor during the pre-update phase /opt/vmware/var/log/vami/updatecli.log /var/log/bootstrap/preupdate.log The below screenshot shows that the preupdate phase has been successful. Now at this phase, we only have to monitor /opt/vmware/var/log/vami/updatecli.log This is where the upgrade phase logs are being logged. Following message is seen once the install phase is complete and post-install or postupdate starts 20/05/2020 10:32:26 [INFO] Update status: Done package installation 20/05/2020 10:32:26 [INFO] Update status: Running post-install scripts Here's a small snippet from postupdate.log Remember this upgrade VA task will be at 67% for a while till the upgrade is complete on the virtual appliance ( might change for a distributed environment ) Below screenshot indicate that the post update is now complete And the final phase of upgrade begins After a few seconds, the whole appliance upgrade is now complete upgradevrava-com.vmware.vrealize.lcm.platform.automata.service.task.FinalTask is now complete as shown below As we selected to take a snapshot after VA upgrade is complete, it would initialize a snapshot task now Here's the task completing on vCenter for IAAS node After snapshot task is complete which finished your Step#3 we now move on to IAAS upgrade phase This is where IAAS upgrade starts As the first step during this phase it would initiate a reboot on the virtual appliance Once the reboot task is complete as shown above, the new 7.6 VAMI would be available One thing to note during this phase we can monitor progress from /opt/vmware/var/log/vami/upgrade-iaas.log Unless all VA services are started it would not move ahead with IAAS components upgrade Here's the snippet which states that all services have been started [UTC: 2020-05-20 10:49:34.204100; Local: 2020-05-20 10:49:34.204108] [INFO]: Starting automatic IaaS upgrade. [UTC: 2020-05-20 10:49:34.204426; Local: 2020-05-20 10:49:34.204430] [INFO]: IaaS upgrade is pending: Waiting for VA services to start... [UTC: 2020-05-20 11:06:09.930810; Local: 2020-05-20 11:06:09.930820] [INFO]: All VA services have started. IAAS server components would now be identified and then upgrade would be initiated As stated above upgrade-iaas.log is one and the other one would be inside the IAAS node itself <>\Program Files (x86) \VMware\vCAC\ManagementAgent\Logs\CommandOutput Starts upgrading server components Once done, it goes ahead with DEM's and completes them Then comes Agents and few final tasks which complete your IAAS upgrade as well Now going back to vRSLCM UI, we can see that the final tasks are complete as well drawing curtains to the automated upgrade from 7.5 to 7.6 The whole request is now marked as completed Going back to the environment, we will now see the version to be 7.6 We can even check the same from VAMI Now let's deep-dive into various phases from logs perspective, this would help in troubleshooting ||| SECTION BEING UPDATED |||

How do you recover / clean when we delete a datastore cluster from vCenter but it still shows up under Reservations of vRA 7.x Sharing the approach taken to get to the solution Note: Below steps have to be performed on IAAS ( MS SQL Database ) Step 1 Below query is to see if there any VM associated with the Storage path SELECT vm.VirtualMachineName, vm.VirtualMachineID, vm.StoragePath, hr.HostReservationName, h.HostName FROM VirtualMachine vm JOIN HostReservationToStorage hrts ON vm.HostStorageReservationID = hrts.HostReservationToStorageID JOIN HostToStorage hts ON hrts.HostToStorageID = hts.HostToStorageID JOIN HostReservation hr ON vm.HostReservationID = hr.HostReservationID JOIN Host h ON vm.HostID = h.HostID Where hts.StoragePath = '<>' <> is the value of the Storage Path which was deleted or decommissioned on vCenter This query returned no results which means that there are no virtual machines on this storage path Step 2 Now since we know that there are no virtual machines using this storage path let's move on to the next phase where we find out on how many tables is this storage path being referenced Using stored procedure "SearchAllTables" we have to find out the references. How we create this stored procedure has been documented here exec SearchAllTables '<>'; There were 4 tables this storage path was part of [dbo].[HostToStorage].[StoragePath] [dbo].[Storage].[StorageName] [dbo].[UserLog].[Message] [dbo].[VirtualMachineHistoryProperties].[PropertyValue] The first two tables are important ones. The last two Userlog and VirtualMachineHistoryProperties are not that important as it would relate more to logging and storing properties. So let's inspect HostToStorage and Storage tables in depth. Step 3 Executing below query on dbo.HostToStorage we would capture few ids of this StoragePath. select * from dbo.HostToStorage where StoragePath = '<>'; HostToStorageID : 147EC6B2-BC8D-4B79-A94D-A1D30833311B HostID : 6896D5CC-ABFD-4719-A254-FFEA79453239 StorageID : FEF71405-D569-4AAF-BEBF-02ACC37BFB73 Step 4 Executing below query on dbo.Storage we would capture few ids of this StoragePath. select * from dbo.Storage where StoragePath = '<>'; StorageID : FEF71405-D569-4AAF-BEBF-02ACC37BFB73 , StorageUniqueID : f264d83b-d7e6-4f1c-a423-ca9173ac80d9/group-p412357 */ Step 5 If you try and delete these from SQL query window it would not let you do it as there are references to this on another table delete from dbo.HostToStorage where StoragePath = '<>'; delete from dbo.Storage where StorageName = '<>'; Msg 547, Level 16, State 0, Line 39 The DELETE statement conflicted with the REFERENCE constraint "HostToStorage_HostReservationToStorage". The conflict occurred in database "vra", table "dbo.HostReservationToStorage", column 'HostToStorageID'. The statement has been terminated. Msg 547, Level 16, State 0, Line 41 The DELETE statement conflicted with the REFERENCE constraint "Storage_HostToStorage". The conflict occurred in database "vra", table "dbo.HostToStorage", column 'StorageID'. The statement has been terminated. Step 6 If we read the above exception dbo.Storage was referring to dbo.HostToStorage and dbo.HostToStorage to dbo.HostReservationToStorage In order to clean this up, we need to find out from dbo.HostReservationToStorage on what other reservations are using this Storage Path To identify this we have to pick up the HostToStorageID from Step#3 and then execute below query select * from HostReservationToStorage where HostToStorageID = '147EC6B2-BC8D-4B79-A94D-A1D30833311B'; This returns three values in my case HostReservationToStorageID : D64058DB-2CE9-4F87-AA2A-3345639B69A4 HostReservationID : F7655D38-5850-4C22-B2E2-A814C2870135 HostReservationToStorageID : 5B444E45-C798-4757-A7ED-9CC5BEEBCA42 HostReservationID : 0AE886F9-44CE-4EE2-96A2-06FE82AB456E HostReservationToStorageID : A4925663-A7CF-4616-8D48-F581756C7009 HostReservationID : 1723EEDE-C95E-47CF-B993-68315AD7298D Step 7 Now that I have HostReservationID it's very easy for me to find out what's the HostReservationName select * from HostReservation where HostReservationID in ( 'F7655D38-5850-4C22-B2E2-A814C2870135','0AE886F9-44CE-4EE2-96A2-06FE82AB456E','1723EEDE-C95E-47CF-B993-68315AD7298D') The above query returns to me HostReservationNames as an output along with other outputs. Step 8 Using these reservation names captured, we would go ahead and then uncheck this storage path Step 9 Peforming data collection after we uncheck the storage path as discussed under Step#8 would remove this patch from all reservations Note: Please take a backup before you perform any changes on the database either SQL or Postgres !!! Hope this helps !!!

Create Stored Procedure CREATE PROC SearchAllTables @SearchStr nvarchar(100) AS BEGIN DECLARE @dml nvarchar(max) = N'' IF OBJECT_ID('tempdb.dbo.#Results') IS NOT NULL DROP TABLE dbo.#Results CREATE TABLE dbo.#Results ([tablename] nvarchar(100), [ColumnName] nvarchar(100), [Value] nvarchar(max)) SELECT @dml += ' SELECT ''' + s.name + '.' + t.name + ''' AS [tablename], ''' + c.name + ''' AS [ColumnName], CAST(' + QUOTENAME(c.name) + ' AS nvarchar(max)) AS [Value] FROM ' + QUOTENAME(s.name) + '.' + QUOTENAME(t.name) + ' (NOLOCK) WHERE CAST(' + QUOTENAME(c.name) + ' AS nvarchar(max)) LIKE ' + '''%' + @SearchStr + '%''' FROM sys.schemas s JOIN sys.tables t ON s.schema_id = t.schema_id JOIN sys.columns c ON t.object_id = c.object_id JOIN sys.types ty ON c.system_type_id = ty.system_type_id AND c .user_type_id = ty .user_type_id WHERE t.is_ms_shipped = 0 AND ty.name NOT IN ('timestamp', 'image', 'sql_variant') INSERT dbo.#Results EXEC sp_executesql @dml SELECT * FROM dbo.#Results END Execute Stored Procedure exec SearchAllTables '<<VALUE-ONE-WOULD-LIKE-TO-SEARCH'; Delete Stored Procedure DROP PROCEDURE dbo.SearchAllTables

Data Collection for a specific compute resource was failing on a given endpoint in vRealize Automation 7.x The exception was as below 2020-05-14T09:31:30.772Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6016"] [sub-thread-Id="32" context="" token=""] Error retrieving properties in task 26394140: System.Web.Services.Protocols.SoapException: Required property obj is missing from data object of type ObjectSpec while parsing serialized DataObject of type vmodl.query.PropertyCollector.ObjectSpec at line 1, column 717 Data Collection for other compute resources on the same endpoint was working as expected. So why wouldn't it work only for one specific compute resource was the question Went into my lab and then thought of checking the behavior to compare working and the non-working one Usecase was to perform an inventory data collection on a selected compute resource Here's the snippet from a working instance 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Info" thread="6276"] [sub-thread-Id="23" context="" token=""] Starting : Processing Workitem ID [e779099f-b1df-4a31-99f9-1169feb35272] [inventory] 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.Hostname=primary 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.ParentIdentity=premvc.prem.com/production/host/primary 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Name=primary 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Identity=premvc.prem.com/production/host/primary 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Endpoint0=10.109.10.84 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Endpoint1=10.109.10.85 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Endpoint2=10.109.10.86 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.Name=inventory 2020-05-18T04:46:16.412Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetHosts: 1631 milliseconds 2020-05-18T04:46:16.537Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetClusters: 123 milliseconds 2020-05-18T04:46:16.568Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetNetworks: 27 milliseconds 2020-05-18T04:46:16.599Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetResourcePools: 28 milliseconds 2020-05-18T04:46:16.631Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetStoragePools: 34 milliseconds 2020-05-18T04:46:16.631Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetTemplates: 0 milliseconds As you can see in the above snippet after it fetches VirtualMachine.Admin.Name=inventory it goes ahead and then gets the information of the inventory like GetHosts , GetClusters, GetNetworks, GetResourcePools, and GetStoragePools, etc... If we check the same on a non-working instance 2020-05-14T09:31:30.631Z <> vcac: [component="iaas:VRMAgent.exe" priority="Info" thread="2064"] [sub-thread-Id="18" context="" token=""] Starting : Processing Workitem ID [38e0898c-4f0a-4f66-bbf0-368fff7c82d1] [inventory] 2020-05-14T09:31:30.631Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.Hostname=<> 2020-05-14T09:31:30.631Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.ParentIdentity=<><<<>/host/<> 2020-05-14T09:31:30.631Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Name=<> 2020-05-14T09:31:30.631Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Identity=<>/<>/host/<> 2020-05-14T09:31:30.631Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Endpoint0=<> 2020-05-14T09:31:30.631Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.Name=inventory 2020-05-14T09:31:30.772Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6016"] [sub-thread-Id="32" context="" token=""] Error retrieving properties in task 26394140: System.Web.Services.Protocols.SoapException: Required property obj is missing from data object of type ObjectSpec while parsing serialized DataObject of type vmodl.query.PropertyCollector.ObjectSpec at line 1, column 717 while parsing property "objectSet" of static type ArrayOfObjectSpec while parsing serialized DataObject of type vmodl.query.PropertyCollector.FilterSpec at line 1, column 315 while parsing call information for method RetrievePropertiesEx at line 1, column 218 while parsing SOAP body at line 1, column 207 while parsing SOAP envelope at line 1, column 38 while parsing HTTP request for method retrievePropertiesEx on object of type vmodl.query.PropertyCollector at line 1, column 0 at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult) at VMware.vSphere.VimService.EndRetrievePropertiesEx(IAsyncResult asyncResult) at Vmware.VSphereInterface.RetryRequestWrapper.Execute[T](Func`1 function) If you look closely and compare it, it's clearly failing to perform retrievePropertiesEx which does your GetHosts, GetClusters, GetNetworks, GetResourcePools, and GetStoragePools So clearly the problem is not on vRealize Automation end, it's more on vSphere. When we went ahead and verified this particular cluster / compute resource where data collection was failing does not even exist on vCenter. Which clearly explains the exception Required property obj is missing from data object of type ObjectSpec Take-Aways The problem does not always have to be on vRealize Automation end, it can be elsewhere. If we have integrated a vCenter as an endpoint when any changes are being made on the endpoint directly as a best practice one has to inform your vRealize Automation admins. This would ensure before off-boarding a cluster on endpoint proper precautions are taken to offboard the same compute resource from the vRA perspective as well.

One of the users of vRA asked a question on why do we see so many DLL's under Temp folder of a user account configured to run DEM's Tried to check the same in my environment and found that it's same behavior on my side as well As shown in the screenshot below DEM Worker and Orchestrator are configured with a service account named "prem\srv-vra-seven" Selected one of the DLL's and opened it with notepad ++, not a right tool to open but it did give enough information on what's creating these files under *Temp/* If we observe the dll , it states that it might be created due to UpdateCatalogWorkflow UpdateCatalogResource workflow is triggered by DEM If we observe timestamps in the above screenshot, it's the same time when the folders with the DLL's in them are created Details of what this UpdateCatalogResource does is present under DEM logs 2020-05-14T08:57:30.473Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="5212"] [sub-thread-Id="15" context="" token=""] WorkflowExecutionUnit: initialize started: 7709 2020-05-14T08:57:32.161Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="5212"] [sub-thread-Id="15" context="" token=""] The definition of workflow UpdateCatalogResource has been cleaned of designer-specific content: 00:00:01.6766975 2020-05-14T08:57:34.458Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="5212"] [sub-thread-Id="15" context="" token=""] WorkflowExecutionUnit: workflow application loaded: 7709 2020-05-14T08:57:34.551Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="5212"] [sub-thread-Id="15" context="" token=""] Worker Controller: initialized instance 7709 - UpdateCatalogResource of the workflow execution unit : 00:00:04.8993202 2020-05-14T08:57:34.551Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Debug" thread="5212"] [sub-thread-Id="15" context="" token=""] Workflow ID: 7709 Name: UpdateCatalogResource App ID: bf67dd56-a92c-4930-8cf1-8c6a3cbb28bf 2020-05-14T08:57:34.551Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Debug" thread="5212"] [sub-thread-Id="15" context="" token=""] Workflow ID: 7709 Argument Name: WorkflowOperation Argument Value: ManagementModelEntities Id 2737 WorkflowOperations 2737 UpdateCatalogResource 2020-05-14T08:57:34.817Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Executing 2020-05-14T08:57:34.911Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Executing 2020-05-14T08:57:34.926Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Executing 2020-05-14T08:57:34.926Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Executing 2020-05-14T08:57:35.551Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Closed 2020-05-14T08:57:35.567Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Debug" thread="8924"] [sub-thread-Id="8" context="" token=""] UpdateCatalogResource Workflow: UpdateCatalogByVirtualMachineId.Execute 2020-05-14T08:57:37.786Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Executing 2020-05-14T08:57:37.786Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Closed 2020-05-14T08:57:37.786Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Management Model: The DeleteWorkflowOperation activity started. 2020-05-14T08:57:38.364Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Debug" thread="8924"] [sub-thread-Id="8" context="" token=""] DeleteWorkflowOperation: Deleting workflow operation with id: 2737 and operation name: UpdateCatalogResource 2020-05-14T08:57:38.677Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Management Model: The DeleteWorkflowOperation activity finished. 2020-05-14T08:57:38.677Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Executing 2020-05-14T08:57:38.677Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Closed 2020-05-14T08:57:38.677Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Closed 2020-05-14T08:57:38.677Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Closed 2020-05-14T08:57:38.692Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Closed 2020-05-14T08:57:38.708Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Worker Controller: WorkflowApplication_Completed in child domain. 2020-05-14T08:57:38.708Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Worker Controller: WorkflowApplication_Completed 2020-05-14T08:57:38.708Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Worker Controller: ProcessWorkflowCompleted 2020-05-14T08:57:38.708Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Debug" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow Complete: 7709 - Successful These folders can be deleted and should not impact any of the underlying components. !!! Hope this helps !!!

There was an interesting scenario where Linux deployments where stuck at a point where it's Installing OS ( this is not a standard template-based deployment but a kickstart deployment ) For more information on Linux Kickstart deployments from vRA click here During this phase of Installing OS, we also install a Gugent ( Guest Agent for Linux ) This gugent installation was throwing exceptions as mentioned below Linux Guest Agent Connection Log /usr/share/gugent/axis2/logs/gugent-axis.log [Wed May 13 22:15:04 2020] [debug] http_transport_sender.c(246) ctx_epr:https://<>:443/VMPS2 [Wed May 13 22:15:04 2020] [debug] http_transport_sender.c(805) using axis2 native http sender. [Wed May 13 22:15:04 2020] [...TRACE...] http_sender.c(252) Entry:axis2_http_sender_send [Wed May 13 22:15:04 2020] [debug] http_sender.c(416) msg_ctx_id:urn:uuid:607582d6-9513-1ea1-30b8-005056889293 [Wed May 13 22:15:04 2020] [info] [ssl client] Client certificate chain filenot specified [Wed May 13 22:15:04 2020] [debug] ssl/ssl_utils.c(190) [ssl client] SSL certificate verified against peer [Wed May 13 22:15:04 2020] [error] http_sender.c(3006) Authtype NTLM is notsupported Linux Guest Agent Log /usr/share/gugent/GuestAgent.log 2020-05-13 11:23:28 Application: [Information] Using the network enabled proxy ... 2020-05-13 11:23:28 Application: [Information] The vCAC endpoint is https://<>:443/VMPS2. 2020-05-13 11:23:28 Application: [Information] The AXIS2C directory is axis2/. 2020-05-13 11:23:28 Application: [Information] Requesting work for agent ID 42088131-66e2-832d-1324-e5b8f05c5ee7. 2020-05-13 11:23:28 Application: [Information] Fetching a work item ... 2020-05-13 11:23:28 Application.Proxy: [Error] 2020-05-13 11:23:28 Application.Proxy: [Error] Error returned by axis2: Error occurred in transport 2020-05-13 11:23:28 Application.Proxy: [Error] This error code may indicate an incorrect endpoint address https://<>:443/VMPS2, or a certificate issue. 2020-05-13 11:23:28 Application.Proxy: [Error] Consult axis2 log file at axis2/logs for more detail As a first step, https://<>:443/VMPS2 tried accessing the following URL. This URL was not returning a proper response. VMPS is a VirtualMachineProvisionService which corresponds to Manager Service but not Web as being shown in the logs This Guest Agent Workitem service configuration is actually stored under ManagerService.exe.Config https://IAAS-MGR-FQDN:443/VMPS2 returns following output This is where we found out that when Guest Agent was being installed it was calling a wrong FQDN to fetch the workitem. As explained above it was trying to reach out to IAAS-WEB-FQDN rather than trying to pull information from IAAS-MGR-FQDN This misconfiguration was done on one of the repositories from where the guest is being downloaded. Once the script which is installing guest agent is fixed with the right URL from where it has to pull the worktems or point it to the manager sevice fqdn , then issues with respect to deployments were resolved.