Search Results

After upgrading to vRealize Automation 8.4.1 , when you click on vRA Migration Assistant and browse to any of the options you would see below exception Http failure response for https://eightfour.nukescloud.com/migration/api/business-groups?page=0&size=20&sort=name,asc&sort=tenantSourceName,asc: 403 OK This issue is seen because of the recent changes performed with roles under version vRA 8.4.1 Previously in 8.3 or 8.4 GA, When we look into Cloud Assembly service roles we used to have roles as below Service Role : Cloud Assembly 1. Cloud Assembly Administrator 2. Cloud Assembly User 3. Cloud Assembly Viewer Service Role: 1. Migration Assist Administrator 2. Migration Assist User In 8.4.1 GA , we have following service roles Service Role : Cloud Assembly 1. Cloud Assembly Administrator 2. Cloud Assembly User 3. Cloud Assembly Viewer 4. Migration Assistant Administrator 5. Migration Assist Viewer Service Role: vRA Migration Assistant 1. Migration Assist Administrator 2. Migration Assist User See the difference here. From 8.4.1 vRA Migration Assistant service roles have been migrated to Cloud Assembly service roles Keeping this scenario in mind, after upgrade we have following service roles assigned 1. SaltStack Config 2. Orchestrator 3. Service Broker 4. Cloud Assembly 5. Code Stream 6. vRA Migration Assistant Now to resolve this issue we need to select migration roles present under Cloud Assembly and ignore or remove vRA Migration Assistant So now remove vRA Migration Assistant service role and select Cloud Assembly based Migration role as shown below Post this , logout and login into the vRA Portal and you do not see the exception any more.

One of the most common issues for Cloud Proxy upgrade failure is due to the provider-runtime.xml file not getting populated correctly or getting corrupted Exception "Failure: Error downloading manifest. Please contact your vendor. The requested Url returned error: 404 Not found url https://vapp-updates.vmware.com" Resolution Check the contents of provider-runtime.xml /opt/vmware/var/lib/vami/update/provider/provider-runtime.xml _ _ If the file has contents similar to the above, manually update the file with the following contents and restart the cloud proxy. This should help in fixing the Cloud Proxy Upgrade

Symptom In ABX service logs you see similar error messages: abx-service [host='abx-service-app-5c578445f7-j2w6m' thread='parallel-47' user='' org='' trace='' component='GatewayActionProvider' method='convertExceptions:437' actionRunI d='8a7480a179b68ffb0179e552568b1e1a' context='87aad62e-0d90-4a61-b265-8b3720239a00' tokenId='8eb4435a-be1f-4921-9ca3-d809625dce4c' parentTokenId='8bdaa86c-1a01-4714-8c8f-9b1d41a5d98f'] c.v.a.a.p.i.g.Gate wayActionProvider.convertExceptions:437 - Converting gateway provider exception to abx provider exception: com.vmware.automation.abx.serverless.gateway.exception.BuildImageException: ontext: [/workspace] ESC[36mINFOESC[0m[0015] COPY . /function/ ESC[37mDEBUESC[0m[0015] Resolved . to . ESC[37mDEBUESC[0m[0015] Resolved /function/ to /function ESC[37mDEBUESC[0m[0015] Getting files and contents at root /workspace/ for /workspace ESC[37mDEBUESC[0m[0015] Copying file /workspace/Dockerfile to /function/Dockerfile ESC[37mDEBUESC[0m[0015] Copying file /workspace/abx_wrapper.py to /function/abx_wrapper.py ESC[36mINFOESC[0m[0015] RUN groupadd function ESC[36mINFOESC[0m[0015] cmd: /bin/sh ESC[36mINFOESC[0m[0015] args: [-c groupadd function] ESC[36mINFOESC[0m[0015] Running: [/bin/sh -c groupadd function] You are required to change your password immediately (password expired) groupadd: PAM: Authentication token is no longer valid; new one required error building image: error building stage: failed to execute command: waiting for process to exit: exit status Cause Root password for base ABX runner image has expired. Affected Version vRealize Automation 8.4 GA vRealize Automation 8.4.1 GA Resolution This problem is addressed in vRA 8.4.2 GA and later versions Workaround 1. Take a snapshot of the appliances 2. Download patch-abx-images.sh on one of the appliances 3. Make the patch file executable: chmod a+x ./patch-abx-images.sh 4. Run the patch file: ./patch-abx-images.sh Click patch-abx0images.sh to download the script

Security build upgrade is failing because of space issue in /boot filesystem. Please follow the below workaround 1. SSH to vRB VA. 2. Create temp directory and move old kernal files. mkdir /tmp/boot cd /boot/ mv vmlinu* initrd* /tmp/boot Then apply the patch again

Finding a working configuration which helps to build a successful vIDM 8.x cluster was a huge problem. So thought of sharing what worked for me while I was trying to build a cluster in my lab. So, Let's start Creating CA Signed Certificates The first step is to keep our vIDM CA-signed certificate ready. For this one can generate CSR ( Certificate Signing Request ) from vRLCM by clicking on Generate CSR option available under Locker --- Certificate --- Generate CSR Then fill in all the details requested in the Generate CSR form. Remember the Common Name should always be your Load Balancer FQDN Server Domain/Hostname should contain all the hostnames which would part of the vIDM cluster All IP addresses corresponding to the vIDM nodes involved in this cluster must be documented One we click on GENERATE after we fill all the details as shown above we get a pem file downloaded. This PEM file must be given to your certificate authority to get this signed and generate a CA-signed certificate which can be used to deploy our vIDM cluster Your authority would give you a Key file and a Secure Certificate Once we have the above information we would use this to import this certificate into vRLCM To Import certificate into vRLCM, we need to go to Locker and then click on Certificate and Import certificate We have two files at hand the first one is vidm.key and the second one is premidm certificate. Open vidm.key using notepad and then paste that inside Private Key section Open premidm certificate using notepad and then copy and paste this content under Certificate Chain section Then click on IMPORT to get this certificate imported into Locker Once we have our CA-signed certificate which will be used in deploying our vIDM cluster imported into vRLCM we will now go ahead and then import this certificate into NSX-V load balancer Upload vIDM certificate chain and the corresponding root CA certificates onto NSX-V Edge Browse to NSX-V Edge, then under the configure tab, click on Certificates and then Add a new Certificate We need to enter the certificate details the same way we did before and that would import the certificate onto the edge Once the above step is done Server Certificate is imported into Edge Root certificate has to be added in the same way by exporting it out of Server certificate Paste this content into a separate file and save it as root.cer and then on the NSX Edge, Configure and then click on Add new CA certificate That's it you will have both ROOT and the SERVER certificates in place Configuring NSX-V Loadbalancer to support a clustered vIDM Application Profiles Here's the configuration which has to be part of Application Profiles which supports vIDM LB Application Profile Type: HTTPS End to End Persistence: Cookie Cookie Name: JSESSIONID Mode: App Session Expires in: 3600 Insert X-Forwarded-For-HTTP header: Enabled Under Client SSL tab Client Authentication: Ignore Under Server SSL, Server Authentication must be enabled Service Monitoring Under Service Monitoring Interval: 5 Timeout: 10 Max Retries: 3 Type: HTTPS Expected: 200 Method: GET URL: /SAAS/API/1.0/REST/system/health/heartbeat Ensure there are no mistakes while typing or copying URL information Pools Algorithm: LEASTCONN IP Filter: IPv4 Monitor Port: 443 Virtual Servers Virtual Server: Enable Protocol: HTTPS Post/Port Range: 443 Creating a Request in vRLCM for a Clustered vIDM deployment This is not a complicated task so I won't be discussing much. One important aspect is DELEGATE IP ensure this IP is not resolvable This will be used during the PGPOOL configuration of your vIDM Cluster. Before we submit the request we need to ensure all pre-validation is successful. If above certificate steps are not done then your vIDM deployment will fail at Stage-3 as shown below com.vmware.vrealize.lcm.common.exception.EngineException: vIDM install prevalidation failed at com.vmware.vrealize.lcm.vidm.core.task.VidmInstallPrecheckTask.execute(VidmInstallPrecheckTask.java:62) at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:45) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Attached a vIDM PreCheck report below for reference. Once we have all the prechecks successful, then we can submit the request. Post-Submission Stages during Cluster Deployment There are 20 stages for a vIDM clustered deployment, they are Stage 1: validateenvironmentdata As the name itself suggests it validates the environment data submitted Stage 2: infraprevalidation Infrastructure details provided will be validated, the same stuff which was performed during prechecks Stage 3: vidmprevalidation vIDM validations will be performed, the same as the ones in prechecks Stage 4: deployvidm deploys vIDM OVA's on vCenter Stage 5: vidmconfiguremaster,vidmprepareslave,vidmprepareslave During this stage, your Master or the First Node in your vIDM is configured and the Slaves, second and the third nodes will be prepared Note: In Stage 5, there is a phase called VidmFQDNUpdate if there is a failure observed at this stage as shown below in the screenshot, then your primary vIDM appliance is trying to open or communicate to your vIDM LB and expecting a valid response which is not happening. Under /opt/vmware/horizon/workspace/configurator.log following exceptions will be seen 2020-09-03T12:37:37,599 INFO (Thread-146) [;;;] com.vmware.horizon.svadmin.service.ApplicationSetupService - Invalid status code validating FQDN: https://premidm.prem.com : 503 2020-09-03T13:06:20,493 INFO (Thread-3) [;;;] com.vmware.horizon.svadmin.service.ApplicationSetupService - Invalid status code validating FQDN: https://premidm.prem.com : 503 2020-09-03T13:14:39,678 INFO (Thread-189) [;;;] com.vmware.horizon.svadmin.service.ApplicationSetupService - Invalid status code validating FQDN: https://premidm.prem.com : 503 2020-09-03T13:14:49,407 INFO (Thread-189) [;;;] com.vmware.horizon.svadmin.service.ApplicationSetupService - Invalid status code validating FQDN: https://premidm.prem.com : 503 2020-09-03T13:17:01,765 INFO (Thread-189) [;;;] com.vmware.horizon.svadmin.service.ApplicationSetupService - Invalid status code validating FQDN: https://premidm.prem.com : 503 At this moment you need to check if your Primary vIDM is responsive I'll check if https://<>/ is responsive and the vIDM landing page opens up If this works then I'll check if https://<>/ responds or redirects me to the available node and give me the vIDM landing page. If this works then you would not see the above failure in Stage 5. You will have to fix your Load Balancer configuration issue to proceed forward. No matter how many retries you perform, you will only see that one line in the configurator.log but nothing else. At this stage, your LB Pool status should show up for your Primary vIDM appliance Stage 6: vidmconfigureslave Stage 7: vidmclusterverify Stage 8: vidmclusterverify Stage 9: vidmenableconnector Stage 10: Stage 11: vidmpreparemasterpgpool Stage 12: vidmconfigurepgpool Note: You might encounter an exception during VidmAddSSHPostgresKeys task during this stage. Actually, your vIDM appliances are rebooted and if the appliances do not come on time then LCM will fail to execute the scripts to complete deployment. All we need to do is that to find out if SSH to the nodes are working and then perform a Retry Step 13: vidmstartmasterpgpoolservices Step 14: vidmstartslavepgpoolservices Step 15: vidminitialconfigprep Stage 16: savevmoidtoinventory Stage 17: environmentupdate Stage 18: notificationschedules Stage 19: setauthprovider Stage 20: vidmClusterHealthScheduler That's it. Your vIDM cluster has been completely deployed Post Deployment Checks When we browse to Environments then you can see the globalenvironment fully configured You may also use "Trigger Cluster Health" request Also on the pool status, you can see that all the nodes of vIDM cluster are fully functional and UP And when you browse to https://<>/ you should be able to browse the landing page !! Happy Learning !!

Seeing exceptions in the integrated components section of the system Diagnostics page.. Exceptions states There was a problem with Messaging service. Error retrieving rabbitmq status This is a known issue and here are the steps to fix it 1. Check and file permissions ls -lrth /etc/rabbitmq ls -lrth /db/rabbitmq 2. Change the ownership of these two folders: if they are not rabbitmq:rabbitmq in step 1 chown -R rabbitmq:rabbitmq /etc/rabbitmq chown -R rabbitmq:rabbitmq /db/rabbitmq 3. Restart the rabbitmq service service rabitmq-server restart

We have a Custom Resource which has a Create and Delete workflows mapped to it in vRealize Automation 8.x Suppose there is a need in organization and you have to modify certain inputs on these workflows. It's very easy to change this in the workflow ( vRealize Orchestrator ) end But, the same new value does not reflect under custom resource. The only solution available is to create a duplicated custom resource that is exactly the same except it has the new value in it. The other drawback is that we cannot deprecate old custom resource as they would be in use. This behavior is expected. The custom resource schema is immutable and can not be changed after the custom resource is created. The reason behind is that we don't want to allow users to break existing blueprints when "Create" vRO workflow inputs change. This is a current limitation in vRealize Automation 8.x and is expected to be fixed in future releases

We had a use-case where user accidentally passed VM name which was already provisioned and existing on vCenter through vRA 8.x The moment this was done, request was anyways expected to fail detecting that there is a VM with the same name already present on the endpoint 2021-04-05T00:03:29.934Z ERROR provisioning [host='provisioning-service-app-67b88d497d-w7cvq' thread='vsphere-io-72' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterInstanceService.log:453 - [8282/provisioning/vsphere/instance-adapter] Error in createInstanceAsync [com.vmware.photon.controller.model.adapters.vsphere.vapi.RpcException: Cannot deploy library item 1f19595c-1770-4288-b76c-d44b0766d232 ( com.vmware.vdcs.util.duplicate_name An object of type "ManagedObjectReference: type = VirtualMachine, value = vm-45, serverGuid = ae737187-1a20-4c5c-a1cd-9d7b3f7e1d8c" named "vcdrphoton" already exists.) at com.vmware.photon.controller.model.adapters.vsphere.vapi.VapiClient.throwIfError(VapiClient.java:261) at com.vmware.photon.controller.model.adapters.vsphere.vapi.LibraryClient.deployOvfLibItem(LibraryClient.java:453) at com.vmware.photon.controller.model.adapters.vsphere.InstanceClient.createInstanceFromLibraryItem(InstanceClient.java:4309) at com.vmware.photon.controller.model.adapters.vsphere.VSphereAdapterInstanceService.createInstanceAsync(VSphereAdapterInstanceService.java:498) at com.vmware.photon.controller.model.adapters.vsphere.VSphereAdapterInstanceService.lambda$handleCreateInstanceAsync$11(VSphereAdapterInstanceService.java:351) at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) ] Example Blueprint formatVersion: 1 inputs: {} resources: VirtualMachine: type: Cloud.vSphere.Machine properties: name: testone image: photonfour flavor: small networks: - network: '${resource.Network.id}' assignment: static Network: type: Cloud.vSphere.Network properties: networkType: existing If this issue occurs when using a normal vSphere template , there is no impact. But if your using a content library based template and then there is a risk that the original virtual machine with the same name get's deleted For vCenter 7.x , please upgrade your vRealize Automation version to 8.3 P1 For vCenter 6.x , fix would be part of vRealize Automation 8.4 Patch 1 All we need to ensure is that do not pass an existing virtual machine name when using Content Library templates

We have a workflow in vRO called "Add Firewall Rules" or anything for that matter. In the input form we have set a validation to one of the fields to check if there is any sort of duplication when user enters a value. This is achieved by using a validation action in the background Now when we use this same workflow as a Day-2 action in a Custom Resource, we do not see the same validation pane while we edit request parameters This is not a bug but an expected behavior, we still don't support external validations for day 2 actions

I delivered a training recently about the process which occurs in the background when we upgrade vRealize Automation 8.2 to 8.3 Thought it would be useful for everyone hence writing this blog As a first step, we need to keep the updaterepo ready and downloaded Then took a manual snapshot , once the snapshot task is complete we shall proceed with the upgrade Now that the snapshot is taken we will go ahead with upgrade Before we proceed, we need to perform inventory sync. Unless this is done, PROCEED option will not be available Once , done we need to run precheck Whole precheck pdf attached here When you click on next check the repo url and then submit , upgrade starts Upgrade successfully completes in 1.5 hours. Here's the list of log files which are to be monitored during upgrade BEHIND THE SCENES The first thing which happens when you click on upgrade is that we download manifest. And this is logged under upgrade-noop.log upgrade-noop.log [INFO][2021-03-30 06:20:47][labvra.nukescloud.com] Downloading manifest ... [INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest downloaded successfully. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 10.109.45.33... * TCP_NODELAY set * Connected to reflcm.nukescloud.com (10.109.45.33) port 443 (#0) * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * ignoring certificate verify locations due to disabled peer verification * TLSv1.2 (OUT), TLS header, Certificate Status (22): } [5 bytes data] * TLSv1.2 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * * * * Connection #0 to host reflcm.nukescloud.com left intact [INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Validating manifest ... [INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest signature validated successfully. [INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Repository product ID matched successfully. [INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest validated successfully. [INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Searching for update bootstrap RPM ... [INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Update bootstrap RPM found. [INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Update bootstrap RPM downloaded successfully. * * * * SSL certificate verify result: self signed certificate (18), continuing anyway. } [5 bytes data] > GET /repo/productBinariesRepo/vra/8.3.0/upgrade/update/package-pool/bootstrap-update-8.3.0.15014-1.noarch.rpm HTTP/1.1 > Host: reflcm.nukescloud.com > User-Agent: curl/7.59.0 > Accept: */* * * < { [15833 bytes data] 100 64516 0 64516 0 0 2250k 0 --:--:-- --:--:-- --:--:-- 2250k * Connection #0 to host reflcm.nukescloud.com left intact Then starts the actual upgrade and logged under upgrade-YYYY-MM-DD-HH-MM-SS.log 1. Searches for the bootstrap package and checks if SSH port is open and it's able to connect [INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Retrieving product version... [INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Product version retrieved successfully. [INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel801 ... [INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel801-hf345 ... [INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel810-gap12 ... [INFO][2021-03-30 06:20:57][labvra.nukescloud.com] Applying configuration profile lcm. [INFO][2021-03-30 06:20:57][labvra.nukescloud.com] Configuration profile lcm applied successfully. [INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Retrieving nodes status... [INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes status retrieval succeeded. [INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Processing nodes data... [INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes data processing succeeded. [INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Verifying nodes... [INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes verification completed successfully. [INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Retrieving services status... [INFO][2021-03-30 06:20:59][labvra.nukescloud.com] Services status retrieval succeeded. [INFO][2021-03-30 06:21:00][labvra.nukescloud.com] Processing services data... [INFO][2021-03-30 06:21:18][labvra.nukescloud.com] Services data processing succeeded. [INFO][2021-03-30 06:21:18][labvra.nukescloud.com] Verifying services... [INFO][2021-03-30 06:21:19][labvra.nukescloud.com] Services verification completed successfully. [INFO][2021-03-30 06:21:21][labvra.nukescloud.com] Creating SSH key pairs [INFO][2021-03-30 06:21:21][labvra.nukescloud.com] SSH key pairs created successfully. [INFO][2021-03-30 06:21:21][labvra.nukescloud.com] Configuring SSH on nodes. [INFO][2021-03-30 06:21:22][labvra.nukescloud.com] Service sshd status: active/enabled [INFO][2021-03-30 06:21:22][labvra.nukescloud.com] SSH port 22 is open. [INFO][2021-03-30 06:21:22][labvra.nukescloud.com] SSH configurated on nodes successfully. [INFO][2021-03-30 06:21:22][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/ssh-noop.sh at host: labvra.nukescloud.com [INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/ssh-noop.sh at host: labvra.nukescloud.com [INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Verifying remote nodes are able to connect to one another and to this node. [INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Verification that nodes are able to connect to one another and to this node succeeded. 2. Retrieves product version on all nodes. In this environment we just have one node [INFO][2021-03-30 06:21:24][labvra.nukescloud.com] Retriving product versions on all nodes. [INFO][2021-03-30 06:21:24][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/vami-save-vers.sh prep at host: labvra.nukescloud.com [INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Retrieving product version. [INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product version retrieved successfully. [INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/vami-save-vers.sh prep at host: labvra.nukescloud.com [INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product versions successfully retrieved from all nodes. [INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Checking that product versions match across all nodes. [INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product versions match across all nodes verified. 3. Performs Infrastructure Health [INFO][2021-03-30 06:21:26][labvra.nukescloud.com] Checking appliance nodes infrastructure health [INFO][2021-03-30 06:21:26][labvra.nukescloud.com] Running remote command: /opt/health/run-once.sh health at host: labvra.nukescloud.com [INFO][2021-03-30 06:21:31][labvra.nukescloud.com] Remote command succeeded: /opt/health/run-once.sh health at host: labvra.nukescloud.com [INFO][2021-03-30 06:21:31][labvra.nukescloud.com] Infrastructure health check passed on all appliance nodes. 4. Set's up update repository on all nodes and also verifies if this is accessible across all nodes [INFO][2021-03-30 06:21:33][labvra.nukescloud.com] Setting up update repository on all nodes. [INFO][2021-03-30 06:21:33][labvra.nukescloud.com] Running remote command: vamicli update --repo https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update at host: labvra.nukescloud.com [INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Remote command succeeded: vamicli update --repo https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update at host: labvra.nukescloud.com [INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Update repository set successfully on all nodes. [INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Verifying the access to update repository on all nodes. [INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/vami-config-repo.sh 'local' at host: labvra.nukescloud.com [INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Verifying the access to the update repository. [INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Access to the update repository verified. [INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Verifying availability of new product version in the repository. [INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Availability of new version in the repository verified. [INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/vami-config-repo.sh 'local' at host: labvra.nukescloud.com [INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Update repository is verified successfully on all nodes. 5. When the above vami-config.sh runs under /opt/vmware/var/log/vami/vami.log , the following repository is set 30/03/2021 06:21:34 [INFO] Setting local repository address. url=https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update, username=, password= 30/03/2021 06:21:34 [INFO] Checking available updates. jobid=1 30/03/2021 06:21:34 [INFO] Downloading latest manifest. jobId=1, url=https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update, username=, password= 30/03/2021 06:21:35 [INFO] Using download cache 30/03/2021 06:21:35 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml 30/03/2021 06:21:35 [INFO] Using download cache 30/03/2021 06:21:35 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml.sig 30/03/2021 06:21:35 [INFO] Signature script output: Verified OK 30/03/2021 06:21:35 [INFO] Signature script output: 30/03/2021 06:21:35 [INFO] Manifest signature verification passed 30/03/2021 06:21:35 [INFO] New manifest. installed-version=8.2.0.12946, downloaded-version=8.3.0.15014 6. Starts to shutdown application or services [INFO][2021-03-30 06:21:38][labvra.nukescloud.com] Shutting down application services [INFO][2021-03-30 06:23:58][labvra.nukescloud.com] Application services shut down successfully. [INFO][2021-03-30 06:23:58][labvra.nukescloud.com] Shutting down infrastructure services [INFO][2021-03-30 06:26:27][labvra.nukescloud.com] Infrastructure services shut down successfully. [INFO][2021-03-30 06:26:29][labvra.nukescloud.com] Saving restore points on all nodes. [INFO][2021-03-30 06:26:29][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-save.sh 'local' sys-config sys-config at host: labvra.nukescloud.com [INFO][2021-03-30 06:26:34][labvra.nukescloud.com] Saving local restore points. ========================= [2021-03-30 06:24:03.425+0000] Waiting for deploy healthcheck ========================= ========================= [2021-03-30 06:24:09.389+0000] Waiting for command execution pods ========================= ========================= [2021-03-30 06:24:15.420+0000] Tear down existing deployment ========================= ========================= [2021-03-30 07:24:29.787+0000] Waiting for command execution pods ========================= ========================= [2021-03-30 07:24:32.368+0000] Tear down existing deployment ========================= 7. Saves local restore point for k8's , processes node data and then activates monitor on all nodes [INFO][2021-03-30 06:26:34][labvra.nukescloud.com] Saving local restore point for Kubernetes object prelude-vaconfig. [INFO][2021-03-30 06:26:35][labvra.nukescloud.com] Local restore point for Kubernetes object prelude-vaconfig saved successfully. Flag --export has been deprecated, This flag is deprecated and will be removed in future. [INFO][2021-03-30 06:26:35][labvra.nukescloud.com] Local restore points saved successfully. [INFO][2021-03-30 06:26:35][labvra.nukescloud.com] Restore points on all nodes saved successfully. [INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Retrieving nodes status... [INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Nodes status retrieval succeeded. [INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Processing nodes data... [INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Nodes data processing succeeded. [INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Verifying nodes... [INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Nodes verification completed successfully. [INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Activating local monitors on all nodes. [INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/mon-activate.sh at host: labvra.nukescloud.com [INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Activating upgrade monitor on the node 8. Once the upgrade monitor is activated on the node and cluster nodes are removed successfully , it will proceed for installation. Remember this is a simple install [INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Upgrade monitor activated successfully on the node. [INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/mon-activate.sh at host: labvra.nukescloud.com [INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Local monitors successfully activated on all nodes. [INFO][2021-03-30 06:28:28][labvra.nukescloud.com] Removing nodes from the cluster [INFO][2021-03-30 06:28:28][labvra.nukescloud.com] Running remote command: vracli cluster leave at host: labvra.nukescloud.com [INFO][2021-03-30 06:44:26][labvra.nukescloud.com] Cluster nodes removed successfully 9. The Installation of the updates start [INFO][2021-03-30 06:46:01][labvra.nukescloud.com] Starting installation of updates. [INFO][2021-03-30 06:46:01][labvra.nukescloud.com] Update installation started successfully. [INFO][2021-03-30 06:47:01][labvra.nukescloud.com] VAMI checking upgrade progress [INFO][2021-03-30 06:47:01][labvra.nukescloud.com] VAMI upgrade in progress. [INFO][2021-03-30 06:47:01][labvra.nukescloud.com] VAMI upgrade still in progress. 10. Just a second after it states Starting installation of updates under the upgrade-xx-xx-xx-xx.log , this is the time you would see activity under vami.log 30/03/2021 06:46:02 [INFO] Installing updates. instanceid=VMware:VMware_8.3.0.15014, jobid=2 30/03/2021 06:46:02 [INFO] Installing update. instanceId=VMware:VMware_8.3.0.15014, jobId=2, url=https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update, username=, password= 30/03/2021 06:46:02 [INFO] Downloading and installing update packages 30/03/2021 06:46:02 [INFO] Using download cache 30/03/2021 06:46:02 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml 30/03/2021 06:46:02 [INFO] Using download cache 30/03/2021 06:46:02 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml.sig 30/03/2021 06:46:02 [INFO] Signature script output: Verified OK 30/03/2021 06:46:02 [INFO] Signature script output: 30/03/2021 06:46:02 [INFO] Manifest signature verification passed 30/03/2021 06:46:02 [INFO] Creating /opt/vmware/var/lib/vami/update/data/update_progress.json 30/03/2021 06:46:02 [INFO] Downloading the following packages for update version 8.3.0.15014 30/03/2021 06:46:02 [INFO] package NEW PACKAGE : VMware-Log-Insight-Agent noarch (none) 8.3.0 17489324 /package-pool/VMware-Log-Insight-Agent-8.3.0-17489324.noarch.rpm rpm 9457625 e744601203ec01d0d637453260324b7b35a872ce 30/03/2021 06:46:02 [INFO] package UPDATE VERSION: abx-common noarch (none) 8.3.0.15014 1 /package-pool/abx-common-8.3.0.15014-1.noarch.rpm rpm 43780508 c134c5476bd805293bdfaf45dc9b6bd9e06bde63 30/03/2021 06:46:02 [INFO] package UPDATE VERSION: bindutils x86_64 (none) 9.16.6 1.ph2 /package-pool/bindutils-9.16.6-1.ph2.x86_64.rpm rpm 3584466 67d101f6718a81a2abc3dfd9b54381cac6948b60 30/03/2021 06:46:02 [INFO] package UPDATE VERSION: bootstrap-prelude noarch (none) 8.3.0.15014 1 /package-pool/bootstrap-prelude-8.3.0.15014-1.noarch.rpm rpm 2629839 14c2906620ddca6690e13867f1e385851b9e06d8 30/03/2021 06:46:02 [INFO] package UPDATE VERSION: cifs-utils x86_64 (none) 6.7 2.ph2 /package-pool/cifs-utils-6.7-2.ph2.x86_64.rpm rpm 38568 2eea29de217fc837792d0c8db9848d240c3b18d5 30/03/2021 06:46:02 [INFO] package NEW PACKAGE : config-xl noarch (none) 8.3.0.15014 1 /package-pool/config-xl-8.3.0.15014-1.noarch.rpm rpm 14255 c32254fd9e2c1074ea1b5e33e9c72af5350c27e4 30/03/2021 06:46:02 [INFO] package UPDATE VERSION: containerd x86_64 (none) 1.2.14 1.ph2 /package-pool/containerd-1.2.14-1.ph2.x86_64.rpm rpm 24221328 227d22a05417f4286e0247ca87ab415f0b095231 * * * 30/03/2021 06:48:34 [INFO] package UPDATE VERSION: vro-migrate noarch (none) 8.3.0.1611732871 17522798 /package-pool/vro-migrate-8.3.0.1611732871-17522798.noarch.rpm rpm 4984 eb517af86432ea5b752627335696eec3509a052c 30/03/2021 06:48:34 [INFO] Install packages set: '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/VMware-Log-Insight-Agent-8.3.0-17489324.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/abx-common-8.3.0.15014-1.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/bindutils-9.16.6-1.ph2.x86_64.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/bootstrap-prelude-8.3.0.15014-1.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/cifs-utils-6.7-2.ph2.x86_64.rpm' * * vmware-studio-appliance-config_3.0.0.7-210201212046.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/vmware-studio-init_3.0.0.7-210201212046.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/vro-migrate-8.3.0.1611732871-17522798.noarch.rpm' 30/03/2021 06:48:34 [INFO] Reboot required when installing linux package 30/03/2021 06:48:34 [INFO] Created reboot required file 30/03/2021 06:48:34 [INFO] Update 8.3.0.15014 manifest is set to be installed 30/03/2021 06:48:34 [INFO] Using update post-install script 30/03/2021 06:48:34 [INFO] Running updatecli to install updates. command={ mkdir -p /usr/share/update-notifier ; ln -s /opt/vmware/share/vami/vami_notify_reboot_required /usr/share/update-notifier/notify-reboot-required ; /opt/vmware/share/vami/update/updatecli '/opt/vmware/var/lib/vami/update/data/job/2' '8.2.0.12946' '8.3.0.15014' ; mv /opt/vmware/var/lib/vami/update/data/update_progress.json /opt/vmware/var/lib/vami/update/data/.update_progress.json ; } >> /opt/vmware/var/log/vami/updatecli.log 2>&1 & 30/03/2021 06:48:36 [INFO] Installation running in the background 11. Once the installation starts in the background , this is the time we need to monitor /var/log/bootstrap/postupdate.log The ones highlighted in the RED are the series of scripts which run in the background. + log 'Main bootstrap postupdate started' ++ date '+%Y-%m-%d %H:%M:%S' + echo '2021-03-30 06:51:46 Main bootstrap postupdate started' 2021-03-30 06:51:46 Main bootstrap postupdate started + for script in "${bootstrap_dir}"/* + echo + log '/etc/bootstrap/postupdate.d/00-clear-services.sh starting...' + log '/etc/bootstrap/postupdate.d/00-disable-set-hostname.sh starting...' + log '/etc/bootstrap/postupdate.d/00-fix-vmtoolsd-deps.sh starting...' + log '/etc/bootstrap/postupdate.d/01-10-reset-k8s starting...' + log '/etc/bootstrap/postupdate.d/02-07-prune-images.sh starting...' * * untagged: photonos3_private:latest deleted: sha256:30ded4d67031ea69c5792250dcd4c466511d940ff92f13cd6cc945c4456596ed untagged: relocation-service:ed72098 untagged: relocation-service_private:latest deleted: sha256:840c0a6cb51902b0bb1d81c7ebb3b7ae233e1666dbe387f39a24d7066ad1781a deleted: sha256:3e737404723de6f8e2a6d8ad85b8cad2b9206e76b40b536766ba6a164a6d24ed deleted: sha256:ea1c6ad6204eb1aa5ffccc44a3890014b52a39bc9ed64de34af88195670196e5 deleted: sha256:3a90641efc2b2704dcfa4a8777a4c8cce7150b683811e315773ce35cb83d2e92 deleted: sha256:fa46acfe7078d8404ebda3e229ae4058e68aa579512454194f3bc6121be0818e untagged: rabbitmq_private:3.8.5-photon_9ff5c8078eac untagged: rabbitmq_private:latest * * + log '/etc/bootstrap/postupdate.d/02-10-load-images.sh starting...' + log '/etc/bootstrap/postupdate.d/02-disable-var-log-cron.sh starting...' + log '/etc/bootstrap/postupdate.d/03-10-setup-k8s.sh starting...' + log '/etc/bootstrap/postupdate.d/10-disable-tzselect.sh starting...' + log '/etc/bootstrap/postupdate.d/20-relocate-db.sh starting...' + log '/etc/bootstrap/postupdate.d/90-disable-default-services.sh starting...' + log '/etc/bootstrap/postupdate.d/91-00-remove-config-xl.sh starting...' + log '/etc/bootstrap/postupdate.d/99-10-handover-prelude starting...' * * # Wait for Kubernetes and other prerequisites vracli status first-boot -w 1800 && \ /opt/scripts/upgrade/vami-set-update.sh "success" || \ /opt/scripts/upgrade/vami-set-update.sh "error" rm -f /etc/bootstrap/everyboot.d/zz-zz-resume-upgrade.sh' + chmod a+x /etc/bootstrap/everyboot.d/zz-zz-resume-upgrade.sh + export -f vami_reboot_background + echo 'Scheduling a post-upgrade reboot...' Scheduling a post-upgrade reboot... + echo 'Post-upgrade reboot scheduled' Post-upgrade reboot scheduled + exit 0 + nohup bash -c vami_reboot_background + log '/etc/bootstrap/postupdate.d/99-10-handover-prelude done, succeeded.' ++ date '+%Y-%m-%d %H:%M:%S' + echo '2021-03-30 07:14:06 /etc/bootstrap/postupdate.d/99-10-handover-prelude done, succeeded.' * * + log '/etc/bootstrap/postupdate.d/README is not executable.' ++ date '+%Y-%m-%d %H:%M:%S' + echo '2021-03-30 07:14:06 /etc/bootstrap/postupdate.d/README is not executable.' 2021-03-30 07:14:06 /etc/bootstrap/postupdate.d/README is not executable. + log 'main bootstrap postupdate done' ++ date '+%Y-%m-%d %H:%M:%S' + echo '2021-03-30 07:14:06 main bootstrap postupdate done' 2021-03-30 07:14:06 main bootstrap postupdate done + '[' postupdate == firstboot ']' + exit 0 Rebooting the appliance after VAMI upgrade ... 12. Finishes the postupdate and the appliance is rebooted 13. As we can see above the appliance was rebooted sometime after 2021-03-30 07:14:06 , under upgrade-2021-03-30-06-20-48.log once the appliance is back you would see the status change [INFO][2021-03-30 07:14:01][labvra.nukescloud.com] VAMI checking upgrade progress [INFO][2021-03-30 07:14:01][labvra.nukescloud.com] VAMI upgrade in progress. [INFO][2021-03-30 07:14:01][labvra.nukescloud.com] VAMI upgrade still in progress. [INFO][2021-03-30 07:14:06][labvra.nukescloud.com] Waiting for VAMI to exit ... [INFO][2021-03-30 07:14:36][labvra.nukescloud.com] Verifying VAMI overall upgrade result ... [INFO][2021-03-30 07:14:36][labvra.nukescloud.com] VAMI upgrade in progress. [INFO][2021-03-30 07:17:01][labvra.nukescloud.com] VAMI checking upgrade progress [INFO][2021-03-30 07:17:01][labvra.nukescloud.com] VAMI upgrade in progress. * * * [INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI checking upgrade progress [INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI upgrade completed succesfully. [INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI upgrade completed successfully. [INFO][2021-03-30 07:23:01][labvra.nukescloud.com] Saving artifacts metadata [INFO][2021-03-30 07:23:02][labvra.nukescloud.com] Artifacts metadata saved. [INFO][2021-03-30 07:24:01][labvra.nukescloud.com] Resolving post-upgrade controller. [INFO][2021-03-30 07:24:01][labvra.nukescloud.com] This node is elected for post-upgrade controller. [INFO][2021-03-30 07:24:03][labvra.nukescloud.com] Resolving post-upgrade nodes quorum... [INFO][2021-03-30 07:24:04][labvra.nukescloud.com] Adding nodes to the cluster. [INFO][2021-03-30 07:24:04][labvra.nukescloud.com] Cluster nodes added successfully. [INFO][2021-03-30 07:24:05][labvra.nukescloud.com] Restoring from restore points on all nodes. [INFO][2021-03-30 07:24:05][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-restore.sh 'local' sys-config sys-config at host: labvra.nukescloud.com [INFO][2021-03-30 07:24:11][labvra.nukescloud.com] Restoring all restore points. 14. Restore point restoration starts and then application and infrastructure services start as well [INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI checking upgrade progress [INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI upgrade completed succesfully. [INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI upgrade completed successfully. [INFO][2021-03-30 07:23:01][labvra.nukescloud.com] Saving artifacts metadata [INFO][2021-03-30 07:23:02][labvra.nukescloud.com] Artifacts metadata saved. [INFO][2021-03-30 07:24:01][labvra.nukescloud.com] Resolving post-upgrade controller. [INFO][2021-03-30 07:24:01][labvra.nukescloud.com] This node is elected for post-upgrade controller. [INFO][2021-03-30 07:24:03][labvra.nukescloud.com] Resolving post-upgrade nodes quorum... [INFO][2021-03-30 07:24:04][labvra.nukescloud.com] Adding nodes to the cluster. [INFO][2021-03-30 07:24:04][labvra.nukescloud.com] Cluster nodes added successfully. [INFO][2021-03-30 07:24:05][labvra.nukescloud.com] Restoring from restore points on all nodes. [INFO][2021-03-30 07:24:05][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-restore.sh 'local' sys-config sys-config at host: labvra.nukescloud.com [INFO][2021-03-30 07:24:11][labvra.nukescloud.com] Restoring all restore points. 15. At this point in time one has to check /var/log/deploy.log to see if the services are coming back as expected ========================= [2021-03-30 07:26:36.605+0000] Creating kubernetes namespaces ========================= ========================= [2021-03-30 07:26:39.435+0000] Applying ingress certificate ========================= ========================= [2021-03-30 07:26:50.589+0000] Updating etcd configuration to include https_proxy if such exists ========================= ========================= [2021-03-30 07:26:57.836+0000] Deploying infrastructure services ========================= ========================= [2021-03-30 07:28:43.477+0000] Clearing liquibase locks ========================= ========================= [2021-03-30 07:28:56.296+0000] DB upgrade schema ========================= ========================= [2021-03-30 07:32:08.619+0000] Populating initial identity-service data ========================= ========================= [2021-03-30 07:32:38.488+0000] Deploying application services ========================= ========================= [2021-03-30 07:42:06.787+0000] Deploying application UI ========================= ========================= [2021-03-30 07:43:28.519+0000] Setting feature UI toggles to Provisioning service ========================= ========================= [2021-03-30 07:43:54.731+0000] Registering embedded vRO ========================= ========================= [2021-03-30 07:43:58.698+0000] Registering embedded ABX on-prem endpoint in organizations ========================= 16. That marks the application and infrastructure start up procedure and then there would be a verification , this procedure is documented under 2 logs Reference: Verifies if all kubernetes related services are up and running upgrade-k8s-2021-03-30-06-20-48.log Reference: Copy of deploy.log upgrade-deploy-2021-03-30-06-20-48.log 17. As the verification is now complete, it will now works towards the last few steps [INFO][2021-03-30 07:44:32][labvra.nukescloud.com] Infrastrucre and application services deployed successfully. [INFO][2021-03-30 07:44:33][labvra.nukescloud.com] Retrieving services status... [INFO][2021-03-30 07:44:33][labvra.nukescloud.com] Services status retrieval succeeded. [INFO][2021-03-30 07:44:33][labvra.nukescloud.com] Processing services data... [INFO][2021-03-30 07:44:54][labvra.nukescloud.com] Services data processing succeeded. [INFO][2021-03-30 07:44:54][labvra.nukescloud.com] Verifying services... [INFO][2021-03-30 07:44:56][labvra.nukescloud.com] Services verification completed successfully. [INFO][2021-03-30 07:44:58][labvra.nukescloud.com] Cleaning up restore point on all nodes. [INFO][2021-03-30 07:44:58][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-clean.sh sys-config at host: labvra.nukescloud.com [INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Cleaning up restore point /data/restorepoint/sys-config . [INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Restore point /data/restorepoint/sys-config cleaned up successfully. [INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Restore point cleaned up on all nodes. [INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Cleaning up restore point on all nodes. [INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-clean.sh artifacts-lastworking at host: labvra.nukescloud.com [INFO][2021-03-30 07:45:08][labvra.nukescloud.com] Cleaning up restore point /data/restorepoint/artifacts-lastworking . [INFO][2021-03-30 07:45:08][labvra.nukescloud.com] Restore point /data/restorepoint/artifacts-lastworking cleaned up successfully. [INFO][2021-03-30 07:45:08][labvra.nukescloud.com] Restore point cleaned up on all nodes. [INFO][2021-03-30 07:45:08][labvra.nukescloud.com] Cleaning up restore point on all nodes. [INFO][2021-03-30 07:45:09][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-clean.sh live-data at host: labvra.nukescloud.com [INFO][2021-03-30 07:45:14][labvra.nukescloud.com] Cleaning up restore point /data/restorepoint/live-data . [INFO][2021-03-30 07:45:14][labvra.nukescloud.com] Restore point /data/restorepoint/live-data cleaned up successfully. [INFO][2021-03-30 07:45:14][labvra.nukescloud.com] Restore point cleaned up on all nodes. [INFO][2021-03-30 07:45:15][labvra.nukescloud.com] Reverting SSH configuration on nodes. [INFO][2021-03-30 07:45:16][labvra.nukescloud.com] SSH configuration reverted on nodes successfully. [INFO][2021-03-30 07:45:24][labvra.nukescloud.com] Cleaning up upgrade runtime state. [INFO][2021-03-30 07:45:25][labvra.nukescloud.com] Archiving upgrade runtime data. [INFO][2021-03-30 07:45:26][labvra.nukescloud.com] Upgrade runtime data archived successfully. [INFO][2021-03-30 07:45:26][labvra.nukescloud.com] Clearing upgrade runtime directory. [INFO][2021-03-30 07:45:26][labvra.nukescloud.com] Upgrade runtime directory cleared successfully. [INFO][2021-03-30 07:45:26][labvra.nukescloud.com] Upgrade runtime clean up completed. This concludes vRealize Automation upgrade As said earlier following logs are the ones which we need to monitor

If there is AD integration inside vRA 8.3 and then you attempt to deploy a machine to an OU relative path that is more than 50 characters long then your deployments would fail. This is a bug that is fixed in vRealize Automation 8.4 GA

We've recently seen issues while trying update resource elements in vRealize Orchestrator 7.x Some of the exceptions you would see when the vRO workflow is executed is as below. Below excerpts are from scripting.log 2021-03-09 11:42:35.070+0000 [WorkflowExecutorPool-Thread-2] ERROR {|__SYSTEM|administrator@vsphere.local:updateResourceEleme:f896e1b8-4b5e-484e-b5d1-db04390a1bcc:token=e98c4043-52d4-4375-96b4-c699da88f34d} [ResourceElementManagementServiceImpl] Failed to commit 'Resource saved.'. Reason: null java.lang.NullPointerException Cannot set content from mime attachment : ch.dunes.util.DunesServerException: org.springframework.orm.jpa.JpaSystemException: Transaction was marked for rollback only; cannot commit; nested exception is org.hibernate.TransactionException: Transaction was marked for rollback only; cannot commit One of the main reasons why this is not working is because the jgit garbage collection is not working as expected in the background 2021-03-09 11:42:23.354+0000 [http-nio-127.0.0.1-8280-exec-6] ERROR {} [ContentVersionManagementServiceImpl] Failed to disable auto push: org.eclipse.jgit.api.errors.TransportException: /storage/db/vcodata/git/__SYSTEM.git: internal server error java.lang.RuntimeException: org.eclipse.jgit.api.errors.TransportException: /storage/db/vcodata/git/__SYSTEM.git: internal server error at com.vmware.o11n.git.StagingRepositoryService.checkout(StagingRepositoryService.java:226) at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl$ReusableStaginRepository.(ContentVersionManagementServiceImpl.java:495) at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl$StagingRepositoryPooledObjectFactory.create(ContentVersionManagementServiceImpl.java:455) at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl$StagingRepositoryPooledObjectFactory.create(ContentVersionManagementServiceImpl.java:445) at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:58) 2021-03-10 03:00:00.000+0000 [vcoSystemTaskScheduler-3] DEBUG {} [ContentVersionManagementServiceImpl] Starting garbage collection in all repositories. 2021-03-10 03:00:05.931+0000 [vcoSystemTaskScheduler-3] INFO {} [BareRepositoryService] Started garbage collection for __SYSTEM.git 2021-03-10 03:00:06.352+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService] Failed to run garbage collection for __SYSTEM.git. Reason: 2021-03-10 03:00:06.352+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService] Garbage collection failed. org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed. at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:229) at com.vmware.o11n.git.BareRepositoryService.runGc(BareRepositoryService.java:109) at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl.lambda$compact$3(ContentVersionManagementServiceImpl.java:260) at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl.wrapInTransaction(ContentVersionManagementServiceImpl.java:111) at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl.compact(ContentVersionManagementServiceImpl.java:256) at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl.lambda$compactAll$2(ContentVersionManagementServiceImpl.java:251) at java.util.ArrayList.forEach(ArrayList.java:1257) Garbage Collection runs everyday at 3 AM. If you notice garbage collection failures as shown below then execute the steps mentioned in the remediation section 2021-03-06 03:00:12.167+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService] Garbage collection failed. org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed. 2021-03-07 03:00:00.545+0000 [vcoSystemTaskScheduler-4] ERROR {} [BareRepositoryService] Garbage collection failed. org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed. 2021-03-08 03:00:03.969+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService] Garbage collection failed. org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed. 2021-03-09 03:00:00.469+0000 [vcoSystemTaskScheduler-4] ERROR {} [BareRepositoryService] Garbage collection failed. org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed. 2021-03-10 03:00:06.352+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService] Garbage collection failed. org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed. To resolve this issue follow these steps Download JGit cli client , below command would work if internet is available, else execute this command on node which has internet access and copy the file into vRA node curl -k --output jgit https://repo.eclipse.org/content/groups/releases//org/eclipse/jgit/org.eclipse.jgit.pgm/5.4.0.201906121030-r/org.eclipse.jgit.pgm-5.4.0.201906121030-r.sh 2. Move jgit to /usr/bin, set it to executable and change its owner to vco and make it executable mv jgit /usr/bin/ chown vco:vco /usr/bin/jgit chmod +x /usr/bin/jgit 3. Remove the HEAD ref of the MASTER mv /storage/db/vcodata/git/__SYSTEM.git/refs/heads/master /var/lib/vco/master_ref.backup 4. Check where the HEAD should be.Get the commit ref sha from the first line. ex: 01bb5f58716dc12016b4fd8798c7fa8c91c76bf3 cd /storage/db/vcodata/git/__SYSTEM.git jgit log 5. Login as vco su - vco 6. Create the master ref echo '01bb5f58716dc12016b4fd8798c7fa8c91c76bf3' > /storage/db/vcodata/git/__SYSTEM.git/refs/heads/master 7. Run jgit garbage collection cd /storage/db/vcodata/git/__SYSTEM.git jgit gc 8. Login as root again and make sure everything in the git folder is owned by vco:vco exit cd /storage/db/vcodata/git/__SYSTEM.git chown -R vco:vco 9. Restart the vRO server service vco-server restart This should fix any garbage collection issues