Search Results

Usecase Explaination of the steps taken if one wants to move VMware Aria Operations which is integrated with vIDM or Globalenvironment from one Suite Lifecycle to another Note: VMware Aria Suite Lifecycle on both Source and Destination should be same , include policy Environment Here are the details of the environment we have on source and destination. The domain names taken here are an example and does not represent any organization Type Source Destination VMware Aria Suite Lifecycle devvasl.cap.org vasl.cap.org VMware Aria Operations Master ​lvrops1.cap.org lvrops1.cap.org VMware Aria Operations Replica lvrops2.cap.org lvrops2.cap.org VMware Aria Operations Cloud Proxy lvrops3.cap.org lvrops3.cap.org VMware Identity Manager vidm.cap.org vidmlb.cap.org Product UI The product has 2 auth sources Local Users and vIDMAuthSource You can see when we select vIDMAuthSource , it does redirect it to vIDM and we can login using configadmin which is my local vIDM based auth account Auth Source is configured to vidm as shown below Procedure Phase 1 : Removal from Source VMware Aria Suite Lifecycle Here's the Operations instance which i'd like to move it to a different Suite Lifecycle We may see that the Operations instance is integrated with VMware Identity Manager On globalenvironment or VMware Identity Manager we can see this environment as a reference In order to move this Operations instance to a different VMware Aria Suite Lifecycle instance, i'll have to remove this from Suite Lifecycle's inventory Remember, I am deleting environment because this is the only product in the environment. If i have multiple products in an environment as shown in the next pane , then i would only delete that specific product Delete environment will present me with the following screen. I shall select the first option where it removes the environment / product from VMware Aria Suite Lifecycle and not from vCenter. If we select "Delete associated VMs from vCenter" it would delete all associated virtual machines/appliances from the vCenter causing an outage. So let's select the first option to delete the environment from just VMware Aria Suite Lifecycle and submit the request Environment is deleted from VMware Aria Suite Lifecycle The request is now complete The references in globalenvironment is removed too This does not mean it will delete the integration it has with the vIDM whcih it had before that shall still remain Before moving to the destination Suite Lifecycle, i shall download the certificate being consumed by Operations in the Source Suite Lifecycle and keep it aside to be imported into the destination Suite Lifecycle Using this information we shall import this key into the destination Suite Lifecycle before we import the Operations product into it. So that when the product is imported , the certificate mapping is perfect. Phase-2 : Importing to Destination VMware Aria Suite Lifecycle Let's import the certificate into Suite Lifecycle first. This is the Operations certificate we downloaded just few steps before Al i did was to point to the downloaded pem file and it automatically detects the cert and it's private key to import Once we click on import the certificate is now imported globalenvironment or VMware Identity Manager on the destination is a distributed node Let's import the product into destination Suite Lifecycle Click on "Create Environment" to start the import process Select VMware Aria Operations as a product to import and then click next Because i am importing an existing product , there not much of information i need to enter in the next pane . All i need to enter is the master node's fqdn , select passwords and then choose the vCenter it's located on Click on next to review the summary and then submit the request Request is submitted Import request is now complete and now we can see the new environment in the destination Suite Lifecycle Remember the certificate we imported just before the product import. It is now marked as used it is mapped to the imported product If you clearly observe , the imported Operations instance is still pointing to the source VMware Identity Manager If we clearly observe the vIDM integration is set to false as it is not integrated with the vIDM in this Suite Lifecycle In the next phase we will add the new vIDM as an auth source and then remove existing vIDM auth source This should be done from VMware Aria Operations UI Phase 3: Replacing Auth Source in VMware Aria Operations Before making any changes take a snapshot Login into VMware Aria Operations as admin Browse to Administration and Authentication sources As you can see it's currently pointing to source vIDM Make sure the roles and the groups to which the roles are given are taken down. So that the same groups can be readded again Delete the authentication source Go back to Suite Lifecycle and perform an inventory sync As one can see the vIDM information is now gone Now let's go and add a new auth source in Operations as shown below. This will be pointing to the new globalenvironment on the destination Suite Lifecycle Enter appropriate information and then click on test Accept Certificate , once test connection is successful. Click on OK to save the config Now the auth is pointing to the new vIDM When i logout and check for the vIDMAuthSource now , it points to the new vIDM I'd now map the group back in Operations and then give the same role or access to the user Phase 4: Inventory Sync to reflect appropriate properties in Suite Lifecycle Get back to Suite Lifecycle and perform inventory sync. After performing inventory sync the properties are now updated The reference is shown as well This concludes the blog

Last night we had ADFS configured in my lab on vIDM ( globalenvironment ) and it was working as expected with vRealize Automation Over a period of time , we did notice that globalenvironment on vRSLCM was not reporting health status This was weird as vIDM was healthy and authentication was going through as expected. When i browsed to Identity and Tenant Management pane , there was a message stating VMware Identity Manager is not available at the moment. There are some requests which are in progress There was something wrong with the new ADFS configuration as the permissions seemed to me messed up as well vIDM Inventory Sync never went past the initial stage *** Request Creation *** 2022-06-15 01:43:32.459 INFO [http-nio-8080-exec-7] c.v.v.l.l.u.RequestSubmissionUtil - -- ++++++++++++++++++ Creating request to Request_Service :::>>> { "vmid" : "7d637ba2-5294-4e92-836a-2610a509fa03", "transactionId" : null, "tenant" : "default", "requestName" : "productinventorysync", "requestReason" : "VIDM in Environment globalenvironment - Product Inventory Sync", "requestType" : "PRODUCT_INVENTORY_SYNC", "requestSource" : "globalenvironment", "requestSourceType" : "user", "inputMap" : { "environmentId" : "globalenvironment", "productId" : "vidm", "tenantId" : "" }, "outputMap" : { }, "state" : "CREATED", "executionId" : null, "executionPath" : null, "executionStatus" : null, "errorCause" : null, "resultSet" : null, "isCancelEnabled" : null, "lastUpdatedOn" : 1655257412458, "createdBy" : null } *** Request Response *** 2022-06-15 01:43:32.466 INFO [http-nio-8080-exec-7] c.v.v.l.l.u.RequestSubmissionUtil - -- Generic Request Response : { "requestId" : "7d637ba2-5294-4e92-836a-2610a509fa03" } 2022-06-15 01:43:34.001 INFO [scheduling-1] c.v.v.l.r.c.RequestProcessor - -- Number of request to be processed : 1 2022-06-15 01:43:34.022 INFO [scheduling-1] c.v.v.l.r.c.p.ProductInventorySyncPlanner - -- Creating spec for inventory sync for product : vidm in environment : globalenvironment 2022-06-15 01:43:34.025 INFO [scheduling-1] c.v.v.l.r.u.InfrastructurePropertiesHelper - -- VCF properties: { "vcfEnabled" : false, "sddcManagerDetails" : [ ] } 2022-06-15 01:43:34.027 INFO [scheduling-1] c.v.v.l.r.c.p.ProductInventorySyncPlanner - -- Found product with id vidm 2022-06-15 01:43:34.038 INFO [scheduling-1] c.v.v.l.r.c.p.CreateEnvironmentPlanner - -- Not a clustered vIDM, fetching the hostname from primary node. *** Suit Request is generated and then set to IN_PROGRESS *** 2022-06-15 01:43:34.785 INFO [scheduling-1] c.v.v.l.r.c.RequestProcessor - -- Processing request with ID : 7d637ba2-5294-4e92-836a-2610a509fa03 with request type PRODUCT_INVENTORY_SYNC with request state INPROGRESS. Browsing to "Authentication Provider " under settings would return an exception stating Failed to fetch Complete Authentication Provider details That networksettings API returns 400 , response was "No Settings " The problem described here could be due to engine is not picking up the request to due to already processing requests (overloaded) or genuine unprocessed but stuck requests. So logged into database and executed a query which would return the number of requests which are stuck in "IN_PROGRESS" state Login into vRSLCM database /opt/vmware/vpostgres/11/bin/psql -U postgres -d vrlcm Check IN_PROGRESS from vm_engine_event table select currentState,status from vm_engine_event where status='IN_PROGRESS'; We do see 50 of them. If this number is greater than or equal to 50 ( > vRSLCM 8.4 ) , then we need to follow below steps to clear this data and bring the system to functional state Remediation Before even proceeding further , please take a snapshot of vRSLCM Appliance. This is mandatory Here's the plan which would be implemented in order to fix this issue *** select count(*) queries. These queries will help you in identifying the number of records per table in a defined state *** select count(*) from vm_rs_request where requestname= 'lcmgenricsetting'; select count(*) from vm_engine_execution_request where enginestatus= 'INITIATED'; select count(*) from vm_engine_statemachine_instance where status= 'CREATED'; select count(*) from vm_engine_event where status= 'IN_PROGRESS'; *** execute these delete queries. This would clear up all stuck queries *** delete from vm_rs_request where requestname= 'lcmgenricsetting'; delete from vm_engine_execution_request where enginestatus= 'INITIATED'; delete from vm_engine_statemachine_instance where status= 'CREATED'; delete from vm_engine_event where status= 'IN_PROGRESS'; *** Perform a full VACUUM of the postgres database *** VACUUM FULL verbose analyze vm_rs_request; *** exit database *** \q *** Restart vRSLCM Service *** systemctl restart vrlcm-server Let's implement this plan at hand in my environment and see if it helps *** select count(*) queries. These queries will help you in identifying the number of records per table in a defined state *** vrlcm=# select count(*) from vm_engine_event where status='IN_PROGRESS'; count ------- 50 ^ vrlcm=# select count(*) from vm_engine_statemachine_instance where status= 'CREATED'; count ------- 1580 vrlcm=# select count(*) from vm_engine_execution_request where enginestatus = 'INITIATED'; count ------- 1616 vrlcm=# select count(*) from vm_rs_request where requestname = 'lcmgenericsetting'; count ------- 0 *** execute these delete queries. This would clear up all stuck queries *** vrlcm=# delete from vm_engine_event where status='IN_PROGRESS'; DELETE 50 vrlcm=# delete from vm_engine_statemachine_instance where status= 'CREATED'; DELETE 1580 vrlcm=# delete from vm_engine_execution_request where enginestatus = 'INITIATED'; DELETE 1616 *** Note , Since select count(*) from vm_rs_request where requestname = 'lcmgenericsetting'; returned 0 records , i am not executing delete statement **** *** Perform a full VACUUM of the postgres database *** vrlcm=# VACUUM FULL verbose analyze vm_rs_request; INFO: vacuuming "public.vm_rs_request" INFO: "vm_rs_request": found 48 removable, 1565 nonremovable row versions in 379 pages DETAIL: 0 dead row versions cannot be removed yet. CPU: user: 0.06 s, system: 0.02 s, elapsed: 0.46 s. INFO: analyzing "public.vm_rs_request" INFO: "vm_rs_request": scanned 339 of 339 pages, containing 1565 live rows and 0 dead rows; 1565 rows in sample, 1565 estimated total rows VACUUM *** exit database *** \q *** Restart vRSLCM Service *** systemctl restart vrlcm-server After this plan implementation , it would take 2 minutes for the server to startup and come back to it's functional state again Then , if we go ahead and check the Authentication setting information globalenvironment or the product inventory sync completed too To conclude , there was something which went wrong in our last nights tests in lab which triggered these queued requests in vRSLCM. Happy to get the environment back in functional state. Ensure a snapshot is taken before making any changes. Can't stress how important is this if things does not work out.

Goal Deploy a clustered VMware Aria Operations for Logs (formerly vRealize Log Insight) Environment Node Role lilb.cap.org Load Balancer FQDN lione.cap.org Operations for Logs Master Node litwo.cap.org Operations for Logs Worker Node lithree.cap.org Operations for Logs Worker Node Logs to monitor VMware Aria Suite Lifecycle ​/var/log/vrlcm/vmware_vrlcm.log VMware Aria Operations for Logs /storage/var/loginsight/runtime.log /storage/var/loginsight/api_audit.log Deployment Process Create an environment in VMware Aria Suite Lifecycle which would host VMware Aria Operations for Logs product Accept the EULA Selected appropriate certificate Provide infrastructure details Enter details for the product to be deployed. We are integrating it with vIDM and also suppluing loadbalancer VIP for a distributed operations for logs configuration. Now run a precheck . Remember, all prechecks should be successful. Review the summary Click on submit to move forward with the product deployment There are 9 stages for a 3 node Operations for Logs deployment Different stages are Stage Description Timing 1 Check Product Prerequisites Start Start pre-requisites check Check environment pre-requisites Finish pre-requisites check 463ms 2 VMware Aria Operations for Logs Master Start Start VMware Aria Operations for Logs Deployment Validate Deployment Inputs Deploy Ovf Upgrade VM compatibility Power On VM Guest Tools Status Check Hostname Ip status check VMware Aria Operations for Logs Post Install check Start new Operations for Logs Install VMware Aria Operations for Logs service status check Extend hard disk for VMware Aria Operations for Logs Configure VMware Aria Operations for Logs Telemetry Final 18m 46s 3 VMware Aria Operations for Logs Worker Node Deployment Start Start VMware Aria Operations for Logs Deployment Validate Deployment Inputs Deploy Ovf Upgrade VM compatibility Power On VM Guest Tools Status Check Hostname Ip status check VMware Aria Operations for Logs Post Install check Start new Operations for Logs Install VMware Aria Operations for Logs service status check Extend hard disk for VMware Aria Operations for Logs VMware Aria Operations for Logs Scale Out Check Final 5m 55s 4 License VMware Aria Operations for Logs Start Start VMware Aria Operations for Logs License VMware Aria Operations for Logs Service Status VMware Aria Operations for Logs Init License License VMware Aria Operations for Logs Configure VMware Aria Operations for Logs Integrated Load Balancer Final 10m 11s 5 Configure VMware Aria Operations for Logs Cluster Start Configure VMware Aria Operations for Logs Cluster Add VMware Aria Operations for Logs to Node to Master Node VMware Aria Operations for Logs to Approve Worker node from Master Node Prepare Nodes Configure VMware Aria Operations for Logs vIDM Configure VMware Aria Operations for Logs vIDM App Catalog vIDM create weblink app Save vIDM catalog app id for VMware Aria Operations for Logs Inventory Add vIDM user Configure VMware Aria Operations for Logs NTP VMware Aria Operations for Logs Certificate Init VMware Aria Operations for Logs Certificate Import Final 4m 18s 6 VMware Aria Operations for Logs enable english usage Start VMware Aria Operations for Logs Service Status Check Configure VMware Operations for Logs for English Usage Finish 2m 28s 7 vCenter Affiinity for VMware Aria Operations for Logs Start vCenter Affinity Init vCenter Affinity Final 11s 8 Update Environment Details 213ms 9 Schedule Notifications 1s Total deployment time taken is around 52 minutes and 2 seconds Behind the scenes VMware Aria Suite Lifecycle perspective Reference: /var/log/vrlcm/vmware_vrlcm.log Checking the certificate replacement task /** Certificate Init Task Starts here **/ 2023-07-24 02:35:55.019 INFO [pool-3-thread-6] c.v.v.l.p.v.VrliCertificateInitTask - -- Starting :: VMware Aria Operations for Logs Install Certificate Init Task.... 2023-07-24 02:35:55.019 INFO [pool-3-thread-6] c.v.v.l.p.v.VrliCertificateInitTask - -- certificate : locker:certificate:f33c54a7-efd1-4609-9b4b-8d1010ce368c:opslogs 2023-07-24 02:35:55.019 INFO [pool-3-thread-6] c.v.v.l.p.a.s.Task - -- Injecting Edge :: OnVrliCertificateInitTaskCompleted /** In the product spec, we have the locker object information for certificate, vidm information and opslogs information **/ 2023-07-24 02:35:55.020 INFO [pool-3-thread-6] c.v.v.l.p.a.s.Task - -- ======================================== { "productSpec" : { "object" : null, "originalObject" : null }, "componentSpec" : { "object" : { "component" : { "symbolicName" : "vrlicluster", "type" : null, "componentVersion" : null, "properties" : { "contentLibraryId" : null, "vidmPrimaryNodeRootPassword" : "JXJXJXJX", "certificateId" : "locker:certificate:f33c54a7-efd1-4609-9b4b-8d1010ce368c:opslogs", "baseTenantId" : "base tenant", "certificate" : "locker:certificate:f33c54a7-efd1-4609-9b4b-8d1010ce368c:opslogs", "workerNodeIP" : "XX.XX.XX.18,XX.XX.XX.19", "vrliAdminEmail" : "conifigadmin@vsphere.local", "__version" : "8.12.0", "nodeSize" : "small", "masterVidmAdminUserName" : "admin", "environmentName" : "Operations for Logs", * * * * "vrliClusterVips" : "XX.XX.XX.16#lilb.cap.org", "isUpgradeVmCompatibility" : "true", "authProviderHostnames" : "vidmlb.cap.org,vidmtwo.cap.org,vidmone.cap.org,vidmthree.cap.org", "vidmPrimaryNodeHostname" : "vidmone.cap.org", "appIcon" : "vidm_driver_images/vrli.png", "appName" : "Operations for Logs_vRLI_8.12.0_XX.XX.XX.17", "targetUrl" : "https://vidmlb.cap.org:443/SAAS/auth/oauth2/authorize?response_type=code&client_id=116b0383-9fc3-43ee-aff6-a1e753584f35&redirect_uri=https://lilb.cap.org/login" } }, "priority" : 4 }, "originalObject" : { "component" : { "symbolicName" : "vrlicluster", "type" : null, "componentVersion" : null, "properties" : { "contentLibraryId" : null, "vidmPrimaryNodeRootPassword" : "JXJXJXJX", "certificateId" : "locker:certificate:f33c54a7-efd1-4609-9b4b-8d1010ce368c:opslogs", "baseTenantId" : "base tenant", "certificate" : "locker:certificate:f33c54a7-efd1-4609-9b4b-8d1010ce368c:opslogs", "workerNodeIP" : "XX.XX.XX.18,XX.XX.XX.19", "vrliAdminEmail" : "conifigadmin@vsphere.local", "__version" : "8.12.0", "nodeSize" : "small", "masterVidmAdminUserName" : "admin", "environmentName" : "Operations for Logs", "vrliHostName" : "lione.cap.org", "masterNodeIP" : "XX.XX.XX.17", "__productId" : "vrli", * * * * "authProviderHostnames" : "vidmlb.cap.org,vidmtwo.cap.org,vidmone.cap.org,vidmthree.cap.org", "vidmPrimaryNodeHostname" : "vidmone.cap.org", "appIcon" : "vidm_driver_images/vrli.png", "appName" : "Operations for Logs_vRLI_8.12.0_XX.XX.XX.17", "targetUrl" : "https://vidmlb.cap.org:443/SAAS/auth/oauth2/authorize?response_type=code&client_id=116b0383-9fc3-43ee-aff6-a1e753584f35&redirect_uri=https://lilb.cap.org/login" } }, "priority" : 4 } } } /** Complete Product Spec **/ 2023-07-24 02:35:55.021 INFO [pool-3-thread-6] c.v.v.l.p.a.s.Task - -- FIELD NAME :: productSpec 2023-07-24 02:35:55.021 INFO [pool-3-thread-6] c.v.v.l.p.a.s.Task - -- KEY PICKER IS NULL 2023-07-24 02:35:55.022 INFO [pool-3-thread-6] c.v.v.l.p.a.s.Task - -- ======================================== { "productSpec" : { "object" : null, "originalObject" : null }, "componentSpec" : { "object" : { "component" : { "symbolicName" : "vrlicluster", "type" : null, "componentVersion" : null, "properties" : { "contentLibraryId" : null, "vidmPrimaryNodeRootPassword" : "JXJXJXJX", "certificateId" : "locker:certificate:f33c54a7-efd1-4609-9b4b-8d1010ce368c:opslogs", "baseTenantId" : "base tenant", "certificate" : "locker:certificate:f33c54a7-efd1-4609-9b4b-8d1010ce368c:opslogs", "workerNodeIP" : "XX.XX.XX.18,XX.XX.XX.19", "vrliAdminEmail" : "conifigadmin@vsphere.local", "__version" : "8.12.0", "nodeSize" : "small", "masterVidmAdminUserName" : "admin", "environmentName" : "Operations for Logs", "vrliHostName" : "lione.cap.org", "masterNodeIP" : "XX.XX.XX.17", "__productId" : "vrli", "ntp" : "****.****", "uberAdminUserType" : "LOCAL", "masterVidmAdminPassword" : "JXJXJXJX", "vrliAdminPassword" : "JXJXJXJX", "uberAdmin" : "configadmin", "masterVidmEnabled" : "true", "uberAdminPassword" : "JXJXJXJX", "masterVidmHostName" : "vidmlb.cap.org", "fipsMode" : "False", "timeSyncMode" : "ntp", "isVcfUser" : "false", "isVcfEnabledEnv" : "false", "vrliClusterVips" : "XX.XX.XX.16#lilb.cap.org", "isUpgradeVmCompatibility" : "true", "authProviderHostnames" : "vidmlb.cap.org,vidmtwo.cap.org,vidmone.cap.org,vidmthree.cap.org", "vidmPrimaryNodeHostname" : "vidmone.cap.org", "appIcon" : "vidm_driver_images/vrli.png", "appName" : "Operations for Logs_vRLI_8.12.0_XX.XX.XX.17", "targetUrl" : "https://vidmlb.cap.org:443/SAAS/auth/oauth2/authorize?response_type=code&client_id=116b0383-9fc3-43ee-aff6-a1e753584f35&redirect_uri=https://lilb.cap.org/login" } }, "priority" : 4 }, "originalObject" : { "component" : { "symbolicName" : "vrlicluster", "type" : null, "componentVersion" : null, "properties" : { "contentLibraryId" : null, * * * * "priority" : 4 } } } /** Component Spec **/ 2023-07-24 02:35:55.023 INFO [pool-3-thread-6] c.v.v.l.p.a.s.Task - -- FIELD NAME :: componentSpec 2023-07-24 02:35:55.023 INFO [pool-3-thread-6] c.v.v.l.p.a.s.Task - -- KEY PICKER IS :: com.vmware.vrealize.lcm.drivers.commonplugin.task.keypicker.GenericComponentSpecKeyPicker 2023-07-24 02:35:55.537 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- INITIALIZING NEW EVENT :: { "vmid" : "91cec54d-57df-4dc5-9f3b-b9a1ea11dda5", "transactionId" : null, "tenant" : "default", "createdBy" : "root", "lastModifiedBy" : "root", "createdOn" : 1690166155030, "lastUpdatedOn" : 1690166155533, "version" : "8.1.0.0", "vrn" : null, "eventName" : "OnVrliCertificateInitTaskCompleted", "currentState" : null, "eventArgument" : "{\"certificate\":{\"name\":\"certificate\",\"type\":\"java.lang.String\",\"value\":\"\\\"locker:certificate:f33c54a7-efd1-4609-9b4b-8d1010ce368c:opslogs\\\"\"},\"productSpec\": * * * * "errorCause" : null, "sequence" : 3454, "eventLock" : 1, "engineNodeId" : "devvasl.cap.org" } /** Certificate Init Task Completes **/ 2023-07-24 02:35:55.546 INFO [scheduling-1] c.v.v.l.a.c.MachineRegistry - -- GETTING MACHINE FOR THE KEY :: vrlicluster 2023-07-24 02:35:55.546 INFO [scheduling-1] c.v.v.l.a.c.MachineRegistry - -- QUERYING CONTENT :: SystemFlowInventory::flows::flow->vrlicluster 2023-07-24 02:35:55.546 INFO [scheduling-1] c.v.v.l.d.i.u.InventorySchemaQueryUtil - -- GETTING ROOT NODE FOR :: SystemFlowInventory 2023-07-24 02:35:55.599 INFO [scheduling-1] c.v.v.l.a.c.MachineRegistry - -- URL :: /system/flow/vrlicluster.vmfx 2023-07-24 02:35:55.602 INFO [scheduling-1] c.v.v.l.c.c.ContentDownloadController - -- INSIDE ContentDownloadControllerImpl 2023-07-24 02:35:55.603 INFO [scheduling-1] c.v.v.l.c.c.ContentDownloadController - -- REPO_NAME :: /systemflowrepo 2023-07-24 02:35:55.603 INFO [scheduling-1] c.v.v.l.c.c.ContentDownloadController - -- CONTENT_PATH :: /system/flow/vrlicluster.vmfx 2023-07-24 02:35:55.603 INFO [scheduling-1] c.v.v.l.c.c.ContentDownloadController - -- URL :: /systemflowrepo/system/flow/vrlicluster.vmfx 2023-07-24 02:35:55.603 INFO [scheduling-1] c.v.v.l.c.c.ContentDownloadController - -- Decoded URL :: /systemflowrepo/system/flow/vrlicluster.vmfx 2023-07-24 02:35:55.605 INFO [scheduling-1] c.v.v.l.c.c.ContentDownloadController - -- ContentDTO{BaseDTO{vmid='vrlicluster', version=8.1.0.0} -> repoName='systemflowrepo', contentState='PUBLISHED', url='/systemflowrepo/system/flow/vrlicluster.vmfx'} 2023-07-24 02:35:55.606 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- Responding for Edge :: OnVrliCertificateInitTaskCompleted 2023-07-24 02:35:55.606 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- State to find :: com.vmware.vrealize.lcm.plugin.vrli.VrliCertificateInitTask 2023-07-24 02:35:55.606 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- State to find :: com.vmware.vrealize.lcm.plugin.vrli.VrliImportCertificateTask 2023-07-24 02:35:55.611 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- Invoking Task :: com.vmware.vrealize.lcm.plugin.vrli.VrliImportCertificateTask 2023-07-24 02:35:55.613 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- Injecting Locker Object :: productSpec 2023-07-24 02:35:55.614 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- Injecting Locker Object :: componentSpec 2023-07-24 02:35:55.614 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- KEY_VARIABLE :: uberAdminPassword<=KXKXKXKX KEY :: XXXXXX 2023-07-24 02:35:55.620 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- KEY_VARIABLE :: vidmPrimaryNodeRootPassword<=KXKXKXKX KEY :: XXXXXX 2023-07-24 02:35:55.622 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- KEY_VARIABLE :: masterVidmAdminPassword<=KXKXKXKX KEY :: XXXXXX 2023-07-24 02:35:55.625 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- KEY_VARIABLE :: vrliAdminPassword<=KXKXKXKX KEY :: XXXXXX 2023-07-24 02:35:55.628 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- Injecting Locker Object :: certificate 2023-07-24 02:35:55.628 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- LOCKER OBJECT: TYPE :: certificate <===> TARGET :: certificateDto 2023-07-24 02:35:55.630 INFO [scheduling-1] c.v.v.l.l.s.p.PKIStoreServiceImpl - -- Reading certificate 'f33c54a7-efd1-4609-9b4b-8d1010ce368c' from [PKIStoreService] 2023-07-24 02:35:55.650 INFO [scheduling-1] c.v.v.l.c.u.EventExecutionTelemetryUtil - -- Start Instrumenting EventMetadata. 2023-07-24 02:35:55.651 INFO [scheduling-1] c.v.v.l.c.u.EventExecutionTelemetryUtil - -- Stop Instrumenting EventMetadata. /** Certificate Import Task starts **/ 2023-07-24 02:35:55.659 INFO [pool-3-thread-8] c.v.v.l.p.v.VrliImportCertificateTask - -- Starting :: VMware Aria Operations for Logs import certificate task 2023-07-24 02:35:55.660 INFO [pool-3-thread-8] c.v.v.l.p.v.VrliImportCertificateTask - -- Called from Install flow 2023-07-24 02:35:56.019 INFO [pool-3-thread-8] c.v.v.l.d.v.InstallConfigureVRLI - -- Return message for VMware Aria Operations for Logs: {"releaseName":"GA","version":"8.12.0-21696970"} 2023-07-24 02:35:56.020 INFO [pool-3-thread-8] c.v.v.l.d.v.InstallConfigureVRLI - -- Return status code for VMware Aria Operations for Logs: 200 2023-07-24 02:35:56.026 INFO [pool-3-thread-8] c.v.v.l.p.v.VrliImportCertificateTask - -- Version >= 4.6.0. Applying certificate for the newly deployed VMware Aria Operations for Logs 2023-07-24 02:35:56.027 INFO [pool-3-thread-8] c.v.v.l.d.v.InstallConfigureVRLI - -- Checking if VMware Aria Operations for Logs instance is running 2023-07-24 02:35:56.105 INFO [pool-3-thread-8] c.v.v.l.d.v.InstallConfigureVRLI - -- The VMware Aria Operations for Logs instance https://XX.XX.XX.17 service is running 2023-07-24 02:36:03.690 INFO [http-nio-8080-exec-9] c.v.v.l.s.n.s.NotificationServiceImpl - -- Authentication object is not null org.springframework.security.authentication.UsernamePasswordAuthenticationToken@fe908ae6: YXYXYXYX org.springframework.security.core.userdetails.User@c220133a: Username: admin@local; Password: YXYXYXYX Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: LCM_ADMIN; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: LCM_ADMIN 2023-07-24 02:37:03.710 INFO [http-nio-8080-exec-5] c.v.v.l.s.n.s.NotificationServiceImpl - -- Authentication object is not null org.springframework.security.authentication.UsernamePasswordAuthenticationToken@fe908ae6: YXYXYXYX org.springframework.security.core.userdetails.User@c220133a: Username: admin@local; Password: YXYXYXYX Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: LCM_ADMIN; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: LCM_ADMIN 2023-07-24 02:37:39.993 INFO [pool-3-thread-8] c.v.v.l.d.v.InstallConfigureVRLI - -- certificate api response: statuscode = 201 2023-07-24 02:37:39.997 INFO [pool-3-thread-8] c.v.v.l.d.v.InstallConfigureVRLI - -- certificate api response: message = Created /** Certificate is uploaded and completion is called **/ 2023-07-24 02:37:39.997 INFO [pool-3-thread-8] c.v.v.l.p.v.VrliImportCertificateTask - -- Certificate uploaded successfully 2023-07-24 02:37:39.997 INFO [pool-3-thread-8] c.v.v.l.p.a.s.Task - -- Injecting Edge :: OnVRLIUpdateCertificateCompletion 2023-07-24 02:37:39.998 INFO [pool-3-thread-8] c.v.v.l.p.a.s.Task - -- ======================================== { "productSpec" : { "object" : null, "originalObject" : null }, "componentSpec" : { "object" : { "component" : { "symbolicName" : "vrlicluster", "type" : null, * * * * Logs_vRLI_8.12.0_XX.XX.XX.17", "targetUrl" : "https://vidmlb.cap.org:443/SAAS/auth/oauth2/authorize?response_type=code&client_id=116b0383-9fc3-43ee-aff6-a1e753584f35&redirect_uri=https://lilb.cap.org/login" } }, "priority" : 4 } } } 2023-07-24 02:37:40.000 INFO [pool-3-thread-8] c.v.v.l.p.a.s.Task - -- FIELD NAME :: productSpec 2023-07-24 02:37:40.000 INFO [pool-3-thread-8] c.v.v.l.p.a.s.Task - -- KEY PICKER IS NULL 2023-07-24 02:37:40.000 INFO [pool-3-thread-8] c.v.v.l.p.a.s.Task - -- ======================================== { "productSpec" : { "object" : null, "originalObject" : null }, "componentSpec" : { "object" : { "component" : { "symbolicName" : "vrlicluster", "type" : null, "componentVersion" : null, * * * * "targetUrl" : "https://vidmlb.cap.org:443/SAAS/auth/oauth2/authorize?response_type=code&client_id=116b0383-9fc3-43ee-aff6-a1e753584f35&redirect_uri=https://lilb.cap.org/login" } }, "priority" : 4 } } } 2023-07-24 02:37:40.002 INFO [pool-3-thread-8] c.v.v.l.p.a.s.Task - -- FIELD NAME :: componentSpec 2023-07-24 02:37:40.002 INFO [pool-3-thread-8] c.v.v.l.p.a.s.Task - -- KEY PICKER IS :: com.vmware.vrealize.lcm.drivers.commonplugin.task.keypicker.GenericComponentSpecKeyPicker 2023-07-24 02:37:40.534 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- INITIALIZING NEW EVENT :: { "vmid" : "e91c6812-e7a5-40b9-aa3d-4b4983369a96", "transactionId" : null, "tenant" : "default", "createdBy" : "root", "lastModifiedBy" : "root", "createdOn" : 1690166260003, "lastUpdatedOn" : 1690166260530, "version" : "8.1.0.0", "vrn" : null, "eventName" : "OnVRLIUpdateCertificateCompletion", "currentState" : null, "eventArgument" : "{\"productSpec\":{\"name\":\"productSpec\",\"type\":\"com.vmware.vrealize.lcm.domain.ProductSpecification\",\"value\":\"null\"},\"componentSpec\":{\"name\":\"componentSpec\",\"type\":\"com.vmware.vrealize.lcm.domain.ComponentDeploymentSpecification\",\"value\":\"{\\\"component\\\":{\\\"symbolicName\\\":\\\"vrlicluster\\\",\\\"type\\\":null,\\\"componentVersion\\\":null,\\\"properties\\\":{\\\"contentLibraryId\\\":null,\\\"vidmPrimaryNodeRootPassword\\\":\\\"JXJXJXJX\\\",\\\"certificateId\\\":\\\"locker:KXKXKXKX\\\",\\\"baseTenantId\\\":\\\"base tenant\\\",\\\"certificate\\\":\\\"locker:certificate:f33c54a7-efd1-4609-9b4b-8d1010ce368c:opslogs\\\",\\\"workerNodeIP\\\":\\\"XX.XX.XX.18,XX.XX.XX.19\\\",\\\"vrliAdminEmail\\\":\\\"conifigadmin@vsphere.local\\\",\\\"__version\\\":\\\"8.12.0\\\",\\\"nodeSize\\\":\\\"small\\\",\\\"masterVidmAdminUserName\\\":\\\"admin\\\",\\\"environmentName\\\":\\\"Operations for Logs\\\",\\\"vrliHostName\\\":\\\"lione.cap.org\\\",\\\"masterNodeIP\\\":\\\"XX.XX.XX.17\\\",\\\"__productId\\\":\\\"vrli\\\",\\\"ntp\\\":\\\"****.****\\\",\\\"uberAdminUserType\\\":\\\"LOCAL\\\",\\\"masterVidmAdminPassword\\\":\\\"JXJXJXJX\\\",\\\"vrliAdminPassword\\\":\\\"JXJXJXJX\\\",\\\"uberAdmin\\\":\\\"configadmin\\\",\\\"masterVidmEnabled\\\":\\\"true\\\",\\\"uberAdminPassword\\\":\\\"JXJXJXJX\\\",\\\"masterVidmHostName\\\":\\\"vidmlb.cap.org\\\",\\\"fipsMode\\\":\\\"False\\\",\\\"timeSyncMode\\\":\\\"ntp\\\",\\\"isVcfUser\\\":\\\"false\\\",\\\"isVcfEnabledEnv\\\":\\\"false\\\",\\\"vrliClusterVips\\\":\\\"XX.XX.XX.16#lilb.cap.org\\\",\\\"isUpgradeVmCompatibility\\\":\\\"true\\\",\\\"authProviderHostnames\\\":\\\"vidmlb.cap.org,vidmtwo.cap.org,vidmone.cap.org,vidmthree.cap.org\\\",\\\"vidmPrimaryNodeHostname\\\":\\\"vidmone.cap.org\\\",\\\"appIcon\\\":\\\"vidm_driver_images/vrli.png\\\",\\\"appName\\\":\\\"Operations for Logs_vRLI_8.12.0_XX.XX.XX.17\\\",\\\"targetUrl\\\":\\\"https://vidmlb.cap.org:443/SAAS/auth/oauth2/authorize?response_type=code&client_id=116b0383-9fc3-43ee-aff6-a1e753584f35&redirect_uri=https://lilb.cap.org/login\\\"}},\\\"priority\\\":4}\"}}", "status" : "CREATED", "stateMachineInstance" : "1adef09c-09d0-4a16-bd6d-036db4eb2ce7", "errorCause" : null, "sequence" : 3456, "eventLock" : 1, "engineNodeId" : "devvasl.cap.org" } 2023-07-24 02:37:40.542 INFO [scheduling-1] c.v.v.l.a.c.MachineRegistry - -- GETTING MACHINE FOR THE KEY :: vrlicluster 2023-07-24 02:37:40.542 INFO [scheduling-1] c.v.v.l.a.c.MachineRegistry - -- QUERYING CONTENT :: SystemFlowInventory::flows::flow->vrlicluster 2023-07-24 02:37:40.542 INFO [scheduling-1] c.v.v.l.d.i.u.InventorySchemaQueryUtil - -- GETTING ROOT NODE FOR :: SystemFlowInventory 2023-07-24 02:37:40.568 INFO [scheduling-1] c.v.v.l.a.c.MachineRegistry - -- URL :: /system/flow/vrlicluster.vmfx 2023-07-24 02:37:40.568 INFO [scheduling-1] c.v.v.l.c.c.ContentDownloadController - -- INSIDE ContentDownloadControllerImpl 2023-07-24 02:37:40.569 INFO [scheduling-1] c.v.v.l.c.c.ContentDownloadController - -- REPO_NAME :: /systemflowrepo 2023-07-24 02:37:40.569 INFO [scheduling-1] c.v.v.l.c.c.ContentDownloadController - -- CONTENT_PATH :: /system/flow/vrlicluster.vmfx 2023-07-24 02:37:40.569 INFO [scheduling-1] c.v.v.l.c.c.ContentDownloadController - -- URL :: /systemflowrepo/system/flow/vrlicluster.vmfx 2023-07-24 02:37:40.569 INFO [scheduling-1] c.v.v.l.c.c.ContentDownloadController - -- Decoded URL :: /systemflowrepo/system/flow/vrlicluster.vmfx 2023-07-24 02:37:40.572 INFO [scheduling-1] c.v.v.l.c.c.ContentDownloadController - -- ContentDTO{BaseDTO{vmid='vrlicluster', version=8.1.0.0} -> repoName='systemflowrepo', contentState='PUBLISHED', url='/systemflowrepo/system/flow/vrlicluster.vmfx'} 2023-07-24 02:37:40.573 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- Responding for Edge :: OnVRLIUpdateCertificateCompletion 2023-07-24 02:37:40.573 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- State to find :: com.vmware.vrealize.lcm.plugin.vrli.VrliImportCertificateTask 2023-07-24 02:37:40.574 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- State to find :: com.vmware.vrealize.lcm.platform.automata.service.task.FinalTask 2023-07-24 02:37:40.581 INFO [scheduling-1] c.v.v.l.a.c.EventProcessor - -- Invoking Task :: com.vmware.vrealize.lcm.platform.automata.service.task.FinalTask Operations for Log perspective api_audit.log [2023-07-24 02:35:56.476+0000] ["application-akka.actor.default-dispatcher-21"/1X.XX.XX.XX INFO] [com.vmware.loginsight.api.Authentication.AuthenticatedActionBase] [admin calls POST /api/v1/certificate] [2023-07-24 02:35:56.485+0000] ["application-akka.actor.default-dispatcher-21"/1X.XX.XX.XX INFO] [com.vmware.loginsight.api.Authentication.AuthenticatedActionBase] [admin calls POST /api/v1/certificate] runtime.log [2023-07-24 02:35:35.445+0000] ["SslCertificateManagerScheduler-thread-1"/1X.XX.XX.XX INFO] [com.vmware.loginsight.daemon.shared.ssl.SslCertificateManager] [checkAndUpdateTruststore--] [2023-07-24 02:35:35.448+0000] ["SslCertificateManagerScheduler-thread-1"/1X.XX.XX.XX INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Loading truststore from path: /usr/java/jre-vmware/lib/security/cacerts] [2023-07-24 02:35:56.486+0000] ["application-akka.actor.default-dispatcher-21"/1X.XX.XX.XXINFO] [com.vmware.loginsight.api.providers.certificate.CertificateProvider] [Saving new custom SSL certificate] [2023-07-24 02:36:35.450+0000] ["SslCertificateManagerScheduler-thread-1"/1X.XX.XX.XX INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Loading truststore from path: /usr/java/jre-vmware/lib/security/cacerts] [2023-07-24 02:37:32.717+0000] ["DaemonCommands-thread-14"/1X.XX.XX.XX INFO] [com.vmware.loginsight.daemon.shared.ssl.SslCertificateManager] [SSL script result: [exitCode=0, stdOut=, stdErr=Importing keystore /usr/lib/loginsight/application/etc/certs/keystore.pkcs12 to /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.87/conf/keystore... Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.87/conf/keystore -destkeystore /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.87/conf/keystore -deststoretype pkcs12". Certificate was added to keystore Importing keystore /usr/lib/loginsight/application/etc/certs/keystore.pkcs12 to /usr/lib/loginsight/application/etc/3rd_config/keystore... Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /usr/lib/loginsight/application/etc/3rd_config/keystore -destkeystore /usr/lib/loginsight/application/etc/3rd_config/keystore -deststoretype pkcs12".]] [2023-07-24 02:37:32.717+0000] ["DaemonCommands-thread-14"/1X.XX.XX.XX WARN] [com.vmware.loginsight.daemon.shared.ssl.SslCertificateManager] [Importing keystore /usr/lib/loginsight/application/etc/certs/keystore.pkcs12 to /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.87/conf/keystore... Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.87/conf/keystore -destkeystore /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.87/conf/keystore -deststoretype pkcs12". Certificate was added to keystore Importing keystore /usr/lib/loginsight/application/etc/certs/keystore.pkcs12 to /usr/lib/loginsight/application/etc/3rd_config/keystore... Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /usr/lib/loginsight/application/etc/3rd_config/keystore -destkeystore /usr/lib/loginsight/application/etc/3rd_config/keystore -deststoretype pkcs12".] [2023-07-24 02:37:32.717+0000] ["DaemonCommands-thread-14"/1X.XX.XX.XX INFO] [com.vmware.loginsight.daemon.shared.ssl.SslCertificateManager] [Syncing certificate] [2023-07-24 02:37:35.445+0000] ["SslCertificateManagerScheduler-thread-1"/1X.XX.XX.XX INFO] [com.vmware.loginsight.daemon.shared.ssl.SslCertificateManager] [checkAndUpdateTruststore--] [2023-07-24 02:37:35.450+0000] ["SslCertificateManagerScheduler-thread-1"/1X.XX.XX.XX INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Loading truststore from path: /usr/java/jre-vmware/lib/security/cacerts]

Goal Upgrade globalenvironment or VMware Identity Manager from version 3.3.6 to 3.3.7 For this demo, we are using vRSLCM 8.10 PSPACK 15 Environment Nodes Hostname Load balancer fqdn vidmlb.cap.org first node vidmone.cap.org second node vidmtwo.cap.org third node vidmthree.cap.org Pre-requisites Take snapshot of VMware Identity Manager globalenvironment product properties take snapshot using "create snapshot" day-2 action of vRSLCM takes around 58 minutes to complete the snapshot task Perform Inventory Sync Upgrade before request submission phase In order to perform an upgrade, we need the upgraderepo downloaded Once downloaded, we will go ahead and trigger upgrade of vIDM from lifecycle operations pane of Suite Lifecycle An inventory sync must be performed here as well in order to proceed Once Inventory Sync is complete we can click on "proceed" to start the process Remember we already have the upgrade repo downloaded so let's point it to the same link You may opt for product snpashot and also to retain the snapshot after an upgrade Clicking on next will ask you to run a precheck SSH connectivity Disk space on / Node health Opened ports Root Password expiration Version check Sizing check are the ones which are verified during precheck Prechecks are successful as shown above upgrade triggered phase Review the summary before submitting the request and then click on submit to initiate uprgade Upgrade is now triggered Multiple stages during upgrade are shown as below. Remember the timings mentioned below are from my lab, these might vary in your environment. Stage Description Time 1 Graceful Shutdown of VMware Identity Manager 16m 40s 2 Create Node Snapshot 16s 3 Power On VMware Identity Manager Nodes 6m 17s 4 Remediate VMware Identity Manager Nodes 27m 53s 5 Enable VMware Identity Manager Health Notification 3m 46s 6 Create Snapshot Inventory 575ms 7 Disable VMware Identity Manager Health Notification 2s 8 Stop VMware Identity Manager Services before starting an upgrade 2m 19s 9 Stop VMware Identity Manager Services before starting an upgrade 2m 14s 10 Stop VMware Identity Manager Services before starting an upgrade 2m 14s 11 Upgrade VMware Identity Manager Offline 30m 34s 12 Assign delegate IP to Postgres primary node of VMware Identity Manager 2m 11s 13 Upgrade VMware Identity Manager Offline 30m 8s 14 Upgrade VMware Identity Manager Offline 28m 43s 15 Configure pgpool on VMware Identity Manager Secondary Nodes 3m 35s 16 Verify VMware Identity Manager Postgres Clustering 32s 17 Configure VMware Identity Manager pgpool post upgrade 5m 58s 18 Stop Postgres on VMware Identity Manager 4s 19 Remediate VMware Identity Manager Nodes 20m 28s 20 Product Upgrade Inventory Update 582ms 21 Enable Log Rotation of pgpool services 3s 22 Snapshot Inventory Update 530ms 23 Snapshot Inventory Compute 206ms 25 Snapshot Inventory Update 530ms Total Time Taken for Upgrade 3 hours 8 mins You might see some steps repeating as it has to perform upgrade on 3 VMware Identity Manager nodes one after another vRSLCM logs deep-dive w.r.t vIDM Upgrade Reference: /var/log/vrlcm/vmware_vrlcm.log We will specifically look into Stages 13 to 21 where the upgrade and post upgrade steps occur //**** Upgrade vIDM Offline state machine ( stage 11 ) **** // To begin with old updateoffline.hzn is deleted and new one is copied and permissions are changed 2023-07-22 12:05:26.565 INFO [pool-3-thread-23] c.v.v.l.v.c.t.VidmUploadUpgradeScriptTask - -- Starting :: vIDM upload upgrade script Task 2023-07-22 12:05:26.567 INFO [pool-3-thread-23] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil - -- vIDM Configuration property vidmUpgradeBasePath value is obtained from Config Service : /usr/local/horizon/update/ 2023-07-22 12:05:26.567 INFO [pool-3-thread-23] c.v.v.l.v.c.t.VidmUploadUpgradeScriptTask - -- Target version is 3.3.3 and above, loading new offline upgrade script. 2023-07-22 12:05:26.568 INFO [pool-3-thread-23] c.v.v.l.v.c.t.VidmUploadUpgradeScriptTask - -- Upgrade script source folder path :: newupgradescript/ 2023-07-22 12:05:26.667 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:05:26.668 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:05:26.668 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Command: ls /usr/local/horizon/update/updateoffline.hzn 2023-07-22 12:05:26.668 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:05:27.260 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:05:27.260 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:05:27.261 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Command execution response: { "exitStatus" : 0, "outputData" : "/usr/local/horizon/update/updateoffline.hzn\n", "errorData" : null, "commandTimedOut" : false } 2023-07-22 12:05:27.268 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Found file for upload: /tmp/updateoffline.hzn 2023-07-22 12:05:27.393 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Uploading file --> ssh://root@vidmone.cap.org/usr/local/horizon/update/ 2023-07-22 12:05:27.740 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Uploaded file sucessfully 2023-07-22 12:05:27.742 INFO [pool-3-thread-23] c.v.v.l.v.c.t.VidmUploadUpgradeScriptTask - -- Uploaded file to the location : /tmp/updateoffline.hzn 2023-07-22 12:05:27.935 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:05:27.937 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:05:27.938 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Command: chmod 500 /usr/local/horizon/update/updateoffline.hzn 2023-07-22 12:05:27.939 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:05:28.520 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:05:28.520 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:05:28.521 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Command execution response: { "exitStatus" : 0, "outputData" : "", "errorData" : null, "commandTimedOut" : false } 2023-07-22 12:05:28.522 INFO [pool-3-thread-23] c.v.v.l.v.c.t.VidmUploadUpgradeScriptTask - -- File: /tmp/updateoffline.hzn deleted successfully. 2023-07-22 12:05:28.612 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:05:28.613 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:05:28.613 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Command: sed -i '/^ *read$/d' /usr/local/horizon/update/updateoffline.hzn 2023-07-22 12:05:28.613 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:05:29.119 INFO [http-nio-8080-exec-2] c.v.v.l.s.n.s.NotificationServiceImpl - -- Authentication object is not null org.springframework.security.authentication.UsernamePasswordAuthenticationToken@fe908ae6: YXYXYXYX org.springframework.security.core.userdetails.User@c220133a: Username: admin@local; Password: YXYXYXYX Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: LCM_ADMIN; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: LCM_ADMIN 2023-07-22 12:05:29.212 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:05:29.212 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:05:29.213 INFO [pool-3-thread-23] c.v.v.l.u.SshUtils - -- Command execution response: { "exitStatus" : 0, "outputData" : "", "errorData" : null, "commandTimedOut" : false } 2023-07-22 12:05:29.213 INFO [pool-3-thread-23] c.v.v.l.p.a.s.Task - -- Injecting Edge :: OnVidmUploadUpgradeScriptCompletion vIDM bundle upload task is initiated and completed 2023-07-22 12:05:29.563 INFO [pool-3-thread-26] c.v.v.l.v.c.t.VidmUploadUpgradeBundleTask - -- Starting :: vIDM upgrade bundle upload Task 2023-07-22 12:05:29.563 INFO [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmInstallTaskUtil - -- VidmInstallTaskUtil.getLeasedLocalFileLocation :: Start 2023-07-22 12:05:29.565 INFO [pool-3-thread-26] c.v.v.l.c.s.ContentLeaseServiceImpl - -- Inside create content lease. 2023-07-22 12:05:29.566 INFO [pool-3-thread-26] c.v.v.l.c.s.ContentLeaseServiceImpl - -- Created lease for the folder with id :: 8a233ccd-ccf9-4ddb-adff-358c20c20fe1. 2023-07-22 12:05:29.577 INFO [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmInstallTaskUtil - -- /data/vm-config/symlinkdir/8a233ccd-ccf9-4ddb-adff-358c20c20fe1 2023-07-22 12:05:29.577 INFO [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmInstallTaskUtil - -- Started Downloading Content Repo 2023-07-22 12:05:29.577 INFO [pool-3-thread-26] c.v.v.l.c.c.ContentDownloadController - -- INSIDE ContentDownloadControllerImpl 2023-07-22 12:05:29.578 INFO [pool-3-thread-26] c.v.v.l.c.c.ContentDownloadController - -- REPO_NAME :: /productBinariesRepo 2023-07-22 12:05:29.578 INFO [pool-3-thread-26] c.v.v.l.c.c.ContentDownloadController - -- CONTENT_PATH :: /vidm/3.3.7/upgrade/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz 2023-07-22 12:05:29.578 INFO [pool-3-thread-26] c.v.v.l.c.c.ContentDownloadController - -- URL :: /productBinariesRepo/vidm/3.3.7/upgrade/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz 2023-07-22 12:05:29.578 INFO [pool-3-thread-26] c.v.v.l.c.c.ContentDownloadController - -- Decoded URL :: /productBinariesRepo/vidm/3.3.7/upgrade/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz 2023-07-22 12:05:29.581 INFO [pool-3-thread-26] c.v.v.l.c.c.ContentDownloadController - -- ContentDTO{BaseDTO{vmid='da06426f-5f18-4985-b640-38d711fd597f', version=8.1.0.0} -> repoName='productBinariesRepo', contentState='PUBLISHED', url='/productBinariesRepo/vidm/3.3.7/upgrade/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz'} 2023-07-22 12:05:29.582 INFO [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmInstallTaskUtil - -- Completed Downloading from Content Repo and starting InputStream 2023-07-22 12:05:29.582 INFO [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmInstallTaskUtil - -- /data/vm-config/symlinkdir/8a233ccd-ccf9-4ddb-adff-358c20c20fe1/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz 2023-07-22 12:09:56.628 INFO [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmInstallTaskUtil - -- Output stream completed * * * * 2023-07-22 12:09:56.637 INFO [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmInstallTaskUtil - -- VidmInstallTaskUtil.getLeasedLocalFileLocation upgradeFileLocation : /data/vm-config/symlinkdir/8a233ccd-ccf9-4ddb-adff-358c20c20fe1/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz 2023-07-22 12:09:56.649 INFO [pool-3-thread-26] c.v.v.l.v.c.t.VidmUploadUpgradeBundleTask - -- Started :: Copying upgrade bundle to vIDM with hostname: vidmone.cap.org 2023-07-22 12:09:56.652 INFO [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil - -- vIDM Configuration property vidmMinDiskspaceFor333UpgradeInKb value is obtained from Config Service : 7340032 2023-07-22 12:09:56.653 INFO [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil - -- vIDM Configuration property vidmOptMinDiskspaceFor333UpgradeInKb value is obtained from Config Service : 7 2023-07-22 12:09:56.654 INFO [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil - -- vIDM Configuration property vidmExpandDiskspaceFor333UpgradeKb value is obtained from Config Service : https://kb.vmware.com/s/article/81220 2023-07-22 12:09:56.655 INFO [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil - -- Checking if required free space of 7340032 GB is available in /var or / partiton in vIDM host vidmone.cap.org 2023-07-22 12:09:56.750 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:09:56.751 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:09:56.751 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- Command: df | grep -w "/" | awk '{print $4}' 2023-07-22 12:09:56.751 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:09:57.340 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:09:57.340 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:09:57.343 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- Command execution response: { "exitStatus" : 0, "outputData" : "8693676\n", "errorData" : null, "commandTimedOut" : false } 2023-07-22 12:09:57.343 INFO [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil - -- Partition / has disk space of 8 GB 2023-07-22 12:09:57.343 INFO [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil - -- Partition / has required disk space of 7340032 GB to store vIDM upgrade bundle 2023-07-22 12:09:57.345 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- vIDM ENDPOINT HOST :: vidmone.cap.org 2023-07-22 12:09:57.345 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- COMMAND :: mkdir -p /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/ 2023-07-22 12:09:57.444 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:09:57.445 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:09:57.445 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- Command: mkdir -p /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/ 2023-07-22 12:09:57.446 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:09:58.528 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:09:58.528 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:09:58.529 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- Command Status code :: 0 2023-07-22 12:09:58.529 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:09:58.530 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- Output Stream :: 2023-07-22 12:09:58.530 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:09:58.530 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- 2023-07-22 12:09:58.530 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:09:58.530 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- Error Stream :: 2023-07-22 12:09:58.531 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:09:58.532 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- null 2023-07-22 12:09:58.532 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:09:58.534 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- Found file for upload: /data/vm-config/symlinkdir/8a233ccd-ccf9-4ddb-adff-358c20c20fe1/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz 2023-07-22 12:09:58.735 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- Uploading file --> ssh://root@vidmone.cap.org/var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/ * * * 2023-07-22 12:15:05.647 INFO [pool-3-thread-26] c.v.v.l.v.c.t.VidmUploadUpgradeBundleTask - -- Completed :: Copying upgrade bundle to vIDM host: vidmone.cap.org 2023-07-22 12:15:05.647 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- vIDM ENDPOINT HOST :: vidmone.cap.org 2023-07-22 12:15:05.647 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- COMMAND :: chmod 644 /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz && chown root:KXKXKXKX /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz 2023-07-22 12:15:06.264 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:15:06.266 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:15:06.266 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- Command: chmod 644 /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz && chown root:KXKXKXKX /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz 2023-07-22 12:15:06.266 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:15:07.356 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:15:07.356 INFO [pool-3-thread-26] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:15:07.356 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- Command Status code :: 0 2023-07-22 12:15:07.356 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:15:07.356 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- Output Stream :: 2023-07-22 12:15:07.356 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:15:07.356 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- 2023-07-22 12:15:07.356 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:15:07.357 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- Error Stream :: 2023-07-22 12:15:07.357 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:15:07.357 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- null 2023-07-22 12:15:07.357 INFO [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:15:07.357 INFO [pool-3-thread-26] c.v.v.l.p.a.s.Task - -- Injecting Edge :: OnVidmUpgradeBundleUploadCompletion Starts vIDM offline upgrade task 2023-07-22 12:15:07.961 INFO [pool-3-thread-27] c.v.v.l.v.c.t.VidmOfflineUpgradeTask - -- Starting :: vIDM offline upgrade Task 2023-07-22 12:15:07.962 INFO [pool-3-thread-27] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil - -- vIDM Configuration property vidmUpgradeBasePath value is obtained from Config Service : /usr/local/horizon/update/ 2023-07-22 12:15:07.963 INFO [pool-3-thread-27] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil - -- vIDM Configuration property timeoutCount value is obtained from Config Service : 45 2023-07-22 12:15:07.963 INFO [pool-3-thread-27] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil - -- vIDM Configuration property serverAliveCountMax value is obtained from Config Service : 3 2023-07-22 12:15:07.964 INFO [pool-3-thread-27] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil - -- vIDM Configuration property serverAliveInterval value is obtained from Config Service : 60000 2023-07-22 12:15:07.964 INFO [pool-3-thread-27] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil - -- vIDM Configuration property timeoutSleep value is obtained from Config Service : 60000 2023-07-22 12:15:07.965 INFO [pool-3-thread-27] c.v.v.l.v.c.t.VidmOfflineUpgradeTask - -- Version to upgrade: 3.3.7 source version :: 3.3.6.0 2023-07-22 12:15:07.965 INFO [pool-3-thread-27] c.v.v.l.v.c.t.VidmOfflineUpgradeTask - -- Offline Upgrade to version 3.3.3 and above. 2023-07-22 12:15:08.090 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:15:08.097 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:15:08.098 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Command: touch /usr/local/horizon/conf/flags/elasticsearchReindexed.flag 2023-07-22 12:15:08.099 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:15:39.128 INFO [http-nio-8080-exec-5] c.v.v.l.s.n.s.NotificationServiceImpl - -- Authentication object is not null org.springframework.security.authentication.UsernamePasswordAuthenticationToken@fe908ae6: YXYXYXYX org.springframework.security.core.userdetails.User@c220133a: Username: admin@local; Password: YXYXYXYX Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: LCM_ADMIN; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: LCM_ADMIN 2023-07-22 12:16:08.184 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:16:08.191 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:16:08.193 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- output content : 2023-07-22 12:16:08.321 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:16:08.324 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:16:08.324 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Command: rm -rf /db/elasticsearch/horizon 2023-07-22 12:16:08.325 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:16:40.145 INFO [http-nio-8080-exec-3] c.v.v.l.s.n.s.NotificationServiceImpl - -- Authentication object is not null org.springframework.security.authentication.UsernamePasswordAuthenticationToken@fe908ae6: YXYXYXYX org.springframework.security.core.userdetails.User@c220133a: Username: admin@local; Password: YXYXYXYX Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: LCM_ADMIN; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: LCM_ADMIN 2023-07-22 12:17:08.420 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:17:08.427 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:17:08.428 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- output content : 2023-07-22 12:17:08.526 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:17:08.527 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:17:08.527 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Command: echo "vidmone.cap.org" > /usr/local/horizon/conf/flags/opensearch.masternode 2023-07-22 12:17:08.528 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:17:41.119 INFO [http-nio-8080-exec-9] c.v.v.l.s.n.s.NotificationServiceImpl - -- Authentication object is not null org.springframework.security.authentication.UsernamePasswordAuthenticationToken@fe908ae6: YXYXYXYX org.springframework.security.core.userdetails.User@c220133a: Username: admin@local; Password: YXYXYXYX Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: LCM_ADMIN; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: LCM_ADMIN 2023-07-22 12:18:08.612 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:18:08.619 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:18:08.619 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- output content : 2023-07-22 12:18:08.620 INFO [pool-3-thread-27] c.v.v.l.v.c.t.VidmOfflineUpgradeTask - -- Manifest cmd formed :: /usr/local/horizon/update/configureupdate.hzn manifest --set-version 3.3.6.0 2023-07-22 12:18:08.824 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:18:08.826 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:18:08.827 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Command: /usr/local/horizon/update/configureupdate.hzn manifest --set-version 3.3.6.0 2023-07-22 12:18:08.827 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:18:42.131 INFO [http-nio-8080-exec-3] c.v.v.l.s.n.s.NotificationServiceImpl - -- Authentication object is not null org.springframework.security.authentication.UsernamePasswordAuthenticationToken@fe908ae6: YXYXYXYX org.springframework.security.core.userdetails.User@c220133a: Username: admin@local; Password: YXYXYXYX Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: LCM_ADMIN; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: LCM_ADMIN 2023-07-22 12:18:46.551 INFO [http-nio-8080-exec-9] c.v.v.l.a.c.AuthznCustomObjectMapper - -- ConfigParamsDTO : ConfigParamsDTO [configname=lcm.vidm.registered, configvalue=true] 2023-07-22 12:18:46.553 INFO [http-nio-8080-exec-9] c.v.v.l.a.c.AuthznCustomObjectMapper - -- ConfigParamsDTO : ConfigParamsDTO [configname=lcm.firstboot.admin.password.changed, configvalue=true] 2023-07-22 12:19:08.920 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Setting backrev version in manifest-installed.xml to: 3.3.6.0 2023-07-22 12:19:08.928 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:19:08.928 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:19:08.929 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- output content : Setting backrev version in manifest-installed.xml to: 3.3.6.0 2023-07-22 12:19:08.929 INFO [pool-3-thread-27] c.v.v.l.v.c.t.VidmOfflineUpgradeTask - -- Fetching upgrade binaries from /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/ 2023-07-22 12:19:08.930 INFO [pool-3-thread-27] c.v.v.l.v.c.u.VidmFileChangeUtil - -- Creating upgrade script - /opt/vmware/horizon/vrslcm_vidm_upgrade.sh on vIDM host - vidmone.cap.org 2023-07-22 12:19:09.022 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:19:09.023 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:19:09.024 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Command: echo "/usr/local/horizon/update/updateoffline.hzn -f /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz echo \$? > /opt/vmware/horizon/vidmUpgradeCommandExitCode.log" > /opt/vmware/horizon/vrslcm_vidm_upgrade.sh && chmod 777 /opt/vmware/horizon/vrslcm_vidm_upgrade.sh 2023-07-22 12:19:09.024 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:19:09.611 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:19:09.612 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:19:09.612 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Command execution response: { "exitStatus" : 0, "outputData" : "", "errorData" : null, "commandTimedOut" : false } 2023-07-22 12:19:09.613 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- Command exit code :: 0 2023-07-22 12:19:09.613 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- ============================================== 2023-07-22 12:19:09.613 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- Command output stream :: 2023-07-22 12:19:09.613 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- ============================================== 2023-07-22 12:19:09.613 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- Command error stream :: null 2023-07-22 12:19:09.613 INFO [pool-3-thread-27] c.v.v.l.v.c.t.VidmOfflineUpgradeTask - -- Invoking upgrade script using the command :: nohup /opt/vmware/horizon/vrslcm_vidm_upgrade.sh > /opt/vmware/horizon/vrslcm_vidm_upgrade.log 2>&1 & 2023-07-22 12:19:09.614 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- vIDM ENDPOINT HOST :: vidmone.cap.org 2023-07-22 12:19:09.614 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- COMMAND :: nohup /opt/vmware/horizon/vrslcm_vidm_upgrade.sh > /opt/vmware/horizon/vrslcm_vidm_upgrade.log 2>&1 & 2023-07-22 12:19:09.697 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:19:09.698 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:19:09.698 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Command: nohup /opt/vmware/horizon/vrslcm_vidm_upgrade.sh > /opt/vmware/horizon/vrslcm_vidm_upgrade.log 2>&1 & 2023-07-22 12:19:09.698 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:20:09.815 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:20:09.822 INFO [pool-3-thread-27] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:20:09.823 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- Command Status code :: 0 2023-07-22 12:20:09.823 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:20:09.823 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- Output Stream :: 2023-07-22 12:20:09.824 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:20:09.824 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- 2023-07-22 12:20:09.824 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:20:09.824 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- Error Stream :: 2023-07-22 12:20:09.824 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:20:09.824 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- null 2023-07-22 12:20:09.825 INFO [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil - -- ==================================================== 2023-07-22 12:20:09.825 INFO [pool-3-thread-27] c.v.v.l.v.c.t.VidmOfflineUpgradeTask - -- Upgrade initiated successfully. 2023-07-22 12:20:09.825 INFO [pool-3-thread-27] c.v.v.l.p.a.s.Task - -- Injecting Edge :: OnVidmOfflineUpgradeCompletion vRSLCM waits for uprgade to complete. There are 45 retries. So we expect upgrade to be completed 2023-07-22 12:20:10.366 INFO [pool-3-thread-25] c.v.v.l.v.c.t.VidmRebootTask - -- Starting :: vIDM reboot Task 2023-07-22 12:20:10.367 INFO [pool-3-thread-25] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil - -- vIDM Configuration property timeoutSleep value is obtained from Config Service : 60000 2023-07-22 12:20:10.368 INFO [pool-3-thread-25] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil - -- vIDM Configuration property timeoutCount value is obtained from Config Service : 45 2023-07-22 12:20:10.368 INFO [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil - -- Checking vIDM upgrade status on vIDM host - vidmone.cap.org 2023-07-22 12:20:10.456 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:20:10.460 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:20:10.461 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command: ls /opt/vmware/horizon/vidmUpgradeCommandExitCode.log 2023-07-22 12:20:10.462 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:20:11.004 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- exit-status: 2 2023-07-22 12:20:11.004 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:20:11.005 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command execution response: { "exitStatus" : 2, "outputData" : "", "errorData" : null, "commandTimedOut" : false } 2023-07-22 12:20:11.005 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command exit code :: 2 2023-07-22 12:20:11.005 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- ============================================== 2023-07-22 12:20:11.006 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command output stream :: 2023-07-22 12:20:11.006 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- ============================================== 2023-07-22 12:20:11.006 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command error stream :: null 2023-07-22 12:20:11.006 INFO [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil - -- Waiting for upgrade to complete, sleeping for 60000 milliseconds 2023-07-22 12:21:11.006 INFO [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil - -- Number of retries left to check vIDM upgrade status on the node - vidmone.cap.org : 44 2023-07-22 12:21:11.164 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:21:11.170 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:21:11.171 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command: ls /opt/vmware/horizon/vidmUpgradeCommandExitCode.log 2023-07-22 12:21:11.172 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:21:11.715 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- exit-status: 2 2023-07-22 12:21:11.716 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:21:11.716 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command execution response: { "exitStatus" : 2, "outputData" : "", "errorData" : null, "commandTimedOut" : false } 2023-07-22 12:21:11.717 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command exit code :: 2 2023-07-22 12:21:11.718 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- ============================================== 2023-07-22 12:21:11.718 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command output stream :: 2023-07-22 12:21:11.718 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- ============================================== 2023-07-22 12:21:11.718 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command error stream :: null 2023-07-22 12:21:11.718 INFO [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil - -- Waiting for upgrade to complete, sleeping for 60000 milliseconds 2023-07-22 12:22:11.718 INFO [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil - -- Number of retries left to check vIDM upgrade status on the node - vidmone.cap.org : 43 vIDM upgrade completes finally. 2023-07-22 12:25:13.798 INFO [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil - -- Number of retries left to check vIDM upgrade status on the node - vidmone.cap.org : 40 2023-07-22 12:25:13.889 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:25:13.889 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:25:13.889 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command: ls /opt/vmware/horizon/vidmUpgradeCommandExitCode.log 2023-07-22 12:25:13.890 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:25:14.439 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- exit-status: 2 2023-07-22 12:25:14.440 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:25:14.440 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command execution response: { "exitStatus" : 2, "outputData" : "", "errorData" : null, "commandTimedOut" : false } 2023-07-22 12:25:14.440 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command exit code :: 2 2023-07-22 12:25:14.441 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- ============================================== 2023-07-22 12:25:14.441 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command output stream :: 2023-07-22 12:25:14.441 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- ============================================== 2023-07-22 12:25:14.441 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command error stream :: null 2023-07-22 12:25:14.441 INFO [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil - -- Waiting for upgrade to complete, sleeping for 60000 milliseconds 2023-07-22 12:26:14.441 INFO [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil - -- Number of retries left to check vIDM upgrade status on the node - vidmone.cap.org : 39 2023-07-22 12:26:14.526 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:26:14.527 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:26:14.527 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command: ls /opt/vmware/horizon/vidmUpgradeCommandExitCode.log 2023-07-22 12:26:14.527 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:26:15.075 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:26:15.076 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:26:15.076 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command execution response: { "exitStatus" : 0, "outputData" : "/opt/vmware/horizon/vidmUpgradeCommandExitCode.log\n", "errorData" : null, "commandTimedOut" : false } 2023-07-22 12:26:15.077 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command exit code :: 0 2023-07-22 12:26:15.077 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- ============================================== 2023-07-22 12:26:15.077 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command output stream ::/opt/vmware/horizon/vidmUpgradeCommandExitCode.log 2023-07-22 12:26:15.077 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- ============================================== 2023-07-22 12:26:15.077 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command error stream :: null 2023-07-22 12:26:15.077 INFO [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil - -- File: /opt/vmware/horizon/vidmUpgradeCommandExitCode.log found 2023-07-22 12:26:15.294 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:26:15.301 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:26:15.302 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command: cat /opt/vmware/horizon/vidmUpgradeCommandExitCode.log 2023-07-22 12:26:15.302 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:26:15.848 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:26:15.848 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:26:15.849 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command execution response: { "exitStatus" : 0, "outputData" : "0\n", "errorData" : null, "commandTimedOut" : false } 2023-07-22 12:26:15.849 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command exit code :: 0 2023-07-22 12:26:15.850 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- ============================================== 2023-07-22 12:26:15.850 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command output stream ::0 2023-07-22 12:26:15.850 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- ============================================== 2023-07-22 12:26:15.850 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command error stream :: null 2023-07-22 12:26:15.850 INFO [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil - -- Upgrade command exit status obtained from /opt/vmware/horizon/vidmUpgradeCommandExitCode.log is 0. vIDM upgrade completed successfully 2023-07-22 12:26:15.851 INFO [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil - -- Number of retries left to check vIDM upgrade status on the node - vidmone.cap.org : 38 2023-07-22 12:26:15.930 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX 2023-07-22 12:26:15.930 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:26:15.930 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command: find /usr/local/horizon/conf -iname update.success 2023-07-22 12:26:15.930 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- ------------------------------------------------------ 2023-07-22 12:26:16.479 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- exit-status: 0 2023-07-22 12:26:16.480 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command executed sucessfully 2023-07-22 12:26:16.480 INFO [pool-3-thread-25] c.v.v.l.u.SshUtils - -- Command execution response: { "exitStatus" : 0, "outputData" : "/usr/local/horizon/conf/update.success\n", "errorData" : null, "commandTimedOut" : false } 2023-07-22 12:26:16.481 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command exit code :: 0 2023-07-22 12:26:16.481 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- ============================================== 2023-07-22 12:26:16.481 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command output stream ::/usr/local/horizon/conf/update.success 2023-07-22 12:26:16.481 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- ============================================== 2023-07-22 12:26:16.481 INFO [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil - -- Command error stream :: null 2023-07-22 12:26:16.481 INFO [pool-3-thread-25] c.v.v.l.v.c.t.VidmRebootTask - -- File: /usr/local/horizon/conf/update.success found. Upgrade completed successfully! 2023-07-22 12:26:16.481 INFO [pool-3-thread-25] c.v.v.l.v.c.t.VidmRebootTask - -- vIDM upgrade command exit status: 0. vIDM upgrade completed successfully Takes around 11 minutes on the primary node There are some post uprgade actions performed and then it goes ahead with further steps VMware Identity Manager Upgrade logs Reference: /opt/vmware/horizon/vrslcm_vidm_upgrade.log / ***** Identifies that the upgrade. isthrough vRSLCM *****/ Upgrade via LCM dualbootupdate-3.3.7.0-21173100.tar.gz identity-manager-3.3.7.0-21173100-updaterepo.zip Archive: /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/identity-manager-3.3.7.0-21173100-updaterepo.zip creating: manifest/ inflating: manifest/manifest-latest.xml inflating: manifest/manifest-repo.xml inflating: manifest/manifest-latest.xml.sha256 extracting: manifest/manifest-latest.xml.sig creating: package-pool/ inflating: package-pool/tar-1.30-5.ph3.x86_64.rpm inflating: package-pool/procps-ng-3.3.15-3.ph3.x86_64.rpm inflating: package-pool/irqbalance-1.4.0-5.ph3.x86_64.rpm inflating: package-pool/iptables-1.8.3-6.ph3.x86_64.rpm inflating: package-pool/device-mapper-event-2.02.187-2.ph3.x86_64.rpm inflating: package-pool/python3-prettytable-0.7.2-6.ph3.noarch.rpm inflating: package-pool/lsscsi-0.30-1.ph3.x86_64.rpm inflating: package-pool/python2-libs-2.7.17-7.ph3.x86_64.rpm inflating: package-pool/python3-xml-3.7.5-23.ph3.x86_64.rpm inflating: package-pool/python3-six-1.12.0-1.ph3.noarch.rpm inflating: package-pool/svadmin-rpm-3.3.7.0-21173100.noarch.rpm inflating: package-pool/python3-requests-2.24.0-1.ph3.noarch.rpm inflating: package-pool/libmnl-1.0.4-4.ph3.x86_64.rpm inflating: package-pool/cracklib-2.9.7-2.ph3.x86_64.rpm inflating: package-pool/sudo-1.9.5-5.ph3.x86_64.rpm inflating: package-pool/bzip2-1.0.8-2.ph3.x86_64.rpm inflating: package-pool/bc-1.07.1-3.ph3.x86_64.rpm inflating: package-pool/filesystem-1.1-4.ph3.x86_64.rpm inflating: package-pool/sed-4.5-2.ph3.x86_64.rpm * * * * inflating: package-pool/horizon-database-rpm-3.3.7.0-21173100.noarch.rpm inflating: package-pool/libcap-2.25-8.ph3.x86_64.rpm inflating: package-pool/python3-lxml-4.9.1-1.ph3.x86_64.rpm inflating: package-pool/python2-2.7.17-7.ph3.x86_64.rpm inflating: package-pool/horizon-preconfig-standard-3.3.7.0-21173100.noarch.rpm inflating: package-pool/libuv-1.34.2-2.ph3.x86_64.rpm The product RID matches so continue Continuing... Server is up, running the update root 18068 8.0 0.0 19112 14548 ? S 12:19 0:00 python -m SimpleHTTPServer 8008 root 18072 0.0 0.0 5484 1820 ? S 12:19 0:00 grep SimpleHTTPServer Checking website Website found, validating RID Website validated Installing custom provider-runtime tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified Updating update rpm to 3.3.7.0 3.3.7.0 tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified Getting the Current version... Checking Space FREESPACE 4715420 KB Checking for updates... /dualbootupdate.tar.gz Current version: 3.3.6.0 Updating all VMs to version: 3.3.7.0 Update started, current version : 3.3.6.0 Time : Sat Jul 22 12:20:01 UTC 2023 %_pkgverify_level digest %_pkgverify_level none Running preupdate Saving manifest.xml This is an LCM deployment. Assuming that the /usr/local/horizon/update/reindexingIndices.hzn script is already run and proceeding... Skipping managenodes.inc in case of Upgrade via LCM LCM added master hostname before upgrade : vidmone.cap.org You are required to change your password immediately (password expired) su: Authentication token is no longer valid; new one required (Ignored) ERROR: KDC configuration not yet initialized. Use the init subcommand. You are required to change your password immediately (password expired) su: Authentication token is no longer valid; new one required (Ignored) horizon-elasticsearch : normal uninstall horizon-elasticsearch : normal uninstall iNode usage on /var at % runSqlVerification 12:20:07.608 [main] INFO com.vmware.horizon.utils.KeystoreUtilities - Failed to load non-fips bc provider, keystore conversions will not be available. 12:20:08.283 [main] DEBUG com.vmware.horizon.utils.KeystoreUtilities - Loading BCFKS stream from BCFIPS SUCCESS: Connection verified for driver version: 42.2.12 Updating the vm. Saving manifest.xml Installing version - 3.3.7.0 Build 21173100 ..................................................Connection to sfcbd lost Attempting to reconnect: 1 .....................................................Waiting on upgrade process to complete: Vami upgrade process completed. Restoring default provider-runtime %_pkgverify_level digest %_pkgverify_level digest Running postupdate 0 false Updating IDM CA keystore with JRE certificates Import command completed: 151 entries successfully imported, 0 entries failed or cancelled Configuring opensearch.yml file : vidmone.cap.org Resetting RabbitMQ Upgrade provisioning adapters %d [%thread] %-5level %logger - %msg%n%d [%thread] %-5level %logger - %msg%nError : Error creating bean with name 'scopedTarget.cacheService' defined in com.vmware.horizon.cache.config.CacheServiceConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.horizon.cache.CacheService]: Factory method 'getCacheService' threw exception; nested exception is org.hibernate.exception.GenericJDBCException: could not prepare statement renamed '/usr/local/horizon/conf/iptables.cfg.rpmsave' -> '/usr/local/horizon/conf/iptables.cfg' <79>Jul 22 12:25:30 root: updateiptables.hzn Discovering IP Address Unable to retrieve IP list, using current ip list No current ip list defined, using own ip <79>Jul 22 12:25:30 root: updateiptables.hzn Processing port roles <79>Jul 22 12:25:31 root: updateiptables.hzn Setup iptables locally <79>Jul 22 12:25:31 root: setupIPTables.hzn begin iptable setup <79>Jul 22 12:25:31 root: setupIPTables.hzn allow any ip to access tcp ports: 22 443 5432 6443 7443 80 8080 8443 9000 9300 9694 9898 9999 <79>Jul 22 12:25:32 root: setupIPTables.hzn vApp access only to tcp ports: 40002 40003 5701 9300 <79>Jul 22 12:25:32 root: setupIPTables.hzn finish with static rules <79>Jul 22 12:25:33 root: setupIPTables.hzn save firewall rules <79>Jul 22 12:25:33 root: setupIPTables.hzn end iptable setup <79>Jul 22 12:25:33 root: updateiptables.hzn finish processing iptables Rebuilding manifest file Update completed, current version : 3.3.7.0 Time : Sat Jul 22 12:25:38 UTC 2023 Update complete, please reboot the VM. Important Logs during Upgrade vRealize Suite Lifecycle /var/log/vrlcm/vmware_vrlcm.log VMware Identity Manager /opt/vmware/var/log/vami/updatecli.log /opt/vmware/var/log/vami/vami.log /opt/vmware/horizon/vrslcm_vidm_upgrade.log

VMware Aria Suite Lifecycle allows us to download product binaries once we have "My VMware Account " mapped into it. What really happens in the backend? Where does it store the product binary? In short , when a download is triggered and mapping is done , there is a content id generated. This content id is something like the one shown below 2023-07-14 03:46:27.764 INFO [pool-3-thread-25] c.v.v.l.c.c.ContentRepositoryController - -- Created content with vmid :: ced3c5b4-1cdc-4107-899c-5261e949c67b. Once it's created for this specific binary download it is stored under /data/vm-config/vmrepo/productBinariesRepo/ Are you interested to know in depth , then let's jump into it. We will understand the whole process by looking into the logs The moment we click on ADD for any product binary,it can be an upgrade or install there is a validation done and a request to request service is generated ****** ADD is clicked here ****** 2023-07-14 03:32:56.415 INFO [http-nio-8080-exec-4] c.v.v.l.l.u.SettingInputValidationUtil - -- Product validation status : true 2023-07-14 03:32:56.415 INFO [http-nio-8080-exec-4] c.v.v.l.l.c.SettingsController - -- Validation result for Setting :myvmwaredownload result true 2023-07-14 03:32:56.416 INFO [http-nio-8080-exec-4] c.v.v.l.l.c.SettingsController - -- Setting Value before save: "" ****** Request to request service is generated ****** 2023-07-14 03:32:56.418 INFO [http-nio-8080-exec-4] c.v.v.l.l.u.RequestSubmissionUtil - -- ++++++++++++++++++ Creating request to Request_Service :::>>> { "vmid" : "9a42d32b-8866-47ab-ba5b-e3bb4e119393", "transactionId" : null, "tenant" : "default", "requestName" : "myvmwaredownload", "requestReason" : "VROPS 8.12.1 Type Install - My VMware Product Binary Download", "requestType" : "MYVMWARE_DOWNLOAD", "requestSource" : null, "requestSourceType" : "user", "inputMap" : { "productId" : "vrops", "productVersion" : "8.12.1", "productBinaryType" : "Install", "productBinaryPath" : null, "componentName" : null, "mappingType" : null, "productName" : "VMware Aria Operations", "requestId" : null, "removeBinary" : null }, "outputMap" : { }, "state" : "CREATED", "executionId" : null, "executionPath" : null, "executionStatus" : null, "errorCause" : null, "resultSet" : null, "isCancelEnabled" : null, "lastUpdatedOn" : 1689305576418, "createdBy" : null } As we can see from the request above, I am trying to download VMware Aria Operations 8.12.1 Install binary It identifies how many requests to be processed. There is an engine request created and it's set to IN PROGRESS 2023-07-14 03:32:56.432 INFO [http-nio-8080-exec-4] c.v.v.l.l.u.SettingsHelper - -- MyVMware download requestIds : 9a42d32b-8866-47ab-ba5b-e3bb4e119393 2023-07-14 03:32:56.830 INFO [scheduling-1] c.v.v.l.r.c.RequestProcessor - -- Processing request with ID : 9a42d32b-8866-47ab-ba5b-e3bb4e119393 with request type MYVMWARE_DOWNLOAD with request state INPROGRESS. The statemachine is now kicked in 2023-07-14 03:32:57.911 INFO [scheduling-1] c.v.v.l.a.c.FlowProcessor - -- Injected OnStart Edge for the Machine ID :: myvmwaredownload Validates MY VMware credentials 2023-07-14 03:32:58.007 INFO [pool-3-thread-20] c.v.v.l.p.c.m.t.StartGenericMyVMwareCredentialsTask - -- Starting :: Start Validating the MyVMware Credentials Starts downloading binaries after fetching the token for the user mapped under MY VMware account 2023-07-14 03:32:58.563 INFO [pool-3-thread-25] c.v.v.l.p.c.m.t.MyVmwareDownloadTask - -- Starting :: Download VMware Aria Product Binaries 2023-07-14 03:32:58.565 INFO [pool-3-thread-25] c.v.v.l.p.c.m.t.MyVmwareDownloadTask - -- Trying to get access token with user arun@arunnukula.com 2023-07-14 03:32:58.566 INFO [pool-3-thread-25] c.v.v.l.c.a.InternalOnlyApiAspect - -- Internal Only Check for: execution(ResponseEntity com.vmware.vrealize.lcm.locker.controller.CredentialController.getPassword(String)) 2023-07-14 03:32:58.568 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Get myvmware access token - Started 2023-07-14 03:32:58.569 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Myvmware base URL: https://apigw.vmware.com 2023-07-14 03:32:58.569 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Get myvmware access token. Current try number : '1' 2023-07-14 03:32:59.964 INFO [pool-3-thread-25] c.v.v.l.u.RestHelper - -- Status code : 200 2023-07-14 03:32:59.965 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Get myvmware access token - Finished 2023-07-14 03:32:59.966 INFO [pool-3-thread-25] c.v.v.l.p.c.m.t.MyVmwareDownloadTask - -- Successfully logged into my vmware with account username Checksum validation is done 2023-07-14 03:32:59.971 INFO [pool-3-thread-25] c.v.v.l.c.c.ContentDownloadController - -- INSIDE ContentDownloadControllerImpl 2023-07-14 03:32:59.971 INFO [pool-3-thread-25] c.v.v.l.c.c.ContentDownloadController - -- REPO_NAME :: /checksumRepo 2023-07-14 03:32:59.972 INFO [pool-3-thread-25] c.v.v.l.c.c.ContentDownloadController - -- CONTENT_PATH :: /vrops/8.12.1/checksum.json 2023-07-14 03:32:59.972 INFO [pool-3-thread-25] c.v.v.l.c.c.ContentDownloadController - -- URL :: /checksumRepo/vrops/8.12.1/checksum.json 2023-07-14 03:32:59.972 INFO [pool-3-thread-25] c.v.v.l.c.c.ContentDownloadController - -- Decoded URL :: /checksumRepo/vrops/8.12.1/checksum.json The initial phase of the download process begins 2023-07-14 03:33:12.263 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadUtil - -- *** Download requested VMware Aria Product : [VROPS-8.12.1]: OPERATIONS-8121 2023-07-14 03:33:12.263 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Get myvmware access token - Started 2023-07-14 03:33:12.264 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Myvmware base URL: https://apigw.vmware.com Download group is identified 2023-07-14 03:45:30.871 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadUtil - -- Download group to be checked OPERATIONS-8121 against OPERATIONS-8121. 2023-07-14 03:45:30.871 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadUtil - -- *** Binary file NAME match happened for : 'vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova' 2023-07-14 03:45:30.872 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadUtil - -- *** Binary file NAME match happened for : 'vRealize_Operations_Manager_With_CP-8.x-to-8.12.1.21952629.pak' 2023-07-14 03:45:30.872 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadUtil - -- *** Binary file NAME match happened for : 'vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova' Starts writing the binary to disk 2023-07-14 03:45:34.357 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Successfully created the metadata file '/data/myvmware/vrops/8.12.1/install/vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.metadata ' with content '886d6d4ce3ff995f1faa24f00e9aec86e2574477'. 2023-07-14 03:45:34.357 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Starting download of 'vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova' to '/data/myvmware/vrops/8.12.1/install' ... 2023-07-14 03:45:34.393 INFO [pool-3-thread-25] c.v.v.l.u.RestHelper - -- Status code : 200 2023-07-14 03:45:34.393 INFO [pool-3-thread-25] c.v.v.l.u.DownloadHelper - -- Started writing input stream to the file '/data/myvmware/vrops/8.12.1/install/vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova'. 2023-07-14 03:46:27.764 INFO [pool-3-thread-25] c.v.v.l.c.c.ContentRepositoryController - -- Created content with vmid :: ced3c5b4-1cdc-4107-899c-5261e949c67b. 2023-07-14 03:46:27.764 INFO [pool-3-thread-25] c.v.v.l.c.c.FileContentDatabase - -- SOURCE LOCATION :: /data/myvmware/vrops/8.12.1/install/vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova 2023-07-14 03:46:27.764 INFO [pool-3-thread-25] c.v.v.l.c.c.FileContentDatabase - -- content.getRepositoryIdentifier() :: productBinariesRepo 2023-07-14 03:46:27.766 INFO [pool-3-thread-25] c.v.v.l.c.c.FileContentDatabase - -- DESTINATION LOCATION :: productBinariesRepo 2023-07-14 03:46:27.768 INFO [pool-3-thread-25] c.v.v.l.c.c.FileContentDatabase - -- DESTINATION FOLDER LOCATION :: /data/vm-config/vmrepo/productBinariesRepo/ce/ced3c5b4-1cdc-4107-899c-5261e949c67b Download is now complete and uploaded to content repo 2023-07-14 03:51:02.014 INFO [pool-3-thread-25] c.v.v.l.c.c.ContentRepositoryController - -- Content uploaded successfully in the content repo 2023-07-14 03:51:02.016 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Download product binary file 'vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova' - Finished. 2023-07-14 03:51:02.028 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadUtil - -- Verifying if binary file 'vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova' of 'VROPS-8.12.1' is fully downloaded. 2023-07-14 03:51:02.151 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Validate Checksum disabled false 2023-07-14 03:51:02.152 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- >> PROCESSING 1 of 1. File Name : vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova 2023-07-14 03:51:02.152 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Starting :: checksum matching for : /data/myvmware/vrops/8.12.1/install/vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova 2023-07-14 03:51:02.155 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Extract directory for source file '/data/myvmware/vrops/8.12.1/install/vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova' is '/data/tempextract/9aeb0d09-be87-446e-bd33-14679eab6738/'. 2023-07-14 03:51:02.156 INFO [pool-3-thread-25] c.v.v.l.u.ShellExecutor - -- Executing shell command: tar -xf /data/myvmware/vrops/8.12.1/install/vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova -C /data/tempextract/9aeb0d0 9-be87-446e-bd33-14679eab6738/ --wildcards *.mf 2023-07-14 03:51:02.156 INFO [pool-3-thread-25] c.v.v.l.u.ProcessUtil - -- Execute tar 2023-07-14 03:51:02.291 INFO [pool-3-thread-25] c.v.v.l.u.ShellExecutor - -- Result: []. 2023-07-14 03:51:02.292 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Extract '/data/myvmware/vrops/8.12.1/install/vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova' to '/data/tempextract/9aeb0d09-be87-446e-bd 33-14679eab6738/' success. 2023-07-14 03:51:02.293 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Binary file NAME match happened for : /data/myvmware/vrops/8.12.1/install/vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova 2023-07-14 03:51:02.295 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Adding result record for :: vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova 2023-07-14 03:51:02.296 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- ProductId: vrops The temporary extract directory is deleted 2023-07-14 03:51:02.296 INFO [pool-3-thread-25] c.v.v.l.u.f.FileUtil - -- Directory is deleted : /data/tempextract/9aeb0d09-be87-446e-bd33-14679eab6738 2023-07-14 03:51:02.300 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Extract directory '/data/tempextract/9aeb0d09-be87-446e-bd33-14679eab6738/' for source file '/data/myvmware/vrops/8.12.1/install/vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova' is deleted. 2023-07-14 03:51:02.300 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadUtil - -- Checksum MATCHED for file vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova of VROPS-8.12.1. Along with VMware Aria Operations 8.12.1,we have the cloud proxy to be downloaded as well 2023-07-14 03:51:02.385 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Get product download url for 'vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova' - Started. 2023-07-14 03:51:04.855 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Get product download url for 'vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova' - Finished. 2023-07-14 03:51:04.856 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadUtil - -- HOST of final download URL :: https://download2.vmware.com 2023-07-14 03:51:04.856 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadUtil - -- SHA1 Checksum check for file 'vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova' with checksum value '193444c7b424f67ab1a46939e4f1a3d7aa6b85af ' - started 2023-07-14 03:51:04.858 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadUtil - -- SHA1 Checksum check for file 'vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova' with checksum value '193444c7b424f67ab1a46939e4f1a3d7aa6b85af ' - Finished 2023-07-14 03:51:04.858 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadUtil - -- Starting download of product binary file 'vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova'. Size is : '1.65 GB'. 2023-07-14 03:51:04.859 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Download product binary file 'vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova' - Started. 2023-07-14 03:51:04.860 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Download product binary file 'vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova'.Current try number : '1' 2023-07-14 03:51:04.860 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Get myvmware access token - Started 2023-07-14 03:51:04.861 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Myvmware base URL: https://apigw.vmware.com 2023-07-14 03:51:04.862 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Get myvmware access token. Current try number : '1' 2023-07-14 03:51:06.201 INFO [pool-3-thread-25] c.v.v.l.u.RestHelper - -- Status code : 200 2023-07-14 03:51:06.273 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Get myvmware access token - Finished 2023-07-14 03:51:06.275 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Successfully created the metadata file '/data/myvmware/vrops/8.12.1/install/vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.metadata' with content '193444c7b424f67ab1a46939e4f1a3d7aa6b85af'. 2023-07-14 03:51:06.277 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Starting download of 'vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova' to '/data/myvmware/vrops/8.12.1/install' ... Starts writing the download to the specific location 2023-07-14 03:51:06.275 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Successfully created the metadata file '/data/myvmware/vrops/8.12.1/install/vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.metadata' with content '193444c7b424f67ab1a46939e4f1a3d7aa6b85af'. 2023-07-14 03:51:06.277 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Starting download of 'vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova' to '/data/myvmware/vrops/8.12.1/install' ... 2023-07-14 03:51:06.363 INFO [pool-3-thread-25] c.v.v.l.u.RestHelper - -- Status code : 200 2023-07-14 03:51:06.363 INFO [pool-3-thread-25] c.v.v.l.u.DownloadHelper - -- Started writing input stream to the file '/data/myvmware/vrops/8.12.1/install/vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova'. 2023-07-14 03:51:40.944 INFO [pool-3-thread-25] c.v.v.l.u.DownloadHelper - -- Finished writing input stream to the file '/data/myvmware/vrops/8.12.1/install/vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova'. 2023-07-14 03:51:41.000 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Writing file to content repo with filename vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova. 2023-07-14 03:51:41.003 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Uploading product binary with filename cloudproxy.ova to content repo. 2023-07-14 03:51:41.003 INFO [pool-3-thread-25] c.v.v.l.c.c.ContentRepositoryController - -- Content delete requested for url /productBinariesRepo/vrops/8.12.1/install/cloudproxy.ova. 2023-07-14 03:51:41.520 INFO [pool-3-thread-25] c.v.v.l.c.c.ContentRepositoryController - -- Content delete is failed with given URL :: /productBinariesRepo/vrops/8.12.1/install/cloudproxy.ova 2023-07-14 03:51:41.521 INFO [pool-3-thread-25] c.v.v.l.c.c.ContentRepositoryController - -- Creating content operation. 2023-07-14 03:51:41.532 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- URL :: /productBinariesRepo/vrops/8.12.1/install/cloudproxy.ova 2023-07-14 03:51:41.536 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- URL :: /productBinariesRepo/vrops/8.12.1/install/cloudproxy.ova 2023-07-14 03:51:41.537 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- PATH LENGTH :: 6 2023-07-14 03:51:41.537 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- PATH LENGTH TEST PASSED 2023-07-14 03:51:41.539 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- SEARCHINE FOR :: REPO -> productBinariesRepo :: NAME -> __ROOT__ :KXKXKXKX PARENT -> 367eb197-9850-4013-b66e-e4c4077b3263 2023-07-14 03:51:41.550 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- PATH :: 2023-07-14 03:51:41.551 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- PATH ::productBinariesRepo 2023-07-14 03:51:41.551 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- PATH ::vrops 2023-07-14 03:51:41.551 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- PATH ::8.12.1 2023-07-14 03:51:41.551 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- PATH ::install 2023-07-14 03:51:41.551 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- PATH ::cloudproxy.ova 2023-07-14 03:51:41.551 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- ADDING NODE - PATH LENGTH :: 6 2023-07-14 03:51:41.551 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- SEARCHINE FOR :: REPO -> productBinariesRepo :: PARENT -> a530119f-d933-4391-9d62-6fe8d40d57f3 :: NAME -> vrops 2023-07-14 03:51:41.554 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- SEARCHINE FOR :: REPO -> productBinariesRepo :: PARENT -> 45ce3af0-0325-4439-af4d-3160366d8bfd :: NAME -> 8.12.1 2023-07-14 03:51:41.557 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- SEARCHINE FOR :: REPO -> productBinariesRepo :: PARENT -> d3e25c69-45db-4e05-b87c-cbf4aacbe24e :: NAME -> install 2023-07-14 03:51:41.559 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- SEARCHINE FOR :: REPO -> productBinariesRepo :: PARENT -> fbc115f0-dc23-4e8b-87a7-5d057f51cfec :: NAME -> cloudproxy.ova 2023-07-14 03:51:41.562 INFO [pool-3-thread-25] c.v.v.l.c.s.ContentDownloadUrlServiceImpl - -- ADDING NODE :: 5 2023-07-14 03:51:41.570 INFO [pool-3-thread-25] c.v.v.l.c.c.ContentRepositoryController - -- Created content with vmid :: 3a051861-9b68-43a2-aa65-71abc24aae59. 2023-07-14 03:51:41.570 INFO [pool-3-thread-25] c.v.v.l.c.c.FileContentDatabase - -- SOURCE LOCATION :: /data/myvmware/vrops/8.12.1/install/vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova 2023-07-14 03:51:41.571 INFO [pool-3-thread-25] c.v.v.l.c.c.FileContentDatabase - -- content.getRepositoryIdentifier() :: productBinariesRepo 2023-07-14 03:51:41.572 INFO [pool-3-thread-25] c.v.v.l.c.c.FileContentDatabase - -- DESTINATION LOCATION :: productBinariesRepo 2023-07-14 03:51:41.573 INFO [pool-3-thread-25] c.v.v.l.c.c.FileContentDatabase - -- DESTINATION FOLDER LOCATION :: /data/vm-config/vmrepo/productBinariesRepo/3a/3a051861-9b68-43a2-aa65-71abc24aae59 2023-07-14 03:51:43.267 INFO [http-nio-8080-exec-2] c.v.v.l.d.i.u.InventoryWriteUtil - -- QUERY MAP LENGTH :: 3 2023-07-14 03:51:43.271 INFO [http-nio-8080-exec-2] c.v.v.l.d.i.u.InventorySchemaQueryUtil - -- GETTING ROOT NODE FOR :: ProductInventory 2023-07-14 03:51:43.592 INFO [http-nio-8080-exec-3] c.v.v.l.d.i.u.InventoryWriteUtil - -- QUERY MAP LENGTH :: 3 2023-07-14 03:51:43.592 INFO [http-nio-8080-exec-3] c.v.v.l.d.i.u.InventorySchemaQueryUtil - -- GETTING ROOT NODE FOR :: ProductInventory 2023-07-14 03:51:46.066 INFO [pool-3-thread-25] c.v.v.l.c.c.ContentRepositoryController - -- Content uploaded successfully in the content repo 2023-07-14 03:51:46.066 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadRestClient - -- Download product binary file 'vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova' - Finished. 2023-07-14 03:51:46.067 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadUtil - -- Verifying if binary file 'vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova' of 'VROPS-8.12.1' is fully downloaded. 2023-07-14 03:51:46.069 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Validate Checksum disabled false 2023-07-14 03:51:46.069 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- >> PROCESSING 1 of 1. File Name : vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova 2023-07-14 03:51:46.069 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Starting :: checksum matching for : /data/myvmware/vrops/8.12.1/install/vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova 2023-07-14 03:51:46.069 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Extract directory for source file '/data/myvmware/vrops/8.12.1/install/vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova' is '/data/tempextract/ b2c7eb4f-e3cc-49bc-b0e8-225d4a9c08ec/'. 2023-07-14 03:51:46.231 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Extract '/data/myvmware/vrops/8.12.1/install/vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova' to '/data/tempextract/b2c7eb4f-e3cc-49bc-b0e8-225 d4a9c08ec/' success. 2023-07-14 03:51:46.231 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Binary file NAME match happened for : /data/myvmware/vrops/8.12.1/install/vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova 2023-07-14 03:51:46.233 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Adding result record for :: vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova 2023-07-14 03:51:46.235 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- ProductId: vrops 2023-07-14 03:51:46.235 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- File match found 2023-07-14 03:51:46.235 INFO [pool-3-thread-25] c.v.v.l.u.f.FileUtil - -- Directory is deleted : /data/tempextract/b2c7eb4f-e3cc-49bc-b0e8-225d4a9c08ec 2023-07-14 03:51:46.240 INFO [pool-3-thread-25] c.v.v.l.d.s.h.SourceMappingUtil - -- Extract directory '/data/tempextract/b2c7eb4f-e3cc-49bc-b0e8-225d4a9c08ec/' for source file '/data/myvmware/vrops/8.12.1/install/vRealize-Operations-Clo ud-Proxy-8.12.1.21952631_OVF10.ova' is deleted. 2023-07-14 03:51:46.240 INFO [pool-3-thread-25] c.v.v.l.d.m.h.MyVmwareDownloadUtil - -- Checksum MATCHED for file vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova of VROPS-8.12.1. 2023-07-14 03:51:46.270 INFO [pool-3-thread-25] c.v.v.l.p.c.m.t.MyVmwareDownloadTask - -- Source mapping results for myvmware download is ::: [ { "productId" : "vrops", "productVersion" : "8.12.1", "productBinaryType" : "install", "productBinaryPath" : "vRealize-Operations-Manager-Appliance-8.12.1.21952151_OVF10.ova", "componentName" : "default", "mappingType" : "MY VMware", "productName" : "VMware Aria Operations", "requestId" : null, "removeBinary" : null }, { "productId" : "vrops", "productVersion" : "8.12.1", "productBinaryType" : "install", "productBinaryPath" : "vRealize-Operations-Cloud-Proxy-8.12.1.21952631_OVF10.ova", "componentName" : "cloudproxy", "mappingType" : "MY VMware", "productName" : "VMware Aria Operations", "requestId" : null, "removeBinary" : null } ]

Upgrading infrastructure and management applications regularly offers several benefits: 1. Security : Newer versions often include security patches and fixes, protecting your systems against vulnerabilities and reducing the risk of cyberattacks or data breaches. 2. Performance: Upgrades often bring performance improvements, optimizing resource utilization, enhancing speed, and enabling your infrastructure to handle increased workloads efficiently. 3. Compatibility: As technology advances, software and hardware requirements change. Upgrading ensures compatibility with newer technologies, preventing compatibility issues and enabling seamless integration with other systems. 4. Features and Functionality: Upgrades frequently introduce new features, functionality, and enhancements that can improve productivity, streamline operations, and provide a competitive edge. Staying up to date allows you to take advantage of these advancements. 5. Support and Maintenance : Software vendors typically provide ongoing support and maintenance for the latest versions, ensuring you have access to assistance when needed. Older versions may have limited support or no longer receive updates. 6. Cost Savings: Maintaining outdated infrastructure and applications can become costly due to increased downtime, security risks, and inefficient processes. Regular upgrades can help minimize these risks and potential expenses. By keeping your infrastructure and management applications up to date, you can leverage the latest technology, enhance security, improve performance, and optimize your overall operations, ultimately benefiting your organization in the long run. VMware Aria Suite Lifecycle provides simplified lifecycle management through streamlined deployment,configuration,patching,upgrades,configuration management and decommissioning process. Suite Lifecycle (formerly vRSLCM) automates most of the repetitive tasks which avoids human errors. So, If you haven't started exploring VMware Aria Suite Lifecycle , it's right time to begin...... 👍

It was all normal till November 8, 2022 till my dad boarded Chicago flight from Abu Dhabi to visit my brother in Austin along with my mother. Flight was about to land in Chicago O'Hare in 30 minutes and all of a sudden he lost his geo-location sense and had a sense of fear. My mom was all alone with him, somehow they managed to get off Chicago flight but they were not allowed to board Austin flight due to aggressive & panicky behaviour of dad. No one knows what's happening at that instance. It gives me goosebumps when I imagine what my mom had to go through during that 12 hours. Fortunately they were already in US which allowed my brother to pick them up from Chicago. Over a period of next few weeks he totally stopped conversations , no appetite , he was lost in his own world. Few days later he was admitted at St. David's Round Rock Medical Center where every possible test was performed and everything turned out negative. They had to extract his cerebrospinal fluid for examination which was sent to Mayo Clinic. After few days at hospital he was discharged and based on doctors suggestion we decided to bring him back to India safely. December 26,2022 we took him to KIMS, Hyderabad for further examinations as we still didn't know the exact cause as we were waiting for results from Mayo Clinic,Cleaveland. Upon careful examination and few MRI scans, it was revealed that he had CJD (Creutzfeldt-Jakob Disease). This was something new to us. Doctors stated that he might have 6 months at hand but he was hardly alive for 40 days. CJD is a rare disease, one in a million gets this and in a million CJD patients , dad's condition was rarest as it was rapidly progressive. The reason I wanted to share this with you all is to bring awareness of CJD, it can happen all of a sudden and we won't even know the changes are occurring in our brain. Early hours of February 1, 2023 dad passed away peacefully with everyone of his family around him. Let's understand CJD in-depth and why does it occur. What's CJD CJD stands for Creutzfeldt-Jakob Disease. It is a rare and degenerative brain disorder that belongs to a group of diseases called prion diseases. Prions are abnormal proteins that can cause normal proteins in the brain to become misfolded, leading to the accumulation of damaged tissue. CJD affects a person's cognitive abilities, causing rapid mental deterioration, memory loss, personality changes, and difficulties with movement. It is a progressive and ultimately fatal condition. Why does it occur? Creutzfeldt-Jakob Disease (CJD) occurs due to the misfolding and accumulation of a specific protein called prion protein. The exact cause of this misfolding is not fully understood, but it is believed to be associated with spontaneous changes in the structure of the protein. In some cases, CJD can be inherited through mutations in the PRNP gene, which provides instructions for making the prion protein. Additionally, CJD can also be acquired through exposure to infected brain or nervous tissue, typically through medical procedures or contaminated surgical instruments. However, such cases are extremely rare. Overall, the underlying mechanisms and triggers for prion misfolding and the development of CJD are still an active area of research, and scientists continue to investigate these aspects in order to better understand and manage this complex disease. I am forever indebted to my uncle Dr Naresh Vadlamani . who was always there for us till the end to make right decisions and guided us appropriately. Columbus hospitals was his home for few days , thanks to all its staff who helped us in crisis. !!!! Bala Sista , Kiran Nukala , Divya . & Vaishnavi thank you !!!! !!! You'll be missed dad , your always with us !!!

A good product manager possesses several key qualities that contribute to their effectiveness in the role. Here are some important qualities of a good product manager: 1. Strong Leadership: A good product manager is capable of leading a cross-functional team, driving collaboration, and inspiring others to achieve shared goals. 2. Strategic Thinking: They have the ability to think strategically and make informed decisions based on market analysis, customer needs, and business goals. They can identify opportunities, set a clear product vision, and devise effective strategies to achieve it. 3. Customer-Centric Approach: A good product manager deeply understands customer needs, pain points, and desires. They prioritize user satisfaction and leverage customer feedback to make informed product decisions. 4. Excellent Communication: Effective communication is crucial for a product manager. They can articulate their vision, share product requirements, and collaborate with various stakeholders, including developers, designers, marketing teams, and executives. 5. Analytical Skills: They possess strong analytical skills to analyze data, identify trends, and make data-driven decisions. They can gather insights from market research, user feedback, and metrics to guide product improvements. 6. Adaptability and Flexibility: Good product managers can adapt to changing market conditions, prioritize tasks, and adjust their strategies accordingly. They are comfortable with ambiguity and can handle evolving priorities. 7. Technical Aptitude: While not always mandatory, a good product manager has a solid understanding of the technical aspects of their product. This helps them effectively communicate with development teams and make informed decisions. 8. Problem-Solving Abilities: They are adept at identifying problems, defining solutions, and overcoming obstacles. Good product managers approach challenges with a creative mindset and find innovative ways to solve them. 9. Empathy and Collaboration: They foster a collaborative and inclusive work environment, showing empathy towards team members' perspectives. They actively listen, understand others' viewpoints, and encourage teamwork. 10. Result Orientation: A good product manager is focused on delivering results and achieving measurable outcomes. They set clear goals, track progress, and take ownership of the product's success. These qualities collectively enable a good product manager to effectively lead product development, align teams, and deliver successful products that meet customer needs and drive business growth.

Upgrading VMware products frequently can have both advantages and challenges. Here are a few points to consider: Advantages: 1. Access to new features and enhancements: Frequent upgrades can provide you with the latest features, performance improvements, security updates, and bug fixes offered by VMware. 2. Better compatibility and support: Upgrading to newer versions can ensure compatibility with other software and hardware, and it may offer improved support from VMware. Challenges: 1. Time and resources: Frequent upgrades may require time and effort to plan, test, and implement, especially if you have a complex IT infrastructure. 2. Disruption to operations: Upgrading software can temporarily disrupt operations as you transition to the new version, potentially requiring downtime or impacting productivity. 3. Compatibility issues: There's a possibility of encountering compatibility issues with existing software, hardware, or custom configurations during the upgrade process. It's important to weigh the benefits against the potential challenges and consider factors such as your specific requirements, the stability of your current environment, and the significance of the new features or fixes in the newer versions.

There was a vRSLCM or VASL environment where the product UI wasn't loading at all after a reboot After checking logs , none of the application services were running as expected. Apart from bootstrap all of them were stopped or in "not running" status What could have happened ? How do i resolve this ? If i look closely , the postgres service is stopped. Unless postgres is up , none of the application services would start. So my starting point would be postgres. Executing status command on vposgres tells us the story /storage/db/pgdata ]# systemctl status vpostgres * vpostgres.service - VMware Postgres database server Loaded: loaded (/etc/systemd/system/vpostgres.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2023-06-29 15:25:54 UTC; 32s ago Process: 27267 ExecStart=/opt/vmware/vpostgres/current/bin/pg_ctl -s -D ${VMWARE_POSTGRES_DATA} -w -t ${VMWARE_POSTGRES_PGCTL_TIMEOUT} start (code=exited, status=1/FAILURE) Jun 29 15:25:54 lcm-fqdn systemd[1]: vpostgres.service: Service RestartSec=100ms expired, scheduling restart. Jun 29 15:25:54 lcm-fqdn systemd[1]: vpostgres.service: Scheduled restart job, restart counter is at 5. Jun 29 15:25:54 lcm-fqdn systemd[1]: Stopped VMware Postgres database server. Jun 29 15:25:54 lcm-fqdn systemd[1]: vpostgres.service: Start request repeated too quickly. Jun 29 15:25:54 lcm-fqdn systemd[1]: vpostgres.service: Failed with result 'exit-code'. Jun 29 15:25:54 lcm-fqdn systemd[1]: Failed to start VMware Postgres database server. root@lcm-fqdn [ /storage/db/pgdata ]# systemctl start vpostgres Job for vpostgres.service failed because the control process exited with error code. See "systemctl status vpostgres.service" and "journalctl -xe" for details. root@lcm-fqdn [ /storage/db/pgdata ]# Checking journalctl logs , we come to know why it was failing Reference Command : journalctl -u vpostgres.service --since "YYYY-MM-DD" Jun 29 12:18:22 lcm-fqdn systemd[1]: vpostgres.service: Control process exited, code=exited status=1 Jun 29 12:18:22 lcm-fqdn systemd[1]: vpostgres.service: Failed with result 'exit-code'. Jun 29 12:18:22 lcm-fqdn systemd[1]: Failed to start VMware Postgres database server. Jun 29 12:18:22 lcm-fqdn systemd[1]: vpostgres.service: Service RestartSec=100ms expired, scheduling restart. Jun 29 12:18:22 lcm-fqdn systemd[1]: vpostgres.service: Scheduled restart job, restart counter is at 1. Jun 29 12:18:22 lcm-fqdn systemd[1]: Stopped VMware Postgres database server. Jun 29 12:18:22 lcm-fqdn systemd[1]: Starting VMware Postgres database server... Jun 29 12:18:22 lcm-fqdn postgres[53444]: pg_ctl: directory "/var/vmware/vpostgres/current/pgdata" does not exist Jun 29 12:18:22 lcm-fqdn systemd[1]: vpostgres.service: Control process exited, code=exited status=1 Jun 29 12:18:22 lcm-fqdn systemd[1]: vpostgres.service: Failed with result 'exit-code'. Jun 29 12:18:22 lcm-fqdn systemd[1]: Failed to start VMware Postgres database server. Jun 29 12:18:22 lcm-fqdn systemd[1]: vpostgres.service: Service RestartSec=100ms expired, scheduling restart. Jun 29 12:18:22 lcm-fqdn systemd[1]: vpostgres.service: Scheduled restart job, restart counter is at 2. Jun 29 12:18:22 lcm-fqdn systemd[1]: Stopped VMware Postgres database server. The whole folder contents of /var/vmware/** was missing on the problematic environment On a working environment like my lab , this is how the folder structure would look Now , if there is no snapshot how do i fix this ? Fortunately , we have the database which was under /storage/db/pgdata and that was intact. So to fix , the thought process was to create missing folders and symbolic links and see if that helps. As during postgres startup it was complaining about only this issue. Here are the steps taken to fix the problem Step-1 : Take snapshot of vRSLCM / VASL appliance Step-2 : Create vpostgres directory mkdir /var/vmware/vpostgres Step-3 : Set appropriate permissions to vpostgres directory chmod 755 /var/vmware/vpostgres Step-4 : Under vpostgres directory create version folder , which is 11 mkdir /var/vmware/vpostgres/11 Step-5 : Assign postgres:users permission to the folder created chown postgres:users /var/vmware/vpostgres/11 Step-6 : Change permissions for version directory "11" chown 700 /var/vmware/vpostgres/11 Step-7 : Create symbolic link "11" as shown below ln -s /var/vmware/vpostgres/11 11 Step-8 : Browse to the directory "11" cd /var/vmware/vpostgres/11 Step-9 : Create directory pgdata mkdir /var/vmware/vpostgres/11/pgdata Step-10 : Browse to directory pgdata cd pgdata Step-11 : Assign postgres:users to pgdata folder chmod postgres:users pgdata Step-12 : Create symbolic link for pgdata which points to the actual db location ln -s /storage/db/pgdata pgdata Step-13 : Restart postgres service vRSLCM systemctl start vpostgres.service Step-13 : Restart postgres service vRSLCM / VASL systemctl stop vrlcm-server.service systemctl start vrlcm-server.service systemctl status vrlcm-server.service Step-14 : Monitor /var/log/vrlcm/vmware_vrlcm.log for status. Ideally if the services are up UI and the whole application should be up After executing these steps , vRSLCM or VASL UI is now up

vRLSCM 8.x deployment through Easy Installer or vCF might fail while initializing postgres database with below exception Below snippet is taken from /var/log/bootstrap/firstboot/firstboot.log 2022-06-22 11:45:03.719 UTC [3053] DEBUG: proc_exit(-1): 0 callbacks to make ok performing post-bootstrap initialization ... ok syncing data to disk ... ok WARNING: enabling "trust" authentication for local connections You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. Success. You can now start the database server using: /opt/vmware/vpostgres/current/bin/pg_ctl -D /storage/db/pgdata -l logfile start Warning: your password will expire in 5 days Failed to start vpostgres.service: Transaction is destructive. See system logs and 'systemctl status vpostgres.service' for details. Ensure PostgreSQL server is fully started... Warning: your password will expire in 5 days PSQL: Sleeping 2 seconds for Postgres to start fully... + set -eu + set -x + log 'main bootstrap firstboot started' ++ date '+%Y-%m-%d %H:%M:%S' + echo '2022-06-22 11:45:28 main bootstrap firstboot started' 2022-06-22 11:45:28 main bootstrap firstboot started + for script in "${BOOTSTRAP_DIR}"/* + echo When it works following output is expected 2022-06-13 01:37:56.259 UTC [3064] DEBUG: proc_exit(-1): 0 callbacks to make ok performing post-bootstrap initialization ... ok syncing data to disk ... ok WARNING: enabling "trust" authentication for local connections You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. Success. You can now start the database server using: /opt/vmware/vpostgres/current/bin/pg_ctl -D /storage/db/pgdata -l logfile start Ensure PostgreSQL server is fully started... List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+----------+----------+---------+-------+----------------------- postgres | postgres | UTF8 | C | C | template0 | postgres | UTF8 | C | C | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | C | C | =c/postgres + | | | | | postgres=CTc/postgres (3 rows) PostgreSQL server started. You are required to change your password immediately (password expired) su: Authentication token is no longer valid; new one required (Ignored) ALTER ROLE CREATE ROLE ALTER ROLE CREATE DATABASE password updated successfully The cause of the problem is that vmware-studio invokes firstboot script twice. This retriggers postgres configuration again while other one (original firstboot) was failed to configure fully. Note: "In case of VCF environment, execute steps 1 to 7" 1. SSH to SDDC Manager appliance and switch to root user 2. Open /etc/vmware/vcf/domainmanager/application-prod.properties in a text editor. 3. Add this line vrslcm.firstboot.timeout.period.in.minutes=15 4. Save the file and close the editor. 5. Restart the Domain Manager with: systemctl restart domainmanager 6. Wait 2 minutes for the service to restart. You can check status using: systemctl status domainmanager 7. Run the "Deploy vRealize Suite Lifecycle Manager" operation from the SDDC Manager UI. 8. Delete the postgres directory and run postgres configuration script. Incase of a vcf environment, customer has to perform this within 5mins or quicker. Once vRSLCM appliance gets deployed, login to Web Console from vCenter Server and perform following steps: 1. rm -rf /storage/db/pgdata/ 2. /etc/bootstrap/firstboot.d/02-configure-postgres.sh 3. /etc/bootstrap/firstboot.d/20-start-services 9. Reboot vRSLCM 10. Check vpostgres status service vpostgres status if in case admin@local password does not work , then use below command to reset it /opt/vmware/share/vami/vami-vlcm-passwd-reset Reboot Appliance and Login Verification After this workaround, vRSLCM UI will be up and running Check systemctl status vpostgres is in running state.

Before upgrading a vRealize product from one version to another, administrator needs to check how many phases or hops it might take for him to get to the desired version Also , check if the compatibility with the other vRealize products will remain intact. This used to be a manual effort from customer / administrator side and was little taxing on their part if you have large environments with multiple products involved Coming vRSLCM 8.8.2 , we introduced a new feature called Upgrade Planner There are two places where you would see an upgrade planner available Under Environments pane which helps customers or administrators to create a report if vRealize products are integrated between multiple vRSLCM environments For example , vRA might be a separate environment and vROPs managing or integrated with this vRA might be in another environment Note: Customer/Administrators should not deviate from the sequence of upgrades provided by Upgrade planner report as this will cause compatibility break Inside an Environment where administrators or customers would like to upgrade vRealize Products within itself You may download attached PDF which contains the procedure mentioned below if there is difficulty in reading screenshots Running an Upgrade Planner inside an vRSLCM environment Firstly , I upgraded my vRSLCM to 8.8.2 which provides me the new feature Upgrade Planner In my lab , apart from the globalenvironment which is my VMware Identity Manager , I do have 2 environments Environment called "Production" has multiple vRealize products in it vRealize Automation vRealize SaltStack Config vRealize Operations vRealize LogInsight vRealize Network Insight Environment called "sevensix" has one vRA 7.x product in it vRealize Automation 7.6 Now let's click on "Production" environment to get further details about it Now , let's click on "Upgrade Planner" , as shown below we are in an environment called "Production" All of the products which are available inside environment "Production" would be listed Upgrade Planner UI displays following data Product: vRealize product name Current Version: Current version of the product in this environment Latest Version: Latest version of the product which has been released by VMware Target Version: Target version of the product which is the desired one to go to If we examine this in detail , this is how we can interpret the UI Production Environment has ..... vRealize Automation 8.8.1 , whereas 8.8.2 is the latest version released. If i have to plan an upgrade , then the obvious target version would be 8.8.2 vRealize Log Insight 8.6.2 , whereas 8.8.2 is the latest version released. If i have to plan an upgrade , then the obvious target version would be 8.8.2 or 8.8.0 vRealize Operations 8.6.2 , whereas 8.6.3 is the latest version released. If i have to plan an upgrade , then the obvious target version would be 8.6.3 vRealize Network Insight 6.6.0 , whereas 6.7.0 is the latest version released. If i have to plan an upgrade , then the obvious target version would be 6.7.0 VMware Identity Manager 3.3.6 , whereas 3.3.6 is the latest version released. There is nothing to upgrade here vRealize Suite Lifecycle Manager 8.8.8 , whereas 8.8.2 is the latest version released. There is nothing to upgrade here The task at hand is to upgrade all of the eligible products to the latest ones. So let's generate our upgrade planner report Remember/Note: While upgrade planner is being generated do not navigate away from this page As seen in the above upgrade plan there is a need for 4 hops or windows needed for me to go from my existing versions to the target version ( latest available ) We can even download Upgrade Plan in PDF format so that it can be shared with wider group or stakeholders Now that we have "Upgrade Plan" , let's understand what it says Upgrade Plan presents me an optimal upgrade sequence of products in an environment. Out of all products selected in the environment , there are 4 eligible products which can be upgraded. vRA 8.8.1 to 8.8.2 and it takes 1 hop or window or attempt to reach target version vRLI 8.6.2 to 8.8.2 and it takes 1 hop or window or attempt to reach target version vROps 8.6.2 to 8.6.3 and it takes 1 hop or window or attempt to reach target version vRNI 6.6.0 to 6.7 and it takes 1 hop or window or attempt to reach target version Using this data , I can now confidently plan my upgrades Remember , Upgrade Planner will only show products which are not EOL or End of Life Running upgrade planner outside vRSLCM environment In this scenario , we will showcase how an upgrade planner can be leveraged outside of an environment This generic upgrade planner option gives us a way to change current version too as it's not dependent on environment Remember , end of life product releases will not be considered or even listed here Let's now generate an upgrade plan by selecting least possible versions of the product and the target version to be same as the latest version Select all of the products and then click on "Generate Upgrade Plan" If I go ahead with all minimum versions of the product then i'd be getting below exception which throws all the details regarding incompatibilities Let's say I have following environment and want to see upgrade path vRA 8.2 --> 8.8.2 vIDM 3.3.2 --> 3.3.6 vRSLCM 8.2 --> 8.8.2 Based on the previous data , here's the upgrade plan i get As seen in the above upgrade plan there is a need for 4 hops or windows needed for me to go from my existing versions to the target version ( latest available ) We can even generate a PDF out of this If we interpret or understand the report There are 4 windows / hops needed for us to reach target versions Window / Hop 1 1. Upgrade vRSLCM 8.2.0 to 8.4.1 2. Upgrade vIDM 3.3.2 to 3.3.4 3. Upgrade vRA 8.2.0 to 8.3.0 Window / Hop 2 1. Upgrade vIDM 3.3.4 to 3.3.5 2. Upgrade vRA 8.3.0 to 8.4.1 Windows / Hop 3 1. Upgrade vRSLCM 8.4.1 to 8.6.2 2. Upgrade vIDM 3.3.5 to 3.3.6 3. Upgrade vRA 8.4.1 to 8.6.2 Windows / Hop 4 1. Upgrade vRSLCM 8.6.2 to 8.8.2 2. Upgrade vRA 8.6.2 to 8.8.2 This is how an upgrade planner report can be generated and consumed in your planning. I have not written about vCF aware vRSLCM based environment's upgrade planner yet as i don't have a lab handy but from functionality wise it's pretty much the same