Search Results

Are you seeing similar exceptions in horizon.log? First step you do to troubleshoot elasticcache problem is to check if runtime-config.properties file is properly configured From first vRA appliance in a 3 node cluster # ehcache configuration properties ehcache.replication.rmi.registry.port=40002 ehcache.replication.rmi.remoteObject.port=40003 # Overrides the list of ehcache replication peers. FQDNs separated by ":", e.g. server1.example.com:server2.example.com ehcache.replication.rmi.servers=node2fqdn:node3fqdn From Second vRA appliance in a 3 node cluster # ehcache configuration properties ehcache.replication.rmi.registry.port=40002 ehcache.replication.rmi.remoteObject.port=40003 # Overrides the list of ehcache replication peers. FQDNs separated by ":", e.g. server1.example.com:server2.example.com ehcache.replication.rmi.servers=node1fqdn:node3fqdn From Third vRA appliance in a 3 node cluster # ehcache configuration properties ehcache.replication.rmi.registry.port=40002 ehcache.replication.rmi.remoteObject.port=40003 # Overrides the list of ehcache replication peers. FQDNs separated by ":", e.g. server1.example.com:server2.example.com ehcache.replication.rmi.servers=node1fqdn:node2fqdn Next , we did check if port connectivity is established and open node1:~ # curl -v telnet://node1fqdn:40003 * Rebuilt URL to: telnet://node1fqdn:40003/ * Trying 10.37.79.15... * TCP_NODELAY set * Connected to node1fqdn (XX.XX.XX.XX) port 40003 (#0) Ran elastic-search health-check and it's output was promising as well https://hostname/SAAS/API/1.0/REST/system/health/ Did approach Engineering and was suggested to perform following steps 1) Backup existing file: cp /opt/vmware/horizon/workspace/bin/setenv.sh /opt/vmware/horizon/workspace/bin/setenv_bak.sh 2) vi /opt/vmware/horizon/workspace/bin/setenv.sh 3) Import utils.inc file by adding this line in setenv.sh: . /usr/local/horizon/scripts/utils.inc 4) Search for JVM_OPTS in setenv.sh file and ensure you have this property set exactly like this: -Djava.rmi.server.hostname=$(myip) 5) Please repeat above steps for all appliances 6) Restart vIDM service on all appliances: service horizon-workspace restart By default this is how it looks... JVM_OPTS="-server -Djdk.tls.ephemeralDHKeySize=1024 -XX:+AggressiveOpts \ -XX:MaxMetaspaceSize=768m -XX:MetaspaceSize=768m \ -Xss1m -Xmx3419m -Xms2564m \ -XX:+UseParallelGC -XX:+UseParallelOldGC \ -XX:NewRatio=3 -XX:SurvivorRatio=12 \ -XX:+DisableExplicitGC \ -XX:+UseBiasedLocking -XX:-LoopUnswitching" and we need to change it to JVM_OPTS="-server -Djdk.tls.ephemeralDHKeySize=1024 -Djava.rmi.server.hostname=$(myip) -XX:+AggressiveOpts \ -XX:MaxMetaspaceSize=768m -XX:MetaspaceSize=768m \ -Xss1m -Xmx3419m -Xms2564m \ -XX:+UseParallelGC -XX:+UseParallelOldGC \ -XX:NewRatio=3 -XX:SurvivorRatio=12 \ -XX:+DisableExplicitGC \ -XX:+UseBiasedLocking -XX:-LoopUnswitching" After making the change on all available vRA appliances and restarting them. There were no more exceptions seen in horizon.log This file is setting the correct hostname and IP address in the java environment for the application to form a cluster correctly. This code has been already fixed in IDM and should be added in vRA 7.4 Finally , root-cause is that using IPv6 address in /etc/hosts file is not setting the hostname and ip-address correctly for the application. #vRealizeAutomation

When provisioning / destroy fails or is stuck due to whatever reason , there is every chance that we have to manually clean them. In our scenario, these stale entries were present under vRA Items -> Configuration Management Note : Before we get into steps to remove these entries from Database , we have to have a full backup of vRA vPostgres database Also ensure , Virtual Machine tagged to this entry is no longer present on the endpoint and managed by vRA Steps to remove these entries from Database After database backup has been taken, take a snapshot of vRA appliance Connect to vRA postgres database These entries would be present under cat_resource and cat_resource_owners tables Filter active Puppet entries using below query select * FROM public.cat_resource WHERE resourcetype_id = 'ConfigManagement.Puppet' AND status = 'ACTIVE'; One above query is executed, you would be presented with all active Puppet entries, this would match with the UI entries Before we delete entries from cat_resource , we need to remove references from cat_resource_owner Using the value present in id column of above query execute following query on cat_resource_owners select * FROM public.cat_resource_owners WHERE resource_id = 'XXXXX'; You would be presented with one result, then delete it delete * FROM public.cat_resource_owners WHERE resource_id = 'XXXXX'; Now for the resource_id which was used in the previous query to remove from cat_resource_owners , select the binding ID and then remove it from cat_resource table select * FROM public.cat_resource WHERE binding_id = 'YYYYY'; then delete this entry from database delete * FROM public.cat_resource WHERE binding_id = 'YYYYY'; Now refresh vRA portal , removed Puppet entry would be no longer present. #vRealizeAutomation

Came across a situation where even though the product was licensed , it was stating that it's not found in the database. Re-applying same license was not working and it was stating that it's invalid While going through /var/log/vmware/vcac/catalina.log found that there jdbc connection errors while trying to access database entry where licensed assets were present. Also, /var/log/vmware/messages was clearly stating that the license is missing or not found Follow below steps to delete an existing license in vRA ** Before Attempting these steps take a snapshot and a valid backup of vRA database ( vPostgres ) ** SSH or putty into the vRealize Automation appliance as root Take a backup of vRealize Automation Database Stop vRealize automation service service vcac-server stop Change directory using below command cd /tmp Run this command to create a copy of the database in /tmp su -m -c "/opt/vmware/vpostgres/current/bin/pg_dumpall -c -f /tmp/vcac.sql" postgres Run this command to compress database bzip2 -z /tmp/vcac.sql Connect to vPostgres database su -postgres psql vcac Verify if embeddedlicenseentry table is present in database \dt embeddedlicenseentry Review how many rows are present inside the table SELECT * FROM embeddedlicenseentry; There should be 36 ~ 37 entries Delete all information from embeddedlicenseentry table delete FROM embeddedlicenseentry; Verify if the table is now empty SELECT * FROM embeddedlicenseentry; Exit from Database \q Restart services on appliance vcac­vami service­manage stop vco­server vcac­server horizon­workspace elasticsearch vcac­vami service­manage start vco­server vcac­server horizon­workspace elasticsearch Once the Services are back , if we go back to VAMI portal , under vRA Settings --> Licensing , you should not see your previous license present Re-Apply your existing license it should be successful. For distributed environments 2 vRA appliances scenario:- Perform database on the Master node , you can get to know the MASTER from vRA Settings --> Database Restart services on both the nodes 3 vRA appliance scenario:- Change database to ASYNCHRONOUS mode Once done , shutdown both the non-MASTER applainces Since your left only with the MASTER node now , follow same steps as above Once the key is accepted , bring the other nodes online Once all services are registered , go ahead and change the database mode to SYNCHRONOUS Ensure , there are proper backups before performing any of these steps. #vRealizeAutomation

Pre-requisites Take a snapshot of all virtual appliances that make up your vROps cluster. Per VMware’s update instructions, you should uncheck the option to snapshot the virtual appliance memory and uncheck the option to quiesce the guest operating system Download the vRealize Operations Manager Virtual Appliance Update vRealize_Operations_Manager-VA-OS-6.7.0.8183616.pak vRealize_Operations_Manager-VA-6.7.0.8183616.pak Procedure One of the many advantages to using the virtual appliance deployment of vRealize Operations is that it’s not only easy to deploy but it’s super-easy to upgrade. Both the operating system AND the vRealize Operations application can be upgraded in a few clicks. Here’s how to do it, step by step Step 1 Login to the vROps admin interface at – https:///admin Step 2 Take the cluster offline Step 3 Install the Operating System Update Go to the Software Update panel and click Install a Software Update. Select the OS PAK update file that you downloaded, as in the graphic below and click Upload Once upload completes , staging starts Accept EULA Read Release and Update Information Click on Install Step 4 Monitor Update Progress After some time, the OS update will be complete and you can move on to the virtual appliance update Step 5 Install the vRealize Operations Manager virtual appliance update As in Step 5, you’ll go to the Software Update panel and click Install Software Update. Follow the same series of steps, shown above but provide the virtual appliance update PAK file this time. Click Upload. PAK upload in progress Staging in progress.... Read Release and Update information and click next .. Click on Install .... Installation in Progress... Step 6 Installation Completed Step 7 Finally , we have our new version of vRealize Operations 6.7

There might be situations where you would not be able to delete directory from vRA portal Recently, was working on a issue where vRA directory deletion process throws an exception First thing what come to one's mind when it's a vIDM or Directory Services issue in vRA is to check horizon.log Checking horizon.log , did see following exception 2018-03-16 03:11:15,121 ERROR (Thread-132) [vsphere.local;3f2680e5-99e4-457d-a080-e3b692c9fd62;10.2.14.176;] com.vmware.horizon.connector.management.DirectoryConfigOperationsImpl - Directory Delete Exception :d025cfc3-3949-4c4b-b630-1beb4b7170bd com.vmware.horizon.common.api.components.exceptions.GroupDeleteException: group.delete.error at com.vmware.horizon.components.authorization.GroupServiceImpl.deleteGroupsByUserStore(GroupServiceImpl.java:1004) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invoke vRA UI states it's System Exception and contact System Administrator We did resolve this issue by connecting to database and manually removing the config Pre-Requisites Before you do this ensure you have a valid snapshot and a recent backup of vRA Postgres database. Click here to learn how to take backup of vPostgres vRA database Once you have satisfied all pre-requisites , take ssh to vRA appliance2 . Connect to vRA database Execute su - postgres /opt/vmware/vpostgres/current/bin/psql vcac Now change schema to "saas" so that vIDM tables can be accessed set schema 'saas'; Find idOrganization and domain name Select * from "Organizations"; Select * from "Domain"; DELETE from “GroupParent” where “groupUuid” in ( select “uuid” from “Group” where “idOrganization” = ‘’ and “domain” = ‘’); DELETE FROM “UserGroup” where “idGroup” in ( select id from “Group” where “idOrganization” = '' and “domain” = ‘’); DELETE from “Group” where “idOrganization” = '' and “domain”=‘’; update “DirectoryConfig” set “deleteInProgress”=‘f’ where “id”=‘’; Delete directory from UI Ensure all of these steps are done in a proper manner. #vRealizeAutomation

The vRealize Automation appliance supports active-active high availability for all components except the appliance database. Starting with the 7.3 release, database failover is automatic if three nodes are deployed and synchronous replication is configured between two nodes. When vRealize Automation appliance detects database failure, it promotes a suitable database server to be the master. You can monitor and manage the appliance database on the Virtual Appliance Management Console vRA Settings > Database tab. vRealize Automation appliance provides the environment where services are running in a pre-configured and isolated environment. If several appliances are joined in a cluster following rules are valid to all of them On each node there is a equal set of running services All services share common database vRA, vRO and Horizon are accessed though external load balancer On one node the database is running in Master mode, while on the rest nodes databases are configured in Replica mode When joining a new node to cluster it accepts the same configuration as the rest nodes in the cluster Services running inside virtual appliance which are managed by automated HA deployment are vRA core services ( vcac-server ) , this one is the heart of vRA and starting this starts all services vRO ( vco-server and vco-configurator ) , vIDM ( horizon-workspace ) , provides authentication framework int he form of VMware Identity Manager which is embedded in appliance vPostgres (vpostgres) , embedded Postgres database that contains tables for vRA, vRO and vIDM Health (vrhb-service), reports functional health of select suite components that's vRA and vRO Setting these services in HA mode is part of automated process whern new vRA appliance is joined to already running vRA. Once the new appliance is joined to cluster, all services are automatically configured in cluster mode and synchronized against the cluster #vRealizeAutomation

Recently I did work on a project to upgrade vRealize Automation from 7.0.1 to 7.3. I would like to share my experience on how the upgrade took place and various issues encountered while implementing it We had to execute upgrade in two phases Upgrade from 7.0.1 to 7.2 Once 7.2 validation is complete upgrade to 7.3 This was done in two phases because in-place upgrade between 7.0.1 to 7.3 is not supported Pre-Requisites Ensure Databases have been backed up. Both IaaS ( Contact your DBA ) and vRA Postgres database. Take snapshots of all virtual machines ( vRA appliances and IaaS machines ). Ensure vRA components are in working and stable condition and correct all issues before starting upgrade Check for any space issues on root partition of appliances Check the /storage/log subfolder and remove any old archived ZIP files to cleanup space Ensure you have access to database , if you have restricted access then ensure DBA is around to help you during situation where you need to revert during failure System must be unavailable for users to submit new request and any applications that query vRealize Automation Upgrade from 7.0.1 to 7.2 Appliance Upgrade Create a file called disable-iaas-upgrade under /tmp on all vRA appliances as we do not want to perform an automated IaaS upgrade. ​/tmp/disable-iaas-upgrade Mount 7.2 vRA iso images to appliances (VMware-vR-Appliance-7.2.0.381-4660246-updaterepo.iso) Ensure under VAMI portal , Update Appliance settings are pointing towards CDRom updates Click on "Check Updates" , then you would get information on existing version and the version your upgrading to , accept and proceed for upgrade You may use following logs to see how upgrade is progressing ​/opt/vmware/var/log/vami/vami.log /opt/vmware/var/log/vami/updatecli.log /var/log/messages Once upgrade is complete, VAMI asks you to reboot appliances once done vRA appliances are upgraded to 7.2 IaaS Upgrade Ensure Java version is Java 8 build 91 or above Download IaaS installer from vRA VAMI ( https://vrapphostname:5480/installer) Execute IaaS install on all the IaaS nodes starting with ​IaaS Websites starting with the one where Model Manager Data is installed and then going ahead with other nodes Manager Services , ensure active one is upgraded before the passive node DEM Orchestrator and workers. Ensure you perform one box after another Agents , one node after another Management Agents on all nodes ( automatically upgraded ) , no manual intervention needed Check cluster status rabbitmqctl cluster_status Ensure all services are in "REGISTERED" status under VAMI portal of vRA nodes Upgrade vRO appliances if it's external Issues encountered during upgrade to 7.2 While starting vRA upgrade by mounting iso image during post upgrade phase of postgresql on appliances we did encounter failure Exception + echo 'Script /etc/bootstrap/postupdate.d/09-90-prepare-psql failed, error status 1' + exit 1 + exit 1 + trapfunc + excode=1 + test 1 -gt 0 + vami_update_msg set post-install 'Post-install: failed' + test -x /usr/sbin/vami-update-msg + /usr/sbin/vami-update-msg set post-install 'Post-install: failed' + sleep 1 + test 1 -gt 0 -o 0 -gt 0 + vami_update_msg set update-status 'Update failed (code 0-1). Check logs in /opt/vmware/var/log/vami or retry update later.' + test -x /usr/sbin/vami-update-msg + /usr/sbin/vami-update-msg set update-status 'Update failed (code 0-1). Check logs in /opt/vmware/var/log/vami or retry update later.' + exit 21/10/2017 01:12:41 [ERROR] Failed with exit code 256 21/10/2017 01:12:41 [INFO] Update status: Running VMware tools reconfiguration 21/10/2017 01:12:41 [INFO] Running /opt/vmware/share/vami/vami_reconfigure_tools vmware-toolbox-cmd is /usr/bin/vmware-toolbox-cmd Configuring VAMI VMware tools service wrapper. 21/10/2017 01:12:41 [INFO] Update status: Done VMware tools reconfiguration 21/10/2017 01:12:41 [INFO] Update status: Error while running post-install scripts 21/10/2017 01:12:41 [ERROR] Failure: updatecli exiting abnormally 21/10/2017 01:12:41 [INFO] Install Finished Resolution Ensure Clustering is working as expected between Primary and Secondary vRA nodes Exception Model Manager Data upgrade fails with an error stating [21/10/2017 8:09:09 PM] "E:\Program Files (x86)\VMware\vCAC\Server\Model Manager Data\RepoUtil.exe" Model-Uninstall -f DynamicOps.ManagementModel.dll -v [21/10/2017 8:09:11 PM] System.InvalidOperationException: File DynamicOps.ManagementModel.dll not found. Upgrade from 7.2 to 7.3 Resolution When above failure occurred , we had to revert IaaS node where Model Manager Data was and then the SQL server which was hosting the database Once done , we had to uninstall and reinstall MSDTC Post these steps , restarted upgrade which helped in fixing upgrade failure on IaaS Upgrade from 7.2 to 7.3 Appliance Upgrade Same steps as the previous upgrade to 7.2. Only difference is that when you start upgrading primary vRA node it automatically upgrades the second one as well. IaaS Upgrade Same steps as previous upgrade to 7.2 Issues encountered during upgrade to 7.3 Exception Though vRA upgrade was successful post reboot of the nodes or appliances apart from component registry , none of the services were showing up as registered Browsing through logs we did find out that horizon.log under vRA appliance ( Primary ) was full of postgres errors org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [spring/datastore-wireup.xml]: Invocation of init method failed; nested exception is liquibase.exception.MigrationFailedException: Migration failed for change set changelog-0008-2015-H2.xml::1::HHW-58146: Reason: liquibase.exception.DatabaseException: Error executing SQL ALTER TABLE saas."EncryptionKeys" DROP CONSTRAINT encryptionkeys_pkey: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist: Caused By: Error executing SQL ALTER TABLE saas."EncryptionKeys" DROP CONSTRAINT encryptionkeys_pkey: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist: Caused By: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist 2017-10-22 01:41:43,410 ERROR (localhost-startStop-1) [;;;] org.springframework.web.context.ContextLoader - Context initialization failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [spring/datastore-wireup.xml]: Invocation of init method failed; nested exception is liquibase.exception.MigrationFailedException: Migration failed for change set changelog-0008-2015-H2.xml::1::HHW-58146: Reason: liquibase.exception.DatabaseException: Error executing SQL ALTER TABLE saas."EncryptionKeys" DROP CONSTRAINT encryptionkeys_pkey: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist: Caused By: Error executing SQL ALTER TABLE saas."EncryptionKeys" DROP CONSTRAINT encryptionkeys_pkey: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist: Caused By: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) Resolution If you encounter this issue it does not mean that your upgrade failed. After the upgrade there are new liquibase files which are downloaded who performs DB changes if any between versions. The reason of failure is because CONSTRAINT was not present Check details below Login into vRA database set schema 'saas' \x to set expanded display ON \d+ "EncrytionKeys" Output of above commands would be Before you create a constraint you need to verify if there are duplicate key id's. If yes , then your alter command will fail with following error ALTER TABLE "EncryptionKeys" ADD CONSTRAINT "encryptionkeys_pkey" PRIMARY KEY(id); ERROR: could not create unique index "encryptionkeys_pkey" DETAIL: Key (id)=(1) is duplicated. So now identify number of id's present in this table. In my case there were 8 where first two had same key ID as described below id | 1 uuid | 6e4c6867-377e-4ce8-9f76-102da90951a7 keyContainer | __GLOBAL_REST_KEY_CONTAINER__ algorithmId | bbdec4a9-dbaa-49bd-95ed-11fd6cff26f3 keySize | 1024 id | 1 uuid | 6d1d29ae-4bb6-487e-a902-2ddb798f9c3c keyContainer | VSPHERE.LOCAL algorithmId | 1f932d40-0204-11e2-a21f-0800200c9a66 keySize | 256 Observe carefully id number is same. Select a number for ID which is not present in the output from previous command and then change the value of key id Select UUID to change the Key ​​select id,uuid from "EncryptionKeys" where uuid = '6e4c6867-377e-4ce8-9f76-102da90951a7'; Remember the value of uuid will vary from environment to environment Update UUID with the new key update saas."EncryptionKeys" set id=8 where uuid='6e4c6867-377e-4ce8-9f76-102da90951a7'; Verify if the new ID is updated select id,uuid from "EncryptionKeys" where uuid = '6e4c6867-377e-4ce8-9f76-102da90951a7'; Once done now run the Alter table command to create constraint for the table ALTER TABLE "EncryptionKeys" ADD CONSTRAINT "encryptionkeys_pkey" PRIMARY KEY(id); Once done verify this using \d+ "EncryptionKeys" command , you would see now the Primary Key is present. Restart vRA services so that all services will now be shown as "REGISTERED" successfully on both the nodes if distributed and on single node if simple installation This issue is pre-dominantly seen on version upgrades from 7.0 and 7.0.1 to 7.3 Happy Upgrades