Search Results

Last week was debugging a problem where user was trying to reconfigure a VM which was imported into vRA Version of vRA was 7.3.1 , There is a possibility for this issue to occur in any release prior to 7.4 Exception :- An item with the same key has already been added Reconfigure actions works as expected for all provisioned virtual machines from vRA , only bulk imported machines experience this problem Reason for Failure With every reconfigure , we do a sync on networks where in we check for network custom properties like VirtualMachine.Network0. While syncing the vRA uses a regex which is suppose to return a unique property key-value for the above property but instead returns more than one value if the machine properties were imported. This causes a .net exception which results in the request being marked as failed even though the reconfigure happens in vSphere Fix for this is present in vRealize Automation 7.4 Workaround Open IaaS database using SQL Management Studio and remove properties which begin with "__preregistration" First verify that these properties exist in the database SELECT * FROM VirtualMachineProperties WHERE PropertyName LIKE '__preregistration.%'; If these do exist, then remove them DELETE FROM VirtualMachineProperties WHERE PropertyName LIKE '__preregistration.%'; !! Hope this helps !!

At the end of this blog post, you should be able t Upgrade vRealize Suite Lifecycle Manager to a newer version Locate upgrade related log files Some useful links vRLCM Upgrade Paths Matrix vRLCM Product Interoperability Matrix Pre-Upgrade Tasks Prior to upgrading vRLCM to a newer version following checks and best practices must be followed Verify if Upgrade Path is supported Review Release Notes of the target version No vRLCM tasks must be in "In Progress" Virtual Hardware Requirements are met Pre-Upgrade Snapshot Mandatory Upgrading vRLCM An upgrade is performed by using Settings --> Update tab under vRLCM UI as shown under below screenshot. We can verify current version of vRLCM installed and select repository type to upgrade the appliance There are three types of Repository options Default repository is an online repository available through VMware over Internet. Needs internet access to use this option CD-ROM options is suitable if vRLCM does not have access to Internet. User has to download update ISO from VMware Downloads page and then attach it to the appliance via CD-ROM Custom Repository URL can be used to upgrade vRLCM to an intermediate release or to use a locally hosted repository containing the extracted contents of vRLCM upgrade iso files Then click on CHECK UPDATES , vRLCM responds back if the updates are available or not Post that click on INSTALL UPGRADES to start the upgrade. During upgrade vRLCM appliance will be rebooted and will be temporarily unavailable After the upgrade is successful, vRLCM displays the message stating it's upgraded successfully in the UI Monitoring Upgrade Since update process restarts vRLCM WebServices and restarts the appliance , the UPDATE tab of vRLCM UI does not provide live information on the progress of the update After the update installation is complete , refresh the page of the browser to see the results. If the update goes wrong , UI would not be available. At that point in time we need to ssh to the appliance and review /opt/vmware/var/log/vami/updatecli.log and find out what went wrong or the reason which led to the failure. Collect the log bundle before you rollback snapshot. Provide the bundle to VMware Support for analysis in case one need's any assistance. Post-Upgrade Tasks Configure new product binaries supported by upgraded version of vRLCM Perform acceptance testing to confirm vRLCM is working as expected When satisfied that the upgrade is successful , delete snapshot

We monitor /opt/vmware/var/log/vami/updatecli.log and upgrade-iaas.log during upgrade of vRealize Automation , it might be any version for that matter. Was working on an upgrade to vRA 7.5 today and decided to look behind the scenes on what exactly does it log under above mentioned logs. So let's start our deep-dive into them. Upgrade Starts Starts Running pre-install scripts Pre-Install is set to "IN PRORGESS" Then it disables database automatic failover ( SYNCHRONOUS TO ASYNCHRONOUS ) Run's abort-on-replica script , at this point it checks if the MASTER version is lesser that it's REPLICA version Then creates a copy the upgrade repository to another location so it can be used in postupgrade scripts. This happens only on the MASTER node The following script checks if the hardware resources are enough for the newly installable vRA version Checks vRA Replica hosts availability for source versions >= 7.1 , if Replica hosts are not available then it would throw an exception to fix them before the upgrade If check's for any vRO duplicates and if none if proceeds with the next step. If it finds any then it would delete them At this point it upgrades MANAGEMENT AGENTS on all IAAS nodes Post Management Agent's upgrade , pre-requisite checks on IAAS node's start. This is the point where it checks if we have included IAAS upgrade or we have excluded it and upgrading it manually. If it finds /tmp/disable-iaas-upgrade , then it's going to disable or skip all pre-req steps on IAAS nodes Identifies or Generates cluster node id Checks if vRB service is registered. If REGISTERED then it's going to let us know if that version of vRB is compatible with the version of vRA we are upgrading to. Else , it would let us know that's there is a compatibility issue but it would not stop the upgrade Validates if there are blueprints in the system that cannot be migrated automatically. If such blueprints are found the upgrade will be blocked Maps LB to localhost Kills all Java Processes. It executes vcac-server , vco-server and vco-configurator stop commands Cleans any temporary files under /var/lib/vco/* Formats /dev/sdd and moves database to it. /opt and /var are moved to the older db partition. Applies few fixes w.r.t extending partition Checks if /dev/sda is 50 GB and resize partitions according to the new existing space Kills Health Broker service and monitor before upgrading Stops IaaS services in the order: Agent, DemWorker, DemOrchestrator, ManagerService Stops vRA services on all the Replica hosts for source versions >= 7.1 Vacuums database only from the primary VA This script is used to dump vco database and import it to vcac. It's only for older versions This script is used to fix location where packages are downloaded because in 6.2.x and earlier 7.x ,versions there is not enough space in the root partition. If there is not enough space , then it's going to create one and move the content into it Uninstalls the artifactory rpms from the appliance This script is used to dump PostgreSQL databases in case of major upgrade. Checks if dump/restore is needed at all and exit if major versions are the same Executes a script which is a workaround for an issue when upgrading vmStudio from versions prior to 3.0.0.0. If the current vmStudio version is prior to 3.0.0.0, then forces deletetion of the vmware-studio-vami-cimom package Removes persistent net rules Artifactory uninstall fix starts Saves the JRE cacerts file as some of them are imported by horizon Removes any resource bundles Stop psql-manager service at the beginning of post update operations, as it might not be able to connect to the database, thus it will not be able to see that it is in async mode and will try to perform a reset. It will be started again at the end of the post update After PostgreSQL database is processed (still in preupdate step), this script will check if there is already exported database (which means major upgrade) then ,Will ensure PostgreSQL is stopped and delete existing data and server directory Prepares various services to stop It mark's Pre-Install tasks as complete Now it start's running installation tests Start's package installation Now that package installation is complete , it would start running post installation scripts Preserves DB settings , copies /etc/vcac/server.xml to /tmp Performs rpm status checks Stop psql-manager service at the beginning of post update operations, as it might not be able to connect to the database, thus it will not be able to see that it is in async mode and will try to perform a reset. It will be started again at the end of the post update. Creates a file /tmp/vra-upgrade-on.txt Checks if there are any external databases to be merged Ensures that all local users will not expire Ensure the keys are not already in the file /etc/sysctl.conf This script is used to fix location where packages are downloaded because in 6.2.x and earlier 7.x ,versions there is not enough space in the root partition.The script will not run (see package-pool.inc) if the /opt is symlink, which means it is moved to /storage/ext Stop PostgreSQL server and ensure data directory is on the correct partition, checks or set's the MASTER in Database Checks recovery.conf and updates other config files on postgres This script is used to restore PostgreSQL databases previously exported (in preupdate step) in case of major upgrade Starts sshd after the update if it is enabled Prepares required services Initialize users and generate encrypted pwd for administrator foe XENON. And it initializes XENON Initialize users and generate pswds for vrhb Performs cleanup of sandbox dir. In upgrade from previous system than 7.5 it's mandatory. Upgrade from 7.5 and later, sandbox folder will contains only the static UI files that are regenerated from the host Calls firstboot scripts for postgres clustering Only if D is external and local database is replica ( in the case of 6.3 with external lb ) , the db replica state will be cleared ,else it would exit Patches the rabbitmq scripts for the sed options Removing persistent net rules Prepares required vcac services Updates database and then creates tables used by vcac-config Removes truststore Adds vCO system properties in vmo.properties and enables vra mode in ControlCenter Reencrypts keystore password Changing the hzn master keystore password if it's been set to a default one Applies fix for issue with value: none for property certificate.store.ssl.rabbit.alias This script will disable particular PIDs from hardening scripts being invoked after upgrade Replaces the update URL Reconfigures vco Add additional lighttpd configurations directory Just logs version Removes old log file that is not used anymore - the same messages are in /var/log/vmware/vcac/vcac-config.log Executes set guest to export vami variables (they were not exported in the old versions) Deploys all vRA services to tomcat Remove the orig file created by the studio build process /etc/init.d/rc Updates the java timezone data Pinning telemetry log collection runs only on master. Disables them on Replica Setting up log symlinks for /storage/log/ Set's coredumps under /storage/core Fixes sshd config Edits sysctl.conf and pushes configurations into it Additional lighttpd configuration goes to the /opt/vmware/etc/lighttpd/conf.d directory - removes old config if there is any Configures allowed services under /etc/hosts.allow dodscript.sh makes a symlink for the /etc/issue file - does not overwrite it instead writes to /etc/issue.ORIG Set's and customizes grub timeout Fixes one of the vami_ovf bugs Fixes for another bug Fixes for another bug Disable screen blanking on tty1 Links vmware-rpctool where vami expects it Patches vami_set_hostname Another bug fix Add user root to wheel group otherwise it cannot login with SSH because of hardening scripts Fixes tcserver startup flles Disable chroot for ntpd Adds lighttpd headers Patches VAMI css Patches vami-deploy.xm Executes haproxy fix Enables haproxy Deletes postgres export directory Deletes legacy services Copies openscap branding Removes multiple tomcat servers if existed Applies default ciphers for SFCB server Starts psql manager Marks to trigger automated IAAS upgrade after node's reboot Checks RabbitMQ node health. Starts upgrade on the replica nodes or VA's. Now you would see that there would be no logging for a long time on the MASTER until the replica's are upgraded Deletes /tmp/disable-iaas-upgrade , if it was created before ( incase of manual iaas upgrade) + for script in '"${bootstrap_dir}"/*' Ciphers updated Flag set for vami_setnetwork Applies workaround for kernel hanging on non-available cdrom device Migrates custom groups if any Posts iaas upgrade messages "After all appliances are upgraded, ssh to the master appliance and go to /usr/lib/vcac/tools/upgrade and execute the "./upgrade" " "Wait for step *Post-install* to complete and then reboot the master appliance. After it is rebooted, IaaS nodes upgrade will commence automatically.Progress will be displayed on *this page* on the master appliance" All Appliances are now upgraded Completes Post Install scripts and finishes reconfiguration Finalizes Installation Complete Upgrade on Appliance successfully on MASTER and REPLICA's Now that Virtual Appliance Upgrade is complete , we will reboot the MASTER node. Once we reboot MASTER while it's starting up at a point where it brings up network interfaces and start's application services , it would initiate a automatic reboot on other REPLICA nodesOnce all Services on the MASTER node are REGISTERED , IAAS Upgrade would kick in IAAS Upgrade starts, upgrades components on my two of IAAS nodes Disables Maintenance mode on second IAAS node Upgrades DEM's Upgrades Proxy Agents Enables Manager Service Automatic Failover mode Finally completes the upgrade and restores Postgres Replication mode back to SYNCHRONOUS mode That's Curtains for your vRA Upgrade...... Download complete updatecli.log here

Recently, was working on a interesting problem with vRA 7.3 where user's request get's into IN PROGRESS state after a successful approval UI does not show any exceptions and you have to dig through the logs to understand what's going on One common point was all of these requests were submitted more than a week ago. Any request's submitted less than a week were successful as expected Let's take an example Request ID: 685 Submitted: 1/30/18 9:26 AM THAI ( 2:26 AM UTC ) Approved: 2/6/18 11:52 AM THAI ( 4:52 AM UTC) RESULT: stuck at IN PROGRESS 1. Request submitted catalina.out.1:2018-01-30 02:26:01,631 vcac: [component="cafe:composition-service" priority="INFO" thread="tomcat-http--46" tenant="scb" context="7fo7jfi1" parent="lDW4Osr4" token="sedoWsTv"] com.vmware.vcac.composition.service.impl.RequestServiceImpl.validate:889 - Received request for requestedObjectId scb!::!RHEL7forTestPending, catalogRequestId a09a9b4a-32da-4bb4-b754-a95f93f8dc38, requestNumber 685, resourceId null 2. RequestedItemApprovalId for this request was 063c130c-e105-43e9-b0a8-b413ca5e7e69 catalina.out.1:2018-01-30 02:26:03,710 vcac: [component="cafe:approvals" priority="INFO" thread="queue-pool-executer-2" tenant="scb" context="7fo7jfi1" parent="we7d9TMH" token="1kGg7dFR"] com.vmware.vcac.core.approvals.service.workitems.ApprovalRequestPublisherImpl.logSuccess:172 - ApprovalRequest [ApprovalRequestId="844a385a-8e07-42eb-90c7-0f805811b3c1" Name="Level#1: L1" Policy="AP Request VM"] : {Level#1: L1} - Created work item for approvers in Approval Request catalina.out.1:2018-01-30 02:26:03,717 vcac: [component="cafe:approvals" priority="INFO" thread="queue-pool-executer-2" tenant="scb" context="7fo7jfi1" parent="we7d9TMH" token="1kGg7dFR"] com.vmware.vcac.core.approvals.service.impl.ApprovalProcessor.processRequestedItemApprovalInternal:159 - RequestedItemApproval [RequestedItemApprovalId="063c130c-e105-43e9-b0a8-b413ca5e7e69" RequestedItemName="" RequestedFor=""] : Finished creating work item and/or notifying requesting service for Requested Item Approval 3. Approval is now complete after 7 days 2018-02-06 04:52:27,751 vcac: [component="cafe:approvals" priority="INFO" thread="queue-pool-executer-1" tenant="scb" context="G629x9nq" parent="69CYz90w" token="YPKbdSDF"] com.vmware.vcac.core.approvals.service.impl.DataUpdateCallbackServiceImpl.update:46 - RequestedItemApproval [RequestedItemApprovalId="063c130c-e105-43e9-b0a8-b413ca5e7e69" RequestedItemName="" RequestedFor=""] : Sending update for approval.. 2018-02-06 04:52:29,752 vcac: [component="cafe:catalog" priority="INFO" thread="tomcat-http--35" tenant="scb" context="G629x9nq" parent="LSHr7E7h" token="fh5G8UZV"] com.vmware.vcac.catalog.service.impl.RequestServiceImpl.requestApprovalEvent:694 - Request [RequestId ="a09a9b4a-32da-4bb4-b754-a95f93f8dc38" TenantName="" SubtenantName=""] : Received approval event {APPROVAL_COMPLETION} for Request with ApprovalId {42efc619-a4a6-43da-88e0-e17269453399}. 4. Final phases of approval 2018-02-06 04:52:30,004 vcac: [component="cafe:event-broker" priority="INFO" thread="ebs-queue-pool-executer-3" tenant="" context="" parent="" token=""] com.vmware.vcac.eventlog.auditing.saveEvent:90 - Approval [Id="42efc619-a4a6-43da-88e0-e17269453399"] : Notified the requesting service {d87206ce-d0e4-4f2b-85a5-601cd0323ed1} about the completion of Approval with final state {APPROVED} 5. Request now submitted to provider 2018-02-06 04:52:31,184 vcac: [component="cafe:catalog" priority="INFO" thread="queue-pool-executer-1" tenant="scb" context="G629x9nq" parent="fh5G8UZV" token="DuQcAKO8"] com.vmware.vcac.catalog.schema.ExternalRequestDataProvider.getData:74 - CatalogItemRequest [RequestId="a09a9b4a-32da-4bb4-b754-a95f93f8dc38" CatalogItemId="4eea22d6-899a-48f2-8b3b-4365184be78a" CatalogItemName="RHEL7 (for Test Pending)" ServiceName="Standard Blueprint" TenantName="SCB" SubtenantName="Test-BG"] : Retrieving field values from provider for component {} of request 6. Exception is seen here 2018-02-06 04:52:31,175 vcac: [component="cafe:catalog" priority="ERROR" thread="queue-pool-executer-4" tenant="scb" context="G629x9nq" parent="fh5G8UZV" token="6akaGyTr"] com.vmware.vcac.catalog.events.RequestLifecycleEventListener.requestApproved:102 - Request [RequestId ="a09a9b4a-32da-4bb4-b754-a95f93f8dc38" TenantName="" SubtenantName=""] : Error processing request approved event for request java.lang.IllegalArgumentException: Delegated token must be instance of class com.vmware.vcac.authentication.http.spring.oauth2.OAuth2Token: null at org.springframework.util.Assert.instanceCheckFailed(Assert.java:389) ~[spring-core-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.util.Assert.isInstanceOf(Assert.java:327) ~[spring-core-4.3.7.RELEASE.jar:4.3.7.RELEASE] at com.vmware.vcac.authentication.sts.OAuth2TokenService.propagatingAuthenticationWithDelegationFor(OAuth2TokenService.java:108) ~[platform-security-sso-client-7.3.0-SNAPSHOT.jar:?] at com.vmware.vcac.core.componentregistry.rest.client.EndPointAuthenticationManager.propagatingAuthenticationWithDelegationFor(EndPointAuthenticationManager.java:185) ~[component-registry-client-rest-service-7.3.0-SNAPSHOT.jar:?] at com.vmware.vcac.core.componentregistry.rest.client.SolutionRestClientManager.getAuthentication(SolutionRestClientManager.java:302) ~[component-registry-client-rest-service-7.3.0-SNAPSHOT.jar:?] Let me explain the reason for failure After approval service notifies catalog of approval completion, the catalog service tries to submit the request to the provider. This submission is eventually failing while getting client token. Error Message: [Delegated token must be instance of class com.vmware.vcac.authentication.http.spring.oauth2.OAuth2Token: null]. Token used in for the request is deleted after a certain period which is 7 days. Once deleted from database it cannot be retrieved. Hence the NULL value is passed and causes request to stuck at "IN PROGRESS" state Token are removed after 7 days or when the operation needing them is completed. This is not a bug but it's a product behavior. If you delay approvals for a long time, then you are bound to hit this issue If any administrator wants to extend the validity of the token used to invoke a request after approval s/he can set persistentTokenManager.staleTokenAgeDays=14 in /etc/vcac/vcac.properties This change to token age has to be done on all the vRA appliances Keep these points in mind 1. The vcac-server service needs to get restarted in order for the setting to be applied 2. Any requests created before the restart will still have the default validity (7 days) You may use this KB https://kb.vmware.com/s/article/2114385 to remove any IN PROGRESS or PENDING APPROVAL requests. If you would like to automate them then you got to build custom workflow.

Recently, bumped into an issue where users were trying to configure a blueprint to use ImportOvfWorkflow. When ImportOvfWorkflow is selected , we have to mention location where OVF/OVA is located. Post that when we click on configure , it was throwing an exception Note: URL field was intentionally removed due to security reasons Network connectivity from vRA appliance to the Repository/Fileserver was good , as we tested it using wget/curl by performing an ssh to appliance. Next thing we did is to check logs on what exactly it's saying Below snippets are taken from one of the vRA node's catalina.out ( /var/log/vmware/vcac/catalina.out ) Non-Working example [UTC:2018-09-19 10:30:23,841 Local:2018-09-19 16:00:23,841] vcac: [component="cafe:iaas-proxy" priority="INFO" thread="tomcat-http--38" tenant="nukescloud" context="vCyddMTo" parent="" token="vCyddMTo"] com.vmware.vcac.iaas.mapper.ovf.OvfXpathParser.retrieveOvfDescriptor:428 - Parsing OVA at url http://<>/<>.ova [UTC:2018-09-19 10:30:23,862 Local:2018-09-19 16:00:23,862] vcac: [component="cafe:iaas-proxy" priority="ERROR" thread="tomcat-http--38" tenant="nukescloud" context="vCyddMTo" parent="" token="vCyddMTo"] com.vmware.vcac.iaas.mapper.ovf.OvfXpathParser.mapOvfToMachine:220 - Could not map ova / ovf at http://<>/<>.ova, to a vRA Machine. [UTC:2018-09-19 10:30:23,863 Local:2018-09-19 16:00:23,863] vcac: [component="cafe:iaas-proxy" priority="ERROR" thread="tomcat-http--38" tenant="nukescloud" context="vCyddMTo" parent="" token="vCyddMTo"] com.vmware.vcac.iaas.service.impl.SourceVMServiceImpl.getOvfMachine:178 - Could not parse OVF / OVA at http://<>/<>.ova [UTC:2018-09-19 10:30:23,865 Local:2018-09-19 16:00:23,865] vcac: [component="cafe:iaas-proxy" priority="ERROR" thread="tomcat-http--38" tenant="nukescloud" context="vCyddMTo" parent="" token="vCyddMTo"] com.vmware.vcac.platform.service.rest.resolver.ApplicationExceptionHandler.handleNPE:454 - null java.lang.NullPointerException at com.vmware.vcac.iaas.mapper.ovf.OvfXpathParser.writeParseErrorToTelemetryLog(OvfXpathParser.java:780) ~[iaas-service-provider-7.4.0-SNAPSHOT.jar:?] at com.vmware.vcac.iaas.mapper.ovf.OvfXpathParser.mapOvfToMachine(OvfXpathParser.java:222) ~[iaas-service-provider-7.4.0-SNAPSHOT.jar:?] This is not a bug. If your OVA/OVF is valid then this exception will never occur Above exception states that it's not able successfully read / parse the OVA/OVF template to give an option to user to configure it to be used via vRA Blueprints Before even thinking of deploying OVA/OVA using vRA , test it directly by deploying it through vCenter to see if it works as expected. In this way you would save time and easily isolate the problem. #vRealizeAutomation

Was working on a brand new installation of vRA 7.4 where we were attempting to patch with the latest one under KB 56618 While implementing , it was throwing a straight exception right after the upload's completed Below was the exception in /var/log/vmware/vcac/vcac-config.log But actual exception on why the failure was under IAAS node's Management Agent All.log Failure was seen because IAAS was unable to fetch binaries from vRA Appliance On vRA Virtual Appliance nodes, open /etc/hosts file, find the entry for IPv4 loopback IP Address (127.0.0.1). Make sure that the ‘Fully Qualified Domain Name’ of the node immediately follows ‘127.0.0.1’, before ‘localhost’. ** Example: 127.0.0.1 FQDN_HOSTNAME_OF_NODE localhost In our case it was as below /etc/hosts was # VAMI_EDIT_BEGIN # Generated by Studio VAMI service. Do not modify manually. 127.0.0.1 localhost 10.36.19.50 vraapp01.pslab.org vraapp01 127.0.0.1 vraapp01.pslab.org load-balancer-host # VAMI_EDIT_END Actually it should be # VAMI_EDIT_BEGIN # Generated by Studio VAMI service. Do not modify manually. 127.0.0.1 vraapp01.pslab.org vraapp01 localhost ::1 vraapp01.pslab.org vraapp01 localhost ipv6-localhost ipv6-loopback # VAMI_EDIT_END 127.0.0.1 localhost.localdom 127.0.0.1 vrava-lb.pslab.org load-balancer-host This change has been documented in the Patch KB as well. Here are the screenshots for failure and successful patch installation #vRealizeAutomation

Just upgraded my lab to RA 7.5, so thought to blog on changes in VAMI from 7.4 to 7.5 TIP 1 When you take an SSH session to a vRA node, now it shows what's MASTER and what's REPLICA on the ssh session itself TIP 2 In Previous versions "Cluster" and "Database" settings were under vRA tab on VAMI portal. In 7.5 "Cluster" is now a separate tab which includes "Database" options as well VAMI 7.4 VAMI 7.5 As you can see above Cluster and Database settings are now merged. You have an additional option for taking a Database Dump. When we click on this option it would generate vRA appliance's postgres database dump under /tmp Dump would be in the format of vcac_dbdump_YYYY_MM_DD_HH_MM_SS.sql.zip This is a good feature from VMware Support perspective and can be taken without any downtime or stopping services. TIP 3 Support bundle option has now been removed from Cluster tab as it was on 7.4 and now moved under Logs tab in 7.5 Log Bundle collection in 7.4 Log Bundle Collection in 7.5 #vRealizeAutomation

vRealize Automation 7.5 released on September 20, 2018. Below is the list of new features and enhancements which came up with this release. Revamped UI and User Experience A complete new look and feel for vRealize Automation along with streamlined flows for common self-service tasks UI based on VMware Clarity standards Larger catalog cards show more of the description Cleaner catalog view Multiple instances of the same catalog item across business groups are now rolled up; the user selects the business group at request time Items and Requests tab merged into new Deployments tab Request details for decommissioned resources moved to the Administration tab Improved status of in-progress requests History view shows all requests associated with a single deployment over time Improved search capabilities across product menus and objects Contextual access to documentation from the product UI Home page and portlets are deprecated in this release Save button on requests is deprecated in this release Improved Integration with vRealize Operations Manager This release introduces deployment dashboards for application owners and enhancements to intelligent workload placement capabilities via integration with vRealize Operations. Show deployment alerts and key metrics (CPU, memory, IOPS, and network) for machines in the deployment details view Enable optimization of vRealize Automation managed workloads to align with vRealize Operations placement policy This builds on an earlier integration for optimizing initial placement to allow ongoing optimization of existing workload Configuration Automation Framework Native integration with external Ansible Tower Management tool. OOTB support for Ansible Tower as first class citizens in vRealize Automation Drag and drop Ansible Tower object in the Blueprint design canvas Parameterize and support early and late binding/request time Dynamically select Ansible job templates,including playbooks, for application configuration Support Day 2 actions to register or decommission machine Troubleshooting Improvements Improvements to Force Delete/Re-Submit ( failed or orphaned deployments) Post-Migration validation Consistent log tracing across solution Expose trace-id to the vRealize Orchestrator plug-in API vRO Database Clustering and Configuration vRO database configuration moved to vRA VAMI Embedded vRO database ( Postgres) is now able to be clustered and supports failover Microsoft Azure Blueprint Enhancements Support for Azure Managed Disks Enhanced Support for Azure regions NSX-T Datacenter Native Integration vRealize Automation now has native integration with NSX-T Datacenter OOTB support for NSX-T Datacenter as first class citizen in vRealize Automation Drag and Drop the following NSX-T Datacenter Services in blueprint design canvas Support for Day-2 actions Event Broker & Custom forms Improvements #vRealizeAutomation

There isn't a great deal of difference in deploying a simple instance of vRA 7.5. It's same as in it's previous versions. Sharing screenshots from my implementation in lab , as it might benefit others. Run Prerequisite Checks once it shows failure , click on Fix so that it runs script on IAAS node mapped to this appliance and get's it ready for the install Now once these pre-reqs are done , move on to enter vRA appliance hostname here and then click on Next Configure SSO ( administrator@vsphere.local ) Configure SQL Server Configure DEM's Configure Agents Generate Certificates Web and Manager are on same node , so no need of a separate certificate in this case ( simple install ) Post Certificates, It would bring up a screen to take snapshots and then the installation starts. It took around 30 minutes for the installation to complete Post Installation , one has to license vRA and it would present post-installation options That's it , it's so simple to install vRA 7.5 ( that's been the case since 7.x was launched ) Will explore various features in my upcoming posts... !! Stay Tuned !! #vRealizeAutomation

Once IaaS upgrade kicks-off in some environments you might experience an error stating it cannot continue upgrading ModelManagerWeb Exception goes as below Executing:E:\Program Files (x86)\VMware\vCAC\Server\Model Manager Data\Cafe\Vcac-Config.exe UpgradeArchiveDayTo62 -v[12:26:33.812] Error: [sub-thread-Id="6" context="" token=""] The archive period of resource NUKES02 is not up to date.VMware.Cafe.JsonResponseException: Child resource must have the same lease as its parent.PUT https://vrasimple/catalog-service/api/provider/providers/<>/resources/<> Request: { "id": "VirtualMachineID", "name": "NUKES02", "description": null, "resourceTypeId": "Infrastructure.Virtual", "catalogItemId": null, "catalogItemRequestId": "<<", "organization": { "tenantLabel": "nukes", "tenantRef": "nukes", * * * Response: {"errors":[{"code":20148,"source":null,"message":"Child resource must have the same lease as its parent.","systemMessage":"Child resource must have the same lease as its parent.","moreInfoUrl":null}]} at VMware.Cafe.JsonRestClient.d__2`1.MoveNext() --- End of stack trace from previous location where exception was thrown --- This exception occurs when there is a discrepancy between SQL and Postgres databases with respect to archive dates. Analysis You would have to find out from IAAS Node / ManagementAgent / All.log , how many Virtual Machines are at fault. Note down the names first , then execute query select * from dbo.VirtualMachine where VirtualMachineName = '<>,<>' Once we have VirtualMachineID of VirtualMachines showing discrepancy, keep an eye on "Expire Days" column in the output make a note of the value Login into vRA postgres database ​ su - postgres /opt/vmware/vpostgres/current/bin/psql vcac \x Then execute select * from cat_resource where name = 'vm01'; Examine archive_days value from the output Expire Days from SQL Database for this Virtual Machine and archive_days from Postgres should be same , else you would have problem during upgrade Remediation Now to come out of this problem, we need to update VirtualMachineID's of the virtual machines which are showing this discrepancy. We have the VirtualMachineID captured through the query shared above, so we need to just go ahead and update's it's value as it is shown in Postgres UPDATE VirtualMachine SET ExpireDays = 0 WHERE VirtualMachineID in ('XXXXX-XXXXX-XXXXX-XXXX') Once done go ahead and retry failed iaas upgrade and it should work like charm. Post this exception , only DEM Orchestrator and Agents would be pending. This error occurs when your almost done upgrading ModelManagerData\ #vRealizeAutomation

This article details on what steps are to be taken in order to migrate vRA IAAS database from one node to another There are three places where you need to make a change ManagerService.exe.config under ...\Program Files (x86)\VMware\vCAC\Server\ ( Manager Service nodes ) Web.Config under ...\Program Files (x86)\VMware\vCAC\Server\Model Manager Web\ ( Web Nodes ) IAAS database Note : If it's a distributed environment , then ensure you make changes across all nodes which have these components installed. Stop all IAAS related services across all nodes before starting this activity Step 1 Making changes to ManagerService.exe.config Take a backup of ManagerService.exe.config file Edit ManagerService.exe.config using word-pad , as seen in the screenshot below. Search for existing database server ,you would find database server name configured under , change the value to "Source=<>" to the new SQL server name 3. Save the file Step 2 Making changes to Web.config under Model Manager Web Take backup of Web.config There would be current database hostname under , edit that to new one and save the file Step 3 Making changes inside your IAAS database Connect to SQL Management Studio and open vRA IaaS database In the VMware vRealize Automation IAAS database on the new server, update the DynamicOps.RepositoryModel.Models table. This table contains loopback connection strings ( ConnectionString column) for each of the VMware vRealize Automation models that require updating with the new Data Source and Initial Catalog values. You will need to edit this table to replace the Data Source with your updated server FQDN and the Initial Catalog with your updated database name (if different). Verify existing values using following query SELECT * FROM [<>].[DynamicOps.RepositoryModel].[Models] Modify values using following query update [<>].[DynamicOps.RepositoryModel].[Models] set ConnectionString='Data Source=<>;Initial Catalog=<>;Integrated Security=True;Pooling=True;Max Pool Size=200;MultipleActiveResultSets=True;Connect Timeout=200' Start all IAAS related services across all nodes before starting this activity Post Change Validations Perform health-checks across on whole environment. Click Here for detailed instructions Ensure data-collection is working as expected Provision a VM and check if it goes through completely You might encounter following issues If there are errors seen similar to below stack Error processing workflow creation Error executing query usp_SearchInitializingRequestVirtualMachines Inner Exception: Error executing query usp_SelectGroup Then it's a problem with MSDTC. Most likely it would be because of SQL nodes. On SQL node you might see MSDTC exception as well 2018-08-21 05:41:31.620 spid62 Enlist operation failed: 0x8004d01c(XACT_E_CONNECTION_DOWN). SQL Server could not register with Microsoft Distributed Transaction Coordinator (MS DTC) as a resource manager for this transaction. The transaction may have been stopped by the client Reconfigure MSDTC on SQL node , these exceptions should stop occurring again. For MSDTC troubleshooting follow KB 2089503 #vRealizeAutomation

Unable to start rabbitmq after an outage? Are you seeing a similar exception as below 2018-07-26T09:39:17.273888+00:00 <> [cluster-rabbitmq-monitor] - ERROR - Rabbitmq reported unrecoverable state: [Error]: {could_not_start,rabbit, {{badmatch, {error, {{{badmatch, {error, {not_a_dets_file, "/var/lib/rabbitmq/mnesia/rabbit@<>/recovery.dets"}}}, [{rabbit_recovery_terms,open_table,0, [{file,"src/rabbit_recovery_terms.erl"},{line,126}]}, {rabbit_recovery_terms,init,1, [{file,"src/rabbit_recovery_terms.erl"},{line,107}]}, {gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,247}]}]}, {child,undefined,rabbit_recovery_terms, {rabbit_recovery_terms,start_link,[]}, transient,30000,worker, [rabbit_recovery_terms]}}}}, [{rabbit_queue_index,start,1, [{file,"src/rabbit_queue_index.erl"},{line,464}]}, {rabbit_variable_queue,start,1, [{file,"src/rabbit_variable_queue.erl"},{line,455}]}, {rabbit_priority_queue,start,1, [{file,"src/rabbit_priority_queue.erl"},{line,92}]}, {rabbit_amqqueue,recover,0, [{file,"src/rabbit_amqqueue.erl"},{line,239}]}, {rabbit,recover,0,[{file,"src/rabbit.erl"},{line,756}]}, {rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1, [{file,"src/rabbit_boot_steps.erl"},{line,49}]}, {rabbit_boot_steps,run_step,2, [{file,"src/rabbit_boot_steps.erl"},{line,49}]}, {rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1, [{file,"src/rabbit_boot_steps.erl"},{line,26}]}]}} 2018-07-26T09:39:17.898241+00:00 vasydp161 su: (to rabbitmq) root on /dev/pts/4 Above exception states that rabbitmq could not start as there was an exception reading recovery.dets file If you browse to /var/lib/rabbitmq/mnesia and perform ls -ltrh You would see that this file recovery.dets is corrupt or 0 bytes recovery.dets file contains recovery metadata if the node was stopped gracefully. There exists a high change of it's corruption if the node rabbitmq is stopped abruptly To remediate , delete or move this 0 byte file to another location ( eg. /tmp/ ) and then reboot the node , in this case vRealize Automation appliance Once done , during boot process we did see all services including rabbitmq started successfully. #vRealizeAutomation