Search Results

Understanding Reverse Proxy ( haproxy ) On vRealize Automation Appliance we have multiple services running simultaneously. An Open-Source application called HA Proxy is used to provide a reverse proxy on port 443 , routing traffic to each service appropriately. Each incoming request is analyzed on where it should be routed. Routing is determined by looking at the configuration files that contains a set of rules which live under /etc/haproxy/conf.d/ These rules outline What url should be directed where Any checks that needs to be done before routing traffic What to do by default In order to understand how reverse proxy functions in vRealize Automation , let's look at the configuration files When haproxy starts each *cfg file is loaded in alphanumeric order. The default and recommended naming convention is to start each file with 2 digit number. These file define what to do with each requests coming in. Let's look at /etc/haproxy/conf.d/20-vcac.cfg , we can learn how routing is determined The snippet shown above defines requests for orchestrator and health-checks , then states what to do with these requests. Lines 25 to 29 starting with acl says what URI to look for , for example /vco or /vcac/services/api/health If /vco is found then it would use backend-vro ( Line#33 ) If /vcac/services/api/health is found then backend-vra-health is used ( Line#34 ) If nothing is found or none of the conditions are met , then default backend is used that's backend-vra If we go down the configuration file , we can find the respective backend code as well. Line#66 in the above screenshot shows us backend-vra-health , Line#80 shows us backend-vro In the above example , if backend-vro is used then the traffic will be routed to port 8280 on the localhost Let's now see how Reverse Proxy is used in vRealize Automation with an example When a user navigates to https:///vcac/ a check is performed for a valid SAML token. In this case , if the user did not login into vRA for a while and does not have a valid token.Reverse Proxy , then forwards the connection to vIDM for authentication. When authentication is complete , a valid SAML token is cached into browser. Browser sends the login request again to Reverse Proxy , now since a valid SAML token is present it forwards connection to vRA CAFE If a valid SAML token was found in the first instance , then Reverse Proxy would have directly forwarded the connection to vRA CAFE Some requests like vRO API calls are done through URI. If request comes in like https:///vco , then Reverse Proxy forwards it to vRO directly Reverse Proxy Logging By default Reverse Proxy ( haproxy ) logging is disabled. In my experience, never came across any issues with this component of vRealize Automation. If you still want to enable logging for haproxy then we can accomplish it at /etc/haproxy/haproxy.cfg Note: Backup /etc/haproxy/haproxy.cfg before making any configuration changes to it To enable logging we need to add following lines to haproxy.cfg and then restart services stats enable stats uri /stats stats realm Strictly\ Private stats auth admin:VMware123! The first line enables stats view Second line set's access URI to stats , that means that there is a new view set at https:///stats Third line set's access to private Fourth line set's the username and password ( you may set password as you want ) Post these additions to the file , you would have to restart the haproxy service service haproxy restart Captured these steps in screenshot for reference To confirm if Reverse proxy is functional We can check it's service status that's service haproxy status Find Listening Ports and Running Process netstat -plnt | grep haproxy ps -ef | grep haproxy haproxy stats page

Test Connection Request while adding an endpoint fails with an exception Unable to connect to endpoint. Timeout occurred while processing the request Even though we verified firewall , ports , vCenter Services status still we were still encountering this exception. When we were looking into the logs For a working environment Manager Service \ All.log As seen below, when a Test Connection is initiated from vRA UI , Manager Service is responsible for initiating this request and it creates a workitem to be processed by underlying vSphereAgent [UTC:2018-11-05 09:25:45 Local:2018-11-05 09:25:45] [Debug]: [sub-thread-Id="67" context="gjbLTAc0" token="gx3pVepj"] TestConnection Request: Name [vc.nukescloud.com], Address: [https://vc.nukescloud.com/sdk], Username: [nukescloud\nukes] [UTC:2018-11-05 09:25:48 Local:2018-11-05 09:25:48] [Debug]: [sub-thread-Id="25" context="" token=""] TestConnection WorkItemResponse: [Test connection failed: Certificate is not trusted (RemoteCertificateChainErrors). Subject: C=US, CN=vc.nukescloud.com Thumbprint: 9C949FAEB87CBF97419CF4BFE70EB3FB8A035173INVALID_CERTIFICATE] vSphereAgent.log Below snippet is from vSphereAgent.log as you can see , once the workitem is created by Manager Service , vSphereAgent will process the workitem and fetch the certificate from endpoint. 2018-11-05T09:25:48.218Z IAAS75 vcac: [component="iaas:VRMAgent.exe" priority="Info" thread="4332"] [sub-thread-Id="5" context="" token=""] Starting : Processing Workitem ID [9bca4157-1fd3-4e19-a7dd-117aacc5e5d6] [testconnection] 2018-11-05T09:25:48.218Z IAAS75 vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="4332"] [sub-thread-Id="5" context="" token=""] [[]] [testconnection] TestConnection.Endpoint.Address=https://vc.nukescloud.com/sdk 2018-11-05T09:25:48.218Z IAAS75 vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="4332"] [sub-thread-Id="5" context="" token=""] [[]] [testconnection] TestConnection.Endpoint.Username=nukescloud\nukes 2018-11-05T09:25:48.218Z IAAS75 vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="4332"] [sub-thread-Id="5" context="" token=""] [[]] [testconnection] TestConnection.Endpoint.TrustThumbprint= 2018-11-05T09:25:48.218Z IAAS75 vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="4332"] [sub-thread-Id="5" context="" token=""] [[]] [testconnection] TestConnection.Endpoint.TrustAllCertificates=False 2018-11-05T09:25:48.218Z IAAS75 vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="4332"] [sub-thread-Id="5" context="" token=""] Begin test connection request.... 2018-11-05T09:25:48.234Z IAAS75 vcac: [component="iaas:VRMAgent.exe" priority="Warning" thread="5364"] [sub-thread-Id="10" context="" token=""] Invalid certificate found: C=US, CN=vc.nukescloud.com, Untrusted certificate chain 2018-11-05T09:25:48.234Z IAAS75 vcac: [component="iaas:VRMAgent.exe" priority="Warning" thread="5364"] [sub-thread-Id="10" context="" token=""] Invalid certificate found: C=US, CN=vc.nukescloud.com, Untrusted certificate chain 2018-11-05T09:25:48.249Z IAAS75 vcac: [component="iaas:VRMAgent.exe" priority="Error" thread="4332"] [sub-thread-Id="5" context="" token=""] false System.AggregateException: One or more errors occurred. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> DynamicOps.Common.Client.UntrustedCertificateException: Certificate is not trusted (RemoteCertificateChainErrors). Subject: C=US, CN=vc.nukescloud.com Thumbprint: 9C949FAEB87CBF97419CF4BFE70EB3FB8A035173 Above "Certificate is not trusted" message is not an exception but a message which states that certificate of the vCenter is not present in vRA yet. For non-working environment Manager Service \ All.log For the environment where we see exception , We so see test connection request being processed but there is no workitem generated for agent to go ahead and fetch certificate [UTC:2018-10-26 06:56:46 Local:2018-10-26 17:56:46] [Trace]: [sub-thread-Id="15" context="fW8t555E" token="3wP5GoMr"] TestConnection Request: Begin creating request record for request id [d2b48795-bdd2-4f9a-b585-2f8be4e33ac9] [UTC:2018-10-26 06:56:46 Local:2018-10-26 17:56:46] [Trace]: [sub-thread-Id="15" context="fW8t555E" token="3wP5GoMr"] TestConnection Request: Finish creating request record for request id [d2b48795-bdd2-4f9a-b585-2f8be4e33ac9] [UTC:2018-10-26 06:56:46 Local:2018-10-26 17:56:46] [Debug]: [sub-thread-Id="15" context="fW8t555E" token="3wP5GoMr"] TestConnection Request: Name [vcenter.nukescloud.com], Address: [https://vcenter.nukescloud.com/sdk], Username: [nukescloud\nukes] Apart from above three lines , there was no other information in the logs to tell us why and where it's stuck. After little bit of research found that dbo.Agent table found that there were multiple entries under AgentName for the same endpoint on the same Agent Machine , Refer the screenshot below. To come out of this problem , we followed these steps. Firstly , Backup IAAS database Then RDP to IAAS Agent nodes and then uninstall proxy agent for the endpoint which cannot be added Once , agent's are uninstalled , browse to the previously installed location and remove any existing traces ( there might be fe log files left ) Now , get back to SQL Database and then execute following query to remove any traces of the endpoint where it's failing to add from dbo.Agent table In my scenario vCenter was vcenter.nukescloud.com dbo.Agent table had following AgentID's for my endpoint vcenter.nukescloud.com AAAAA , CCCCC and DDDDD Now , we need to go ahead and delete these entries from Database, we may use below SQL query to do this delete from dbo.Agent where AgentID in ('AAAAA','CCCCC','DDDDD'); Post deletion , go ahead and reinstall the agent Once done your expected to see one entry for this endpoint and the agent in dbo.Agent table and it's AgentAlive value would be set to 1 As you can see above if it is not working it's AgentAlive status would be set to 0 Now adding endpoint through vRA UI , it should work as expected Reason why this exception occurs is very simple , when an agent is removed , it does not remove it from IaaS database. When we add an endpoint now , it always looks for the top entry for the agent in the table to assign a workitem. So it hit's the one which is not a proper proxy agent for this endpoint it would not be able to process it further causing a timeout in the UI So when Proxy Agent is uninstalled , ensure you perform a check in dbo.Agent table under IAAS database if it's completely removed and there isn't any stale entry || Happy Learning ||

Recently stumbled upon a problem where "Last Connected " status of IaaS nodes under VAMI was way off. Ideally "Last Connected" Status of any IaaS nodes should be under 10 ~ 15 minutes The first step to diagnose the problem was to look into the Management Agent/All.log of that node Found following exception , it was the only one inside these All.log's Exception :- [UTC:2018-10-28 23:45:35 Local:2018-10-29 10:45:35] [Error]: [sub-thread-Id="7" context="" token=""] Microsoft.Practices.Unity.ResolutionFailedException: Resolution of the dependency failed, type = "VMware.Cafe.IManagementEndpointClient", name = "(none)". Exception occurred while: Calling constructor VMware.Cafe.ManagementEndpointClient(System.Uri baseAddress, VMware.Cafe.ManagementEndpointSecurityContext authenticationContext, VMware.Cafe.TrustedCertificatePredicate trustCertificatePredicate). Exception is: CryptographicException - Keyset does not exist ----------------------------------------------- At the time of the exception, the container was: Resolving VMware.Cafe.ManagementEndpointClient,(none) (mapped from VMware.Cafe.IManagementEndpointClient, (none)) Calling constructor VMware.Cafe.ManagementEndpointClient(System.Uri baseAddress, VMware.Cafe.ManagementEndpointSecurityContext authenticationContext, VMware.Cafe.TrustedCertificatePredicate trustCertificatePredicate) ---> System.Security.Cryptography.CryptographicException: Keyset does not exist Management agent does not function properly when running under non-administrative account Management agent needs to be started using an administrative account. Even though you have it configured during installation if it is removed later you would end up in this problem.

Last week was debugging a problem where user was trying to reconfigure a VM which was imported into vRA Version of vRA was 7.3.1 , There is a possibility for this issue to occur in any release prior to 7.4 Exception :- An item with the same key has already been added Reconfigure actions works as expected for all provisioned virtual machines from vRA , only bulk imported machines experience this problem Reason for Failure With every reconfigure , we do a sync on networks where in we check for network custom properties like VirtualMachine.Network0. While syncing the vRA uses a regex which is suppose to return a unique property key-value for the above property but instead returns more than one value if the machine properties were imported. This causes a .net exception which results in the request being marked as failed even though the reconfigure happens in vSphere Fix for this is present in vRealize Automation 7.4 Workaround Open IaaS database using SQL Management Studio and remove properties which begin with "__preregistration" First verify that these properties exist in the database SELECT * FROM VirtualMachineProperties WHERE PropertyName LIKE '__preregistration.%'; If these do exist, then remove them DELETE FROM VirtualMachineProperties WHERE PropertyName LIKE '__preregistration.%'; !! Hope this helps !!

At the end of this blog post, you should be able t Upgrade vRealize Suite Lifecycle Manager to a newer version Locate upgrade related log files Some useful links vRLCM Upgrade Paths Matrix vRLCM Product Interoperability Matrix Pre-Upgrade Tasks Prior to upgrading vRLCM to a newer version following checks and best practices must be followed Verify if Upgrade Path is supported Review Release Notes of the target version No vRLCM tasks must be in "In Progress" Virtual Hardware Requirements are met Pre-Upgrade Snapshot Mandatory Upgrading vRLCM An upgrade is performed by using Settings --> Update tab under vRLCM UI as shown under below screenshot. We can verify current version of vRLCM installed and select repository type to upgrade the appliance There are three types of Repository options Default repository is an online repository available through VMware over Internet. Needs internet access to use this option CD-ROM options is suitable if vRLCM does not have access to Internet. User has to download update ISO from VMware Downloads page and then attach it to the appliance via CD-ROM Custom Repository URL can be used to upgrade vRLCM to an intermediate release or to use a locally hosted repository containing the extracted contents of vRLCM upgrade iso files Then click on CHECK UPDATES , vRLCM responds back if the updates are available or not Post that click on INSTALL UPGRADES to start the upgrade. During upgrade vRLCM appliance will be rebooted and will be temporarily unavailable After the upgrade is successful, vRLCM displays the message stating it's upgraded successfully in the UI Monitoring Upgrade Since update process restarts vRLCM WebServices and restarts the appliance , the UPDATE tab of vRLCM UI does not provide live information on the progress of the update After the update installation is complete , refresh the page of the browser to see the results. If the update goes wrong , UI would not be available. At that point in time we need to ssh to the appliance and review /opt/vmware/var/log/vami/updatecli.log and find out what went wrong or the reason which led to the failure. Collect the log bundle before you rollback snapshot. Provide the bundle to VMware Support for analysis in case one need's any assistance. Post-Upgrade Tasks Configure new product binaries supported by upgraded version of vRLCM Perform acceptance testing to confirm vRLCM is working as expected When satisfied that the upgrade is successful , delete snapshot

We monitor /opt/vmware/var/log/vami/updatecli.log and upgrade-iaas.log during upgrade of vRealize Automation , it might be any version for that matter. Was working on an upgrade to vRA 7.5 today and decided to look behind the scenes on what exactly does it log under above mentioned logs. So let's start our deep-dive into them. Upgrade Starts Starts Running pre-install scripts Pre-Install is set to "IN PRORGESS" Then it disables database automatic failover ( SYNCHRONOUS TO ASYNCHRONOUS ) Run's abort-on-replica script , at this point it checks if the MASTER version is lesser that it's REPLICA version Then creates a copy the upgrade repository to another location so it can be used in postupgrade scripts. This happens only on the MASTER node The following script checks if the hardware resources are enough for the newly installable vRA version Checks vRA Replica hosts availability for source versions >= 7.1 , if Replica hosts are not available then it would throw an exception to fix them before the upgrade If check's for any vRO duplicates and if none if proceeds with the next step. If it finds any then it would delete them At this point it upgrades MANAGEMENT AGENTS on all IAAS nodes Post Management Agent's upgrade , pre-requisite checks on IAAS node's start. This is the point where it checks if we have included IAAS upgrade or we have excluded it and upgrading it manually. If it finds /tmp/disable-iaas-upgrade , then it's going to disable or skip all pre-req steps on IAAS nodes Identifies or Generates cluster node id Checks if vRB service is registered. If REGISTERED then it's going to let us know if that version of vRB is compatible with the version of vRA we are upgrading to. Else , it would let us know that's there is a compatibility issue but it would not stop the upgrade Validates if there are blueprints in the system that cannot be migrated automatically. If such blueprints are found the upgrade will be blocked Maps LB to localhost Kills all Java Processes. It executes vcac-server , vco-server and vco-configurator stop commands Cleans any temporary files under /var/lib/vco/* Formats /dev/sdd and moves database to it. /opt and /var are moved to the older db partition. Applies few fixes w.r.t extending partition Checks if /dev/sda is 50 GB and resize partitions according to the new existing space Kills Health Broker service and monitor before upgrading Stops IaaS services in the order: Agent, DemWorker, DemOrchestrator, ManagerService Stops vRA services on all the Replica hosts for source versions >= 7.1 Vacuums database only from the primary VA This script is used to dump vco database and import it to vcac. It's only for older versions This script is used to fix location where packages are downloaded because in 6.2.x and earlier 7.x ,versions there is not enough space in the root partition. If there is not enough space , then it's going to create one and move the content into it Uninstalls the artifactory rpms from the appliance This script is used to dump PostgreSQL databases in case of major upgrade. Checks if dump/restore is needed at all and exit if major versions are the same Executes a script which is a workaround for an issue when upgrading vmStudio from versions prior to 3.0.0.0. If the current vmStudio version is prior to 3.0.0.0, then forces deletetion of the vmware-studio-vami-cimom package Removes persistent net rules Artifactory uninstall fix starts Saves the JRE cacerts file as some of them are imported by horizon Removes any resource bundles Stop psql-manager service at the beginning of post update operations, as it might not be able to connect to the database, thus it will not be able to see that it is in async mode and will try to perform a reset. It will be started again at the end of the post update After PostgreSQL database is processed (still in preupdate step), this script will check if there is already exported database (which means major upgrade) then ,Will ensure PostgreSQL is stopped and delete existing data and server directory Prepares various services to stop It mark's Pre-Install tasks as complete Now it start's running installation tests Start's package installation Now that package installation is complete , it would start running post installation scripts Preserves DB settings , copies /etc/vcac/server.xml to /tmp Performs rpm status checks Stop psql-manager service at the beginning of post update operations, as it might not be able to connect to the database, thus it will not be able to see that it is in async mode and will try to perform a reset. It will be started again at the end of the post update. Creates a file /tmp/vra-upgrade-on.txt Checks if there are any external databases to be merged Ensures that all local users will not expire Ensure the keys are not already in the file /etc/sysctl.conf This script is used to fix location where packages are downloaded because in 6.2.x and earlier 7.x ,versions there is not enough space in the root partition.The script will not run (see package-pool.inc) if the /opt is symlink, which means it is moved to /storage/ext Stop PostgreSQL server and ensure data directory is on the correct partition, checks or set's the MASTER in Database Checks recovery.conf and updates other config files on postgres This script is used to restore PostgreSQL databases previously exported (in preupdate step) in case of major upgrade Starts sshd after the update if it is enabled Prepares required services Initialize users and generate encrypted pwd for administrator foe XENON. And it initializes XENON Initialize users and generate pswds for vrhb Performs cleanup of sandbox dir. In upgrade from previous system than 7.5 it's mandatory. Upgrade from 7.5 and later, sandbox folder will contains only the static UI files that are regenerated from the host Calls firstboot scripts for postgres clustering Only if D is external and local database is replica ( in the case of 6.3 with external lb ) , the db replica state will be cleared ,else it would exit Patches the rabbitmq scripts for the sed options Removing persistent net rules Prepares required vcac services Updates database and then creates tables used by vcac-config Removes truststore Adds vCO system properties in vmo.properties and enables vra mode in ControlCenter Reencrypts keystore password Changing the hzn master keystore password if it's been set to a default one Applies fix for issue with value: none for property certificate.store.ssl.rabbit.alias This script will disable particular PIDs from hardening scripts being invoked after upgrade Replaces the update URL Reconfigures vco Add additional lighttpd configurations directory Just logs version Removes old log file that is not used anymore - the same messages are in /var/log/vmware/vcac/vcac-config.log Executes set guest to export vami variables (they were not exported in the old versions) Deploys all vRA services to tomcat Remove the orig file created by the studio build process /etc/init.d/rc Updates the java timezone data Pinning telemetry log collection runs only on master. Disables them on Replica Setting up log symlinks for /storage/log/ Set's coredumps under /storage/core Fixes sshd config Edits sysctl.conf and pushes configurations into it Additional lighttpd configuration goes to the /opt/vmware/etc/lighttpd/conf.d directory - removes old config if there is any Configures allowed services under /etc/hosts.allow dodscript.sh makes a symlink for the /etc/issue file - does not overwrite it instead writes to /etc/issue.ORIG Set's and customizes grub timeout Fixes one of the vami_ovf bugs Fixes for another bug Fixes for another bug Disable screen blanking on tty1 Links vmware-rpctool where vami expects it Patches vami_set_hostname Another bug fix Add user root to wheel group otherwise it cannot login with SSH because of hardening scripts Fixes tcserver startup flles Disable chroot for ntpd Adds lighttpd headers Patches VAMI css Patches vami-deploy.xm Executes haproxy fix Enables haproxy Deletes postgres export directory Deletes legacy services Copies openscap branding Removes multiple tomcat servers if existed Applies default ciphers for SFCB server Starts psql manager Marks to trigger automated IAAS upgrade after node's reboot Checks RabbitMQ node health. Starts upgrade on the replica nodes or VA's. Now you would see that there would be no logging for a long time on the MASTER until the replica's are upgraded Deletes /tmp/disable-iaas-upgrade , if it was created before ( incase of manual iaas upgrade) + for script in '"${bootstrap_dir}"/*' Ciphers updated Flag set for vami_setnetwork Applies workaround for kernel hanging on non-available cdrom device Migrates custom groups if any Posts iaas upgrade messages "After all appliances are upgraded, ssh to the master appliance and go to /usr/lib/vcac/tools/upgrade and execute the "./upgrade" " "Wait for step *Post-install* to complete and then reboot the master appliance. After it is rebooted, IaaS nodes upgrade will commence automatically.Progress will be displayed on *this page* on the master appliance" All Appliances are now upgraded Completes Post Install scripts and finishes reconfiguration Finalizes Installation Complete Upgrade on Appliance successfully on MASTER and REPLICA's Now that Virtual Appliance Upgrade is complete , we will reboot the MASTER node. Once we reboot MASTER while it's starting up at a point where it brings up network interfaces and start's application services , it would initiate a automatic reboot on other REPLICA nodesOnce all Services on the MASTER node are REGISTERED , IAAS Upgrade would kick in IAAS Upgrade starts, upgrades components on my two of IAAS nodes Disables Maintenance mode on second IAAS node Upgrades DEM's Upgrades Proxy Agents Enables Manager Service Automatic Failover mode Finally completes the upgrade and restores Postgres Replication mode back to SYNCHRONOUS mode That's Curtains for your vRA Upgrade...... Download complete updatecli.log here

Recently, was working on a interesting problem with vRA 7.3 where user's request get's into IN PROGRESS state after a successful approval UI does not show any exceptions and you have to dig through the logs to understand what's going on One common point was all of these requests were submitted more than a week ago. Any request's submitted less than a week were successful as expected Let's take an example Request ID: 685 Submitted: 1/30/18 9:26 AM THAI ( 2:26 AM UTC ) Approved: 2/6/18 11:52 AM THAI ( 4:52 AM UTC) RESULT: stuck at IN PROGRESS 1. Request submitted catalina.out.1:2018-01-30 02:26:01,631 vcac: [component="cafe:composition-service" priority="INFO" thread="tomcat-http--46" tenant="scb" context="7fo7jfi1" parent="lDW4Osr4" token="sedoWsTv"] com.vmware.vcac.composition.service.impl.RequestServiceImpl.validate:889 - Received request for requestedObjectId scb!::!RHEL7forTestPending, catalogRequestId a09a9b4a-32da-4bb4-b754-a95f93f8dc38, requestNumber 685, resourceId null 2. RequestedItemApprovalId for this request was 063c130c-e105-43e9-b0a8-b413ca5e7e69 catalina.out.1:2018-01-30 02:26:03,710 vcac: [component="cafe:approvals" priority="INFO" thread="queue-pool-executer-2" tenant="scb" context="7fo7jfi1" parent="we7d9TMH" token="1kGg7dFR"] com.vmware.vcac.core.approvals.service.workitems.ApprovalRequestPublisherImpl.logSuccess:172 - ApprovalRequest [ApprovalRequestId="844a385a-8e07-42eb-90c7-0f805811b3c1" Name="Level#1: L1" Policy="AP Request VM"] : {Level#1: L1} - Created work item for approvers in Approval Request catalina.out.1:2018-01-30 02:26:03,717 vcac: [component="cafe:approvals" priority="INFO" thread="queue-pool-executer-2" tenant="scb" context="7fo7jfi1" parent="we7d9TMH" token="1kGg7dFR"] com.vmware.vcac.core.approvals.service.impl.ApprovalProcessor.processRequestedItemApprovalInternal:159 - RequestedItemApproval [RequestedItemApprovalId="063c130c-e105-43e9-b0a8-b413ca5e7e69" RequestedItemName="" RequestedFor=""] : Finished creating work item and/or notifying requesting service for Requested Item Approval 3. Approval is now complete after 7 days 2018-02-06 04:52:27,751 vcac: [component="cafe:approvals" priority="INFO" thread="queue-pool-executer-1" tenant="scb" context="G629x9nq" parent="69CYz90w" token="YPKbdSDF"] com.vmware.vcac.core.approvals.service.impl.DataUpdateCallbackServiceImpl.update:46 - RequestedItemApproval [RequestedItemApprovalId="063c130c-e105-43e9-b0a8-b413ca5e7e69" RequestedItemName="" RequestedFor=""] : Sending update for approval.. 2018-02-06 04:52:29,752 vcac: [component="cafe:catalog" priority="INFO" thread="tomcat-http--35" tenant="scb" context="G629x9nq" parent="LSHr7E7h" token="fh5G8UZV"] com.vmware.vcac.catalog.service.impl.RequestServiceImpl.requestApprovalEvent:694 - Request [RequestId ="a09a9b4a-32da-4bb4-b754-a95f93f8dc38" TenantName="" SubtenantName=""] : Received approval event {APPROVAL_COMPLETION} for Request with ApprovalId {42efc619-a4a6-43da-88e0-e17269453399}. 4. Final phases of approval 2018-02-06 04:52:30,004 vcac: [component="cafe:event-broker" priority="INFO" thread="ebs-queue-pool-executer-3" tenant="" context="" parent="" token=""] com.vmware.vcac.eventlog.auditing.saveEvent:90 - Approval [Id="42efc619-a4a6-43da-88e0-e17269453399"] : Notified the requesting service {d87206ce-d0e4-4f2b-85a5-601cd0323ed1} about the completion of Approval with final state {APPROVED} 5. Request now submitted to provider 2018-02-06 04:52:31,184 vcac: [component="cafe:catalog" priority="INFO" thread="queue-pool-executer-1" tenant="scb" context="G629x9nq" parent="fh5G8UZV" token="DuQcAKO8"] com.vmware.vcac.catalog.schema.ExternalRequestDataProvider.getData:74 - CatalogItemRequest [RequestId="a09a9b4a-32da-4bb4-b754-a95f93f8dc38" CatalogItemId="4eea22d6-899a-48f2-8b3b-4365184be78a" CatalogItemName="RHEL7 (for Test Pending)" ServiceName="Standard Blueprint" TenantName="SCB" SubtenantName="Test-BG"] : Retrieving field values from provider for component {} of request 6. Exception is seen here 2018-02-06 04:52:31,175 vcac: [component="cafe:catalog" priority="ERROR" thread="queue-pool-executer-4" tenant="scb" context="G629x9nq" parent="fh5G8UZV" token="6akaGyTr"] com.vmware.vcac.catalog.events.RequestLifecycleEventListener.requestApproved:102 - Request [RequestId ="a09a9b4a-32da-4bb4-b754-a95f93f8dc38" TenantName="" SubtenantName=""] : Error processing request approved event for request java.lang.IllegalArgumentException: Delegated token must be instance of class com.vmware.vcac.authentication.http.spring.oauth2.OAuth2Token: null at org.springframework.util.Assert.instanceCheckFailed(Assert.java:389) ~[spring-core-4.3.7.RELEASE.jar:4.3.7.RELEASE] at org.springframework.util.Assert.isInstanceOf(Assert.java:327) ~[spring-core-4.3.7.RELEASE.jar:4.3.7.RELEASE] at com.vmware.vcac.authentication.sts.OAuth2TokenService.propagatingAuthenticationWithDelegationFor(OAuth2TokenService.java:108) ~[platform-security-sso-client-7.3.0-SNAPSHOT.jar:?] at com.vmware.vcac.core.componentregistry.rest.client.EndPointAuthenticationManager.propagatingAuthenticationWithDelegationFor(EndPointAuthenticationManager.java:185) ~[component-registry-client-rest-service-7.3.0-SNAPSHOT.jar:?] at com.vmware.vcac.core.componentregistry.rest.client.SolutionRestClientManager.getAuthentication(SolutionRestClientManager.java:302) ~[component-registry-client-rest-service-7.3.0-SNAPSHOT.jar:?] Let me explain the reason for failure After approval service notifies catalog of approval completion, the catalog service tries to submit the request to the provider. This submission is eventually failing while getting client token. Error Message: [Delegated token must be instance of class com.vmware.vcac.authentication.http.spring.oauth2.OAuth2Token: null]. Token used in for the request is deleted after a certain period which is 7 days. Once deleted from database it cannot be retrieved. Hence the NULL value is passed and causes request to stuck at "IN PROGRESS" state Token are removed after 7 days or when the operation needing them is completed. This is not a bug but it's a product behavior. If you delay approvals for a long time, then you are bound to hit this issue If any administrator wants to extend the validity of the token used to invoke a request after approval s/he can set persistentTokenManager.staleTokenAgeDays=14 in /etc/vcac/vcac.properties This change to token age has to be done on all the vRA appliances Keep these points in mind 1. The vcac-server service needs to get restarted in order for the setting to be applied 2. Any requests created before the restart will still have the default validity (7 days) You may use this KB https://kb.vmware.com/s/article/2114385 to remove any IN PROGRESS or PENDING APPROVAL requests. If you would like to automate them then you got to build custom workflow.

Recently, bumped into an issue where users were trying to configure a blueprint to use ImportOvfWorkflow. When ImportOvfWorkflow is selected , we have to mention location where OVF/OVA is located. Post that when we click on configure , it was throwing an exception Note: URL field was intentionally removed due to security reasons Network connectivity from vRA appliance to the Repository/Fileserver was good , as we tested it using wget/curl by performing an ssh to appliance. Next thing we did is to check logs on what exactly it's saying Below snippets are taken from one of the vRA node's catalina.out ( /var/log/vmware/vcac/catalina.out ) Non-Working example [UTC:2018-09-19 10:30:23,841 Local:2018-09-19 16:00:23,841] vcac: [component="cafe:iaas-proxy" priority="INFO" thread="tomcat-http--38" tenant="nukescloud" context="vCyddMTo" parent="" token="vCyddMTo"] com.vmware.vcac.iaas.mapper.ovf.OvfXpathParser.retrieveOvfDescriptor:428 - Parsing OVA at url http://<>/<>.ova [UTC:2018-09-19 10:30:23,862 Local:2018-09-19 16:00:23,862] vcac: [component="cafe:iaas-proxy" priority="ERROR" thread="tomcat-http--38" tenant="nukescloud" context="vCyddMTo" parent="" token="vCyddMTo"] com.vmware.vcac.iaas.mapper.ovf.OvfXpathParser.mapOvfToMachine:220 - Could not map ova / ovf at http://<>/<>.ova, to a vRA Machine. [UTC:2018-09-19 10:30:23,863 Local:2018-09-19 16:00:23,863] vcac: [component="cafe:iaas-proxy" priority="ERROR" thread="tomcat-http--38" tenant="nukescloud" context="vCyddMTo" parent="" token="vCyddMTo"] com.vmware.vcac.iaas.service.impl.SourceVMServiceImpl.getOvfMachine:178 - Could not parse OVF / OVA at http://<>/<>.ova [UTC:2018-09-19 10:30:23,865 Local:2018-09-19 16:00:23,865] vcac: [component="cafe:iaas-proxy" priority="ERROR" thread="tomcat-http--38" tenant="nukescloud" context="vCyddMTo" parent="" token="vCyddMTo"] com.vmware.vcac.platform.service.rest.resolver.ApplicationExceptionHandler.handleNPE:454 - null java.lang.NullPointerException at com.vmware.vcac.iaas.mapper.ovf.OvfXpathParser.writeParseErrorToTelemetryLog(OvfXpathParser.java:780) ~[iaas-service-provider-7.4.0-SNAPSHOT.jar:?] at com.vmware.vcac.iaas.mapper.ovf.OvfXpathParser.mapOvfToMachine(OvfXpathParser.java:222) ~[iaas-service-provider-7.4.0-SNAPSHOT.jar:?] This is not a bug. If your OVA/OVF is valid then this exception will never occur Above exception states that it's not able successfully read / parse the OVA/OVF template to give an option to user to configure it to be used via vRA Blueprints Before even thinking of deploying OVA/OVA using vRA , test it directly by deploying it through vCenter to see if it works as expected. In this way you would save time and easily isolate the problem. #vRealizeAutomation

Was working on a brand new installation of vRA 7.4 where we were attempting to patch with the latest one under KB 56618 While implementing , it was throwing a straight exception right after the upload's completed Below was the exception in /var/log/vmware/vcac/vcac-config.log But actual exception on why the failure was under IAAS node's Management Agent All.log Failure was seen because IAAS was unable to fetch binaries from vRA Appliance On vRA Virtual Appliance nodes, open /etc/hosts file, find the entry for IPv4 loopback IP Address (127.0.0.1). Make sure that the ‘Fully Qualified Domain Name’ of the node immediately follows ‘127.0.0.1’, before ‘localhost’. ** Example: 127.0.0.1 FQDN_HOSTNAME_OF_NODE localhost In our case it was as below /etc/hosts was # VAMI_EDIT_BEGIN # Generated by Studio VAMI service. Do not modify manually. 127.0.0.1 localhost 10.36.19.50 vraapp01.pslab.org vraapp01 127.0.0.1 vraapp01.pslab.org load-balancer-host # VAMI_EDIT_END Actually it should be # VAMI_EDIT_BEGIN # Generated by Studio VAMI service. Do not modify manually. 127.0.0.1 vraapp01.pslab.org vraapp01 localhost ::1 vraapp01.pslab.org vraapp01 localhost ipv6-localhost ipv6-loopback # VAMI_EDIT_END 127.0.0.1 localhost.localdom 127.0.0.1 vrava-lb.pslab.org load-balancer-host This change has been documented in the Patch KB as well. Here are the screenshots for failure and successful patch installation #vRealizeAutomation

Just upgraded my lab to RA 7.5, so thought to blog on changes in VAMI from 7.4 to 7.5 TIP 1 When you take an SSH session to a vRA node, now it shows what's MASTER and what's REPLICA on the ssh session itself TIP 2 In Previous versions "Cluster" and "Database" settings were under vRA tab on VAMI portal. In 7.5 "Cluster" is now a separate tab which includes "Database" options as well VAMI 7.4 VAMI 7.5 As you can see above Cluster and Database settings are now merged. You have an additional option for taking a Database Dump. When we click on this option it would generate vRA appliance's postgres database dump under /tmp Dump would be in the format of vcac_dbdump_YYYY_MM_DD_HH_MM_SS.sql.zip This is a good feature from VMware Support perspective and can be taken without any downtime or stopping services. TIP 3 Support bundle option has now been removed from Cluster tab as it was on 7.4 and now moved under Logs tab in 7.5 Log Bundle collection in 7.4 Log Bundle Collection in 7.5 #vRealizeAutomation

vRealize Automation 7.5 released on September 20, 2018. Below is the list of new features and enhancements which came up with this release. Revamped UI and User Experience A complete new look and feel for vRealize Automation along with streamlined flows for common self-service tasks UI based on VMware Clarity standards Larger catalog cards show more of the description Cleaner catalog view Multiple instances of the same catalog item across business groups are now rolled up; the user selects the business group at request time Items and Requests tab merged into new Deployments tab Request details for decommissioned resources moved to the Administration tab Improved status of in-progress requests History view shows all requests associated with a single deployment over time Improved search capabilities across product menus and objects Contextual access to documentation from the product UI Home page and portlets are deprecated in this release Save button on requests is deprecated in this release Improved Integration with vRealize Operations Manager This release introduces deployment dashboards for application owners and enhancements to intelligent workload placement capabilities via integration with vRealize Operations. Show deployment alerts and key metrics (CPU, memory, IOPS, and network) for machines in the deployment details view Enable optimization of vRealize Automation managed workloads to align with vRealize Operations placement policy This builds on an earlier integration for optimizing initial placement to allow ongoing optimization of existing workload Configuration Automation Framework Native integration with external Ansible Tower Management tool. OOTB support for Ansible Tower as first class citizens in vRealize Automation Drag and drop Ansible Tower object in the Blueprint design canvas Parameterize and support early and late binding/request time Dynamically select Ansible job templates,including playbooks, for application configuration Support Day 2 actions to register or decommission machine Troubleshooting Improvements Improvements to Force Delete/Re-Submit ( failed or orphaned deployments) Post-Migration validation Consistent log tracing across solution Expose trace-id to the vRealize Orchestrator plug-in API vRO Database Clustering and Configuration vRO database configuration moved to vRA VAMI Embedded vRO database ( Postgres) is now able to be clustered and supports failover Microsoft Azure Blueprint Enhancements Support for Azure Managed Disks Enhanced Support for Azure regions NSX-T Datacenter Native Integration vRealize Automation now has native integration with NSX-T Datacenter OOTB support for NSX-T Datacenter as first class citizen in vRealize Automation Drag and Drop the following NSX-T Datacenter Services in blueprint design canvas Support for Day-2 actions Event Broker & Custom forms Improvements #vRealizeAutomation

There isn't a great deal of difference in deploying a simple instance of vRA 7.5. It's same as in it's previous versions. Sharing screenshots from my implementation in lab , as it might benefit others. Run Prerequisite Checks once it shows failure , click on Fix so that it runs script on IAAS node mapped to this appliance and get's it ready for the install Now once these pre-reqs are done , move on to enter vRA appliance hostname here and then click on Next Configure SSO ( administrator@vsphere.local ) Configure SQL Server Configure DEM's Configure Agents Generate Certificates Web and Manager are on same node , so no need of a separate certificate in this case ( simple install ) Post Certificates, It would bring up a screen to take snapshots and then the installation starts. It took around 30 minutes for the installation to complete Post Installation , one has to license vRA and it would present post-installation options That's it , it's so simple to install vRA 7.5 ( that's been the case since 7.x was launched ) Will explore various features in my upcoming posts... !! Stay Tuned !! #vRealizeAutomation