Search Results

Create Stored Procedure CREATE PROC SearchAllTables @SearchStr nvarchar(100) AS BEGIN DECLARE @dml nvarchar(max) = N'' IF OBJECT_ID('tempdb.dbo.#Results') IS NOT NULL DROP TABLE dbo.#Results CREATE TABLE dbo.#Results ([tablename] nvarchar(100), [ColumnName] nvarchar(100), [Value] nvarchar(max)) SELECT @dml += ' SELECT ''' + s.name + '.' + t.name + ''' AS [tablename], ''' + c.name + ''' AS [ColumnName], CAST(' + QUOTENAME(c.name) + ' AS nvarchar(max)) AS [Value] FROM ' + QUOTENAME(s.name) + '.' + QUOTENAME(t.name) + ' (NOLOCK) WHERE CAST(' + QUOTENAME(c.name) + ' AS nvarchar(max)) LIKE ' + '''%' + @SearchStr + '%''' FROM sys.schemas s JOIN sys.tables t ON s.schema_id = t.schema_id JOIN sys.columns c ON t.object_id = c.object_id JOIN sys.types ty ON c.system_type_id = ty.system_type_id AND c .user_type_id = ty .user_type_id WHERE t.is_ms_shipped = 0 AND ty.name NOT IN ('timestamp', 'image', 'sql_variant') INSERT dbo.#Results EXEC sp_executesql @dml SELECT * FROM dbo.#Results END Execute Stored Procedure exec SearchAllTables '<<VALUE-ONE-WOULD-LIKE-TO-SEARCH'; Delete Stored Procedure DROP PROCEDURE dbo.SearchAllTables

Data Collection for a specific compute resource was failing on a given endpoint in vRealize Automation 7.x The exception was as below 2020-05-14T09:31:30.772Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6016"] [sub-thread-Id="32" context="" token=""] Error retrieving properties in task 26394140: System.Web.Services.Protocols.SoapException: Required property obj is missing from data object of type ObjectSpec while parsing serialized DataObject of type vmodl.query.PropertyCollector.ObjectSpec at line 1, column 717 Data Collection for other compute resources on the same endpoint was working as expected. So why wouldn't it work only for one specific compute resource was the question Went into my lab and then thought of checking the behavior to compare working and the non-working one Usecase was to perform an inventory data collection on a selected compute resource Here's the snippet from a working instance 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Info" thread="6276"] [sub-thread-Id="23" context="" token=""] Starting : Processing Workitem ID [e779099f-b1df-4a31-99f9-1169feb35272] [inventory] 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.Hostname=primary 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.ParentIdentity=premvc.prem.com/production/host/primary 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Name=primary 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Identity=premvc.prem.com/production/host/primary 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Endpoint0=10.109.10.84 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Endpoint1=10.109.10.85 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Endpoint2=10.109.10.86 2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.Name=inventory 2020-05-18T04:46:16.412Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetHosts: 1631 milliseconds 2020-05-18T04:46:16.537Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetClusters: 123 milliseconds 2020-05-18T04:46:16.568Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetNetworks: 27 milliseconds 2020-05-18T04:46:16.599Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetResourcePools: 28 milliseconds 2020-05-18T04:46:16.631Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetStoragePools: 34 milliseconds 2020-05-18T04:46:16.631Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetTemplates: 0 milliseconds As you can see in the above snippet after it fetches VirtualMachine.Admin.Name=inventory it goes ahead and then gets the information of the inventory like GetHosts , GetClusters, GetNetworks, GetResourcePools, and GetStoragePools, etc... If we check the same on a non-working instance 2020-05-14T09:31:30.631Z <> vcac: [component="iaas:VRMAgent.exe" priority="Info" thread="2064"] [sub-thread-Id="18" context="" token=""] Starting : Processing Workitem ID [38e0898c-4f0a-4f66-bbf0-368fff7c82d1] [inventory] 2020-05-14T09:31:30.631Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.Hostname=<> 2020-05-14T09:31:30.631Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.ParentIdentity=<><<<>/host/<> 2020-05-14T09:31:30.631Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Name=<> 2020-05-14T09:31:30.631Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Identity=<>/<>/host/<> 2020-05-14T09:31:30.631Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Endpoint0=<> 2020-05-14T09:31:30.631Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.Name=inventory 2020-05-14T09:31:30.772Z <> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6016"] [sub-thread-Id="32" context="" token=""] Error retrieving properties in task 26394140: System.Web.Services.Protocols.SoapException: Required property obj is missing from data object of type ObjectSpec while parsing serialized DataObject of type vmodl.query.PropertyCollector.ObjectSpec at line 1, column 717 while parsing property "objectSet" of static type ArrayOfObjectSpec while parsing serialized DataObject of type vmodl.query.PropertyCollector.FilterSpec at line 1, column 315 while parsing call information for method RetrievePropertiesEx at line 1, column 218 while parsing SOAP body at line 1, column 207 while parsing SOAP envelope at line 1, column 38 while parsing HTTP request for method retrievePropertiesEx on object of type vmodl.query.PropertyCollector at line 1, column 0 at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult) at VMware.vSphere.VimService.EndRetrievePropertiesEx(IAsyncResult asyncResult) at Vmware.VSphereInterface.RetryRequestWrapper.Execute[T](Func`1 function) If you look closely and compare it, it's clearly failing to perform retrievePropertiesEx which does your GetHosts, GetClusters, GetNetworks, GetResourcePools, and GetStoragePools So clearly the problem is not on vRealize Automation end, it's more on vSphere. When we went ahead and verified this particular cluster / compute resource where data collection was failing does not even exist on vCenter. Which clearly explains the exception Required property obj is missing from data object of type ObjectSpec Take-Aways The problem does not always have to be on vRealize Automation end, it can be elsewhere. If we have integrated a vCenter as an endpoint when any changes are being made on the endpoint directly as a best practice one has to inform your vRealize Automation admins. This would ensure before off-boarding a cluster on endpoint proper precautions are taken to offboard the same compute resource from the vRA perspective as well.

One of the users of vRA asked a question on why do we see so many DLL's under Temp folder of a user account configured to run DEM's Tried to check the same in my environment and found that it's same behavior on my side as well As shown in the screenshot below DEM Worker and Orchestrator are configured with a service account named "prem\srv-vra-seven" Selected one of the DLL's and opened it with notepad ++, not a right tool to open but it did give enough information on what's creating these files under *Temp/* If we observe the dll , it states that it might be created due to UpdateCatalogWorkflow UpdateCatalogResource workflow is triggered by DEM If we observe timestamps in the above screenshot, it's the same time when the folders with the DLL's in them are created Details of what this UpdateCatalogResource does is present under DEM logs 2020-05-14T08:57:30.473Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="5212"] [sub-thread-Id="15" context="" token=""] WorkflowExecutionUnit: initialize started: 7709 2020-05-14T08:57:32.161Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="5212"] [sub-thread-Id="15" context="" token=""] The definition of workflow UpdateCatalogResource has been cleaned of designer-specific content: 00:00:01.6766975 2020-05-14T08:57:34.458Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="5212"] [sub-thread-Id="15" context="" token=""] WorkflowExecutionUnit: workflow application loaded: 7709 2020-05-14T08:57:34.551Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="5212"] [sub-thread-Id="15" context="" token=""] Worker Controller: initialized instance 7709 - UpdateCatalogResource of the workflow execution unit : 00:00:04.8993202 2020-05-14T08:57:34.551Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Debug" thread="5212"] [sub-thread-Id="15" context="" token=""] Workflow ID: 7709 Name: UpdateCatalogResource App ID: bf67dd56-a92c-4930-8cf1-8c6a3cbb28bf 2020-05-14T08:57:34.551Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Debug" thread="5212"] [sub-thread-Id="15" context="" token=""] Workflow ID: 7709 Argument Name: WorkflowOperation Argument Value: ManagementModelEntities Id 2737 WorkflowOperations 2737 UpdateCatalogResource 2020-05-14T08:57:34.817Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Executing 2020-05-14T08:57:34.911Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Executing 2020-05-14T08:57:34.926Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Executing 2020-05-14T08:57:34.926Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Executing 2020-05-14T08:57:35.551Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Closed 2020-05-14T08:57:35.567Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Debug" thread="8924"] [sub-thread-Id="8" context="" token=""] UpdateCatalogResource Workflow: UpdateCatalogByVirtualMachineId.Execute 2020-05-14T08:57:37.786Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Executing 2020-05-14T08:57:37.786Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Closed 2020-05-14T08:57:37.786Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Management Model: The DeleteWorkflowOperation activity started. 2020-05-14T08:57:38.364Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Debug" thread="8924"] [sub-thread-Id="8" context="" token=""] DeleteWorkflowOperation: Deleting workflow operation with id: 2737 and operation name: UpdateCatalogResource 2020-05-14T08:57:38.677Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Management Model: The DeleteWorkflowOperation activity finished. 2020-05-14T08:57:38.677Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Executing 2020-05-14T08:57:38.677Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Closed 2020-05-14T08:57:38.677Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Closed 2020-05-14T08:57:38.677Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Closed 2020-05-14T08:57:38.692Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow ID: 7709 Activity : State: Closed 2020-05-14T08:57:38.708Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Worker Controller: WorkflowApplication_Completed in child domain. 2020-05-14T08:57:38.708Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Worker Controller: WorkflowApplication_Completed 2020-05-14T08:57:38.708Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Trace" thread="8924"] [sub-thread-Id="8" context="" token=""] Worker Controller: ProcessWorkflowCompleted 2020-05-14T08:57:38.708Z SEVENIAAS vcac: [component="iaas:DynamicOps.DEM.exe" priority="Debug" thread="8924"] [sub-thread-Id="8" context="" token=""] Workflow Complete: 7709 - Successful These folders can be deleted and should not impact any of the underlying components. !!! Hope this helps !!!

There was an interesting scenario where Linux deployments where stuck at a point where it's Installing OS ( this is not a standard template-based deployment but a kickstart deployment ) For more information on Linux Kickstart deployments from vRA click here During this phase of Installing OS, we also install a Gugent ( Guest Agent for Linux ) This gugent installation was throwing exceptions as mentioned below Linux Guest Agent Connection Log /usr/share/gugent/axis2/logs/gugent-axis.log [Wed May 13 22:15:04 2020] [debug] http_transport_sender.c(246) ctx_epr:https://<>:443/VMPS2 [Wed May 13 22:15:04 2020] [debug] http_transport_sender.c(805) using axis2 native http sender. [Wed May 13 22:15:04 2020] [...TRACE...] http_sender.c(252) Entry:axis2_http_sender_send [Wed May 13 22:15:04 2020] [debug] http_sender.c(416) msg_ctx_id:urn:uuid:607582d6-9513-1ea1-30b8-005056889293 [Wed May 13 22:15:04 2020] [info] [ssl client] Client certificate chain filenot specified [Wed May 13 22:15:04 2020] [debug] ssl/ssl_utils.c(190) [ssl client] SSL certificate verified against peer [Wed May 13 22:15:04 2020] [error] http_sender.c(3006) Authtype NTLM is notsupported Linux Guest Agent Log /usr/share/gugent/GuestAgent.log 2020-05-13 11:23:28 Application: [Information] Using the network enabled proxy ... 2020-05-13 11:23:28 Application: [Information] The vCAC endpoint is https://<>:443/VMPS2. 2020-05-13 11:23:28 Application: [Information] The AXIS2C directory is axis2/. 2020-05-13 11:23:28 Application: [Information] Requesting work for agent ID 42088131-66e2-832d-1324-e5b8f05c5ee7. 2020-05-13 11:23:28 Application: [Information] Fetching a work item ... 2020-05-13 11:23:28 Application.Proxy: [Error] 2020-05-13 11:23:28 Application.Proxy: [Error] Error returned by axis2: Error occurred in transport 2020-05-13 11:23:28 Application.Proxy: [Error] This error code may indicate an incorrect endpoint address https://<>:443/VMPS2, or a certificate issue. 2020-05-13 11:23:28 Application.Proxy: [Error] Consult axis2 log file at axis2/logs for more detail As a first step, https://<>:443/VMPS2 tried accessing the following URL. This URL was not returning a proper response. VMPS is a VirtualMachineProvisionService which corresponds to Manager Service but not Web as being shown in the logs This Guest Agent Workitem service configuration is actually stored under ManagerService.exe.Config https://IAAS-MGR-FQDN:443/VMPS2 returns following output This is where we found out that when Guest Agent was being installed it was calling a wrong FQDN to fetch the workitem. As explained above it was trying to reach out to IAAS-WEB-FQDN rather than trying to pull information from IAAS-MGR-FQDN This misconfiguration was done on one of the repositories from where the guest is being downloaded. Once the script which is installing guest agent is fixed with the right URL from where it has to pull the worktems or point it to the manager sevice fqdn , then issues with respect to deployments were resolved.

Patching vRealize Automation is no less an effort. It needs planning and appropriate pre-requisite steps to be taken before one implements it. vRealize Suite Life Cycle Manager eases this process by helping us to manage and patch vRA environments. Now let's understand how we use vRLCM to patch vRA, In this case, I am using the following versions vRA 7.5 GA which will be patched with HF16 vRLCM 2.1 Patch 2 Below screenshots shows my vRA environment managed by vRLCM It's a vRealize Automation 7.5 environment Before we start thinking about patching vRLCM provides us a feature to take a snapshot on the vRA environment being managed by it. Note: It does not take a snapshot on the database node ( IAAS DB ), this is something we have to take manually. Once the snapshots are taken head to Product Support pane and then download the patch we would like to apply. In our case, it would be vRealize Automation 7.5 HF16 Once the download is complete, we can now go ahead and start the patch installation process Click on the three dots on the right side of the environment and then click on Patches --> History This will not show anything as there are no patches installed currently. Click on the three dots on the right side of the environment and then click on Patches -> Install Patch Select the patch which is being installed, that's HF16, this is the only one we downloaded Review and then click on Install Under requests, you can see one for patching just started Finally after a 1 hour and 7 minutes the patching completes The request is marked as completed inside vRLCM Same is seen in VAMI where the patch history pane would show that the environment is running on HF16 Then environment pane is updated as well This is how a vRA environment is patched using vRLCM. The logs we can use to monitor while this patch is being implemented are vRA Appliance: MASTER node /var/log/vmware/vcac/vcac-config.log /var/log/messages To filter only patch logs one can execute tailf /var/log/vmware/vcac/vcac-config.log | grep -i cluster.patch Note: Ensure timezone on the appliance is set to UTC, this is mandatory Below .txt file has complete snippet right from the point the patch activity started and till the point, it's finished.

Environment vIDM 3.3.1 /vIDM 3.3.2 integrated with vRA 8.0/vRA 8.1 and LCM Cause The initial analysis indicates there could be a potential issue around internal postgres query cache, a considerable number of SCIM APIs triggered by vRA “DB Cache replication Thread” Also, the number of concurrent logins and Number of users synched Resolution For the OutofMemory issue, the immediate fix is to increase the memory size and number of CPUs. We found that the vIDM setup is stable with 4 number of CPUs and 16 GB of RAM/Memory. Advice to make 4 CPUs ad 16 GB RAM as default configuration instead of 2 CPUs and 6 GB ram (which is existing default hardware configuration in vIDM 3.3.1 and vIDM 3.3.2 OVAs).

I have been attempting to install vRA 8.x for quite a number of times but I've never been successful due to a simple problem. Let me explain what was that. Every time I used to install it used to fail at this point where it was installing client-secrets Release "client-secrets" does not exist. Installing it now. Error: Job failed: BackoffLimitExceeded helm failed to upgrade 'client-secrets' in namespace 'prelude' Note: Above snippet has been taken from deploy.log When we check csp-fixture-job-XXXX.log under /services-logs/csp-clients-fixture we see that the curl timed out Logging in % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: premvra.prem.com But before we started the install we did cross-check that nslookup to my DNS was working absolutely fine, so why this problem? premvra which is our vRA node premidm which is our vIDM node premlcm which is out vRLCM node When you trigger the easy installer it would ask you for Netowrk Information as you can see in the below screenshot The first DNS server in my case is my Windows Active Directory which has forward and reverse lookup zones configured and contains all the DNS records for premlcm, premidm and premvra as well as the rest of the VMware environment. The second DNS server 10.yy.yy.yy is our router which also functions as a DNS server for all other systems outside my lab environment. This router will not be able to resolve anything within the dns zone hosted in the MS DNS Server, but is reachable for all systems. When vRA installation is in progress during this stage when client-secrets are being installed there are certain POST calls made for few registrations in the background Form my research looks like we perform a ROUND-ROBIN load balancing mechanism when multiple DNS servers are configured. In my case , servers ( premlcm , premvra, and premidm ) will only be resolved through my primary DNS. If in case the POST calls go through the secondary DNS for the name resolution it would fail and throw below exception 2020-04-28 10:03:41.430+0000 ERROR 43 --- [or-http-epoll-1] c.v.i.common.util.HealthUtilComponent : premidm.prem.com: Name or service not known java.net.UnknownHostException: premidm.prem.com: Name or service not known at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.8.0_241] Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: Error has been observed at the following site(s): |_ checkpoint ⇢ Request to POST https://premidm.prem.com/SAAS/API/1.0/oauth2/token?grant_type=client_credentials [DefaultWebClient] Stack trace: at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.8.0_241] at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[na:1.8.0_241] at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[na:1.8.0_241] at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[na:1.8.0_241] After scrapping out this existing deployment, I went ahead and started the installation with only 1 DNS which was able to resolve all the nodes and has entries, and finally, the installation was successful. This scenario might occur in LAB where not all DNS servers are configured for name resolutions or even in production environments where DNS replications have few issues After numerous attempts, it was so heartening to see this screen where it says "INSTALLED" Every DNS Servers mentioned during installation should be able to resolve all the three nodes else installation failures will happen.

Recently we've seen a situation where the root partition was full on vRLI appliance. This was part of a vRLI 3 node cluster. When this issue occurs, the cassandra service gets into a hung state and then this issue starts impacting other nodes in the cluster as well. cassandra.log shows service unresponsive due to space issue on the root partition INFO [HANDSHAKE-XXXXXXX] 2020-03-04 10:47:57,384 OutboundTcpConnection.java:560 - Handshaking version with XXXXXXX INFO [RequestResponseStage-3] 2020-03-04 10:47:57,400 Gossiper.java:1019 - InetAddress /ZZZZZZZ is now UP INFO [GossipStage:1] 2020-03-04 10:47:58,379 StorageService.java:2292 - Node /ZZZZZZZ state jump to NORMAL ERROR [HintsWriteExecutor:1] 2020-03-04 10:48:24,194 CassandraDaemon.java:228 - Exception in thread Thread[HintsWriteExecutor:1,5,main] org.apache.cassandra.io.FSWriteError: java.io.IOException: No space left on device at org.apache.cassandra.hints.HintsWriteExecutor.flushInternal(HintsWriteExecutor.java:232) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.hints.HintsWriteExecutor.flush(HintsWriteExecutor.java:203) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.hints.HintsWriteExecutor.lambda$flush$1(HintsWriteExecutor.java:195) ~[apache-cassandra-3.11.2.jar:3.11.2] The root partition was occupied by a .hprof file along with multiple .hints file and crc32 file getting created in /usr/lib/loginsight/application/lib/apache-cassandra-*/data/hints directory Background on hints Hints are one of three ways to support consistency in the system. When replica node is not available coordinator stores mutating data in temporary hint files to proceed as replica is available. For details look here - https://cassandra.apache.org/doc/latest/operating/hints.html Ideally, in all vRLI deployments, it's configured that they are deleted after the default 3 hours. But somehow it's not working and hint files stay there seems forever in some environments. Repairing runs automatically that is an addition way to support consistency in the system. Manual deletion is solution in this situation. This is a bug and will be addressed in upcoming releases of vRLI

Roles define a set of privileges associated with users. These privileges are tasks that a user can perform. Depending on responsibilities, a specific role will be assigned to a user. Organization roles are defined at the topmost vRealize Automation layer. The following organization roles exist in vRealize Automation: Organization Owner Organization Member Service roles define user access to individual services offered by vRealize Automation. The following services exist in vRealize Automation: Cloud Assembly Code Stream Orchestrator Service Broker Organization and Services All services offered by vRealize Automation belong to the only possible global organization. Any given service role must be associated with an organization role. Only one organization can exist in a vRealize Automation deployment. Each service in vRealize Automation has more than one role. For example, the Cloud Assembly service includes the following roles: Cloud Assembly Administrator Cloud Assembly User Any user in vRealize Automation must be assigned an organization role with an associated service role.

The identity service runs as a pod in Kubernetes. If a user tries to log in to vRealize Automation The identity service redirects the request to the VMware Identity Manager URL The Identity Manager appliance validates the user credentials with Active Directory The user can log in to vRealize Automation console The identity-db is a dedicated PostgreSQL database for the identity service The URL to access the VMware Identity Manager appliance is set as a VIDM_HOST environment variable during installation. All requests to authorize credentials are forwarded to the VMware Identity Manager appliance. Administrators can use access policies to configure features, such as mobile single sign-on (SSO), conditional access to applications based on enrollment and compliance status, and multifactor authentication. VMware products can use VMware Identity Manager as an enterprise SSO solution VMware Identity Manager is based on the OAuth 2.0 authorization framework.

Authentication Confirms your identity Verifies who you are Requires login credentials Authorization Determines what you have access to Grants permission to access a resource Requires a user role

I was working on a vRA upgrade that was being performed using vRLCM Most of the components were upgraded apart from DEM Orchestrator. This one was constantly failing due to some unknown issues. Since this was the last component we decided to go ahead and install it manually. It did install and the whole upgrade was complete. But when we go to VAMI and check the Cluster tab and the versions, DEM Orchestrator status on the node was set to "Upgrade Failed" This information under the Cluster tab of VAMI is actually coming from two places 1. Management Agent installed on the IAAS node where the respective components are installed 2. cluster_nodes table of vPostgres Inspecting cluster_nodes we do find that there is a ping_info column has the above information {    "installationDrive":"C:\\",      "name":"XXXXXXX.adDEO", "certificate":{ },   "metrics":{         },          "metrics":{ "cpuPercentage":0.0, "memory":52379648     }, "state":"Started",   "version":"7.6.0.16195", "type":"DemOrchestrator", "condition":"Failed"       } As you can see above there is this "condition" tag inside the ping_info column of cluster_nodes table. This is coming from or being updated inside this table from the Management agent. In order to resolve this issue, we have to remove this condition tag inside this table. Management Agent pulls this information from the registry of the IAAS node where the components are installed. To remediate Go to following path HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\VMware, Inc.\VMware vCloud Automation Center DEM Click on DemInstance01 or DemInstance02 ( depends on your environment ) On the right side check if you have ComponentCondition registry key If yes , delete the registry key Restart Management Agent on the node Now if you go back to VAMI the version column and the data inside ping_info column should have appropriate info and VAMI should know correct data