Search Results

I was reading this book "The Influential Product Manager" and came across few points which i thought are really important and it would be awesome to share this knowledge to others via a blog. What are the important qualities of being a Great Product Manager ? Curiosity Sense of Ownership Effective Communication Strategic Thinking Humble / Modest Existence Let's discuss each quality at a time and why it matters Curiosity Curiosity means a strong desire to know or willing to learn about something interesting or new fact or object etc.. Great Product Managers are incredibly curious. If something isn't working , they are relentless in trying to understand why They continuously ask questions and zero in on the problem Most of the times customers might think they need a certain feature or a product but they might need something else. Some Product Managers might just go by what customers say but a great product manager would analyze the data or fact behind the ask , then take a decision based on the dynamics A Great Product Manager is always outbound. He always speaks to customers , consumers or the users of the product to understand their problems in a deeper level. What's important here is not to cater just the top customers or existing ones but to speak to anyone they can as long as they are in their target market segment Curious Product Managers are more interested in outcomes than outputs because outcomes is a great way to measure how customer's value the product and the company As a PM , one must never stop learning and ask questions . This is the best and must trait to possess more to come.................

ORIGINAL CONTENT Based on vRealize Automation 8.8 There was an ask on if we have any queries handy which can help us track certain actions in vRealize Automation through vRealize Log Insight That's when i thought of drafting all of these together. Here's a blog post which describes these in detail Note: Certain sections are empty , that would be updated pretty soon. Work in Progress Blueprint Create text: Created blueprint namespace: prelude Update text: Updated blueprint namespace: prelude Delete text: Deleted blueprint namespace: prelude Tag Create Update text: Successfully updated tag namespace: prelude Delete text: Deleting tag namespace: prelude vCenter ( following statement would be seen on vCenter when a tag is assigned to a VM text: attached tag namespace: prelude text: detached tag namespace: prelude WF/Asset configuration modification Create text: Workflow ' text: created namespace: prelude Update text: Workflow ' text: updated namespace: prelude Delete text: Workflow ' text: deleted namespace: prelude Integration Create text: [/provisioning/requests/endpoint-flow-orchestrator-tasks/ or text: POST /iaas/api/integrations namespace: prelude text: Moving from STARTED(CREATING_ENDPOINT) to STARTED(ENABLE_REGIONS) Update text: PATCH /iaas/api/integrations/ text: HTTP/1.1 202 namespace: prelude Delete text: DELETE /iaas/api/integrations/ text: HTTP/1.1 202 namespace: prelude Cloud Accounts Create namespace: prelude text: targetType:EndpointAdapterService text: description:CREATE Update namespace: prelude Delete namespace: prelude text: DeleteEndpointDocumentsDeletion for DELETED On-boarding Plans Create namespace: prelude text: POST /relocation/onboarding/plan text: HTTP/1.1" 200 Update namespace: prelude text: PATCH /relocation/onboarding/plan text: HTTP/1.1" 200 Delete namespace: prelude text: DELETE /relocation/onboarding/plan/ HTTP/1.1" 202 Compute Create namespace: prelude Update namespace: prelude Delete namespace: prelude Cloud Zone Create namespace: prelude text: POST /provisioning/uerp/provisioning/resources/placement-zones Update namespace: prelude PUT /provisioning/uerp/provisioning/resources/placement-zones Delete namespace: prelude text: DELETE /provisioning/uerp/provisioning/resources/placement-zones Flavor Create namespace: prelude text: POST /provisioning/mgmt/instance-names Update namespace: prelude text: PUT /provisioning/mgmt/instance-names Delete namespace: prelude text: PUT /provisioning/mgmt/instance-names Image Mappings Create namespace: prelude text: POST /provisioning/uerp/provisioning/mgmt/image-names Update namespace: prelude text: PUT /provisioning/uerp/provisioning/mgmt/image-names Delete namespace: prelude text: PUT /provisioning/uerp/provisioning/mgmt/image-names Network Profile Create namespace: prelude text: POST /provisioning/resources/network-profiles Update namespace: prelude text: PUT /provisioning/resources/network-profiles Delete namespace: prelude text: DELETE /provisioning/resources/network-profiles Network Modification Create namespace: prelude Update namespace: prelude Delete namespace: prelude Projects Create namespace: prelude text: Created project with id Update namespace: prelude text: Update project event received Delete namespace: prelude text: Delete project event received Custom Roles Create namespace: prelude text: role with name Update namespace: prelude text: Update of role with id Delete namespace: prelude text: Deleted role with id Requests Create namespace: prelude text: Created request tracker Update namespace: prelude Delete namespace: prelude text: DELETE /provisioning/request-status/ Subscriptions Create namespace: prelude text: POST /event-broker/api/subscriptions Update namespace: prelude text: POST /event-broker/api/subscriptions Delete namespace: prelude text: DELETE /event-broker/api/subscriptions Packages Create namespace: prelude text: PUT /vco/api/packages/ Update namespace: prelude text: PATCH/vco/api/packages Delete namespace: prelude text: DELETE /vco/api/packages Resources Create namespace: prelude Update namespace: prelude Delete namespace: prelude Actions Create namespace: prelude text: Creating new action Update namespace: prelude text: Action updated Delete namespace: prelude text: DELETE /abx/api/resources/actions Policies Create namespace: prelude text: POST /policy/api/policies Update namespace: prelude text: POST /policy/api/policies Delete namespace: prelude text: DELETE /policy/api/policies

A Machine blueprint is a standalone blueprint that other architects can reuse as a component in application blueprints, and catalog administrators can include in catalog services. You can specify settings that apply to the entire blueprint by using the Blueprint Properties page when you create the blueprint. After you create the blueprint, you can edit these settings on the Blueprint Properties page. Settings on the General tab apply to the overall blueprint. Lease (days) Minimum and Maximum Enter a minimum and maximum value to allow users to choose from within a range of lease lengths. When the lease ends, the deployment is either destroyed or archived. If you do not specify a minimum or maximum value, the lease is set to never expire. Enter lease information for your machines in your vRealize Automation blueprint, not in the source endpoint application. If you specify lease information in an external application, that lease information is not recognized or used in vRealize Automation. Archive days You can specify an archival period to temporarily retain deployments instead of destroying deployments as soon as their lease expires. Specify 0 to destroy the deployment when its lease expires. The archive period begins on the day the lease expires. When the archive period ends, the deployment is destroyed. The default is 0. In my example above I've set the minimum to 40 days and Maximum to 80 days Propagate updates to existing deployments Broadened minimum-maximum ranges for CPU, memory, or storage are pushed to active deployments that were provisioned from the blueprint. The new range must fully encompass the old range. For example, for an original minimum 32 and maximum 128 (32, 128), a change such as (16, 128) or (32, 256) or (2, 1000) can take effect upon reconfiguration or scale-out, but a change of (33, 512) or (4, 64) cannot. The changes take effect upon the next reconfigure or scale-out action When I deploy VM's out of this blueprint it would set these values to them as you can see in the screenshot As you can see the values for these deployments is set to expiring in 1 month If there are a handful of virtual machines then changing lease period to "Never Expires" is pretty easy Remove the values of Lease (days) under blueprint properties and set Archive (days) to 0 Once done, click on OK, then Save and Finish to close the blueprint pane. This will ensure that all future deployments are set to "Never Expires" as soon as they are deployed. How about existing deployment's then? One can browse to Deployment's tab and then click on "Change Lease", a day 2 action and then set it to "Never Expires" How to execute "Change Lease" is present in this short recording below. What if you have hundreds of deployment's and you would like to change them in one go? Practically heading to every deployment and changing lease manually is an impossible task. The only way available to change lease on multiple deployments in one go is to modify vRA's Postgres database tables So let's provision a few virtual machines where the lease is set and we will use queries to make changes to these deployment's to set lease to never expire As you can see in the screenshot above, I've made changes to Lease (days) to start with 1 and to go up to 6 days. I have three deployments in my lab to play around with. As you may see in the below screenshot lease expires in about a months time Pre-Requisites before making any changes to Database Backup of vRA Postgres database Snapshots of vRA appliances Let's now connect to vRA's Postgres database. You may use a shell session to connect or use the pgAdmin tool. Anything works. Use below query to find out how many deployments we have to modify whose expiration date is in future select name,status,leasestate,resourcetype_id,lease_end_date from cat_resource where 1=1   and status = 'ACTIVE'   and leasestate in ('ACTIVE','ACTIVE_REMINDER_SENT')   and resourcetype_id = 'composition.resource.type.deployment'   and lease_end_date is not null   and lease_end_date > now(); Note: In my lab, since expiration is less than 24 hours, leasestate is set to ACTIVE_REMINDER_SENT , normally if the expiration is more than 24 hours it would be set to ACTIVE Step#1 Begin a transaction begin; Step#2 Modify deployment records -- Step 2 -- --DEPLOYMENT records -- update cat_resource set lease_end_date = '2025-12-06 01:23:45.678' --temporary "flag" value...used in last update to identify correct Deployments -- where 1=1 and status = 'ACTIVE' and leasestate in ('ACTIVE','ACTIVE_REMINDER_SENT') and resourcetype_id = 'composition.resource.type.deployment' and lease_end_date is not null and lease_end_date > now(); --Expiring in the future. Step#3 You're modifying any dependent children or any of sub dependent records --Step 3 -- --update ALL the VIRTUAL MACHINE children records of the above DEPLOYMENTS -- update cat_resource crvm set lease_end_date = null from (select id, lease_end_date from cat_resource where lease_end_date = '2025-12-06 01:23:45.678' and status = 'ACTIVE' and leasestate in ('ACTIVE','ACTIVE_REMINDER_SENT') and resourcetype_id = 'composition.resource.type.deployment' ) x where crvm.parentresource_id = x.id; Step#4 Updating software components if it exists --Step 4 -- --update ALL the SOFTWARE COMPONENTS that are --children of VIRTUAL MACHINE children records of the above DEPLOYMENTS -- update cat_resource crsw --software components of the VMs -- set lease_end_date = null from (select id, lease_end_date from cat_resource where lease_end_date = '2025-12-06 01:23:45.678' and status = 'ACTIVE' and leasestate in ('ACTIVE','ACTIVE_REMINDER_SENT') and resourcetype_id = 'composition.resource.type.deployment' ) x join cat_resource crvm on crvm.parentresource_id = x.id where crsw.parentresource_id = crvm.id; Step#5 Changing cat_resource table with a null value under lease_end_date to change expiration to "NEVER EXPIRES" --Step 5 -- update cat_resource set lease_end_date = null where 1=1 and status = 'ACTIVE' and leasestate in ('ACTIVE','ACTIVE_REMINDER_SENT') and resourcetype_id = 'composition.resource.type.deployment' and lease_end_date is not null and lease_end_date = '2025-12-06 01:23:45.678'; --very important to ensure only updating the correct Deployments. If you're satisfied with the output and it matches the number of deployments then commit; otherwise rollback; Step#6 If committed, then you would find the lease status of the VM's changed to "NEVER EXPIRES" Click here to get the whole query Happy Learning. As stated above before making any changes to database ensure appropriate backups are taken.

Last week was involved on a use-case where our team was helping one of our customer to onboard virtual machines to vRealize Automation 8.x Once virtual machine is onboarded , there is an extensibility configured where a vRO workflow is triggered which get's all of the machines properties from vRealize Automation 7.x and then updates the machine's records in version 8.x But for some reason we were hitting an exception stating 2022-06-02T08:16:07.454Z ERROR vco [host='vco-app-69b4969975-hvgbp' thread='WorkflowExecutorPool-Thread-65' user='vro-gateway-U9ITj3QActbfOqea' org='-' trace='-'] {|__SYSTEM|vro-gateway-U9ITj3QActbfOqea:Migration Set VM Properties :339411e8-ae83-4318-8e0f-924d3456a591:token=92f0de3f-07e7-475b-affd-3a2439d4f1d1} SCRIPTING_LOG - Workflow execution stack: *** item: 'Migration Set VM Properties /item4', state: 'failed', business state: '', exception: 'Error: Unable to set properties for machine 2ef5658e-3a51-381d-9cfb-9d1ff3531095. Error: 404 Service not found: http://10.244.2.28:8282/resources/compute/<> (Dynamic Script Module name : updateMachineProperties#26)' workflow: 'Migration Set VM Properties ' (339****91) So going by the exception it clearly states that it's unable to set the properties of the machine as it get's 404 returned by the service ( provisioning-service ). Which means that there is some discrepancy in the machine's data / records Investigation Based on the timestamp mentioned above , we would start look at the provisioning service to start with. Based on the trace id , we just see below 5 statements 2022-06-02T08:16:07.412Z WARN provisioning [host='provisioning-service-app-5d7b9467b7-lz5j9' thread='reactor-http-epoll-9' user='configadmin' org='<>' trace='9272***2986' parent='' span='c4e755***43e2'] c.v.i.api.common.ControllerRoutingUtil.handleOperationInternal:464 - [Username=configadmin][OrgID=<>] The request is missing mandatory query parameter apiVersion. 2022-06-02T08:16:07.414Z WARN provisioning [host='provisioning-service-app-5d7b9467b7-lz5j9' thread='xn-index-queries-19' user='configadmin' org='<>' trace='927****2986' parent='' span='c4e75****3e2'] c.v.xenon.common.ServiceErrorResponse.create:83 - message: Service not found: http://10.244.2.28:8282/resources/compute/<>, statusCode: 404, serverErrorId: ff2f34d3-bdab-4b1c-b4f5-378f3b46849d 2022-06-02T08:16:07.414Z WARN provisioning [host='provisioning-service-app-5d7b9467b7-lz5j9' thread='xn-index-queries-19' user='configadmin' org='<>' trace='92724***72986' parent='' span='c4e75***43e2'] c.v.a.s.w.p.x.c.StatefulServiceController.lambda$loadService$20:704 - Failed to start service /resources/compute/<> with 404 status code. 2022-06-02T08:16:07.414Z WARN provisioning [host='provisioning-service-app-5d7b9467b7-lz5j9' thread='xn-index-queries-19' user='configadmin' org='<>' trace='927****72986' parent='' span='c4e75***3e2'] c.v.xenon.common.ServiceErrorResponse.create:83 - message: Service not found: http://10.244.2.28:8282/resources/compute/<>, statusCode: 404, serverErrorId: 1a207e42-2daa-48dd-bc4b-6c5f7599da47 2022-06-02T08:16:07.416Z INFO provisioning [host='provisioning-service-app-5d7b9467b7-lz5j9' thread='reactor-http-epoll-9' user='configadmin' org='<>' trace='92724***2986' parent='' span='c4e755***43e2'] reactor.netty.http.server.AccessLog.info:270 - 10.244.2.11 - - [02/Jun/2022:08:16:07 +0000] "PATCH /iaas/api/machines/<> HTTP/1.1" 404 269 8282 31 ms Replaced orgID and resourceid for security reasons As we see , we get a 404 exception which means resource isn't found after a PATCH call is made PATCH /iaas/api/machines/<> HTTP/1.1" 404 Now that we know that there is an issue somewhere with the data of the resources. Let's login into database and see if we can find a clue Note: The search_columns is a function. This can be created by executing below code in postgres CREATE OR REPLACE FUNCTION search_columns( needle text, haystack_tables name[] default '{}', haystack_schema name[] default '{public}' ) RETURNS table(schemaname text, tablename text, columnname text, rowctid text) AS $$ begin FOR schemaname,tablename,columnname IN SELECT c.table_schema,c.table_name,c.column_name FROM information_schema.columns c JOIN information_schema.tables t ON (t.table_name=c.table_name AND t.table_schema=c.table_schema) WHERE (c.table_name=ANY(haystack_tables) OR haystack_tables='{}') AND c.table_schema=ANY(haystack_schema) AND t.table_type='BASE TABLE' LOOP EXECUTE format('SELECT ctid FROM %I.%I WHERE cast(%I as text)=%L', schemaname, tablename, columnname, needle ) INTO rowctid; IF rowctid is not null THEN RETURN NEXT; END IF; END LOOP; END; $$ language plpgsql; After function is created, then we will use this to identify what all tables is the resource id present based on the machine name select * from search_columns("nameoftheresourceonboarded"); We will be connecting to catalog database and execute the queries. There result returned would be something like below Taking this result into consideration , let's search both the tables . Starting with the dep_request_event , we get 4 records. 2 ONBOARDED records and 2 ACTION records. select * from dep_request_event where resource_name = 'nameoftheresourceonboaded'; Two onboarded records indicate that the machine was onboarded twice . See the difference in the timestamp The next query would be to search dep_resource table select id,name,deployment_id,summary_properties,origin,provider_resource_name,accounts from dep_resource where name = 'nameoftheresourceonboaded'; So we have two records. One for DISCOVERED and the next one for ONBOARDED Keep the id's from this table handy. For security reasons i've wiped it out but the id's here should be mapped to document_self_link under compute_state table of provisioning database Let's now connect to provisioning database and identify what tables is the resource name part of select * from search_columns('nameoftheresourceonboaded'); Result would be as shown below Keep this in mind when we review the records in the table compute_description select document_self_link,document_update_action,TO_TIMESTAMP(document_update_time_micros/1000000),id,TO_TIMESTAMP(document_creation_time_micros/1000000) from compute_description where name = 'nameoftheresourceonboarded'; There is a record for DELETE and one more for patch Checking other table called compute_state select document_self_link,document_update_action,TO_TIMESTAMP(document_update_time_micros/1000000),id,TO_TIMESTAMP(document_creation_time_micros/1000000) from compute_state where name = 'nameoftheresourceonboarded'; There is just one record returned. What's important to understand here is that the id from dep_resource should be matching with the document_self_link of the onboarded resource But what we observed here is that it's being matched with the discovered document_self_link. See the snippet below 2022-06-02T08:15:37.690Z INFO tango-blueprint [host='tango-blueprint-service-app-5b46464947-xtkw8' thread='http-nio-8080-exec-11' user='relocation-8d1Zeo8aFltm4u8j(configadmin)' org='replacedorgid' project='2f***82' deployment='a68***270' trace='ad9***78744'] com.vmware.tango.blueprint.service.BlueprintDeploymentService - Sending read tile operation for resource 2ef5***095 with resource link /resources/compute/284ac****aaca9 Ideally it should be as below Sending read tile operation for resource 2ef5***095 with resource link /resources/compute/2ef5***095 So we now know that there is no record in the database for /resources/compute/2ef5***095 This is the reason for the 404 exception seen in the snippet Remediation When onboarding a machine a duplicate entry for the machine is created in the resource center, with one entry in the 'discovered' state and one in the 'onboarded' state. This is a regression caused by a fix for an onboarding failure when machines have legacy IDs in the provisioning service inventory Applying KB: 88162 should fix this issue seen. Once done , we should be able to onboard the machines as expected and no duplicate records would be identified. Note: Ensure a snapshot is taken from vRSLCM before implementing the patch mentioned in the KB Article After the fix is implemented , offboard the machines onboarded previously ,for us the count was very low. This issue has been documented in the release notes too

NO , Virtual Machines which are / were being onboarded will never be deleted automatically. Onboarding feature in vRealize Automation does not have intelligence or capability to delete the virtual machines on it's own I was working on one such scenario where users were claiming that the machines which were being onboarded were deleted. Not one or two but hundreds. This was a good case-study to understand what really happened in the background Here goes he approach during my investigation and how we found out who did it Ideal Onboarding Procedure These are the two phases which constitute onboarding procedure ---------------------------------------------------- PlanExecutionTask ---------------------------------------------------- 2022-05-25T06:02:37.382Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Starting PlanExecutionTask. Plan link: /relocation/onboarding/plan/15b6c88f-00eb-41f2-9bf8-631de817f95b, execution ID: 21a6c776-34c1-44b1-87cb-e67e91d202a2 2022-05-25T06:02:37.389Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-14' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Updating plan status. 2022-05-25T06:02:37.391Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Old status: OK, new status: EXECUTING. 2022-05-25T06:02:37.489Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Querying for first batch of deployments. 2022-05-25T06:02:37.497Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Getting batch. 2022-05-25T06:02:37.500Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Executing batch. 2022-05-25T06:02:37.508Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Waiting for batch to finish. 2022-05-25T06:02:38.514Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-14' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Waiting for batch to finish. 2022-05-25T06:02:39.523Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Updating plan status. 2022-05-25T06:02:39.526Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Old status: EXECUTING, new status: OK. 2022-05-25T06:02:39.536Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Finished. ---------------------------------------------------- DeploymentExecutionTask ** This task would begin between Executing batch and Updating Plan status of PlanExecutionTask. During this phase deployment get's onboarded ** ---------------------------------------------------- 2022-05-25T06:02:37.502Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Starting DeploymentExecutionTask. Deployment link: /relocation/onboarding/deployment/cd312a40-dc55-43a0-ad07-3c6d2efada67, execution ID: 21a6c776-34c1-44b1-87cb-e67e91d202a2 2022-05-25T06:02:37.511Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Identifying eligibility to onboard the deployment. 2022-05-25T06:02:37.514Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Deployment /relocation/onboarding/deployment/cd312a40-dc55-43a0-ad07-3c6d2efada67 eligible to onboard : true 2022-05-25T06:02:37.516Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Initializing DeploymentExecutionTask. 2022-05-25T06:02:37.523Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Updating resource placements for project /provisioning/resources/projects/e14d678c-f1c1-4a0d-9359-7c2aadbb3736 2022-05-25T06:02:37.528Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating blueprint. 2022-05-25T06:02:37.532Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating deployment. 2022-05-25T06:02:37.615Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating networks. 2022-05-25T06:02:38.142Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating machines. 2022-05-25T06:02:38.574Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Allocating IP addresses. 2022-05-25T06:02:38.602Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating load balancers. 2022-05-25T06:02:38.608Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating disks. 2022-05-25T06:02:38.616Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating securityGroups. 2022-05-25T06:02:38.620Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating network gateways. 2022-05-25T06:02:38.624Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Notify deployment onboarded. 2022-05-25T06:02:38.686Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Finished. Investigation Onboarding feature logs it's information under relocation-service logs After we create a deployment plan, when we execute to onboard as discussed before there is a PlanExecutionTask which is triggered Each PlanExecutionTask is associated with an execution id 2022-05-10T01:51:22.198Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/cd323-4e6b-4c0d-b972-3abc123cba] Starting PlanExecutionTask. Plan link: /relocation/onboarding/plan/5a3d36ba-5df9-4g8c-a665-c7sh3k4kff, execution ID: 72dhrd45-3aa2-4ff9-9941-a4fjhfjfjfja3c What we saw from the logs was that there were multiple executions but the last one or the attempt failed with an exception 2022-05-10T02:03:40.394Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Starting PlanExecutionTask. Plan link: /relocation/onboarding/plan/4b427637-6121-49cc-a1cc-5fe2e4581ea5, execution ID: bef9fc06-1ecf-4c21-a04f-bc53ed881f48 2022-05-10T02:03:40.403Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b69bd8e8-b641-4f9a-b44a-0b022ae5055e] Updating plan status. 2022-05-10T02:03:40.404Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Old status: OK, new status: EXECUTING. 2022-05-10T02:03:40.484Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055ee] Updating plan status. 2022-05-10T02:03:40.485Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-14' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Old status: EXECUTING, new status: ERROR. 2022-05-10T02:03:40.492Z ERROR relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-updates-21' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] java.lang.RuntimeException: Validation failed. 2022-05-10T02:03:40.493Z ERROR relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Failed: RuntimeException: Validation failed. Searching with the deployment id which was deleted from tango-blueprint logs we found out that the resource under the deployment ( virtual machine) was onboard and then deleted 2022-05-10T02:03:10.264Z INFO tango-blueprint [host='tango-blueprint-service-app-c6b9f9d9f-4h9wp' thread='tasks-2' user='relocation-8NNrKCAYnu5oyMLf(configadmin)' org='***' project='67e0331a-70ea-47ed-b9f0-994eb965e552' deployment='67***b-4279-4***-b***-1*******d' tile='311b69a3-5378-4da2-bc32-a6b870ed305e' trace='0f9d6925-904f-4202-b8c9-dd90a8722fe2'] com.vmware.tango.blueprint.gateway.DeploymentGateway - Deployment resource Cloud.vSphere.Machine successful. DeploymentId:67***b-4279-4***-b***-1*******d, ResourceId:f72b***-f***-3***-9***-2*****1 * * * 2022-05-10T02:07:09.188Z INFO tango-blueprint [host='tango-blueprint-service-app-c6b9f9d9f-kdxqc' thread='tasks-5' user='configadmin' org='***' project='67e0331a-70ea-47ed-b9f0-994eb965e552' deployment='67***b-4279-4***-b***-1*******d' request='0cc4024b-bb26-4d3c-a37c-82e31512a2fd' flow='9ff5dc4e-d6f6-4d7a-a826-8390603dc34a' task='f13077fc-103f-4f38-a89c-e60971fc96b0' tile='a5322a42-0cdd-4f9c-86d1-4e3748dea2ac' trace='b6a259b0-9816-4b0d-82fc-822af5a287c8'] com.vmware.tango.blueprint.gateway.DeploymentGateway - Deployment resource deleted successfully. DeploymentId:67***b-4279-4***-b***-1*******d, ResourceId:f72b***-f***-3***-9***-2*****1 Here comes the question , why would someone delete a deployment once it was onboarded You would find the answer above , when the onboarding execution failed , few of the resources which were selected for this process were partially onboarded Now since every resource is part of same deployment , if one deletes the deployment , all the underlying resources would be deleted. This is the reason hundreds of virtual machines were deleted How do you find out who deleted the deployment is from the below snippet under catalog-service-app logs 2022-05-10T02:04:36.340Z INFO catalog-service-app [host='cata log-service-app-***-***' thread='http-nio-8000-exec-18' user='configadmin' org='*****' trace='b6a259b0-9816-4b0d-82fc-822af5a287c8'] c.v.t.d.s.ResourceActionRequestService - Submitted action Delete for deployment 67825eab-4279-4ccd-b100-1fe929623a9d resource null The user configadmin is the one who submitted the action called delete For example , if i submit delete in my lab for a deployment then it would be shown as below *** user here is catalog-xxxxxx* which is a system/service user. Which states that this resource was deleted by vRA as it was expired *** 2022-05-27T03:06:21.115Z INFO catalog-service-app [host='catalog-service-app-75f5666cf-78ncp' thread='http-nio-8000-exec-19' user='catalog-VclHatUt32e13g3w' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='3b0cc70f-8852-475b-975a-43a962d2e7c2'] c.v.t.d.s.ResourceActionRequestService - Submitted action Delete for deployment 59befaa7-876d-480f-9bca-1c64f1c83e87 resource null *** If it was a user initiated deletion , like me triggering one then it would be like below *** 2022-05-26T03:59:56.591Z INFO catalog-service-app [host='catalog-service-app-75f5666cf-78ncp' thread='http-nio-8000-exec-19' user='arun' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='a26fc3d9-7824-41b2-afee-d8d08bf6fd36'] c.v.t.d.s.ResourceActionRequestService - Submitted action Delete for deployment 6eb088c3-d9e6-4580-8e27-d924e93f6681 resource null Note: configadmin is a local account which has privileges of highest level. It's recommended to use individual accounts with certain roles assigned for proper governance There is an enhancement which is being made in the product which would inform user how many resources ( virtual machine , network , disks ...etc... ) would be impacted when he / she deletes the deployment. This will give them a clue for not to go and directly submit a deletion . This should be out in the next release of vRA 8.8.1

I shall be sharing a comprehensive guide for upgrading your existing vRealize Automation 7.3 to 7.4 Let's start by describing my environment. I have setup a 3 node vRealize Automation distributed deployment. Environment vRealize Automation Appliances ( using an embedded vRO ) vraapp01.pslab.org, vraapp02.pslab.org and vraapp03.pslab.org IAAS Nodes iaas01.pslab.org : Database , Website , ModelManagerData , ModelManagerWeb , ManagerService , ManagementAgent , DemOrchestrator and WAPI iaas02.pslab.org : Website, ModelManagerWeb , ManagerService , ManagementAgent , DemOrchestrator and WAPI iaas03.pslab.org : Management Agent , DEMWorker and vSphereAgent iaas04.pslab.org : Management Agent, DEMWorker and vSphereAgent lb aliases vRA : vrava-lb.pslab.org IaaS Web : iaasweb-lb IaaS Manager : iaasmgr-lb Health-checks before upgrade You have to ensure environment is healthy before you start an upgrade of your infrastructure 1. Let's start by checking cluster "Last Connected" status, This IaaS nodes should have last connected time not more than 30 seconds. Appliances should not be more than 10 minutes. Take an ssh to Master/any other vRA appliance and then execute following command vcac-config cluster-config -list | sed -e 's/[{}]/''/g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | grep -E 'nodeHost|lastUpdateRelative' This is how it's output would like 2. Verify if rabbitmq cluster status is intact by running below command rabbitmqctl cluster_status Ensure all three vRA appliances are available under running_nodes section cluster_name is set to current MASTER node and there are no partitions ( should be blank as above ) 3. Verify IaaS service hosted in Microsoft IIS is running Start browser and enter URL https://webhostname/Repository/Data/MetaModel.svc to verify that Web Repository is running. If successful , no errors are returned and you see a list of models in XML format Log in to the IaaS Website and check that the status recorded in the Repository.log file reports OK. The file is located in the VCAC home folder at /Server/Model Manager Web/Logs/Repository.log iaas01 iaas02 Start browser and enter url https://IaaS-Web-FQDN/WAPI/api/status/web , the service must be "REGISTERED" in the output Verify Iaas Manager Service https://IaaS-Mgr-FQDN/VMPS2Proxy 4. Ensure all Prerequisites for upgrade vRealize Automation are met 5. Follow Checklist religiously Run NSX and Security Data collection Backup existing vRA environment ( It's a MUST ) Change database replication mode to "ASYNCHRONOUS" to disable automatic failover Download vRealize Automation updates from repository or use iso image Disable Load balancer Note : In my environment, I have configured NSX Load-balancers. I have it disabled at the moment , see the below screenshot and I am using CNAMES in AD for LB purposes till upgrade is complete. These would be pointing to the primary node 6. Backup vRA database ( vPostgres ) by following KB 7. Backup IAAS database (SQL) with the help of your DBA Download vRealize Automation 7.4 update repository from VMware Downloads Note : Take snapshots before you upgrade , Ensure there are no In-Progress or Stuck request. If there are any then wait till they finished or clean them up Let's kick off upgrade now. We will be performing an upgrade with IAAS nodes inclusive or excluding IAAS nodes Identify your current Master node among all three appliances and mount ISO image to it In my case it was vraapp01.pslab.org as seen in the screenshot below Now go back to VAMI and under Update tab change Settings to Use CD-Rom updates and click on Save Settings Now go back to Status tab under Update and click on Check Updates. Appliance should detect the ISO mapped to it and then show it under Available Updates Now that we have everything ready let's press Install Updates and start the process Under /opt/vmware/var/log/vami/ we can see that the installation is proceeding in background once it started installer-page.log is something we can monitor while this upgrade is in progress , it's available under /opt/vmware/var/log/vami/ Another log to monitor is updatecli.log under same path For a change upgrade did fail on second node ( vrapp02.pslab.org ) , it did upgrade Master ( vraapp02.pslab.org ) and third node ( vraapp03.pslab.org ) As stated under this document reverted the snapshot and updated iso image and started upgrade on this node , vraapp02.pslab.org again After a while , even this node upgrade is now complete Then verified that all nodes were in cluster After verifying cluster status , went ahead and executed reboot on the master node As stated in the documentation , rebooting MASTER will automatically reboot it's peers As soon as MASTER node is back after rebooting it's peers and all services are started , we would see IAAS upgrade being kicked off. Above information can be reviewed in /opt/vmware/var/log/vami/upgrade-iaas.log as well As stated earlier all of the upgrade were going as expected Finally , it states that the upgrade is now complete Now that the upgrade is complete , let's perform a quick health check Last Connected status looks good Services on all of the nodes were registered as expected. So no failures observed here. I am showing only the MASTER node screenshot below. Performed checks on IAAS as well , all of them passed with flying colors No Errors in repository.log as well. Since every check has now been cleared. Now it's time to enable NSX load-balancer back . As a first step, I would go ahead and remove the CNAMES configured on my AD Re-create entries for your LB VIP's Let's go back to NSX and enable virtual servers Once we enable virtual servers , we see pool and member status changed to UP as well. Which is what we look for Once all the checks from LB perspective are now completed. It's time to login into vRA 7.4 . As you can see we are using LB VIP , not a individual node anymore. That's it. This concludes article on upgrading your vRealize Automation to 7.4 Leave your feedback and do not hesitate to ask questions. !!! Happy Learning !!! #vRealizeAutomation

This article is being written to help my team to understand what goes on behind the scenes when you install vRealize Automation 7.4 My environment consists of 3 vRA appliances with 1 IAAS node. vRA nukesvra01.nukescloud.com nukesvra02.nukescloud.com nukesvra03.nukescloud.com IAAS nukesiaas.nukescloud.com I am using NSX Load Balancer in my Lab , so let me show you the configuration which goes into it before we begin vRA installation Configure Edge with the VIP's used for vRA Web , IaaS Web , IaaS Manager and vRO Control Center Then , create application profiles Post that configure Health Monitors or Service Monitoring Then create pools , which has only primary appliance for vRA and IaaS node as it's members. We would be adding other vRA appliances , post installation After pools , then create virtual servers by selecting VIP and mapping them to appropriate pools Now that we have completed all configuration with respect to load balancer , let's kick in the installation part. After deploying vRA and IaaS VM's let's now login to primary appliance's VAMI , you would be now presented with the installation wizard Accept EULA Select the type of installation you would like to perform in your environment Download IaaS ManagementAgent and install it on all of the IAAS nodes present. I have only one of them so , once installed that would be displayed as shows below. Now define all vRA appliances which are supposed to be in a cluster Accept the certificate fetched from second and third appliance by the first or the Master appliance Define roles for various IAAS nodes in the environment. I am using only one IAAS node as i stated above hence all Web , Manager Service , DEM's and Agent's would all go into one box Run Pre-Requisite checker , this would fail as we have not performed any manual configuration in the IAAS node. So let the pre-req-check fail and then click on Fix to automatically fix the IAAS machines thorough script Looks like I did miss taking screenshot of two steps vRA host and SSO , that's where you define vRA LB and SSO default tenant password. Post that you would then configure IaaS Web and IaaS Manager LB's Mention Database information on the page. Perform validation here itself so that you would not have any surprises later during installation In the next pane , we have to provide IIS Website name and port for Web components. Also , we got to mention credentials for the IIS application pools which would be used during deployment. As there is only one IAAS node that would be the active one. Mention the credentials which would be used for the service startup and configuration. Configure DEM's Configure Agent's and mention endpoint information. This same information must be used while configuring in UI , post installation. Configure vRA , Web and Manager certificates Next pane would give you the information on what are the LB addresses being used in the installation and the nodes which has have to present or configured on LB. As stated in the UI , during installation process only the Load Balancer Members shown in this pane must be added to LB. Other's can be added post installation which i would show later. Once done , validation of all configurations performed till now would occur and then wizard would remind you to take snapshots of all vm's and appliances involved in installation. Once snapshots are taken , by clicking on the "Install" option below installer would begin a series of steps. Below are the steps involved during installation. Remember, installation steps would vary depending on the scale of the environment. Configure SSO It would first check if there are any horizon services running Since this is a greenfield installation , it will not be able to find the pid for horizon and it would go ahead with enabling services Runs Workspace Portal Initial Configuration utility Configures certificate Recalculates Memory for various components in the appliance list network elasticsearch vco-server vco-configurator rabbitmq-server vcac-server & vrhb-service Initializes horizon Initiates default tenant or organization Logs in as administrator Disables single tenant mode Enables advanced user role feature Enables tenant features Enables local users support and user store Creates first connector and get's it's activation token to activate it Updates user attribute definitions After successful initialization of horizon , vcac service is registered with horizon Configure Postgres Finds all applications that contains DB scripts and then executes them to configure primary appliance's postgres database These applications or services are It updates all the services mentioned in the screenshot above and then identifies that this node is configured to be in clustered mode Post that it starts vcac service Verify that all services are started Perform a health check to see if all services are started on the primary appliance Install Web certificate on IAAS Certificate installed successfully Configure SQL Runs vCloud Automation Software Configuration Wizard in silent mode and configures IaaS database Configure IaaS Web Configures or Installs vCAC IaaS core components Model Manager Web Service Model Manager Data Administration Portal Website Configures WAPI and runs RegisterCatalogTypes Configure IaaS Manager Service Installs Manager Service Updates Manager Service config file and then changes startup type to "Automatic" VMware vCloud Automation Center Service is stopped and started to ensure changes take place immediately Configure DEO Silent Installation of DEM Orchestrator is performed Configure DEM Silent Installation of DEM Worker is performed Configure Agent Installs vCAC vSphere Agent Service Joins second and third appliance to the Master ( first appliance where the installation was initiated ) This part of installation takes very long time , approximately 90 minutes There would be service restarts on the Master and the Replica nodes Once all above steps are completed successfully, that marks the end of installation Configure Licensing After Licensing , If needed you may run initial content configuration or run migration or leave it as it is and manually configure your vRA environment. In the next pane , you would be presented with the environment details , that's Post Installation configuration. It would show LB addresses along with health monitors which we already configured on NSX That's it from vRA installation perspective. Now we need to go back to NSX and add the other two vRA appliances to vRA Web and vRO CC LB Pools Finally , we got to see all pools in healthy state on NSX This sums up the installation for vRA 7.4. Next blog would be on configuration.

Click here for detailed analysis vRealize Automation 8.4.2 was release few days back and it had some fixes which we were looking forward to w.r.t custom resources and dynamic types. I was testing this behavior in my lab and came across an exception which led to upgrade failure. Others might bump into this as well. So let me explain what it was... My environment was built on vRA 8.2 , where a migration was performed , specific to Custom Resources and then we did upgrade to 8.4.1. There is a known issue in version vRA 8.4.1 with Customer Resources where we get 500 OK exception when we try and execute Day-2 actions. After 8.4.1 upgrade , we then wanted to 8.4.2 Upgrade goes through but the exception is seen when identity-service-app is being initialized 2021-06-26T07:31:56.205Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] com.zaxxer.hikari.HikariDataSource.getConnection:110 - HikariPool-1 - Starting... 2021-06-26T07:31:56.677Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] com.zaxxer.hikari.HikariDataSource.getConnection:123 - HikariPool-1 - Start completed. 2021-06-26T07:31:57.474Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] liquibase.lockservice.log:23 - Successfully acquired change log lock 2021-06-26T07:31:58.508Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - Reading from public.databasechangelog 2021-06-26T07:31:58.683Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] liquibase.lockservice.log:23 - Successfully released change log lock 2021-06-26T07:31:58.689Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] liquibase.lockservice.log:23 - Successfully acquired change log lock 2021-06-26T07:31:58.690Z WARN identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] liquibase.hub.log:23 - Skipping auto-registration 2021-06-26T07:31:58.806Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - Table identity_lock created 2021-06-26T07:31:58.814Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - ChangeSet db/changelog/28-add-lock-table.xml::1619005213735-1::zshindova (generated) ran successfully in 115ms 2021-06-26T07:31:58.834Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - Marking ChangeSet: db/changelog/29-update-migration-roles.xml::1622846852-1::tairskiv ran despite precondition failure due to onFail='MARK_RAN': db/changelog/db.changelog-master.xml : SQL Precondition failed. Expected '0' got '1' 2021-06-26T07:31:58.845Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - Data updated in identity_user_service_role 2021-06-26T07:31:58.847Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - Data updated in identity_user_service_role 2021-06-26T07:31:58.852Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - Data updated in identity_group_role 2021-06-26T07:31:58.853Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - Data updated in identity_group_role 2021-06-26T07:31:58.874Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] liquibase.lockservice.log:23 - Successfully released change log lock 2021-06-26T07:31:58.876Z WARN identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] o.s.b.w.r.c.AnnotationConfigReactiveWebServerApplicationContext.refresh:559 - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [org/springframework/boot/autoconfigure/liquibase/LiquibaseAutoConfiguration$LiquibaseConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for change set db/changelog/29-update-migration-roles.xml::1622846852-2::tairskiv: Reason: liquibase.exception.DatabaseException: ERROR: update or delete on table "identity_service_role" violates foreign key constraint "fkhcfahltn20109qfxny3vpogm" on table "identity_group_service_role" Detail: Key (id)=(6799b5da-19f5-4aa3-9bd3-9376b3ac4890) is still referenced from table "identity_group_service_role". [Failed SQL: (0) DELETE FROM public.identity_service_role WHERE service_definition_id = (SELECT id FROM identity_service_definition WHERE name = 'Migration')] 2021-06-26T07:31:58.876Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] com.zaxxer.hikari.HikariDataSource.close:350 - HikariPool-1 - Shutdown initiated... 2021-06-26T07:31:58.898Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] com.zaxxer.hikari.HikariDataSource.close:352 - HikariPool-1 - Shutdown completed. 2021-06-26T07:31:58.905Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] org.ehcache.core.EhcacheManager.removeCache:216 - Cache 'users' removed from EhcacheManager. 2021-06-26T07:31:58.906Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] org.ehcache.core.EhcacheManager.removeCache:216 - Cache 'user_groups' removed from EhcacheManager. 2021-06-26T07:32:03.963Z INFO identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] o.s.b.a.l.ConditionEvaluationReportLoggingListener.logMessage:136 - Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled. 2021-06-26T07:32:03.993Z ERROR identity-service [host='identity-service-app-568d9fbbc-8zfns' thread='main' user='' org='' trace=''] o.s.boot.SpringApplication.reportFailure:834 - Application run failed Note: 1. Below command will help you to connect to postgres database vracli dev psql \c identity-db **** STRONGLY ENCOURAGED NOT TO CONNECT UNLESS INSTRUCTED BY VMWARE **** 2. The best way would be to take a backup , restore it on a postgres server installed on Linux or Windows VM , then execute your queries on it... vracli db backup identity-db > /root/identity-db-<>.psql Above command will help you to take a backup. <> is a variable , substitute it accordingly. 3. To restore psql -U postgres -d **destinationdb** < **path-to-the-sql-file** Create the dummy database first to restore As part of investigation connected to identity service database and then found out that this id referenced in the snippet is part of three tables ID : 6799b5da-19f5-4aa3-9bd3-9376b3ac4890 Then went ahead and looked into each of those tables select * from identity_service_role where id = '6799b5da-19f5-4aa3-9bd3-9376b3ac4890'; As shown in the above screenshot this belonged to service role " Migration Assistant Administrator" The identity-service-app was stuck in "CrashLoopBackOff" state After a while upgrade did fail with following exception Exception is nothing different a generic one. com.vmware.vrealize.lcm.common.exception.EngineException: Upgrade on vRealize Automation VA nbnvra.nukescloud.com failed with state fatal. To know more about the failure, run command "vracli upgrade status --details" on the vRealize Automation VA nbnvra.nukescloud.com. If user wants to revert snapshot and trigger upgrade again, click RETRY with revertSnapshotNRetryUpgrade property set to true (or) If user wants to cancel the whole uprgade and revert to the state before upgrade click RETRY with cancelUpgradeNRevertBack property set to true. If both the retry properties are set to true, revertSnapshotNRetryUpgrade property will take precedence and will be honoured at com.vmware.vrealize.lcm.plugin.core.vra80.task.VraVaUpgradeStatusCheckTask.execute(VraVaUpgradeStatusCheckTask.java:329) at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:45) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Since the logs from identity-service-app were stating that it's unable to make a update or change to postgres database to a certain ID which belongs to Migration Assistant Administrator , this clearly gave an indication to a recent change made to this service role in 8.4.1 We have Migration Assistant Administrator recently moved into Cloud Assembly role. This change was introduced in 8.4.1. And 8.4.2 was supposed to bring in some enhancements to this. Had no choice but to revert this failed upgrade back to 8.4.1. Then the first thing I did is to go and remove the vRA Migration Assistant Administrator role and then add the new Cloud Assembly based role in place This is how it looks after the change Post that another upgrade attempt has been made and it definitely succeeded 2021-06-26T12:19:16.708Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - ChangeSet db/changelog/28-add-lock-table.xml::1619005213735-1::zshindova (generated) ran successfully in 86ms 2021-06-26T12:19:16.724Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - Marking ChangeSet: db/changelog/29-update-migration-roles.xml::1622846852-1::tairskiv ran despite precondition failure due to onFail='MARK_RAN': db/changelog/db.changelog-master.xml : SQL Precondition failed. Expected '0' got '1' 2021-06-26T12:19:16.738Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - Data updated in identity_user_service_role 2021-06-26T12:19:16.739Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - Data updated in identity_user_service_role 2021-06-26T12:19:16.744Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - Data updated in identity_group_role 2021-06-26T12:19:16.746Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - Data updated in identity_group_role 2021-06-26T12:19:16.749Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - Data deleted from identity_service_role 2021-06-26T12:19:16.750Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - Data deleted from identity_service_definition 2021-06-26T12:19:16.754Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] liquibase.changelog.log:23 - ChangeSet db/changelog/29-update-migration-roles.xml::1622846852-2::tairskiv ran successfully in 25ms 2021-06-26T12:19:16.760Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] liquibase.lockservice.log:23 - Successfully released change log lock 2021-06-26T12:19:17.653Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] o.hibernate.jpa.internal.util.LogHelper.logPersistenceUnitInformation:31 - HHH000204: Processing PersistenceUnitInfo [name: default] 2021-06-26T12:19:18.383Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] org.hibernate.Version.logVersion:44 - HHH000412: Hibernate ORM core version 5.4.30.Final 2021-06-26T12:19:19.657Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] o.hibernate.annotations.common.Version.:56 - HCANN000001: Hibernate Commons Annotations {5.1.2.Final} 2021-06-26T12:19:20.489Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] org.hibernate.dialect.Dialect.:175 - HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL94Dialect 2021-06-26T12:19:21.922Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] o.h.e.t.j.p.i.JtaPlatformInitiator.initiateService:52 - HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2021-06-26T12:19:21.937Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] o.s.o.j.LocalContainerEntityManagerFactoryBean.buildNativeEntityManagerFactory:419 - Initialized JPA EntityManagerFactory for persistence unit 'default' 2021-06-26T12:19:22.052Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] c.v.i.s.i.PropertyConfigurationService.init:70 - Loaded classpath:config_service.properties 2021-06-26T12:19:22.095Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] c.v.i.config.SecurityConfiguration.readKeyStorePasswordFromFile:110 - Loaded /secrets/keystore.pass 2021-06-26T12:19:22.095Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] c.v.i.config.SecurityConfiguration.setKeyStorePassword:80 - Setting the javax.net.ssl.keyStorePassword system property from a file. 2021-06-26T12:19:23.589Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] c.v.i.service.impl.CryptoServiceImpl.loadKeystore:472 - Loaded keystore 2021-06-26T12:19:24.330Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] c.v.a.s.w.platform.server.util.AppUtil.readTrustStorePasswordFromFile:126 - Loaded /jdk/lib/security/cacerts.pass 2021-06-26T12:19:24.330Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] c.v.a.s.w.platform.server.util.AppUtil.configureTrustStorePassword:95 - Setting the javax.net.ssl.trustStorePassword system property from file. 2021-06-26T12:19:24.335Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] c.v.a.s.w.platform.server.util.AppUtil.configureTrustStorePassword:90 - The javax.net.ssl.trustStorePassword is already set. 2021-06-26T12:19:26.316Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] c.v.identity.rest.EbsRestClientFactory.:77 - Extensibility is configured on 'null' URL with clientID 'prelude-user-eFWWZ4dVxU' 2021-06-26T12:19:26.426Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] o.s.b.a.s.r.ReactiveUserDetailsServiceAutoConfiguration.getOrDeducePassword:88 - Using generated security password: 326c6151-5283-48a7-93dc-90ed354fd40e 2021-06-26T12:19:27.347Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] com.vmware.identity.rest.RestClient.lambda$logRequest$1:72 - POST https://nbnidm.nukescloud.com/SAAS/API/1.0/oauth2/token?grant_type=client_credentials 2021-06-26T12:19:27.708Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] com.vmware.identity.rest.RestClient.lambda$logRequest$1:72 - POST https://nbnidm.nukescloud.com/SAAS/API/1.0/oauth2/token?grant_type=client_credentials 2021-06-26T12:19:27.910Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='main' user='' org='' trace=''] c.v.a.s.w.p.s.util.SqlRetryableUtil.:131 - Calculated value for sql-retry-policy.max-retries: 8 2021-06-26T12:19:29.166Z INFO identity-service [host='identity-service-app-568d9fbbc-949ft' thread='reactor-http-epoll-1' user='' org='' trace=''] com.vmware.identity.rest.RestClient.lambda$logRequest$1:72 - GET https://nbnidm.nukescloud.com/SAAS/jersey/manager/api/scim/Users?filter=userName%20eq%20%22configadmin%22 Post this all the pods were initialized as expected and upgrade was successful Lesson Learnt Remove the standalone vRA Migration Assistant service role and then use Cloud Assembly based vRA Migration Assistant Perform this check before you start an upgrade to vRealize Automation 8.4.2 to ensure there are no failures As one can see , once we login into vRA UI of 8.4.2 and check the roles , there is no standalone vRA Migration assistant role anymore Remember vRA MIGRATION ASSISTANT role should be removed for all the AD groups added under IDENTITY & ACCESS MANAGEMENT After upgrade you may add Cloud Assembly based Migration roles back to these groups If you are already in a situation where identity-service has crashed and you have not read the release notes of 8.4.2 or the KB before ,then follow these steps 1. ssh root@FQDN 2. vracli dev psql 3. \c identity-db 4. copy and paste the following: DO $$ BEGIN IF NOT EXISTS (SELECT def.id from identity_service_role AS role INNER JOIN identity_service_definition AS def ON role.service_definition_id = def.id WHERE role.name = 'migration:admin' AND def.name = 'Cloud Assembly') THEN INSERT INTO identity_service_role (id, is_default, created_millis, hidden, display_name, name, service_definition_id) VALUES ((SELECT uuid_in(md5(random()::text || clock_timestamp()::text)::cstring)), TRUE, (SELECT round(date_part('epoch', now() ) * 1000 )), FALSE, 'Migration Assistant Administrator', 'migration:admin', (SELECT id from identity_service_definition where name = 'Cloud Assembly')); INSERT INTO identity_service_role (id, is_default, created_millis, hidden, display_name, name, service_definition_id) VALUES ((SELECT uuid_in(md5(random()::text || clock_timestamp()::text)::cstring)), FALSE, (SELECT round(date_part('epoch', now() ) * 1000 )), FALSE, 'Migration Assistant Viewer', 'migration:viewer', (SELECT id from identity_service_definition where name = 'Cloud Assembly')); END IF; UPDATE identity_group_service_role SET role_id = (SELECT id FROM identity_service_role WHERE service_definition_id = (SELECT id FROM identity_service_definition WHERE name = 'Cloud Assembly') AND name = 'migration:viewer') WHERE role_id = (SELECT id FROM identity_service_role WHERE service_definition_id = (SELECT id FROM identity_service_definition WHERE name = 'Migration') AND name = 'migration:viewer'); UPDATE identity_group_service_role SET role_id = (SELECT id FROM identity_service_role WHERE service_definition_id = (SELECT id FROM identity_service_definition WHERE name = 'Cloud Assembly') AND name = 'migration:admin') WHERE role_id = (SELECT id FROM identity_service_role WHERE service_definition_id = (SELECT id FROM identity_service_definition WHERE name = 'Migration') AND name = 'migration:admin'); END $$ LANGUAGE plpgsql; The output of above query would just say DO Then run a command to restart the services /opt/scripts/deploy.sh Once all the pods are up , then perform inventory sync on the environment which should now show up as 8.4.2 If you've done this directly through vRA without completing the upgrade workflow through LCM, you have to clean the upgrade in vRA state manually with this command by running it on one of the nodes vracli cluster exec -- bash -c 'rm -rf /data/restorepoint /var/vmware/prelude/upgrade /var/log/vmware/prelude/upgrade-report-latest*; crontab -u root -l | grep -v -F "/opt/scripts/upgrade/upg-mon.sh" | crontab -u root -'

Few days back I did blog on the issue i encountered during my lab upgrade where identity-service-app does not initialize due to vRA Migration Assistant service role assigned to AD users. Click on the link for the blog : https://www.nukescloud.com/post/identity-service-app-initialization-failure-after-upgrading-to-vrealize-automation-8-4-2 Important points to note Remove the standalone vRA Migration Assistant service role before you upgrade for all AD groups mapped Read the release notes : https://docs.vmware.com/en/vRealize-Automation/8.4.2/rn/vRealize-Automation-842-releasenotes.html If you are stuck with this issue then instead of reverting perform these steps. Your connecting to 1. ssh root@FQDN 2. vracli dev psql 3. \c identity-db 4. copy and paste the following: DO $$ BEGIN IF NOT EXISTS (SELECT def.id from identity_service_role AS role INNER JOIN identity_service_definition AS def ON role.service_definition_id = def.id WHERE role.name = 'migration:admin' AND def.name = 'Cloud Assembly') THEN INSERT INTO identity_service_role (id, is_default, created_millis, hidden, display_name, name, service_definition_id) VALUES ((SELECT uuid_in(md5(random()::text || clock_timestamp()::text)::cstring)), TRUE, (SELECT round(date_part('epoch', now() ) * 1000 )), FALSE, 'Migration Assistant Administrator', 'migration:admin', (SELECT id from identity_service_definition where name = 'Cloud Assembly')); INSERT INTO identity_service_role (id, is_default, created_millis, hidden, display_name, name, service_definition_id) VALUES ((SELECT uuid_in(md5(random()::text || clock_timestamp()::text)::cstring)), FALSE, (SELECT round(date_part('epoch', now() ) * 1000 )), FALSE, 'Migration Assistant Viewer', 'migration:viewer', (SELECT id from identity_service_definition where name = 'Cloud Assembly')); END IF; UPDATE identity_group_service_role SET role_id = (SELECT id FROM identity_service_role WHERE service_definition_id = (SELECT id FROM identity_service_definition WHERE name = 'Cloud Assembly') AND name = 'migration:viewer') WHERE role_id = (SELECT id FROM identity_service_role WHERE service_definition_id = (SELECT id FROM identity_service_definition WHERE name = 'Migration') AND name = 'migration:viewer'); UPDATE identity_group_service_role SET role_id = (SELECT id FROM identity_service_role WHERE service_definition_id = (SELECT id FROM identity_service_definition WHERE name = 'Cloud Assembly') AND name = 'migration:admin') WHERE role_id = (SELECT id FROM identity_service_role WHERE service_definition_id = (SELECT id FROM identity_service_definition WHERE name = 'Migration') AND name = 'migration:admin'); END $$ LANGUAGE plpgsql; Then exit database by executing \q Post that restart the services manually and then wait for services to initialize and complete /opt/scripts/deploy.sh If you've done this directly through vRA without completing the upgrade workflow through LCM, you have to clean the upgrade in vRA state manually with this command by running it on one of the nodes vracli cluster exec -- bash -c 'rm -rf /data/restorepoint /var/vmware/prelude/upgrade /var/log/vmware/prelude/upgrade-report-latest*; crontab -u root -l | grep -v -F "/opt/scripts/upgrade/upg-mon.sh" | crontab -u root -' Post this perform inventory sync on vRSLCM for this vRA environment To make it simple Then after upgrade choose Cloud Assembly based Migration Role

What happens when your adding an Cloud Account ? After we enter the information what's needed to create an endpoint and then we click on validate following information is seen under provisioning-service-app.log 2021-08-02T08:59:29.030Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-34' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='bbff78e2-d5c4-497d-9e76-8ed6a135ece3' parent='7f557c51-fa95-4d7d-9e54-6a34266ed0b2' span='40566716-182d-42df-8623-568eff25812e'] c.v.a.c.endpoint.EndpointAdapterService.log:453 - [8282/provisioning/mgmt/endpoints] Retrieving endpoint null 2021-08-02T08:59:29.063Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-33' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='bbff78e2-d5c4-497d-9e76-8ed6a135ece3' parent='7f557c51-fa95-4d7d-9e54-6a34266ed0b2' span='40566716-182d-42df-8623-568eff25812e'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/06ca473d-1a6f-43b1-9804-a73119a07f08] Moving from POPULATE_ADAPTER_REFERENCE(CREATED) to POPULATE_ADAPTER_REFERENCE(STARTED) 2021-08-02T08:59:29.064Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-9' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='bbff78e2-d5c4-497d-9e76-8ed6a135ece3' parent='7f557c51-fa95-4d7d-9e54-6a34266ed0b2' span='40566716-182d-42df-8623-568eff25812e'] c.vmware.admiral.host.ProvisioningHost.log:453 - Scheduling operation retry in 1000 ms, 599 retries left 2021-08-02T08:59:29.071Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-5' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='bbff78e2-d5c4-497d-9e76-8ed6a135ece3' parent='7f557c51-fa95-4d7d-9e54-6a34266ed0b2' span='40566716-182d-42df-8623-568eff25812e'] c.vmware.admiral.host.ProvisioningHost.log:453 - [getAdapterUri] [/provisioning/endpoint-tasks/06ca473d-1a6f-43b1-9804-a73119a07f08] adapterRef = http://provisioning-service.prelude.svc.cluster.local:8282/provisioning/vsphere/endpoint-config-adapter: SUCCESS Moves from POPULATE_ADAPTER_REFERENCE(STARTED) to VALIDATE_CREDENTIALS(STARTED) 2021-08-02T08:59:29.089Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-21' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='bbff78e2-d5c4-497d-9e76-8ed6a135ece3' parent='7f557c51-fa95-4d7d-9e54-6a34266ed0b2' span='40566716-182d-42df-8623-568eff25812e'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/06ca473d-1a6f-43b1-9804-a73119a07f08] Moving from POPULATE_ADAPTER_REFERENCE(STARTED) to VALIDATE_CREDENTIALS(STARTED) Validates credentials 2021-08-02T08:59:29.104Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-21' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='bbff78e2-d5c4-497d-9e76-8ed6a135ece3' parent='7f557c51-fa95-4d7d-9e54-6a34266ed0b2' span='40566716-182d-42df-8623-568eff25812e'] c.v.p.c.m.a.v.VSphereEndpointAdapterService.log:453 - [8282/provisioning/vsphere/endpoint-config-adapter] Validating credentials for operation (57242): STARTING Resolves certificate 2021-08-02T08:59:29.130Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-26' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='bbff78e2-d5c4-497d-9e76-8ed6a135ece3' parent='7f557c51-fa95-4d7d-9e54-6a34266ed0b2' span='40566716-182d-42df-8623-568eff25812e'] c.v.p.c.m.security.util.CertificateUtil.resolveCertificate:496 - resolveCertificate with proxyUsername: null 2021-08-02T08:59:29.210Z WARN provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-26' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='bbff78e2-d5c4-497d-9e76-8ed6a135ece3' parent='7f557c51-fa95-4d7d-9e54-6a34266ed0b2' span='40566716-182d-42df-8623-568eff25812e'] c.vmware.admiral.host.ProvisioningHost.log:453 - Certificate with thumbprint 5D:1D:42:8F:D9:6A:DF:A6:02:30:5B:14:04:78:E6:98:3D:AB:26:3C not trusted. Checking the db... 2021-08-02T08:59:29.215Z WARN provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-25' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='bbff78e2-d5c4-497d-9e76-8ed6a135ece3' parent='7f557c51-fa95-4d7d-9e54-6a34266ed0b2' span='40566716-182d-42df-8623-568eff25812e'] c.vmware.admiral.host.ProvisioningHost.log:453 - Certificate with thumbprint 5D:1D:42:8F:D9:6A:DF:A6:02:30:5B:14:04:78:E6:98:3D:AB:26:3C not found 2021-08-02T08:59:29.496Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-26' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='bbff78e2-d5c4-497d-9e76-8ed6a135ece3' parent='7f557c51-fa95-4d7d-9e54-6a34266ed0b2' span='40566716-182d-42df-8623-568eff25812e'] c.v.p.c.m.a.v.VSphereEndpointAdapterService.log:453 - [8282/provisioning/vsphere/endpoint-config-adapter] Validating credentials for operation (57242): SUCCESS with CertificateInfoServiceErrorResponse 2021-08-02T08:59:29.507Z WARN provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-8' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='bbff78e2-d5c4-497d-9e76-8ed6a135ece3' parent='7f557c51-fa95-4d7d-9e54-6a34266ed0b2' span='40566716-182d-42df-8623-568eff25812e'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/06ca473d-1a6f-43b1-9804-a73119a07f08] Referer http://10.244.0.42:8282/provisioning/endpoint-tasks/06ca473d-1a6f-43b1-9804-a73119a07f08 is patching us to failure during subStage VALIDATE_CREDENTIALS: { "certificateInfo": { "documentKind": "com:vmware:photon:controller:model:support:CertificateInfo", "certificate": "-----BEGIN CERTIFICATE-----\nMIID9zCCAt+gAwIBAgIJAMUYdQTxcp75MA0GCSqGSIb3DQEBCwUAMIGYMQswCQYD\nVQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZ\nFgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExGjAYBgNV\nBAoMEXZjLm51a2VzY2xvdWQuY29tMRswGQYDVQQLDBJWTXdhcmUgRW5naW5lZXJp\nbmcwHhcNMjEwMzAzMTUyNTE4WhcNMjMwMzA0MDMyNTE4WjApMRowGAYDVQQDDBF2\nYy5udWtlc2Nsb3VkLmNvbTELMAkGA1UEBhMCVVMwggEiMA0GCSqGSIb3DQEBAQUA\nA4IBDwAwggEKAoIBAQDSgWFL09VkVvVg1GLcsG7uXuhDmfOekRF+7z/bAfjt4Av7\nOtKa1Yq9foIputhEHfQZdmaGpguEcZW8EoKxvMMMZj2EBWbliT2B6kEbSVWk4fi+\nv8aKGmuHF+zZg5KC8eoOt+yVlYpmaR0nUCVja10MoW8dZUjT2C/eW7+U4gUKfNXj\nAHXU4I3XCkXfE2iTDfDBMM+GqsuL0ngaGzjtBG0MzZSt+i9fzDKTT+24BxSV7zLv\nkeKpS4DlvBGsdHH75bhskAXbjV/fCheOJNvi4DgKcstehACQt9y+7MkNHWacbpuv\nnWxy1dZhc+8xJvo7kCxCvxpYkdEhgcjOPWNU/6CVAgMBAAGjgbEwga4wCwYDVR0P\nBAQDAgOoMBwGA1UdEQQVMBOCEXZjLm51a2VzY2xvdWQuY29tMB0GA1UdDgQWBBQl\n9puInH/VuQXoLbO0s1ycWw1xyzAfBgNVHSMEGDAWgBR9//blgcP/+bH0loJBqwoc\n04D47zBBBggrBgEFBQcBAQQ1MDMwMQYIKwYBBQUHMAKGJWh0dHBzOi8vdmMubnVr\nZXNjbG91ZC5jb20vYWZkL3ZlY3MvY2EwDQYJKoZIhvcNAQELBQADggEBAKgcQvGV\nKeZ0YwFc0yHxJFMWo859kTg7bHKEGidX6Tdw4Oee3ftpSH76f5klRNJVwzIUkbzz\nm+4gRTrG6N0QD3rVdncf6qxxLNEVu+1gPNvydSn0IAf3AAJWpGi7ly/+I85gJL5D\n7fibaafpsjXZi8S4p+3n1uNVCOHzxyiAHbwllniqPafyuapXabVR8mdY9NsGXzBG\nGwJGR8278WqGsVZFkHit9RpN8tIK5U/VbpRLewKVjcg4Vd5FvSnbPMesK4sH2t6w\nycCJcLzX3YEtg5kxcAhoAy7q1772R6r6trReb7fLXkT5MFaONs0Gk9xe7Z86Jv8r\nkrleysU4o3QBB+w=\n-----END CERTIFICATE-----\n", "properties": { "commonName": "vc.nukescloud.com", "validSince": "1614785118000", "serial": "14202230089003998969", "issuerName": "CA", "fingerprint": "5D:1D:42:8F:D9:6A:DF:A6:02:30:5B:14:04:78:E6:98:3D:AB:26:3C", "validTo": "1677900318000" } }, "message": "unable to find valid certification path to requested target", "statusCode": 202, "errorCode": -1879048191, "serverErrorId": "5dbf036d-e1f4-4b28-970d-47f447204103", "documentKind": "com:vmware:photon:controller:model:support:CertificateInfoServiceErrorResponse" } 2021-08-02T08:59:30.086Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='reactor-http-epoll-16' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='bbff78e2-d5c4-497d-9e76-8ed6a135ece3' parent='7f557c51-fa95-4d7d-9e54-6a34266ed0b2' span='40566716-182d-42df-8623-568eff25812e'] reactor.netty.http.server.AccessLog.info:274 - 10.244.0.42 - - [02/Aug/2021:08:59:29 +0000] "PUT /provisioning/mgmt/endpoints?validate HTTP/1.1" 202 2067 8282 1065 ms 2021-08-02T08:59:30.088Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='reactor-http-epoll-16' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='bbff78e2-d5c4-497d-9e76-8ed6a135ece3' parent='' span='bd20152d-467c-4958-9e22-6759692e080a'] reactor.netty.http.server.AccessLog.info:274 - 10.244.0.33 - - [02/Aug/2021:08:59:28 +0000] "PUT /provisioning/uerp/provisioning/mgmt/endpoints?validate HTTP/1.1" 202 2067 8282 1126 ms 2021-08-02T08:59:32.350Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='reactor-http-epoll-1' user='' org='' trace='5b757275-e58e-4215-b7ad-891ceb80bc0d' parent='' span='232cac34-f41c-4fdf-a3d9-205643b8ed04'] reactor.netty.http.server.AccessLog.info:274 - 10.244.0.1 - - [02/Aug/2021:08:59:32 +0000] "GET /provisioning/mgmt/status?k8s-probe HTTP/1.1" 200 0 8282 2 ms 2021-08-02T08:59:32.386Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-10' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.a.c.endpoint.EndpointAdapterService.log:453 - [8282/provisioning/mgmt/endpoints] Retrieving endpoint null 2021-08-02T08:59:32.425Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-30' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.vmware.admiral.host.ProvisioningHost.log:453 - Scheduling operation retry in 1000 ms, 599 retries left 2021-08-02T08:59:32.428Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-17' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/2be7741c-d082-4bba-bd6e-5fbfad41a61e] Moving from POPULATE_ADAPTER_REFERENCE(CREATED) to POPULATE_ADAPTER_REFERENCE(STARTED) 2021-08-02T08:59:32.443Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-32' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.vmware.admiral.host.ProvisioningHost.log:453 - [getAdapterUri] [/provisioning/endpoint-tasks/2be7741c-d082-4bba-bd6e-5fbfad41a61e] adapterRef = http://provisioning-service.prelude.svc.cluster.local:8282/provisioning/vsphere/endpoint-config-adapter: SUCCESS 2021-08-02T08:59:32.461Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-28' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/2be7741c-d082-4bba-bd6e-5fbfad41a61e] Moving from POPULATE_ADAPTER_REFERENCE(STARTED) to VALIDATE_CREDENTIALS(STARTED) 2021-08-02T08:59:32.466Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-28' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.p.c.m.a.v.VSphereEndpointAdapterService.log:453 - [8282/provisioning/vsphere/endpoint-config-adapter] Validating credentials for operation (57286): STARTING 2021-08-02T08:59:32.478Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-19' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.p.c.m.security.util.CertificateUtil.resolveCertificate:496 - resolveCertificate with proxyUsername: null 2021-08-02T08:59:32.500Z WARN provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-19' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.vmware.admiral.host.ProvisioningHost.log:453 - Certificate with thumbprint 5D:1D:42:8F:D9:6A:DF:A6:02:30:5B:14:04:78:E6:98:3D:AB:26:3C not trusted. Checking the db... 2021-08-02T08:59:32.504Z WARN provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-38' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.vmware.admiral.host.ProvisioningHost.log:453 - Certificate with thumbprint 5D:1D:42:8F:D9:6A:DF:A6:02:30:5B:14:04:78:E6:98:3D:AB:26:3C not found Since the certificate is not found it adds certificate to the truststore and validates credentials 2021-08-02T08:59:32.541Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-19' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.p.c.m.security.util.CertificateUtil.registerCertificate:625 - Register certificate with common name: vc.nukescloud.com and fingerprint: 5D:1D:42:8F:D9:6A:DF:A6:02:30:5B:14:04:78:E6:98:3D:AB:26:3C in trust store 2021-08-02T08:59:32.549Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-19' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.p.c.m.a.v.ClientCertificateUtils.trustCertificateIfNeeded:239 - Check if certificate thumbprint needs to be added to db: 5D:1D:42:8F:D9:6A:DF:A6:02:30:5B:14:04:78:E6:98:3D:AB:26:3C 2021-08-02T08:59:32.566Z WARN provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-40' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.p.c.m.a.v.ClientCertificateUtils.lambda$loadCertificates$0:219 - Certificate with thumbprint 5D:1D:42:8F:D9:6A:DF:A6:02:30:5B:14:04:78:E6:98:3D:AB:26:3C not found in loadCertificates 2021-08-02T08:59:32.567Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-19' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.p.c.m.a.v.ClientCertificateUtils.trustCertificateIfNeeded:251 - Certificate with thumbprint 5D:1D:42:8F:D9:6A:DF:A6:02:30:5B:14:04:78:E6:98:3D:AB:26:3C not trusted. Adding certificate to the database 2021-08-02T08:59:32.586Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-updates-48' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.p.c.m.a.v.VSphereEndpointAdapterService.log:453 - [8282/provisioning/vsphere/endpoint-config-adapter] handleCertificate completionHandler success. Certificate stored for host: vc.nukescloud.com /config/trusted-certificates/7adadece-1f01-4fde-8176-3bebeb8814fc 2021-08-02T08:59:33.091Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-updates-48' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.p.c.m.a.v.VSphereEndpointAdapterService.log:453 - [8282/provisioning/vsphere/endpoint-config-adapter] Validating credentials for vCenter [https://vc.nukescloud.com/sdk], user [vrealizesrv@nukescloud.com]. Once the credentials are validated the task is complete 2021-08-02T08:59:37.964Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-updates-48' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.p.c.m.a.v.VSphereEndpointAdapterService.log:453 - [8282/provisioning/vsphere/endpoint-config-adapter] Validating credentials for operation (57286): SUCCESS 2021-08-02T08:59:37.977Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-15' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/2be7741c-d082-4bba-bd6e-5fbfad41a61e] Moving from VALIDATE_CREDENTIALS(STARTED) to COMPLETED(STARTED) 2021-08-02T08:59:38.001Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-6' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/2be7741c-d082-4bba-bd6e-5fbfad41a61e] Moving from COMPLETED(STARTED) to COMPLETED(FINISHED) 2021-08-02T08:59:38.004Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-6' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='697e4145-9935-46d9-b8b2-338092ae6686' parent='2d4cc5dc-fc3c-44b0-be1d-f359ab691679' span='4656bbf1-216c-45e9-814c-360e70b101dc'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/2be7741c-d082-4bba-bd6e-5fbfad41a61e] Task is complete Eventually the Cloud Account creation task is complete 2021-08-02T08:59:45.086Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-5' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.p.c.m.a.v.VSphereEndpointAdapterService.log:453 - [8282/provisioning/vsphere/endpoint-config-adapter] Validating credentials for vCenter [https://vc.nukescloud.com/sdk], user [vrealizesrv@nukescloud.com]. 2021-08-02T08:59:45.188Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-5' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.p.c.m.a.v.VSphereEndpointAdapterService.log:453 - [8282/provisioning/vsphere/endpoint-config-adapter] Validating vAPI endpoint for vCenter [https://vc.nukescloud.com/sdk], user [vrealizesrv@nukescloud.com]. 2021-08-02T08:59:45.460Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-5' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.p.c.m.a.v.VSphereEndpointAdapterService.log:453 - [8282/provisioning/vsphere/endpoint-config-adapter] Validating credentials for operation (57388): SUCCESS 2021-08-02T08:59:45.477Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-26' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/e34e1a61-33bf-4df6-815a-2252b614834e] Moving from VALIDATE_CREDENTIALS(STARTED) to CHECK_IF_ACCOUNT_EXISTS(STARTED) 2021-08-02T08:59:45.503Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-8' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/e34e1a61-33bf-4df6-815a-2252b614834e] Moving from CHECK_IF_ACCOUNT_EXISTS(STARTED) to CREATE_UPDATE_ENDPOINT(STARTED) 2021-08-02T08:59:45.607Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-32' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/e34e1a61-33bf-4df6-815a-2252b614834e] Moving from CREATE_UPDATE_ENDPOINT(STARTED) to INVOKE_ADAPTER(STARTED) 2021-08-02T08:59:45.750Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-38' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.s.t.encryption.SecretEncryptorImpl.:228 - Using SecureRandom provider: SUN version 11 2021-08-02T08:59:45.759Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-19' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.p.c.model.resources.ComputeService.log:453 - [8282/resources/compute/95f0c10a-ef7b-4fe7-b73e-d7a2b2cc485f] Patching compute [/resources/compute/95f0c10a-ef7b-4fe7-b73e-d7a2b2cc485f] to power state [ON]. Referer: http://10.244.0.42:8282/provisioning/vsphere/endpoint-config-adapter 2021-08-02T08:59:45.772Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-updates-46' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.p.c.model.adapters.util.TaskManager.log:453 - Patching task http://provisioning-service.prelude.svc.cluster.local:8282/tasks/sub-tasks/2e5679bd-6fd9-4208-8a3c-4f781d66bb30 to stage: FINISHED 2021-08-02T08:59:45.775Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-updates-46' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.p.c.model.adapters.util.TaskManager.log:453 - Sending with Retry to http://provisioning-service.prelude.svc.cluster.local:8282/tasks/sub-tasks/2e5679bd-6fd9-4208-8a3c-4f781d66bb30, authContext.isSystemUser=false, expiration: 1627922040 2021-08-02T08:59:45.785Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-42' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/e34e1a61-33bf-4df6-815a-2252b614834e] Moving from INVOKE_ADAPTER(STARTED) to TRIGGER_ENUMERATION(STARTED) 2021-08-02T08:59:45.803Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-20' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/e34e1a61-33bf-4df6-815a-2252b614834e] Moving from TRIGGER_ENUMERATION(STARTED) to COMPLETED(STARTED) 2021-08-02T08:59:45.827Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-14' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/e34e1a61-33bf-4df6-815a-2252b614834e] Moving from COMPLETED(STARTED) to COMPLETED(FINISHED) 2021-08-02T08:59:45.832Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-14' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.p.c.m.t.EndpointAllocationTaskService.log:453 - [8282/provisioning/endpoint-tasks/e34e1a61-33bf-4df6-815a-2252b614834e] Task is complete 2021-08-02T08:59:46.028Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-24' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.a.c.endpoint.EndpointAdapterService.log:453 - [8282/provisioning/mgmt/endpoints] Syncing association links for endpoint: /resources/endpoints/1663dbad-82c5-481f-b624-0cb80d725f3e, as part of endpoint create/patch/delete: STARTED 2021-08-02T08:59:46.030Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-24' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.a.c.endpoint.EndpointAdapterService.log:453 - [8282/provisioning/mgmt/endpoints] Creating 'vc.nukescloud.com' vsphere Endpoint/Post setup: SUCCESS 2021-08-02T08:59:46.032Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-24' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='2fa2972a-2759-44b6-865f-61effeb7fba7' span='cf513f45-8d1f-47e8-bc2b-5fe5e28d2c5d'] c.v.a.c.endpoint.EndpointAdapterService.log:453 - [8282/provisioning/mgmt/endpoints] Creating 'vc.nukescloud.com' vsphere Endpoint: COMPLETED with { "endpointType": "vsphere", "authCredentialsLink": "/core/auth/credentials/87e2e8e3-be47-4996-9a63-886c73babc45", "computeLink": "/resources/compute/95f0c10a-ef7b-4fe7-b73e-d7a2b2cc485f", "computeDescriptionLink": "/resources/compute-descriptions/27d9795f-99f1-4de8-bfbe-8072b5cf2ff1", "resourcePoolLink": "/resources/pools/a4be4bfaf9cee675-7f703c5265a63d87", "associatedEndpointLinks": [], "endpointProperties": { "hostName": "vc.nukescloud.com", "certificate": "-----BEGIN CERTIFICATE-----\nMIID9zCCAt+gAwIBAgIJAMUYdQTxcp75MA0GCSqGSIb3DQEBCwUAMIGYMQswCQYD\nVQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZ\nFgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExGjAYBgNV\nBAoMEXZjLm51a2VzY2xvdWQuY29tMRswGQYDVQQLDBJWTXdhcmUgRW5naW5lZXJp\nbmcwHhcNMjEwMzAzMTUyNTE4WhcNMjMwMzA0MDMyNTE4WjApMRowGAYDVQQDDBF2\nYy5udWtlc2Nsb3VkLmNvbTELMAkGA1UEBhMCVVMwggEiMA0GCSqGSIb3DQEBAQUA\nA4IBDwAwggEKAoIBAQDSgWFL09VkVvVg1GLcsG7uXuhDmfOekRF+7z/bAfjt4Av7\nOtKa1Yq9foIputhEHfQZdmaGpguEcZW8EoKxvMMMZj2EBWbliT2B6kEbSVWk4fi+\nv8aKGmuHF+zZg5KC8eoOt+yVlYpmaR0nUCVja10MoW8dZUjT2C/eW7+U4gUKfNXj\nAHXU4I3XCkXfE2iTDfDBMM+GqsuL0ngaGzjtBG0MzZSt+i9fzDKTT+24BxSV7zLv\nkeKpS4DlvBGsdHH75bhskAXbjV/fCheOJNvi4DgKcstehACQt9y+7MkNHWacbpuv\nnWxy1dZhc+8xJvo7kCxCvxpYkdEhgcjOPWNU/6CVAgMBAAGjgbEwga4wCwYDVR0P\nBAQDAgOoMBwGA1UdEQQVMBOCEXZjLm51a2VzY2xvdWQuY29tMB0GA1UdDgQWBBQl\n9puInH/VuQXoLbO0s1ycWw1xyzAfBgNVHSMEGDAWgBR9//blgcP/+bH0loJBqwoc\n04D47zBBBggrBgEFBQcBAQQ1MDMwMQYIKwYBBQUHMAKGJWh0dHBzOi8vdmMubnVr\nZXNjbG91ZC5jb20vYWZkL3ZlY3MvY2EwDQYJKoZIhvcNAQELBQADggEBAKgcQvGV\nKeZ0YwFc0yHxJFMWo859kTg7bHKEGidX6Tdw4Oee3ftpSH76f5klRNJVwzIUkbzz\nm+4gRTrG6N0QD3rVdncf6qxxLNEVu+1gPNvydSn0IAf3AAJWpGi7ly/+I85gJL5D\n7fibaafpsjXZi8S4p+3n1uNVCOHzxyiAHbwllniqPafyuapXabVR8mdY9NsGXzBG\nGwJGR8278WqGsVZFkHit9RpN8tIK5U/VbpRLewKVjcg4Vd5FvSnbPMesK4sH2t6w\nycCJcLzX3YEtg5kxcAhoAy7q1772R6r6trReb7fLXkT5MFaONs0Gk9xe7Z86Jv8r\nkrleysU4o3QBB+w=\n-----END CERTIFICATE-----\n", "privateKeyId": "vrealizesrv@nukescloud.com", "supportTagging": "{\"documentKinds\":[\"com:vmware:photon:controller:model:resources:ComputeService:ComputeState\"],\"enabled\":true}", "supportDatastores": "true" }, "id": "7f71f149-4387-4af8-a500-4472f6b8cc21", "name": "vc.nukescloud.com", "desc": "vCenter for nukescloud", "customProperties": { "dcId": "onprem", "vcUuid": "ce88c15a-f576-4d48-b567-47ff72f32aab", "hostName": "vc.nukescloud.com", "isExternal": "false", "certificate": "-----BEGIN CERTIFICATE-----\nMIID9zCCAt+gAwIBAgIJAMUYdQTxcp75MA0GCSqGSIb3DQEBCwUAMIGYMQswCQYD\nVQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZ\nFgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExGjAYBgNV\nBAoMEXZjLm51a2VzY2xvdWQuY29tMRswGQYDVQQLDBJWTXdhcmUgRW5naW5lZXJp\nbmcwHhcNMjEwMzAzMTUyNTE4WhcNMjMwMzA0MDMyNTE4WjApMRowGAYDVQQDDBF2\nYy5udWtlc2Nsb3VkLmNvbTELMAkGA1UEBhMCVVMwggEiMA0GCSqGSIb3DQEBAQUA\nA4IBDwAwggEKAoIBAQDSgWFL09VkVvVg1GLcsG7uXuhDmfOekRF+7z/bAfjt4Av7\nOtKa1Yq9foIputhEHfQZdmaGpguEcZW8EoKxvMMMZj2EBWbliT2B6kEbSVWk4fi+\nv8aKGmuHF+zZg5KC8eoOt+yVlYpmaR0nUCVja10MoW8dZUjT2C/eW7+U4gUKfNXj\nAHXU4I3XCkXfE2iTDfDBMM+GqsuL0ngaGzjtBG0MzZSt+i9fzDKTT+24BxSV7zLv\nkeKpS4DlvBGsdHH75bhskAXbjV/fCheOJNvi4DgKcstehACQt9y+7MkNHWacbpuv\nnWxy1dZhc+8xJvo7kCxCvxpYkdEhgcjOPWNU/6CVAgMBAAGjgbEwga4wCwYDVR0P\nBAQDAgOoMBwGA1UdEQQVMBOCEXZjLm51a2VzY2xvdWQuY29tMB0GA1UdDgQWBBQl\n9puInH/VuQXoLbO0s1ycWw1xyzAfBgNVHSMEGDAWgBR9//blgcP/+bH0loJBqwoc\n04D47zBBBggrBgEFBQcBAQQ1MDMwMQYIKwYBBQUHMAKGJWh0dHBzOi8vdmMubnVr\nZXNjbG91ZC5jb20vYWZkL3ZlY3MvY2EwDQYJKoZIhvcNAQELBQADggEBAKgcQvGV\nKeZ0YwFc0yHxJFMWo859kTg7bHKEGidX6Tdw4Oee3ftpSH76f5klRNJVwzIUkbzz\nm+4gRTrG6N0QD3rVdncf6qxxLNEVu+1gPNvydSn0IAf3AAJWpGi7ly/+I85gJL5D\n7fibaafpsjXZi8S4p+3n1uNVCOHzxyiAHbwllniqPafyuapXabVR8mdY9NsGXzBG\nGwJGR8278WqGsVZFkHit9RpN8tIK5U/VbpRLewKVjcg4Vd5FvSnbPMesK4sH2t6w\nycCJcLzX3YEtg5kxcAhoAy7q1772R6r6trReb7fLXkT5MFaONs0Gk9xe7Z86Jv8r\nkrleysU4o3QBB+w=\n-----END CERTIFICATE-----\n", "privateKeyId": "vrealizesrv@nukescloud.com", "acceptSelfSignedCertificate": "false" }, "tenantLinks": [ "/tenants/organization/ae2ac46a-281f-49ec-a2f8-559dfc834eb0", "/tenants/project/a4be4bfaf9cee675-7f703c5265a63d87" ], "tagLinks": [], "orgAuthLink": "/tenants/organization/ae2ac46a-281f-49ec-a2f8-559dfc834eb0", "computeHostLink": "/resources/compute/95f0c10a-ef7b-4fe7-b73e-d7a2b2cc485f", "expandedTags": [], "documentCreationTimeMicros": 1627894785580000, "documentVersion": 1, "documentKind": "com:vmware:photon:controller:model:resources:EndpointService:EndpointState", "documentSelfLink": "/resources/endpoints/1663dbad-82c5-481f-b624-0cb80d725f3e", "documentUpdateTimeMicros": 1627894785748000, "documentUpdateAction": "PATCH", "documentExpirationTimeMicros": 0, "documentAuthPrincipalLink": "/provisioning/auth/csp/users/arun" }

There are 2 types of enumerations in vRealize Automation 8.x for vSphere Cloud Account vSphere enumeration Image enumeration After endpoint is added, then we have the Cloud Zone created or enabled 2021-08-02T08:59:46.183Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-23' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='9c1ddf19-a381-4afd-bc02-148ee7bbd4dd' span='d070688a-c25d-4865-ba2f-8787e6bfd4ee'] c.v.a.c.endpoint.EndpointRegionsService.log:453 - [8282/provisioning/mgmt/endpoint-regions] Enabling region Datacenter:datacenter-3 in endpoint /resources/endpoints/1663dbad-82c5-481f-b624-0cb80d725f3e Image Enumeration Deep-Dive Log Reference: provisioning-service-app.log *** Schedules Image enumeration *** 2021-08-02T08:59:46.267Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-updates-49' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='9c1ddf19-a381-4afd-bc02-148ee7bbd4dd' span='d070688a-c25d-4865-ba2f-8787e6bfd4ee'] c.vmware.admiral.host.ProvisioningHost.log:453 - Scheduling public image-enumeration ID: vsphere-Datacenter:datacenter-3-public-img-enum: not supported 2021-08-02T08:59:46.271Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-44' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='9c1ddf19-a381-4afd-bc02-148ee7bbd4dd' span='d070688a-c25d-4865-ba2f-8787e6bfd4ee'] c.v.a.compute.PlacementZoneService.log:453 - [8282/provisioning/resources/placement-zones/9f90631b-6e64-4f50-933c-44af981d7986] validatePlacementPolicyAvailability : Selected placement policy : null : Placement zone : vc.nukescloud.com / nukescloud *** Starting image enumeration *** *** One thing to note, because this is the first image enumeration, there is an initial=true tag below *** 2021-08-02T08:59:46.273Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-18' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='9c1ddf19-a381-4afd-bc02-148ee7bbd4dd' span='d070688a-c25d-4865-ba2f-8787e6bfd4ee'] c.v.p.c.m.tasks.ScheduledTaskService.log:453 - [8282/provisioning/scheduled-tasks/vsphere-Datacenter:datacenter-3-e8ed41-img-enum] Scheduled '/provisioning/image-enumeration-tasks' execution (initial=true): STARTING *** Schedules private image enumeration *** 2021-08-02T08:59:46.290Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-updates-51' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='9c1ddf19-a381-4afd-bc02-148ee7bbd4dd' span='d070688a-c25d-4865-ba2f-8787e6bfd4ee'] c.vmware.admiral.host.ProvisioningHost.log:453 - Scheduling private image-enumeration ID: vsphere-Datacenter:datacenter-3-e8ed41-img-enum: SUCCESS *** Publishes a CUD event , nothing but a Create/Update/Delete event *** 2021-08-02T08:59:46.299Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='ebs-1' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='523d6952-edae-49f8-8494-5be4bedb6ba9' parent='cd3e7c79-e02c-40fc-be5b-939ae17f75ce' span='3bf617c0-4f78-401c-9f4f-15e15118d5ce'] c.v.p.e.EventBrokerSubscriptionManager.lambda$publishEvent$18:386 - [ebs-integration] Published event Event[id='8f139703-e426-f2f0-2372-a34dc02fff3a'(endpoint.cud), orgId='ae2ac46a-281f-49ec-a2f8-559dfc834eb0', correlation='null'(null), userName='N/A'], Data= {endpointType=vsphere, customProperties={hostName=vc.nukescloud.com, acceptSelfSignedCertificate=false, dcId=onprem, isExternal=false, certificate=-----BEGIN CERTIFICATE----- MIID9zCCAt+gAwIBAgIJAMUYdQTxcp75MA0GCSqGSIb3DQEBCwUAMIGYMQswCQYD VQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZ FgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExGjAYBgNV BAoMEXZjLm51a2VzY2xvdWQuY29tMRswGQYDVQQLDBJWTXdhcmUgRW5naW5lZXJp bmcwHhcNMjEwMzAzMTUyNTE4WhcNMjMwMzA0MDMyNTE4WjApMRowGAYDVQQDDBF2 Yy5udWtlc2Nsb3VkLmNvbTELMAkGA1UEBhMCVVMwggEiMA0GCSqGSIb3DQEBAQUA A4IBDwAwggEKAoIBAQDSgWFL09VkVvVg1GLcsG7uXuhDmfOekRF+7z/bAfjt4Av7 OtKa1Yq9foIputhEHfQZdmaGpguEcZW8EoKxvMMMZj2EBWbliT2B6kEbSVWk4fi+ v8aKGmuHF+zZg5KC8eoOt+yVlYpmaR0nUCVja10MoW8dZUjT2C/eW7+U4gUKfNXj AHXU4I3XCkXfE2iTDfDBMM+GqsuL0ngaGzjtBG0MzZSt+i9fzDKTT+24BxSV7zLv keKpS4DlvBGsdHH75bhskAXbjV/fCheOJNvi4DgKcstehACQt9y+7MkNHWacbpuv nWxy1dZhc+8xJvo7kCxCvxpYkdEhgcjOPWNU/6CVAgMBAAGjgbEwga4wCwYDVR0P BAQDAgOoMBwGA1UdEQQVMBOCEXZjLm51a2VzY2xvdWQuY29tMB0GA1UdDgQWBBQl 9puInH/VuQXoLbO0s1ycWw1xyzAfBgNVHSMEGDAWgBR9//blgcP/+bH0loJBqwoc 04D47zBBBggrBgEFBQcBAQQ1MDMwMQYIKwYBBQUHMAKGJWh0dHBzOi8vdmMubnVr ZXNjbG91ZC5jb20vYWZkL3ZlY3MvY2EwDQYJKoZIhvcNAQELBQADggEBAKgcQvGV KeZ0YwFc0yHxJFMWo859kTg7bHKEGidX6Tdw4Oee3ftpSH76f5klRNJVwzIUkbzz m+4gRTrG6N0QD3rVdncf6qxxLNEVu+1gPNvydSn0IAf3AAJWpGi7ly/+I85gJL5D 7fibaafpsjXZi8S4p+3n1uNVCOHzxyiAHbwllniqPafyuapXabVR8mdY9NsGXzBG GwJGR8278WqGsVZFkHit9RpN8tIK5U/VbpRLewKVjcg4Vd5FvSnbPMesK4sH2t6w ycCJcLzX3YEtg5kxcAhoAy7q1772R6r6trReb7fLXkT5MFaONs0Gk9xe7Z86Jv8r krleysU4o3QBB+w= -----END CERTIFICATE----- , privateKeyId=vrealizesrv@nukescloud.com, vcUuid=ce88c15a-f576-4d48-b567-47ff72f32aab}} serviceCallback=null, documentSelfLink=null, taskStateClassName=EndpointAdapterService. *** Post call for image enumeration tasks starts *** 2021-08-02T08:59:46.306Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='reactor-http-epoll-13' user='provisioning-FkAbXIHIXWYeerqm' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='6aaa8b38-dd94-4224-90a7-a77b94070b67' span='cd3610af-51b6-4f9a-b90b-94ff906976d8'] reactor.netty.http.server.AccessLog.info:274 - 10.244.0.42 - - [02/Aug/2021:08:59:46 +0000] "POST /provisioning/image-enumeration-tasks HTTP/1.1" 200 868 8282 30 ms 2021-08-02T08:59:46.307Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='reactor-http-epoll-12' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='9c1ddf19-a381-4afd-bc02-148ee7bbd4dd' span='d070688a-c25d-4865-ba2f-8787e6bfd4ee'] c.v.p.c.m.tasks.ScheduledTaskService.log:453 - [8282/provisioning/scheduled-tasks/vsphere-Datacenter:datacenter-3-e8ed41-img-enum] Scheduled '/provisioning/image-enumeration-tasks' execution: SUCCESS *** Status changes from Started to Created *** 2021-08-02T08:59:46.320Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-21' user='provisioning-FkAbXIHIXWYeerqm' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='6aaa8b38-dd94-4224-90a7-a77b94070b67' span='cd3610af-51b6-4f9a-b90b-94ff906976d8'] c.v.a.c.ImageEnumerationTaskService.log:453 - [8282/provisioning/image-enumeration-tasks/vsphere-Datacenter:datacenter-3-e8ed41-img-enum] Moving from CREATED to STARTED (referer: http://10.244.0.42:8282/provisioning/image-enumeration-tasks/vsphere-Datacenter:datacenter-3-e8ed41-img-enum) 2021-08-02T08:59:46.347Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='reactor-http-epoll-16' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='9c1ddf19-a381-4afd-bc02-148ee7bbd4dd' span='d070688a-c25d-4865-ba2f-8787e6bfd4ee'] reactor.netty.http.server.AccessLog.info:274 - 10.244.0.42 - - [02/Aug/2021:08:59:46 +0000] "POST /provisioning/mgmt/endpoint-regions?enumerate HTTP/1.1" 200 256 8282 180 ms 2021-08-02T08:59:46.348Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='reactor-http-epoll-16' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='' span='57bd4faa-5427-4d59-b3f2-f88c3840f426'] reactor.netty.http.server.AccessLog.info:274 - 10.244.0.33 - - [02/Aug/2021:08:59:46 +0000] "POST /provisioning/uerp/provisioning/mgmt/endpoint-regions?enumerate HTTP/1.1" 200 256 8282 192 ms 2021-08-02T08:59:46.370Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-26' user='provisioning-FkAbXIHIXWYeerqm' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='6aaa8b38-dd94-4224-90a7-a77b94070b67' span='cd3610af-51b6-4f9a-b90b-94ff906976d8'] c.v.a.c.ImageEnumerationTaskService.log:453 - [8282/provisioning/image-enumeration-tasks/vsphere-Datacenter:datacenter-3-e8ed41-img-enum] [getEndpointStateByLink] 'vc.nukescloud.com / /resources/endpoints/1663dbad-82c5-481f-b624-0cb80d725f3e' endpoint of (type=vsphere, region=Datacenter:datacenter-3): SUCCESS 2021-08-02T08:59:46.377Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-8' user='provisioning-FkAbXIHIXWYeerqm' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='6aaa8b38-dd94-4224-90a7-a77b94070b67' span='cd3610af-51b6-4f9a-b90b-94ff906976d8'] c.v.a.c.ImageEnumerationTaskService.log:453 - [8282/provisioning/image-enumeration-tasks/vsphere-Datacenter:datacenter-3-e8ed41-img-enum] [getImageEnumerationAdapterReference] adapterRef = http://provisioning-service.prelude.svc.cluster.local:8282/provisioning/vsphere/image-enumeration-adapter: SUCCESS 2021-08-02T08:59:46.379Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-updates-11' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='9c1ddf19-a381-4afd-bc02-148ee7bbd4dd' span='d070688a-c25d-4865-ba2f-8787e6bfd4ee'] c.v.a.c.e.EndpointRegionsTaskService.validateStageTransitionAndState:1428 - [/provisioning/requests/endpoint-regions-tasks/1b59e091-3d5b-4ac8-9d2b-76f09ddc15a8] Moving from STARTED(CREATED) to STARTED(CREATED). 2021-08-02T08:59:46.390Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-8' user='provisioning-FkAbXIHIXWYeerqm' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='6aaa8b38-dd94-4224-90a7-a77b94070b67' span='cd3610af-51b6-4f9a-b90b-94ff906976d8'] c.v.a.c.ImageEnumerationTaskService.log:453 - [8282/provisioning/image-enumeration-tasks/vsphere-Datacenter:datacenter-3-e8ed41-img-enum] [callImageEnumerationAdapter] adapterRef = http://provisioning-service.prelude.svc.cluster.local:8282/provisioning/vsphere/image-enumeration-adapter: SUCCESS 2021-08-02T08:59:46.391Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-8' user='provisioning-FkAbXIHIXWYeerqm' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='6aaa8b38-dd94-4224-90a7-a77b94070b67' span='cd3610af-51b6-4f9a-b90b-94ff906976d8'] c.v.a.c.ImageEnumerationTaskService.log:453 - [8282/provisioning/image-enumeration-tasks/vsphere-Datacenter:datacenter-3-e8ed41-img-enum] sendImageEnumerationAdapterRequest: SUCCESS *** Adapter call starts *** 2021-08-02T08:59:46.404Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-27' user='provisioning-FkAbXIHIXWYeerqm' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='6aaa8b38-dd94-4224-90a7-a77b94070b67' span='cd3610af-51b6-4f9a-b90b-94ff906976d8'] c.v.a.h.i.DecryptingInterceptor.logInterceptingInfo:436 - Intercepting adapter call (action: GET, dest: http://10.244.0.42:8282/resources/compute/95f0c10a-ef7b-4fe7-b73e-d7a2b2cc485f, referer: http://10.244.0.42:8282/provisioning/vsphere/image-enumeration-adapter) 2021-08-02T08:59:46.407Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-updates-35' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='9c1ddf19-a381-4afd-bc02-148ee7bbd4dd' span='d070688a-c25d-4865-ba2f-8787e6bfd4ee'] c.v.admiral.service.TaskExecutionLimit.acquire:125 - Acquiring permit for task: /provisioning/requests/endpoint-regions-tasks/1b59e091-3d5b-4ac8-9d2b-76f09ddc15a8 STARTED 2021-08-02T08:59:46.408Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-updates-35' user='arun' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='9c1ddf19-a381-4afd-bc02-148ee7bbd4dd' span='d070688a-c25d-4865-ba2f-8787e6bfd4ee'] c.v.admiral.service.TaskExecutionLimit.acquire:133 - Acquiring permit for task: /provisioning/requests/endpoint-regions-tasks/1b59e091-3d5b-4ac8-9d2b-76f09ddc15a8 SUCCESS *** Moves from Started created to Started start enumeration *** 2021-08-02T08:59:46.408Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-27' user='provisioning-FkAbXIHIXWYeerqm' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='6aaa8b38-dd94-4224-90a7-a77b94070b67' span='cd3610af-51b6-4f9a-b90b-94ff906976d8'] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] Image enumeration Context: [vc.nukescloud.com], [/resources/endpoints/1663dbad-82c5-481f-b624-0cb80d725f3e], [https://vc.nukescloud.com/sdk], [vrealizesrv@nukescloud.com] 2021-08-02T08:59:46.437Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-32' user='provisioning-FkAbXIHIXWYeerqm' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='6aaa8b38-dd94-4224-90a7-a77b94070b67' span='cd3610af-51b6-4f9a-b90b-94ff906976d8'] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: complete on processing host cluster *** Get's a successful login *** 2021-08-02T08:59:46.545Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-28' user='provisioning-FkAbXIHIXWYeerqm' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='6aaa8b38-dd94-4224-90a7-a77b94070b67' span='cd3610af-51b6-4f9a-b90b-94ff906976d8'] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: Successful Login 2021-08-02T08:59:46.790Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-28' user='provisioning-FkAbXIHIXWYeerqm' org='ae2ac46a-281f-49ec-a2f8-559dfc834eb0' trace='23fe8e95-369f-4aaa-b72c-33c3c8b8c4ef' parent='6aaa8b38-dd94-4224-90a7-a77b94070b67' span='cd3610af-51b6-4f9a-b90b-94ff906976d8'] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: complete on processing host cluster *** Fetches the vSphere templates *** 2021-08-02T08:59:47.021Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: Updating clone template in database: [photonfour] 2021-08-02T08:59:47.025Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: Updating clone template in database: [sixteen] 2021-08-02T08:59:47.025Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: Updating clone template in database: [rheleight] *** Fetches content library templates *** 2021-08-02T08:59:47.062Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: Begin processing all library items. Enable vm template enumeration: true 2021-08-02T08:59:47.351Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='reactor-http-epoll-10' user='' org='' trace='f064745f-7f82-4bf5-957f-237b5259e439' parent='' span='b590e1ee-b782-4e18-ad50-f177b483979d'] reactor.netty.http.server.AccessLog.info:274 - 10.244.0.1 - - [02/Aug/2021:08:59:47 +0000] "GET /provisioning/mgmt/status?k8s-probe HTTP/1.1" 200 0 8282 3 ms 2021-08-02T08:59:47.545Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: local publisher library set [089472c1-1a59-40f1-96b7-495dfa7e9d72] 2021-08-02T08:59:47.547Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: remote library set [] 2021-08-02T08:59:47.548Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: Begin processing items in content library [nukescloud-cl, 089472c1-1a59-40f1-96b7-495dfa7e9d72] 2021-08-02T08:59:47.800Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: Retrieving disk config for item [nukescloud-cl / sixteen],[0001b57d-4a0a-4f7e-9342-b4dada6e949c], populate disk [true] 2021-08-02T08:59:47.801Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: [nukescloud-cl / sixteen] local content item, ignore populate publisher 2021-08-02T08:59:48.363Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.vsphere.vapi.LibraryClient.loadItem:153 - Loading library item 192b568d-b9d1-4394-bc55-12fa2365581d : 2021-08-02T08:59:48.364Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: Retrieving disk config for item [nukescloud-cl / rheleight],[192b568d-b9d1-4394-bc55-12fa2365581d], populate disk [true] 2021-08-02T08:59:48.364Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: [nukescloud-cl / rheleight] local content item, ignore populate publisher 2021-08-02T08:59:49.012Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: Finish processing items in content library [nukescloud-cl, 089472c1-1a59-40f1-96b7-495dfa7e9d72] 2021-08-02T08:59:53.090Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-service-scheduled-56' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: item [0001b57d-4a0a-4f7e-9342-b4dada6e949c], disk id : [vmdisk1], capacity MB :[92160] 2021-08-02T08:59:53.099Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-service-scheduled-56' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: item [0001b57d-4a0a-4f7e-9342-b4dada6e949c,nukescloud-cl / sixteen], os family: [WINDOWS] *** Maps disks to the library items enumerated *** 2021-08-02T08:59:54.524Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-service-scheduled-55' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: item [192b568d-b9d1-4394-bc55-12fa2365581d], disk id : [vmdisk1], capacity MB :[61440] 2021-08-02T08:59:54.530Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-service-scheduled-55' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: item [192b568d-b9d1-4394-bc55-12fa2365581d,nukescloud-cl / rheleight], os family: [LINUX] 2021-08-02T08:59:54.556Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: complete processing all library items. 2021-08-02T08:59:54.557Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: begin processing image datastore map for publisher/subscriber library items. 2021-08-02T08:59:54.564Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-7' user='' org='' trace='' parent='' span=''] c.v.a.c.ImageEnumerationTaskService.log:453 - [8282/provisioning/image-enumeration-tasks/vsphere-Datacenter:datacenter-3-e8ed41-img-enum] Moving from STARTED to FINISHED (referer: http://10.244.0.42:8282/provisioning/vsphere/image-enumeration-adapter) 2021-08-02T08:59:54.568Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] [vc.nukescloud.com]: complete processing image datastore map for publisher/subscriber library items. 2021-08-02T08:59:54.569Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-57' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.VSphereAdapterImageEnumerationService.log:453 - [8282/provisioning/vsphere/image-enumeration-adapter] End of adapter image enumeration, close connection, [vc.nukescloud.com] *** Private images enumeration is now complete *** 2021-08-02T08:59:54.603Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-updates-22' user='' org='' trace='' parent='' span=''] c.v.a.c.ImageEnumerationTaskService.log:453 - [8282/provisioning/image-enumeration-tasks/vsphere-Datacenter:datacenter-3-e8ed41-img-enum] Private images enum (type=null, region=Datacenter:datacenter-3, endpointLink=/resources/endpoints/1663dbad-82c5-481f-b624-0cb80d725f3e): SUCCESS 2021-08-02T08:59:54.603Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-updates-22' user='' org='' trace='' parent='' span=''] c.v.a.c.ImageEnumerationTaskService.log:453 - [8282/provisioning/image-enumeration-tasks/vsphere-Datacenter:datacenter-3-e8ed41-img-enum] Private images enum (type=null, region=Datacenter:datacenter-3, endpointLink=/resources/endpoints/1663dbad-82c5-481f-b624-0cb80d725f3e): SUCCESS 2021-08-02T08:59:54.621Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-updates-52' user='' org='' trace='' parent='' span=''] c.v.a.c.ImageEnumerationTaskService.log:453 - [8282/provisioning/image-enumeration-tasks/vsphere-Datacenter:datacenter-3-e8ed41-img-enum] Self-delete upon FINISHED stage: SUCCESS

Detailed Analysis available here :https://tinyurl.com/vmvtrn66 There are 2 Types of enumerations which occur Full Enumeration Incremental Enumeration After a Cloud Account is added , a full enumeration is performed. Post successful initial enumeration , every other enumeration which is performed after an account is added is an incremental one. Full enumeration occurs every 24 hours and Incremental occurs every 10 minutes Activities performed during Full Enumeration and Incremental enumeration is pretty much the same. Just that we collect only changes when we perform the latter action. If we do math.... 143 Attempts of Full Enumeration in 24 hours vSphere Enumeration Deep-Dive ( First one after creating Cloud Account , A Full Enumeration ) Prepares for Enumeration Starts direct Resource Enumeration by sending request to endpoint adapter Starts collecting tags from vSphere endpoint After tags collection , it starts collecting supervisor cluster information Then comes Datacenter enumeration Fetches and creates folder objects Fetches and creates Storage Pod objects Fetches and creates Datastore objects Fetches and creates Network Objects Opaque Network objects are created Fetches Cluster information and creates it Hosts information is collected and created Resource Pool objects are created Collects the VM's information Creates VM Objects Storage Policy information would be collected Enumeration is marked as finished Incremental Enumeration Tasks performed during incremental enumeration is pretty much the same. But only the changes are collected from vCenter Few example statements during incremental enumeration are below 021-08-02T14:12:53.727Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='scheduling-1' user='' org='' trace='cf6b1ebb-6b1c-4f98-a0c8-a494e27c24e0' parent='' span='575e1683-be35-48e7-bb05 -84f6e7887d95'] c.v.a.s.v.VSphereIncrementalEnumerationController.doIncrementalEnumeration:325 - Starting incremental enumeration of /provisioning/vsphere/incremental-enumeration-adapter/1663dbad-82c5-481f-b624-0cb 80d725f3e 2021-08-02T14:12:53.745Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-30' user='' org='' trace='cf6b1ebb-6b1c-4f98-a0c8-a494e27c24e0' parent='' span='575e1683-be35-48 e7-bb05-84f6e7887d95'] c.v.a.s.v.VSphereIncrementalEnumerationController.lambda$doIncrementalEnumeration$8:337 - /resources/endpoints/1663dbad-82c5-481f-b624-0cb80d725f3e: The following regions will be collected: [ ["Datacenter:datacenter-3"]] 2021-08-02T14:12:53.745Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-30' user='' org='' trace='cf6b1ebb-6b1c-4f98-a0c8-a494e27c24e0' parent='' span='575e1683-be35-48 e7-bb05-84f6e7887d95'] c.v.a.s.v.VSphereIncrementalEnumerationController.lambda$doIncrementalEnumeration$6:353 - Successfully started incremental enumeration of /provisioning/vsphere/incremental-enumeration-adapter/1663dbad-82c5-481f-b624-0cb80d725f3e * * 2021-08-02T14:12:54.049Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-2' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.TagEnumerationService.collectIdentifiers:113 - [vc.nukescloud.com]: Begin collecting tags [com.vmware.photon.controller.model.adapters.vsphere.enumeration.VsphereEnumerationAdapterService$EnumerationContext@49071a12]: Start collecting supervisor resources. * * * 2021-08-02T14:12:54.506Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.VsphereEnumerationAdapterService.lambda$enumerateDatacenter$25:717 - [vc.nukescloud.com][nukescloud] Starting datacenter enumeration. 2021-08-02T14:12:54.506Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.enumerate:287 - [[vc.nukescloud.com]:[nukescloud]] Starting datacenter enumeration. * * * 2021-08-02T14:12:54.710Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-3' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:17 9 - [vc.nukescloud.com]:[nukescloud]:[is-tse-d12]: Successfully updated resource [Name: [is-tse-d12], mo-ref: [datastore-1014], capacity: [461977419776] bytes, free capacity: [460461178880] bytes] 2021-08-02T14:12:54.715Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-1' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:17 9 - [vc.nukescloud.com]:[nukescloud]:[is-tse-d14]: Successfully updated resource [Name: [is-tse-d14], mo-ref: [datastore-1024], capacity: [461977419776] bytes, free capacity: [460461178880] bytes] 2021-08-02T14:12:54.715Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-6' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:17 9 - [vc.nukescloud.com]:[nukescloud]:[is-tse-d11]: Successfully updated resource [Name: [is-tse-d11], mo-ref: [datastore-1005], capacity: [461977419776] bytes, free capacity: [460461178880] bytes] 2021-08-02T14:12:54.720Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-4' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:179 - [vc.nukescloud.com]:[nukescloud]:[is-tse-d13]: Successfully updated resource [Name: [is-tse-d13], mo-ref: [datastore-1015], capacity: [461977419776] bytes, free capacity: [460461178880] bytes] 2021-08-02T14:12:54.720Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-7' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:179 - [vc.nukescloud.com]:[nukescloud]:[shareddsone]: Successfully updated resource [Name: [shareddsone], mo-ref: [datastore-46], capacity: [6596801331200] bytes, free capacity: [2023744339968] bytes] * * * 2021-08-02T14:12:54.825Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-6' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$processPortgroups$15:591 - [vc.nukescloud.com]:[nukescloud]: Total portgroups: [2] 2021-08-02T14:12:54.846Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-4' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.ProcessorDelegate.lambda$getExistingResources$15:354 - [[vc.nukescloud.com]:[nukescloud]]: Fetched 2 existing resources of DistributedVirtualPortgroup... 2021-08-02T14:12:54.847Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.SubnetProcessor.processUpdate:62 - [vc.nukescloud.com]:[nukescloud]: No change detected in subnet [DSwitch-DVUplinks-1218]. Skipping update. 2021-08-02T14:12:54.847Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-4' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.SubnetProcessor.processUpdate:62 - [vc.nukescloud.com]:[nukescloud]: No change detected in subnet [DPortGroup]. Skipping update. 2021-08-02T14:12:54.847Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-4' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.ResourceBatchProcessor.fetchToBeDeletedResources:193 - [[vc.nukescloud.com]:[nukescloud]] Will delete all resources of type [DistributedVirtualPortgroup] that do not have self links in the list [["/resources/sub-networks/beef0b4a-140e-48ff-937b-2b96d1914974","/resources/sub-networks/1e031928-4404-4eb0-a9dd-25241b7b9975"]]. 2021-08-02T14:12:54.849Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-2' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.ResourceBatchProcessor.lambda$deleteResources$2:124 - [vc.nukescloud.com:nukescloud Deleting 0 resources of type DistributedVirtualPortgroup.] 2021-08-02T14:12:54.985Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-2' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$processNetworks$19:633 - [vc.nukescloud.com]:[nukescloud]: Total networks: [2] 2021-08-02T14:12:55.005Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.ProcessorDelegate.lambda$getExistingResources$15:354 - [[vc.nukescloud.com]:[nukescloud]]: Fetched 2 existing resources of Network... 2021-08-02T14:12:55.005Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-3' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.NetworkProcessor.processUpdate:93 - [vc.nukescloud.com]:[nukescloud]: No change detected in network [VM Network]. Skipping update. 2021-08-02T14:12:55.005Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.NetworkProcessor.processUpdate:93 - [vc.nukescloud.com]:[nukescloud]: No change detected in network [VM Network 2]. Skipping update. * * * 2021-08-02T14:12:55.369Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-6' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:179 - [vc.nukescloud.com]:[nukescloud]:[myworld]: Successfully updated resource [Name: [myworld], mo-ref: [domain-c8], isDRSEnabled: [true], max vcpu count: [48], numCpuCores: [96], total cpuMhz: [210624] totalMemory: [1097922330624] bytes, effectiveMemory: [1050290421760] bytes] * * * 2021-08-02T14:12:55.604Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-8' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:179 - [vc.nukescloud.com]:[nukescloud]:[hostfour.nukescloud.com]: Successfully updated resource [Name: [hostfour.nukescloud.com], mo-ref: [host-1023], parent: [ClusterComputeResource:domain-c8], numCpuThreads: [48], cpuMhz: [2194], totalMemory: [274480496640] bytes, availableMemory: [124851847168] bytes] 2021-08-02T14:12:55.609Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:179 - [vc.nukescloud.com]:[nukescloud]:[hosttwo.nukescloud.com]: Successfully updated resource [Name: [hosttwo.nukescloud.com], mo-ref: [host-1010], parent: [ClusterComputeResource:domain-c8], numCpuThreads: [48], cpuMhz: [2194], totalMemory: [274480447488] bytes, availableMemory: [127551881216] bytes] 2021-08-02T14:12:55.609Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-6' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:179 - [vc.nukescloud.com]:[nukescloud]:[hostthree.nukescloud.com]: Successfully updated resource [Name: [hostthree.nukescloud.com], mo-ref: [host-1013], parent: [ClusterComputeResource:domain-c8], numCpuThreads: [48], cpuMhz: [2194], totalMemory: [274480447488] bytes, availableMemory: [134771326976] bytes] 2021-08-02T14:12:55.609Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-1' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:179 - [vc.nukescloud.com]:[nukescloud]:[hostone.nukescloud.com]: Successfully updated resource [Name: [hostone.nukescloud.com], mo-ref: [host-1004], parent: [ClusterComputeResource:domain-c8], numCpuThreads: [48], cpuMhz: [2194], totalMemory: [274480939008] bytes, availableMemory: [126811029504] bytes] * * * c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$processMachines$46:1015 - [[vc.nukescloud.com]:[nukescloud]] Successfully processed VMs via incremental enumeration, next update version [nothing-changed], incremental enumeration iterations performed [30] 2021-08-02T14:12:56.807Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-7' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.processFirstClassDisks:1200 - [[vc.nukescloud.com]:[nukescloud]] Starting FCD enumeration. 2021-08-02T14:12:56.818Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-23' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$getExistingDisks$73:1363 - fcd-enum for endpoint [/resources/endpoints/1663dbad-82c5-481f-b624-0cb80d725f3e]: Time Taken to query existing disks, in millis is 6 2021-08-02T14:12:56.841Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-8' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$fetchDatastores$75:1398 - fcd-enum: Time to fetch datastores in millis in 23 2021-08-02T14:12:56.841Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-8' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$fetchDatastores$75:1404 - [vc.nukescloud.com] Successfully fetched [5] datastores for [5] datastore IDs... 2021-08-02T14:12:56.947Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-8' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$enumerateFirstClassDisks$66:1271 - [[vc.nukescloud.com]:[nukescloud]] Successfully created FCDs via enumeration 2021-08-02T14:12:56.947Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-8' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.FirstClassDiskProcessorUtils.deleteDisks:304 - /resources/endpoints/1663dbad-82c5-481f-b624-0cb80d725f3e No FCDs to be deleted 2021-08-02T14:12:56.947Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-8' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$enumerateFirstClassDisks$68:1285 - fcd-enum: Time taken to complete delete of FCDs in millis is 0 2021-08-02T14:12:56.947Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-8' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$enumerateFirstClassDisks$68:1294 - [[vc.nukescloud.com]:[nukescloud]] Successfully deleted FCDs via enumeration 2021-08-02T14:12:56.986Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-25' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.ClusterReconciliationService.lambda$getUpdatedClusterState$6:138 - [vc.nukescloud.com]: Successfully updated cluster [myworld] in database with selflink [/resources/compute/0da008cc-1069-4f68-a039-1ce17bef9626] 2021-08-02T14:12:56.987Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-25' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.ClusterReconciliationService.lambda$reconcileClusterCPUs$4:91 - [vc.nukescloud.com]: Finished updating 1 clusters in database. 2021-08-02T14:12:57.002Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-2' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$enumerate$0:308 - [vc.nukescloud.com]:[nukescloud]: Finish enumerating datacenter. 2021-08-02T14:12:57.002Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-2' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$enumerate$0:313 - [vc.nukescloud.com]:[nukescloud]: Total elapsed time metrics: { "vc.nukescloud.com.nukescloud.Dvs.processing.total.elapsedTime.millis": "158", "vc.nukescloud.com.nukescloud.machines.processing.total.elapsedTime.millis": "1004", "vc.nukescloud.com.nukescloud.enumerate-fcd.processing.total.elapsedTime.millis": "135", "vc.nukescloud.com.nukescloud.storagepods.processing.total.elapsedTime.millis": "5", "vc.nukescloud.com.nukescloud.resourcepools.processing.total.elapsedTime.millis": "189", "vc.nukescloud.com.nukescloud.hosts.processing.total.elapsedTime.millis": "238", "vc.nukescloud.com.nukescloud.opaque-networks.processing.total.elapsedTime.millis": "3", "vc.nukescloud.com.nukescloud.clusters.processing.total.elapsedTime.millis": "202", "vc.nukescloud.com.nukescloud.folders.processing.total.elapsedTime.millis": "39", "vc.nukescloud.com.nukescloud.portgroups.processing.total.elapsedTime.millis": "125", "vc.nukescloud.com.nukescloud.datastores.processing.total.elapsedTime.millis": "170", "vc.nukescloud.com.nukescloud.networks.processing.total.elapsedTime.millis": "159" } 2021-08-02T14:12:57.003Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-3' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.VsphereEnumerationAdapterService.handleUnmanagedDatacenters:753 - [vc.nukescloud.com]: No unmanaged datacenter found this enumeration. 2021-08-02T14:12:57.004Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-3' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.StoragePolicyEnumerationService.enumerate:130 - [vc.nukescloud.com]: Begin storage policy enumeration. * * * 2021-08-02T14:12:57.634Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-7' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.StoragePolicyEnumerationService.lambda$enumerate$0:142 - [vc.nukescloud.com]: Finish storage policy enumeration. 2021-08-02T14:12:57.641Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.VsphereEnumerationAdapterService.lambda$handlePatch$6:319 - vc.nukescloud.com: Total elapsed time metrics: { "vc.nukescloud.com.tags.processing.total.elapsedTime.millis": "244", "vc.nukescloud.com.storage-policies.processing.total.elapsedTime.millis": "630", "vc.nukescloud.com.supervisor-resources.processing.total.elapsedTime.millis": "140", "vc.nukescloud.com.enumerate-datacenters.processing.total.elapsedTime.millis": "2496" } 2021-08-02T14:12:57.641Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.model.adapters.util.TaskManager.log:453 - Patching task http://10.244.0.42:8282/provisioning/vsphere/incremental-enumeration-adapter/callback/1663dbad-82c5-481f-b624-0cb80d725f3e to stage: FINISHED 2021-08-02T14:12:57.641Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.model.adapters.util.TaskManager.log:453 - Sending with Retry to http://10.244.0.42:8282/provisioning/vsphere/incremental-enumeration-adapter/callback/1663dbad-82c5-481f-b624-0cb80d725f3e, authContext.isSystemUser=true, expiration: 31556889864403199 2021-08-02T14:12:57.641Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.a.s.v.VSphereIncrementalEnumerationController.handlePatch:234 - Callback received on /provisioning/vsphere/incremental-enumeration-adapter/callback/1663dbad-82c5-481f-b624-0cb80d725f3e: FINISHED If you read the above snippets Only changes are collected during incremental enumeration We do mention time taken for each enumeration , this can be used for any performance troubleshooting if needed