Having Multi-Tenancy enabled in vRO causes vRA Upgrade failures during the pre-update phase
Failures are seen during vRA upgrade through vRSLCM 2.1during preupdate phase of vRA appliance upgrade
Upgrade was being performed from vRA 7.4 to 7.6
vRLCM was on version 2.1 Patch 2
/var/log/bootstrap/preupdate.log
Running a check on replicas:
Executing a script on multiple cluster nodes...
2020-05-28 05:34:18+00:00 /etc/bootstrap/preupdate.d/00-00-01-va-resources-check done, succeeded.
2020-05-28 05:34:18+00:00 /etc/bootstrap/preupdate.d/00-00-02-check-replica-availability starting...
Executing a script on multiple cluster nodes...
2020-05-28 05:34:26+00:00 /etc/bootstrap/preupdate.d/00-00-02-check-replica-availability done, succeeded.
2020-05-28 05:34:26+00:00 /etc/bootstrap/preupdate.d/00-00-02-check-vro-duplicates starting...
<!--: spam
Content-Type: text/html
<body bgcolor="#f0f0f8"><font color="#f0f0f8" size="-5"> -->
<body bgcolor="#f0f0f8"><font color="#f0f0f8" size="-5"> --> -->
</font> </font> </font> </script> </object> </blockquote> </pre>
</table> </table> </table> </table> </table> </font> </font> </font><body bgcolor="#f0f0f8">
<table width="100%" cellspacing=0 cellpadding=2 border=0 summary="heading">
<tr bgcolor="#6622aa">
<td valign=bottom> <br>
<font color="#ffffff" face="helvetica, arial"> <br><big><big><strong><type 'exceptions.TypeError'></strong></big></big></font></td
<td align=right valign=bottom
<font color="#ffffff" face="helvetica, arial">Python 2.7.14: /usr/bin/python<br>Thu May 28 05:34:28 2020</font></td></tr></table>
<p>A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.</p>
<table width="100%" cellspacing=0 cellpadding=0 border=0>
<tr><td bgcolor="#d8bbff"><big> </big><a href="file:///etc/bootstrap/preupdate.d/00-00-02-check-vro-duplicates">/etc/bootstrap/preupdate.d/00-00-02-check-vro-duplicates</a> in <strong><module></strong>()</td></tr>
<tr><td><font color="#909090"><tt> <small> 51</small> for dup in duplicates_check_result:<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt> <small> 52</small> dup_line = "{} items in table {}: {} {}".format(str(dup.get('?
column?')), dup.get('type'),<br>
</tt></font></td></tr>
<tr><td bgcolor="#ffccee"><tt>=><small> 53</small>  
; '' if len(dup.get('categoryid')) == 0 
;else "ID=" + dup.get('categoryid'),<br>
</tt></td></tr>
<tr><td><font color="#909090"><tt> <small> 54</small>
'' if len(dup.get('name')) == 0
else "NAME=" + dup.get('name'))<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt> <small> 55</small> msg_html += "<p style='margin-left:20px'>- {}</p>".format(dup_line)<br
</tt></font></td></tr>
<tr><td><small><font color="#909090"><em>builtin</em> <strong>len</strong> = <built-in function len>, <strong>dup</strong> = {'?column?': 1L, 'type': 'vmo_configelementcategory', 'categoryid': None, 'name': 'wbg_SQL'}, du
p.<strong>get</strong> = <built-in method get of RealDictRow object></font></small></td></tr></table><p><strong><type 'exceptions.TypeError'></strong>: object of type 'NoneType' has no len()
<br><tt><small> </small> </tt>args =
("object of type 'NoneType' has no len()",)
<br><tt><small> </small> </tt>message =
"object of type 'NoneType' has no len()"
<!-- The above is a description of an error in a Python program, formatted
for a Web browser because the 'cgitb' module was enabled. In case you
are not reading this in a Web browser, here is the original traceback:
Traceback (most recent call last):/var/log/messages
2020-05-28T05:34:29.194212+00:00 sevenfour kernel: [ 5275.524647] martian source 10.63.249.208 from 0.0.0.0, on dev eth0
2020-05-28T05:34:29.194252+00:00 sevenfour kernel: [ 5275.524657] ll header: ff:ff:ff:ff:ff:ff:00:00:00:00:7b:00:08:00
2020-05-28T05:34:31.588050+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info .................................................................................................................................................
.............................................................................................................................................................................................................................................
...................................................................
2020-05-28T05:34:31.588183+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Failure: Error while running pre-install scripts and post-install scripts
2020-05-28T05:34:31.588194+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info
2020-05-28T05:34:31.603338+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>VA-check: finished</p>
2020-05-28T05:34:31.603561+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>Pre-install: failed (code p-1)</p>
2020-05-28T05:34:31.603870+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>RPM Status 1: <span style='color: red'>Pre install script failed, package test and installation skipped</span>.</p>
2020-05-28T05:34:31.604050+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>Post-install: failed</p>
2020-05-28T05:34:31.604227+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>Update failed (code 1-1). Check logs in /opt/vmware/var/log/vami or retry update later.</p>
2020-05-28T05:34:31.604354+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <br />
2020-05-28T05:34:31.607027+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Finished..
2020-05-28T05:34:31.607088+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Executing shell command...
2020-05-28T05:34:35.725031+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Calling cluster-command-get-next
2020-05-28T05:34:35.725081+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Executing shell command...Executing KB 54987 shows that there is no database configuration update needed. Which means there are actually no duplicates
sevenfour:/tmp # /usr/lib/vco-cli/bin/vro-configure.sh db-validate
Orchestrator's root folder: /var/lib/vco
Orchestrator Configuration Tool. Version: 7.4.0.8989539 Build: 8989539
ValidateDbOptions{dbUsername='vcac', dbPassword='****', jdbcUrl='jdbc:postgresql://127.0.0.1:5433/vcac?sslmode=verify-ca&sslrootcert=/var/vmware/vpostgres/current/.postgresql/root.crt'}
The command does not need a database configuration update.Even then the precheck fails with the exception stating there are duplicates. This is because of Multi-Tenancy enabled in vRO
The duplicates in vRO are coming from the fact that they are using a multi-tenanted environment and vro duplicates script is not designed for that.
Remediation Plan
This is how you remediate this problem so that a successful upgrade can be performed
Step:1
Extract the attached archive 00-00-00-00-remove-check-vro-duplicates.zip into the VA you are going to upgrade In the folder: /etc/bootstrap/preupdate.d
00-00-00-00-remove-check-vro-duplicates
Download • 785B
Step:2
Execute below command to change permissions
chmod 744 /etc/bootstrap/preupdate.d/00-00-00-00-remove-check-vro-duplicatesThen start the upgrade either through vRLCM or through VAMI.
This should help in progressing upgrade further without any failures.
What is this additional script doing?
This script 00-00-00-00-remove-check-vro-duplicates deletes 00-00-02-check-vro-duplicates
when triggered
#!/bin/sh
rm -f /etc/bootstrap/preupdate.d/00-00-02-check-vro-duplicatesIdeally /etc/bootstrap/preupdate.d/* contains all the scripts which has to run during preupdate phase of vRA upgrade
[master] sevenvra:/etc/bootstrap/preupdate.d # ls -l
total 160
-rwxr--r-- 1 root root 2284 Mar 28 2019 00-00-00-prereq-checks
-rwxr--r-- 1 root root 781 Mar 28 2019 00-00-00-wizard-completed
-rwxr--r-- 1 root root 655 Mar 28 2019 00-00-00-z-disable-db-failover
-rwxr--r-- 1 root root 1883 Mar 28 2019 00-00-01-abort-on-replica
-rwxr--r-- 1 root root 853 Mar 28 2019 00-00-01-check-replica-availability
-rwxr--r-- 1 root root 918 Mar 28 2019 00-00-01-copy-repository
-rwxr--r-- 1 root root 1377 Mar 28 2019 00-00-01-va-resources-check
-rwxr--r-- 1 root root 3685 Mar 28 2019 00-00-02-check-vro-duplicates
-rwxr--r-- 1 root root 2598 Mar 28 2019 00-00-03-upgrade-management-agents
-rwxr--r-- 1 root root 12433 Mar 28 2019 00-00-04-prereq-iaas
-rwxr--r-- 1 root root 724 Mar 28 2019 00-02-generate-vra-cluster-node-id
-rwxr--r-- 1 root root 1815 Mar 28 2019 00-10-vrb-version-check.sh
-rwxr--r-- 1 root root 1563 Mar 28 2019 00-20-check-blueprints
-rwxr--r-- 1 root root 600 Mar 28 2019 00-20-map-lb-to-localhost
-rwxr--r-- 1 root root 149 Mar 28 2019 00-50-kill-all-java-processes
-rwxr--r-- 1 root root 472 Mar 28 2019 00-60-clean-resources
-rwxr--r-- 1 root root 1685 Mar 28 2019 00-70-va-resources-disk-resize
-rwxr--r-- 1 root root 298 Mar 28 2019 00-71-PR-1590946-ext-partition-fix
-rwxr--r-- 1 root root 2723 Mar 28 2019 00-71-va-root-disk-resize
-rwxr--r-- 1 root root 352 Mar 28 2019 05-stop-vrhb-service
-rwxr--r-- 1 root root 1488 Mar 28 2019 06-stop-iaas-services
-rwxr--r-- 1 root root 3347 Mar 28 2019 06-stop-vra-replica-services
-rwxr--r-- 1 root root 698 Mar 28 2019 07-vacuum-db
-rwxr--r-- 1 root root 2114 Mar 28 2019 08-vco-database
-rwxr--r-- 1 root root 1000 Mar 28 2019 09-60-package-pool
-rwxr--r-- 1 root root 819 Mar 28 2019 09-70-remove-artifactory
-rwx------ 1 root root 5152 Mar 21 2019 10-postgres
-rwx------ 1 root root 1297 Mar 21 2019 10-prepare-vmstudio-update
-rwxr--r-- 1 root root 1547 Mar 28 2019 14-remove-persistent-net-rules
-rwxr--r-- 1 root root 529 Mar 28 2019 20-artifactory-uninstall-fix
-rwxr--r-- 1 root root 253 Mar 28 2019 80-horizon-certs
-rwxr--r-- 1 root root 57 Mar 28 2019 90-remove-resource-bundles
-rwxr--r-- 1 root root 750 Mar 28 2019 98-psql-manager-stop
-rwx------ 1 root root 1291 Mar 21 2019 99-backup-config-files
-rwx------ 1 root root 602 Mar 21 2019 99-postgres
-rwx------ 1 root root 555 Mar 21 2019 99-prepare-servicesSo when an upgrade is executed after vRA rpm's are copied these scripts are downloaded under preupdate/ path
They are executed sequentially
All we are trying to do is a workaround for an upgrade not to check vro duplicates while running an upgrade resulting in a failure. Hence we are deleting 00-00-02-check-vro-duplicates by inducing another script which will be executed before anything else is executed. As we named it as 00-00-00-00-remove-check-vro-duplicates