I delivered a training recently about the process which occurs in the background when we upgrade vRealize Automation 8.2 to 8.3
Thought it would be useful for everyone hence writing this blog
As a first step, we need to keep the updaterepo ready and downloaded
Then took a manual snapshot , once the snapshot task is complete we shall proceed with the upgrade
Now that the snapshot is taken we will go ahead with upgrade
Before we proceed, we need to perform inventory sync. Unless this is done, PROCEED option will not be available
Once , done we need to run precheck
Whole precheck pdf attached here
When you click on next check the repo url and then submit , upgrade starts
Upgrade successfully completes in 1.5 hours.
Here's the list of log files which are to be monitored during upgrade
BEHIND THE SCENES
The first thing which happens when you click on upgrade is that we download manifest. And this is logged under upgrade-noop.log
upgrade-noop.log
[INFO][2021-03-30 06:20:47][labvra.nukescloud.com] Downloading manifest ...
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest downloaded successfully.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 10.109.45.33...
* TCP_NODELAY set
* Connected to reflcm.nukescloud.com (10.109.45.33) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* ignoring certificate verify locations due to disabled peer verification
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*
*
*
* Connection #0 to host reflcm.nukescloud.com left intact
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Validating manifest ...
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest signature validated successfully.
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Repository product ID matched successfully.
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest validated successfully.
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Searching for update bootstrap RPM ...
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Update bootstrap RPM found.
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Update bootstrap RPM downloaded successfully.
*
*
*
* SSL certificate verify result: self signed certificate (18), continuing anyway.
} [5 bytes data]
> GET /repo/productBinariesRepo/vra/8.3.0/upgrade/update/package-pool/bootstrap-update-8.3.0.15014-1.noarch.rpm HTTP/1.1
> Host: reflcm.nukescloud.com
> User-Agent: curl/7.59.0
> Accept: */*
*
*
<
{ [15833 bytes data]
100 64516 0 64516 0 0 2250k 0 --:--:-- --:--:-- --:--:-- 2250k
* Connection #0 to host reflcm.nukescloud.com left intactThen starts the actual upgrade and logged under upgrade-YYYY-MM-DD-HH-MM-SS.log
1. Searches for the bootstrap package and checks if SSH port is open and it's able to connect
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Retrieving product version...
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Product version retrieved successfully.
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel801 ...
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel801-hf345 ...
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel810-gap12 ...
[INFO][2021-03-30 06:20:57][labvra.nukescloud.com] Applying configuration profile lcm.
[INFO][2021-03-30 06:20:57][labvra.nukescloud.com] Configuration profile lcm applied successfully.
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Retrieving nodes status...
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes status retrieval succeeded.
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Processing nodes data...
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes data processing succeeded.
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Verifying nodes...
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes verification completed successfully.
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Retrieving services status...
[INFO][2021-03-30 06:20:59][labvra.nukescloud.com] Services status retrieval succeeded.
[INFO][2021-03-30 06:21:00][labvra.nukescloud.com] Processing services data...
[INFO][2021-03-30 06:21:18][labvra.nukescloud.com] Services data processing succeeded.
[INFO][2021-03-30 06:21:18][labvra.nukescloud.com] Verifying services...
[INFO][2021-03-30 06:21:19][labvra.nukescloud.com] Services verification completed successfully.
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] Creating SSH key pairs
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] SSH key pairs created successfully.
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] Configuring SSH on nodes.
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] Service sshd status: active/enabled
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] SSH port 22 is open.
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] SSH configurated on nodes successfully.
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/ssh-noop.sh at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/ssh-noop.sh at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Verifying remote nodes are able to connect to one another and to this node.
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Verification that nodes are able to connect to one another and to this node succeeded.
2. Retrieves product version on all nodes. In this environment we just have one node
[INFO][2021-03-30 06:21:24][labvra.nukescloud.com] Retriving product versions on all nodes.
[INFO][2021-03-30 06:21:24][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/vami-save-vers.sh prep at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Retrieving product version.
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product version retrieved successfully.
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/vami-save-vers.sh prep at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product versions successfully retrieved from all nodes.
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Checking that product versions match across all nodes.
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product versions match across all nodes verified.
3. Performs Infrastructure Health
[INFO][2021-03-30 06:21:26][labvra.nukescloud.com] Checking appliance nodes infrastructure health
[INFO][2021-03-30 06:21:26][labvra.nukescloud.com] Running remote command: /opt/health/run-once.sh health at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:31][labvra.nukescloud.com] Remote command succeeded: /opt/health/run-once.sh health at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:31][labvra.nukescloud.com] Infrastructure health check passed on all appliance nodes.
4. Set's up update repository on all nodes and also verifies if this is accessible across all nodes
[INFO][2021-03-30 06:21:33][labvra.nukescloud.com] Setting up update repository on all nodes.
[INFO][2021-03-30 06:21:33][labvra.nukescloud.com] Running remote command: vamicli update --repo https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Remote command succeeded: vamicli update --repo https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Update repository set successfully on all nodes.
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Verifying the access to update repository on all nodes.
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/vami-config-repo.sh 'local' at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Verifying the access to the update repository.
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Access to the update repository verified.
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Verifying availability of new product version in the repository.
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Availability of new version in the repository verified.
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/vami-config-repo.sh 'local' at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Update repository is verified successfully on all nodes.
5. When the above vami-config.sh runs under /opt/vmware/var/log/vami/vami.log , the following repository is set
30/03/2021 06:21:34 [INFO] Setting local repository address. url=https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update, username=, password=<empty>
30/03/2021 06:21:34 [INFO] Checking available updates. jobid=1
30/03/2021 06:21:34 [INFO] Downloading latest manifest. jobId=1, url=https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update, username=, password=
30/03/2021 06:21:35 [INFO] Using download cache
30/03/2021 06:21:35 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml
30/03/2021 06:21:35 [INFO] Using download cache
30/03/2021 06:21:35 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml.sig
30/03/2021 06:21:35 [INFO] Signature script output: Verified OK
30/03/2021 06:21:35 [INFO] Signature script output:
30/03/2021 06:21:35 [INFO] Manifest signature verification passed
30/03/2021 06:21:35 [INFO] New manifest. installed-version=8.2.0.12946, downloaded-version=8.3.0.15014
6. Starts to shutdown application or services
[INFO][2021-03-30 06:21:38][labvra.nukescloud.com] Shutting down application services
[INFO][2021-03-30 06:23:58][labvra.nukescloud.com] Application services shut down successfully.
[INFO][2021-03-30 06:23:58][labvra.nukescloud.com] Shutting down infrastructure services
[INFO][2021-03-30 06:26:27][labvra.nukescloud.com] Infrastructure services shut down successfully.
[INFO][2021-03-30 06:26:29][labvra.nukescloud.com] Saving restore points on all nodes.
[INFO][2021-03-30 06:26:29][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-save.sh 'local' sys-config sys-config at host: labvra.nukescloud.com
[INFO][2021-03-30 06:26:34][labvra.nukescloud.com] Saving local restore points.
=========================
[2021-03-30 06:24:03.425+0000] Waiting for deploy healthcheck
=========================
=========================
[2021-03-30 06:24:09.389+0000] Waiting for command execution pods
=========================
=========================
[2021-03-30 06:24:15.420+0000] Tear down existing deployment
=========================
=========================
[2021-03-30 07:24:29.787+0000] Waiting for command execution pods
=========================
=========================
[2021-03-30 07:24:32.368+0000] Tear down existing deployment
=========================
7. Saves local restore point for k8's , processes node data and then activates monitor on all nodes
[INFO][2021-03-30 06:26:34][labvra.nukescloud.com] Saving local restore point for Kubernetes object prelude-vaconfig.
[INFO][2021-03-30 06:26:35][labvra.nukescloud.com] Local restore point for Kubernetes object prelude-vaconfig saved successfully.
Flag --export has been deprecated, This flag is deprecated and will be removed in future.
[INFO][2021-03-30 06:26:35][labvra.nukescloud.com] Local restore points saved successfully.
[INFO][2021-03-30 06:26:35][labvra.nukescloud.com] Restore points on all nodes saved successfully.
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Retrieving nodes status...
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Nodes status retrieval succeeded.
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Processing nodes data...
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Nodes data processing succeeded.
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Verifying nodes...
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Nodes verification completed successfully.
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Activating local monitors on all nodes.
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/mon-activate.sh at host: labvra.nukescloud.com
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Activating upgrade monitor on the node
8. Once the upgrade monitor is activated on the node and cluster nodes are removed successfully , it will proceed for installation. Remember this is a simple install
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Upgrade monitor activated successfully on the node.
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/mon-activate.sh at host: labvra.nukescloud.com
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Local monitors successfully activated on all nodes.
[INFO][2021-03-30 06:28:28][labvra.nukescloud.com] Removing nodes from the cluster
[INFO][2021-03-30 06:28:28][labvra.nukescloud.com] Running remote command: vracli cluster leave at host: labvra.nukescloud.com
[INFO][2021-03-30 06:44:26][labvra.nukescloud.com] Cluster nodes removed successfully
9. The Installation of the updates start
[INFO][2021-03-30 06:46:01][labvra.nukescloud.com] Starting installation of updates.
[INFO][2021-03-30 06:46:01][labvra.nukescloud.com] Update installation started successfully.
[INFO][2021-03-30 06:47:01][labvra.nukescloud.com] VAMI checking upgrade progress
[INFO][2021-03-30 06:47:01][labvra.nukescloud.com] VAMI upgrade in progress.
[INFO][2021-03-30 06:47:01][labvra.nukescloud.com] VAMI upgrade still in progress.
10. Just a second after it states Starting installation of updates under the upgrade-xx-xx-xx-xx.log , this is the time you would see activity under vami.log
30/03/2021 06:46:02 [INFO] Installing updates. instanceid=VMware:VMware_8.3.0.15014, jobid=2
30/03/2021 06:46:02 [INFO] Installing update. instanceId=VMware:VMware_8.3.0.15014, jobId=2, url=https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update, username=, password=
30/03/2021 06:46:02 [INFO] Downloading and installing update packages
30/03/2021 06:46:02 [INFO] Using download cache
30/03/2021 06:46:02 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml
30/03/2021 06:46:02 [INFO] Using download cache
30/03/2021 06:46:02 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml.sig
30/03/2021 06:46:02 [INFO] Signature script output: Verified OK
30/03/2021 06:46:02 [INFO] Signature script output:
30/03/2021 06:46:02 [INFO] Manifest signature verification passed
30/03/2021 06:46:02 [INFO] Creating /opt/vmware/var/lib/vami/update/data/update_progress.json
30/03/2021 06:46:02 [INFO] Downloading the following packages for update version 8.3.0.15014
30/03/2021 06:46:02 [INFO] package NEW PACKAGE : VMware-Log-Insight-Agent noarch (none) 8.3.0 17489324 /package-pool/VMware-Log-Insight-Agent-8.3.0-17489324.noarch.rpm rpm 9457625 e744601203ec01d0d637453260324b7b35a872ce
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: abx-common noarch (none) 8.3.0.15014 1 /package-pool/abx-common-8.3.0.15014-1.noarch.rpm rpm 43780508 c134c5476bd805293bdfaf45dc9b6bd9e06bde63
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: bindutils x86_64 (none) 9.16.6 1.ph2 /package-pool/bindutils-9.16.6-1.ph2.x86_64.rpm rpm 3584466 67d101f6718a81a2abc3dfd9b54381cac6948b60
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: bootstrap-prelude noarch (none) 8.3.0.15014 1 /package-pool/bootstrap-prelude-8.3.0.15014-1.noarch.rpm rpm 2629839 14c2906620ddca6690e13867f1e385851b9e06d8
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: cifs-utils x86_64 (none) 6.7 2.ph2 /package-pool/cifs-utils-6.7-2.ph2.x86_64.rpm rpm 38568 2eea29de217fc837792d0c8db9848d240c3b18d5
30/03/2021 06:46:02 [INFO] package NEW PACKAGE : config-xl noarch (none) 8.3.0.15014 1 /package-pool/config-xl-8.3.0.15014-1.noarch.rpm rpm 14255 c32254fd9e2c1074ea1b5e33e9c72af5350c27e4
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: containerd x86_64 (none) 1.2.14 1.ph2 /package-pool/containerd-1.2.14-1.ph2.x86_64.rpm rpm 24221328 227d22a05417f4286e0247ca87ab415f0b095231
*
*
*
30/03/2021 06:48:34 [INFO] package UPDATE VERSION: vro-migrate noarch (none) 8.3.0.1611732871 17522798 /package-pool/vro-migrate-8.3.0.1611732871-17522798.noarch.rpm rpm 4984 eb517af86432ea5b752627335696eec3509a052c
30/03/2021 06:48:34 [INFO] Install packages set: '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/VMware-Log-Insight-Agent-8.3.0-17489324.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/abx-common-8.3.0.15014-1.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/bindutils-9.16.6-1.ph2.x86_64.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/bootstrap-prelude-8.3.0.15014-1.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/cifs-utils-6.7-2.ph2.x86_64.rpm'
*
*
vmware-studio-appliance-config_3.0.0.7-210201212046.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/vmware-studio-init_3.0.0.7-210201212046.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/vro-migrate-8.3.0.1611732871-17522798.noarch.rpm'
30/03/2021 06:48:34 [INFO] Reboot required when installing linux package
30/03/2021 06:48:34 [INFO] Created reboot required file
30/03/2021 06:48:34 [INFO] Update 8.3.0.15014 manifest is set to be installed
30/03/2021 06:48:34 [INFO] Using update post-install script
30/03/2021 06:48:34 [INFO] Running updatecli to install updates. command={ mkdir -p /usr/share/update-notifier ; ln -s /opt/vmware/share/vami/vami_notify_reboot_required /usr/share/update-notifier/notify-reboot-required ; /opt/vmware/share/vami/update/updatecli '/opt/vmware/var/lib/vami/update/data/job/2' '8.2.0.12946' '8.3.0.15014' ; mv /opt/vmware/var/lib/vami/update/data/update_progress.json /opt/vmware/var/lib/vami/update/data/.update_progress.json ; } >> /opt/vmware/var/log/vami/updatecli.log 2>&1 &
30/03/2021 06:48:36 [INFO] Installation running in the background
11. Once the installation starts in the background , this is the time we need to monitor /var/log/bootstrap/postupdate.log
The ones highlighted in the RED are the series of scripts which run in the background.
+ log 'Main bootstrap postupdate started'
++ date '+%Y-%m-%d %H:%M:%S'
+ echo '2021-03-30 06:51:46 Main bootstrap postupdate started'
2021-03-30 06:51:46 Main bootstrap postupdate started
+ for script in "${bootstrap_dir}"/*
+ echo
+ log '/etc/bootstrap/postupdate.d/00-clear-services.sh starting...'
+ log '/etc/bootstrap/postupdate.d/00-disable-set-hostname.sh starting...'
+ log '/etc/bootstrap/postupdate.d/00-fix-vmtoolsd-deps.sh starting...'
+ log '/etc/bootstrap/postupdate.d/01-10-reset-k8s starting...'
+ log '/etc/bootstrap/postupdate.d/02-07-prune-images.sh starting...'
*
*
untagged: photonos3_private:latest
deleted: sha256:30ded4d67031ea69c5792250dcd4c466511d940ff92f13cd6cc945c4456596ed
untagged: relocation-service:ed72098
untagged: relocation-service_private:latest
deleted: sha256:840c0a6cb51902b0bb1d81c7ebb3b7ae233e1666dbe387f39a24d7066ad1781a
deleted: sha256:3e737404723de6f8e2a6d8ad85b8cad2b9206e76b40b536766ba6a164a6d24ed
deleted: sha256:ea1c6ad6204eb1aa5ffccc44a3890014b52a39bc9ed64de34af88195670196e5
deleted: sha256:3a90641efc2b2704dcfa4a8777a4c8cce7150b683811e315773ce35cb83d2e92
deleted: sha256:fa46acfe7078d8404ebda3e229ae4058e68aa579512454194f3bc6121be0818e
untagged: rabbitmq_private:3.8.5-photon_9ff5c8078eac
untagged: rabbitmq_private:latest
*
*
+ log '/etc/bootstrap/postupdate.d/02-10-load-images.sh starting...'
+ log '/etc/bootstrap/postupdate.d/02-disable-var-log-cron.sh starting...'
+ log '/etc/bootstrap/postupdate.d/03-10-setup-k8s.sh starting...'
+ log '/etc/bootstrap/postupdate.d/10-disable-tzselect.sh starting...'
+ log '/etc/bootstrap/postupdate.d/20-relocate-db.sh starting...'
+ log '/etc/bootstrap/postupdate.d/90-disable-default-services.sh starting...'
+ log '/etc/bootstrap/postupdate.d/91-00-remove-config-xl.sh starting...'
+ log '/etc/bootstrap/postupdate.d/99-10-handover-prelude starting...'
*
*
# Wait for Kubernetes and other prerequisites
vracli status first-boot -w 1800 && \
/opt/scripts/upgrade/vami-set-update.sh "success" || \
/opt/scripts/upgrade/vami-set-update.sh "error"
rm -f /etc/bootstrap/everyboot.d/zz-zz-resume-upgrade.sh'
+ chmod a+x /etc/bootstrap/everyboot.d/zz-zz-resume-upgrade.sh
+ export -f vami_reboot_background
+ echo 'Scheduling a post-upgrade reboot...'
Scheduling a post-upgrade reboot...
+ echo 'Post-upgrade reboot scheduled'
Post-upgrade reboot scheduled
+ exit 0
+ nohup bash -c vami_reboot_background
+ log '/etc/bootstrap/postupdate.d/99-10-handover-prelude done, succeeded.'
++ date '+%Y-%m-%d %H:%M:%S'
+ echo '2021-03-30 07:14:06 /etc/bootstrap/postupdate.d/99-10-handover-prelude done, succeeded.'
*
*
+ log '/etc/bootstrap/postupdate.d/README is not executable.'
++ date '+%Y-%m-%d %H:%M:%S'
+ echo '2021-03-30 07:14:06 /etc/bootstrap/postupdate.d/README is not executable.'
2021-03-30 07:14:06 /etc/bootstrap/postupdate.d/README is not executable.
+ log 'main bootstrap postupdate done'
++ date '+%Y-%m-%d %H:%M:%S'
+ echo '2021-03-30 07:14:06 main bootstrap postupdate done'
2021-03-30 07:14:06 main bootstrap postupdate done
+ '[' postupdate == firstboot ']'
+ exit 0
Rebooting the appliance after VAMI upgrade ...
12. Finishes the postupdate and the appliance is rebooted
13. As we can see above the appliance was rebooted sometime after 2021-03-30 07:14:06 , under upgrade-2021-03-30-06-20-48.log once the appliance is back you would see the status change
[INFO][2021-03-30 07:14:01][labvra.nukescloud.com] VAMI checking upgrade progress
[INFO][2021-03-30 07:14:01][labvra.nukescloud.com] VAMI upgrade in progress.
[INFO][2021-03-30 07:14:01][labvra.nukescloud.com] VAMI upgrade still in progress.
[INFO][2021-03-30 07:14:06][labvra.nukescloud.com] Waiting for VAMI to exit ...
[INFO][2021-03-30 07:14:36][labvra.nukescloud.com] Verifying VAMI overall upgrade result ...
[INFO][2021-03-30 07:14:36][labvra.nukescloud.com] VAMI upgrade in progress.
[INFO][2021-03-30 07:17:01][labvra.nukescloud.com] VAMI checking upgrade progress
[INFO][2021-03-30 07:17:01][labvra.nukescloud.com] VAMI upgrade in progress.
*
*
*
[INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI checking upgrade progress
[INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI upgrade completed succesfully.
[INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI upgrade completed successfully.
[INFO][2021-03-30 07:23:01][labvra.nukescloud.com] Saving artifacts metadata
[INFO][2021-03-30 07:23:02][labvra.nukescloud.com] Artifacts metadata saved.
[INFO][2021-03-30 07:24:01][labvra.nukescloud.com] Resolving post-upgrade controller.
[INFO][2021-03-30 07:24:01][labvra.nukescloud.com] This node is elected for post-upgrade controller.
[INFO][2021-03-30 07:24:03][labvra.nukescloud.com] Resolving post-upgrade nodes quorum...
[INFO][2021-03-30 07:24:04][labvra.nukescloud.com] Adding nodes to the cluster.
[INFO][2021-03-30 07:24:04][labvra.nukescloud.com] Cluster nodes added successfully.
[INFO][2021-03-30 07:24:05][labvra.nukescloud.com] Restoring from restore points on all nodes.
[INFO][2021-03-30 07:24:05][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-restore.sh 'local' sys-config sys-config at host: labvra.nukescloud.com
[INFO][2021-03-30 07:24:11][labvra.nukescloud.com] Restoring all restore points.
14. Restore point restoration starts and then application and infrastructure services start as well
[INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI checking upgrade progress
[INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI upgrade completed succesfully.
[INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI upgrade completed successfully.
[INFO][2021-03-30 07:23:01][labvra.nukescloud.com] Saving artifacts metadata
[INFO][2021-03-30 07:23:02][labvra.nukescloud.com] Artifacts metadata saved.
[INFO][2021-03-30 07:24:01][labvra.nukescloud.com] Resolving post-upgrade controller.
[INFO][2021-03-30 07:24:01][labvra.nukescloud.com] This node is elected for post-upgrade controller.
[INFO][2021-03-30 07:24:03][labvra.nukescloud.com] Resolving post-upgrade nodes quorum...
[INFO][2021-03-30 07:24:04][labvra.nukescloud.com] Adding nodes to the cluster.
[INFO][2021-03-30 07:24:04][labvra.nukescloud.com] Cluster nodes added successfully.
[INFO][2021-03-30 07:24:05][labvra.nukescloud.com] Restoring from restore points on all nodes.
[INFO][2021-03-30 07:24:05][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-restore.sh 'local' sys-config sys-config at host: labvra.nukescloud.com
[INFO][2021-03-30 07:24:11][labvra.nukescloud.com] Restoring all restore points.
15. At this point in time one has to check /var/log/deploy.log to see if the services are coming back as expected
=========================
[2021-03-30 07:26:36.605+0000] Creating kubernetes namespaces
=========================
=========================
[2021-03-30 07:26:39.435+0000] Applying ingress certificate
=========================
=========================
[2021-03-30 07:26:50.589+0000] Updating etcd configuration to include https_proxy if such exists
=========================
=========================
[2021-03-30 07:26:57.836+0000] Deploying infrastructure services
=========================
=========================
[2021-03-30 07:28:43.477+0000] Clearing liquibase locks
=========================
=========================
[2021-03-30 07:28:56.296+0000] DB upgrade schema
=========================
=========================
[2021-03-30 07:32:08.619+0000] Populating initial identity-service data
=========================
=========================
[2021-03-30 07:32:38.488+0000] Deploying application services
=========================
=========================
[2021-03-30 07:42:06.787+0000] Deploying application UI
=========================
=========================
[2021-03-30 07:43:28.519+0000] Setting feature UI toggles to Provisioning service
=========================
=========================
[2021-03-30 07:43:54.731+0000] Registering embedded vRO
=========================
=========================
[2021-03-30 07:43:58.698+0000] Registering embedded ABX on-prem endpoint in organizations
=========================
16. That marks the application and infrastructure start up procedure and then there would be a verification , this procedure is documented under 2 logs
Reference: Verifies if all kubernetes related services are up and running upgrade-k8s-2021-03-30-06-20-48.log
Reference: Copy of deploy.log upgrade-deploy-2021-03-30-06-20-48.log
17. As the verification is now complete, it will now works towards the last few steps
[INFO][2021-03-30 07:44:32][labvra.nukescloud.com] Infrastrucre and application services deployed successfully.
[INFO][2021-03-30 07:44:33][labvra.nukescloud.com] Retrieving services status...
[INFO][2021-03-30 07:44:33][labvra.nukescloud.com] Services status retrieval succeeded.
[INFO][2021-03-30 07:44:33][labvra.nukescloud.com] Processing services data...
[INFO][2021-03-30 07:44:54][labvra.nukescloud.com] Services data processing succeeded.
[INFO][2021-03-30 07:44:54][labvra.nukescloud.com] Verifying services...
[INFO][2021-03-30 07:44:56][labvra.nukescloud.com] Services verification completed successfully.
[INFO][2021-03-30 07:44:58][labvra.nukescloud.com] Cleaning up restore point on all nodes.
[INFO][2021-03-30 07:44:58][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-clean.sh sys-config at host: labvra.nukescloud.com
[INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Cleaning up restore point /data/restorepoint/sys-config .
[INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Restore point /data/restorepoint/sys-config cleaned up successfully.
[INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Restore point cleaned up on all nodes.
[INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Cleaning up restore point on all nodes.
[INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-clean.sh artifacts-lastworking at host: labvra.nukescloud.com
[INFO][2021-03-30 07:45:08][labvra.nukescloud.com] Cleaning up restore point /data/restorepoint/artifacts-lastworking .
[INFO][2021-03-30 07:45:08][labvra.nukescloud.com] Restore point /data/restorepoint/artifacts-lastworking cleaned up successfully.
[INFO][2021-03-30 07:45:08][labvra.nukescloud.com] Restore point cleaned up on all nodes.
[INFO][2021-03-30 07:45:08][labvra.nukescloud.com] Cleaning up restore point on all nodes.
[INFO][2021-03-30 07:45:09][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-clean.sh live-data at host: labvra.nukescloud.com
[INFO][2021-03-30 07:45:14][labvra.nukescloud.com] Cleaning up restore point /data/restorepoint/live-data .
[INFO][2021-03-30 07:45:14][labvra.nukescloud.com] Restore point /data/restorepoint/live-data cleaned up successfully.
[INFO][2021-03-30 07:45:14][labvra.nukescloud.com] Restore point cleaned up on all nodes.
[INFO][2021-03-30 07:45:15][labvra.nukescloud.com] Reverting SSH configuration on nodes.
[INFO][2021-03-30 07:45:16][labvra.nukescloud.com] SSH configuration reverted on nodes successfully.
[INFO][2021-03-30 07:45:24][labvra.nukescloud.com] Cleaning up upgrade runtime state.
[INFO][2021-03-30 07:45:25][labvra.nukescloud.com] Archiving upgrade runtime data.
[INFO][2021-03-30 07:45:26][labvra.nukescloud.com] Upgrade runtime data archived successfully.
[INFO][2021-03-30 07:45:26][labvra.nukescloud.com] Clearing upgrade runtime directory.
[INFO][2021-03-30 07:45:26][labvra.nukescloud.com] Upgrade runtime directory cleared successfully.
[INFO][2021-03-30 07:45:26][labvra.nukescloud.com] Upgrade runtime clean up completed.
This concludes vRealize Automation upgrade
As said earlier following logs are the ones which we need to monitor
IN : upgrade-noop.log [INFO][2021-03-3006:20:47][labvra.nukescloud.com]Downloading manifest ...[INFO][2021-03-3006:20:48][labvra.nukescloud.com] Manifest downloaded
Can you please tell the location from where the manifest is downloaded ? Also should it be containing manifest of current version or the target version of the upgrade ?