NSX-T 3.2.2.1 upgrade to 4.1.1.0 fails with Search Service and UI, HTTPS are unresponsive and not starting

During a customer case I had we were doing upgrade of NSX-T from 3.2.2.1 to 4.1.1.0 and in the last moment when the NSX-T Manager node upgrade was being performed the NSX UI, HTTPS service is not starting.

During the last step when the UI is being Unpinned from the NSX Managers the search service do not come back up.

After a lot of troubleshooting it was found that the solution to this problem is that the permissions for the NSX-T Search Service is set to the wrong expected user:group in each of the NSX Managers during the upgrade. A Workaround to get this to work is by doing the following.

SSH as root into each of the NSX-T Managers.

Look in the opensearch.log at /var/log/search folder after the follwing Errors:

[2024-01-03T14:42:48,519][ERROR][o.o.b.Bootstrap] [nsx_node] Exception org.opensearch.OpenSearchException: failed to bind service

[2024-01-03T14:42:48,551][WARN ][stderr                   ] [nsx_node] Caused by: org.opensearch.bootstrap.StartupException: OpenSearchException[failed to bind service]; nested: AccessDeniedException[/nonconfig/search/nodes/0];

If we look above on the error it says that the opensearch service have AccessDenied against the following path /nonconfig/search/nodes/0

Go to the following directory:

root@nsx-mgr-1:/#
root@nsx-mgr-1:/#cd /nonconfig/search
root@nsx-mgr-1:/#ls -l
drwxr-x--- 3 elasticsearch elasticsearch 4096 Apr 25  2023 nodes
drwxr-xr-x 2 nsx-search    nsx-search    4096 Jan  4 13:13 tmp

We here see that the folder nodes and all its subfolders and files have the elasticsearch user:group as the owner. The Search Service expect this to be nsx-search

Change by issuing the following:

root@nsx-mgr-1:/nonconfig/search# chown -R nsx-search:nsx-search nodes

Now exit as root and login as Admin and restart the search service

nsx-mgr-3> start service search
nsx-mgr-3> get cluster status

Get the cluster status and wait a few minutes for the service to start.

Then the UI, HTTPS should start working again and be available from the browser.