If you run your VMware vSphere or VMware Cloud Foundation (VCF) stack on Dell EMC VxRail (VCFonVxRail), you are maybe familiar with the vxverify and vcfverify tools (requires login), which can come in handy for checking the health of your VCF and VxRail systems, both ad-hoc and before doing an upgrade.

I won’t go into detail about the tools in this short post, but rather show how to fix a warning I got from the output of said tools when I was checking a customer’s VCF 4.3.1.1 health.

Please note that VCF 4.4.1.0 should be out shortly (early March 2022 hopefully) to include NSX-T 3.1.3.7 hotfix for problems in 3.1.3.5 and 3.1.3.6. (VCF 4.4.0.0 includes NSX-T 3.1.3.5)

When I ran the latest version of vcfverify, I stumbled up on a few identical warnings in regards to some of the VxRail nodes (ESXi hosts). The warning was:

“Warn 000196167 | vxnode: vxnode.config drives missing"

in an (truncated) output like this:

VCFVerify 4.20.225 cluster analysis on VCF: 4.3.1.1  2022-03-01 15:38:46 GMT
#=====================#======#=========#==============================================#==============#
| Hostname / Category |Status  Dell_KB |  Warnings or Failures, unless tests Passed   | Product S.N. |
#=====================#======#=========#==============================================#==============#
| w1c1-esxi01         | Warn 000196167 | vxnode: vxnode.config drives missing          DE600XXXXXXXX1|
| w1c1-esxi02         | Pass           | Tests all passed successfully (36 tests ran)  DE600XXXXXXXX2|
| w1c1-esxi03         | Warn 000196167 | vxnode: vxnode.config drives missing          DE600XXXXXXXX3|
| w1c1-esxi04         | Pass           | Tests all passed successfully (36 tests ran)  DE600XXXXXXXX4|
| w1c1-esxi05         | Warn 000196167 | vxnode: vxnode.config drives missing          DE600XXXXXXXX5|
| w1c1-esxi06         | Pass           | Tests all passed successfully (36 tests ran)  DE600XXXXXXXX6|

The numbers after the ‘Warn’ is actually a Dell EMC KB article number 000196167, which tells you to gather the vxnode.config and vxnode.verify file together with the output of the vxverify output (which you will find in the folder with vcfverify) and contact Dell EMC Support and give them the files.

If your VxRail hosts are under support you should probably do that; contact Dell EMC Support.

Or you can try fixing it yourself. While I give NO GUARANTEES for this solution, this method was used by an Dell EMC Support technician in a remote session with me a month ago in regards to another problem – I had removed a disk from the server after removing it from the vSAN setup, and VxRail plugin in vCenter complained about the missing disk. Support did the same fix then, as I’m suggesting now, for this problem: recreate the vxnode.config file and restart a service.

The fix

Log into the ESXi host using ssh with root user and run the following commands:

cd $(find /vmfs/volumes/*-service-datastore1/ -iname vxrail-datastore)/vxnode_config
mv vxnode.config vxnode.config.bak_$(date +%Y%m%d)
python /opt/vxrail/bin/vxtools/vxt-gennodeconfig
/etc/init.d/vxrail-pservice restart

Line #1 changes directory to where the vxnode.config files should be. If that doesn’t work, you could do a more generic search for the file, which should still be under the vxnode_config directory: “find / -inode vxnode_config“

Line #2 renames the file so you have it as backup

Line #3 generates a new vxnode.config file

Line #4 restarts the service that uses this file for reporting into vCenter among other things.

When I ran vcfverify again, the problem was gone 🥳

VCFVerify 4.20.225 cluster analysis on VCF: 4.3.1.1  2022-03-01 15:38:46 GMT
#=====================#======#=========#==============================================#==============#
| Hostname / Category |Status  Dell_KB |  Warnings or Failures, unless tests Passed   | Product S.N. |
#=====================#======#=========#==============================================#==============#
| w1c1-esxi01         | Pass           | Tests all passed successfully (36 tests ran)  DE600XXXXXXXX1|
| w1c1-esxi02         | Pass           | Tests all passed successfully (36 tests ran)  DE600XXXXXXXX2|
| w1c1-esxi03         | Pass           | Tests all passed successfully (36 tests ran)  DE600XXXXXXXX3|
| w1c1-esxi04         | Pass           | Tests all passed successfully (36 tests ran)  DE600XXXXXXXX4|
| w1c1-esxi05         | Pass           | Tests all passed successfully (36 tests ran)  DE600XXXXXXXX5|
| w1c1-esxi06         | Pass           | Tests all passed successfully (36 tests ran)  DE600XXXXXXXX6|