This is the first of a couple of posts on how I’ve cobbled together some basic monitoring of Neverfail’s  Neverfail Heartbeat H/A software which is also now the basis for VMWare’s vCenter Server Heartbeat. Since Neverfail seems to consider their command lines privileged information I will only cover how to do some simple monitoring using the registry. When starting on this effort internally I was only interested initially in figuring out a quick and simple way to get the info I needed and not so much on the how to get it into something part.

I’ve been working with another team where I work to look at Zabbix as an alternative for some of the monitoring we do in our environment. We use Microsoft Operations Manager 2005 (MOM) but haven’t fully cut over from out previous monitoring solution. I had looked at Zabbix earlier as a potential solution for monitoring a bunch of VMware ESX boxes but another team ended up getting tasked with that particular duty. So I had had some experience with Zabbix but hadn’t done too much with it since.

One of the things that’d been rattling around in my brain is using the capabilities of using the zabbix_sender feature/client to monitor some of other components/things we can’t easily get into MOM.  Zabbix_Sender is a utility that is available for use with Zabbix that allows one to “send” information to Zabbix. In my case it was appealing because we’re already running two different monitoring agents on the Exchange servers where we have Neverfail installed.  Since I only wanted to use Zabbix to monitor a small set of data related specifically to Neverfail zabbix_sender lets me do that without having to run the fullblown zabbix_agent as a service on the boxes.

Getting the Data

Neverfail (at least the versions we have installed) doesn’t obviously expose performance data. However if you look in the registry on each Neverfail server you will find some registry values (see HKLM\Software\Neverfail\R2\Performance) that get updated on a regular and frequent basis that correspond to data presented in the Neverfail GUI . Because of the way Neverfail works some of this data (Unsafe Queue info) is available on the Active node and some of it (Safe Queue info) is in the registry on the Passive node. This presents a couple of issues when trying to put together the solution (at least in my environment).

The first of these is trying to find a single consistent way to get the data out of the registry, especially since all the counters involved are of the REG_DWORD_BIG_ENDIAN variety (you can see a previous entry related to BIG_ENDIAN here).  I ended up settling on using the Reg.exe util available in Windows.  This utility let’s you manipulate the registry locally and remotely. While it doesn’t necessarily deal happily with REG_DWORD_BIG_ENDIAN (RDBE) entries in the registry it is able to extract the data which we can then manipulate to get the correct values.

As an example if I have the following two values in the registry as shown by RegEdit

When I run reg.exe I get the following output…

So while Dword_example and DWORD_BE_Example nominally have the same value reg.exe doesn’t get the data out correctly for the latter. However as I said earlier once we have the data out we can actually do some magic to get the right value.

We can also use reg.exe to get values on a remote machine (i.e. our Passive Neverfail node) by pre-pending the host info to the query registry path. So in this case to reach the passive secondary node over the private channel at 10.0.0.2 I can do something like  reg.exe Query \\10.0.0.2\CRT_CORP\Performance. Testing this out leads us to  a second issue. Getting an  ”Acces is denied” error.

Since my passive Neverfail node is essentially off-net but still thinks the network cables is live I can’t use a domain based account to run the reg.exe command because it can’t contact a domain controller to authenticate my domain account. However if I use the local Administrator account which has a common password on both nodes I can get this work just fine. (It may be possible to use an account other than the local Administrator but in my case where I also run some Neverfail command lines I need an account that’s authorized in Neverfail)

Given this info I was able to put together a vbscript that takes two arguments: a reg path and a value name;  and it returns the data value to the console converting REG_DWORD and REG_DWORD_BIG_ENDIAN to the correct decimal value. Using this script it’s then possible to get  any of the counters we’re interested in on either the active or passive node.  So based on the example above where I ran reg.exe hklm\software\CRT_CORP\Performance /s we can run the script for each of the values and see that we do in fact get the right decimal value for each one.

So now the trick is to figure out which of the registry based perf values we want to use and which host we need to draw them from.  Each of the Neverfail nodes has the same set of values present even though they’re not all populated the same way. That is to say that the counters related to the Safe Queue are not updated on the Active node since the Safe Queue exists on the passive node. And the converse is true with regard to the UnsafeQueue counters.  As I was mostly interested in alerting related to an issue we have occur occassionally I really wanted to get the SafeQueue and UnsafeQueue related counters (OldestSafeUpdateQueueEntry, SafeUpdateQueueSize etc). But since the other counters are also equally easy to get I decided I to include several more.  The image below shows the available values.

So now that I have a simple way of getting the information I want I can focus on how to get it into whatever system I want to monitor with whether it’s Zabbix (now) or Systems Center Operations Manager 2007 (later).  In the next article(s) I’ll talk about setting up the Zabbix part of this monitoring.

Acknowledgement: The hex to decimal routine in the GetRegValue.vbs script is lifted directly from http://www.sonofsofaman.com/hobbies/code/hextodec.asp Thanks to Joel for keeping me from having to reinvent the wheel. -crt

Addendum: While traipsing through the registry in figuring this stuff out I also discovered that there’s a bunch of configuration information stored in a whole different key under HKLM\Software\Javasoft\Prefs\neverfail\current\* It’s also possible to watch a few entries here to help monitor the  file and registry synchronization status even though it’s not as granular/descriptive/timely as can be obtained by using the command line.

The two items I’ve found that might be of interest are the /Registry/State/Manager\/Status Key and the /Value entry

and  the /New/File/State/Mgr\/Synchronization/Status key and /Tag entry

So what it it turns out I was getting from the GetValue method was a sequence of 4 bytes as an array. This made me wonder what the heck was going on so I fired up RegEdit to take a look. The contents of the Data column looked okay but the Type was different, so I expanded the column to see what was different and discovered REG_DWORD_BIG_ENDIAN as a type. This was one I don’t recall having seen before.

TechNet has the following to say about DWORD and it’s brethren:

REG_DWORD
A 32-bit (4-byte) number. Boolean (“True” or “False”) values and many entries for device drivers and services use this data type. REG_DWORD data can be displayed and entered in hexadecimal or decimal format in the registry editor Regedit.exe. For an example, see the ActivityLogFlag entry.

REG_DWORD_BIG_ENDIAN
Same as REG_DWORD. A 32-bit number in which the most significant byte is displayed as the leftmost (or high-order) byte. This is the most common format for storing numbers in computers that are running Windows Server 2003.

REG_DWORD_LITTLE_ENDIAN
A 32-bit number in which the most significant byte is displayed as the rightmost (or low-order) byte. This is opposite of the order in which bytes are stored in the REG_DWORD and REG_DWORD_BIG_ENDIAN data types.

If you’re not paying attention it could be easy to miss the difference when using RegEedit since they appear almost identical to DWORD values. The only obvious difference is the “Type” field.

So in the example above GetValue returns different values for “PerfCtr2″ and “PerfCtr2 DWORD” which are nominally the same value (at least according to RegEdit).

To help me figure out how to get the info I was looking for I put together a test and created a couple of dummy registry keys with each of the types of reg keys and some examples.

If we try to see what Powershell tells us about each of these keys we see that for our BIG_ENDIAN friend GetValueKind returns “unknown.”

GetValue converts each byte to a decimal value. Our key REG_DWORD_BIG_ENDIAN (0xa1b2c3d4) can be expressed as 4 bytes “a1″ “b2″ “c3″ “d4″ which when converted become the values “161″ “178″ “195″ “212″. While this is mildly useful it doesn’t help us easily get the value we want 2712847316. While it is possible to get the right value by doing some math [ (byte1 * 256^3) + (byte2 * 256^2) + (byte3 * 256) + byte4 ] I thought my resulting attempts to write a snippet to do this were ugly since it doesn’t appear Powershell has any easy way to do exponentiation.

The third method was to write out the formula a little more explicitly so that (byte1 * 256^3) + (byte2 * 256^2) + (byte3 * 256^1) + (byte4 *256^0) becomes (byte1 * 16777216) + (byte2 * 65536) + (byte3 * 256) + byte4

I was curious though as to why it was we could get the individual bytes converted but there wasn’t (to me) an obvious way to do the whole value. I came across a mention of the Convert Class on MSDN which does make it possible. Convert class in the .Net framework. After playing around some I was able to come up with a different way using the Convert Class that to me seems a little cleaner. The Convert class has several overloaded methods. One version of the ToInt64 method converts a string version of a number into 64-bit signed integer. Either this or ToUInt32 will work for our purposes here. ToInt32 won’t work because of the value may incorrectly (for our purposed) return a negative (i.e. signed) integer. The call to the ToInt64 method requires an argument that specifies the base of the number the string represents (in this case hex= base 16). So in theory that would work if we were able to represent the bytes as a hex string.. i.e. 0xA1B2C3D4. That’s when I came across a VBScript to Powershell page describing converting numbers from decimal to hex that helped fill in the missing piece. Putting these two things together I was able to come up with a function to use in my script which seemed to work.

The operative word here was “seemed”. I noticed in testing that there were instances where I got a very wrong answer. As an example if the Registry value were (0xA102C4D4, or 2701312980 decimal) the function would return. 169001940 decimal as the value…I realized that in building $tmpString if the value of a particular byte was less than 16 it’d spit out a single character 0xC rather than 0x0C. This was fine if I was interested in the value of a single byte but when concatenating the values together makes for a big difference in the resulting value as 0xA102C3D4 would become 0xA12C3D4. The resolution for this was to change the format string to pad the value with a leading 0 if necessary. So $tmpString += “{0:X}” -f $byte became $tmpString += “{0:X2}” -f $byte.
So the final function I ended up using looks like this: