Carlos' Corner

Recap

So in the previous post I put together a simple script for getting the data out of a specified registry entry that handled the REG DWORD BIG ENDIAN data type.  In this one I’ll go over the general process of getting the registry based perf data into Zabbix and setting up alerting based on it.

Setting up Zabbix

I won’t cover the actual installation of Zabbix here, but before we can put data into Zabbix we need to add the counters/items that I will be populating in the future. The first thing I need to do is determine exactly what those counters are and which of the nodes they need to come from.

Because I have multiple Neverfail clusters in my environment I will create a template in Zabbix that has all the necessary counters associated with it that I can then apply to the hosts rather than adding them manually to each host.  Since a host can have multiple templates assigned to it I’ll also include a new “application” called Neverfail to help with separating Neverfail counters from any other counters that might be associated with a host (ex: Exchange counters).

To help with some of the drudgery associated with manually creating all the items, I’ve provided  a version of the template that can simply be imported into Zabbix. The template includes all of the counters from above as well as a couple of basic triggers for alerting.

Here are a couple of short videos that walk through manually creating a template, and importing the one I’ve provided.

Creating a template

Importing the Neverfail Template into Zabbix

Sharp eyes might notice that I’m capturing  bothUnsafeUpdateQueueSize and safeUpdateQueueSize twice.  In doing so these values are being treated differently. The first is a simple measurement of how much data is in the queue.

About Zabbix_sender

Now turning our attention to how we get the info into Zabbix let’s look at Zabbix_sender.  It’s available a pre-compiled binary for Windows from Zabbix’s website. Getting it ready is as simple as unzipping the download and putting the executable somewhere. By running zabbix_sender -h we can see it can take a number of options.

The ones I use  are -s, -z, -k and -o.  So a typical command line for me would look something like:

Breaking down the command line:

• zabbix.crtcorp.com is the Zabbix server we’re sending this data to
• neverfail01 is the Neverfail node we’re sending information about
• the key for the Zabbix item (i.e. counter) we want the information associated with is nf_cluster[file_sync_status];
• the value we want in the key is  ”/Synchronized“

In the example the value we’re putting into Zabbix is a string rather than a numerical value. Here’s an example with a numeric value being put into Zabbix:

C:\temp\Zabbix_sender -z zabbix.crtcorp.com -s neverfail01 -k”nf_cluster[throughput]” -o “103453″

Here we’re specifying the item with key nf_cluster[throughput] and giving it a value of 103453.

Adding Zabbix_Sender

Now what I needed to do is to combine the script I wrote earlier with zabbix_sender to actually put the registry data into Zabbix. So  I added a new function to the GetRegValue.vbs script to execute the actual zabbix_send. It is pretty straightforward it builds a formulaic command line and then executes it. You’ll notice there is no error checking.

The next step is to modify the main body of the original GetRegValue script to turn it into a function. I then changed the WScript.Echos so that we were returning the registry value rather than simply writing it to the console.  (WScript.Echo HexToDec(HexValue) -> GetRegValue = HexToDec(HexValue) , Wscript.Echo strValue -> GetRegValue=strValue, and so on).  At the end we have this script which is good for reading one specified registry value and then inserting it into Zabbix.

Since there are a number of values we want to put into Zabbix we need to think about how to approach this given that the script only handles one value at a time.  What I settled on was a a batch file that used a for loop to go through a file with a list of registry based perf counters related to Neverfail.  The script as it now stands needs three arguments passed to it. It needs the ZabbixKey, the registry key path, and the registry value .  For values I want to get from the passive node the registry path needs to include the private IP address of the passive node (ex: \\10.0.0.2\HKLM\Software\Neverfail\R2\Performance) so that reg.exe knows where to go get them from.  The script can then query the registry using the path and value combination to get the data which it can then send to Zabbix using the key specified on the command line.  So having the list of registry values from the part 1 post I’m able to put together my file.

Because  I need to specify a delimiter to the for command and I use commas ‘,’ in the Zabbix keys that I’ve defined, I need to use something else as a delimiter for my input file, so I’ve settled on using a pipe symbol as shown below.

While my batch file  is about 35 lines, it really boils down to one line which does all the real work:

With the environment variables expanded it would look more like;

This for loop reads in each line of  the text file zbx_keys_to_send.txt and using the pipe symbol as a delimiter reads in the first three tokens/strings of each line and call the SENDVALUES.vbs script with the three tokens/strings as arguments.  The script and input file worked fine when I ran them on the primary node, but not so well when I ran them while the secondary was active. After some troubleshooting I realized  that one thing I didn’t think through at first wat that I actually need two lists/input files. Since the private IP address I want to use to get data from the passive node’s registry will change depending on which node is active I’ll need one list for when the script is sending from the primary node (10.0.0.1)  and another for when the secondary (10.0.0.2) is active. The lists should essentially be identical with the only difference being the IP address specifed for the passive node.

Now all that is left to do is copy the batch file, the vbscript file and the approapirate inputer to each node in the cluster. Prior to setting up the scehiled task I like to manually run the batch file a few time to make sure  the data is getting populated into Zabbix. To do this I need to use a local account that exists on both nodes (in my case I use the local Administrator account). This is so that the reg.exe util can seamlessly get values from the passive node (assuming the account has the same password on both nodes).

A little troubleshooting hint.

When running the script manually I can see each time the VBScript file calls zabbix_sender and whether or not that submission was successful. Running zabbix_sender and mistyping the key was not an common issue when I was putting this together. Fortunately zabbix_sender lets me know what happened when I attempted to submit data.  As an example, below is the output I get when trying to submit a value for the nf_q[safe,size] key, if I mistype the key as nfq[safe,size]

I can see that it reports that I have 1 failed item, and no Processed items. When I run it without any typos (intentional or otherwise) I get:

Now I can see that I had 1 item processed successfully and no Failed ones.

Setting up Alerting

If you import the template I’ve provided it should have also created four triggers that can be used to generate actions within Zabbix.

These triggers are based on situations I’ve run into in my environment that I want to be aware of.  The first is when the size of either the Safe or Unsafe queue has been above 2GB for over an hour. Neverfail was great at letting me know the queue was full and it was going to stop replicating but not so much on the warning me it was happening front.  I generally wanted to be aware well before we got to that state where it stopped replicating and these triggers are a way of warning me something is going on.  The second situation is when data to be replicated has been sitting in one of the queue’s for more than a specified amount of time.  This is similar to watching the queue get beyond a certain size as the first two triggers do but is helpful in situations where there isn’t a whole of data changing on the active node(i.e. over weekends).

It is of course  possible to change these and set them to what fits for your environment and even to add other triggers. In later versions of this monitoring I’ve added some other counters/keys related to the task state using the nfcmd.exe command line tool. This allows me to see when a server is doing a full system check or even the dreaded “internal system task” as well as how much progress it’s made.  Some example screenshots are included below.

Sample Data for one cluster

Cluster Overview

Sample Graph of the Safe Queue size

Enhanced view

The three files I use are included here:

• DO_Zabbix.cmd – The batch file that reads the input file with reg values & zabbix keys and calls SendRegValue.vbs
• SendRegValue.vbs – The vbscript file that actually reads the registry entry and does any necessary conversions to send the value to Zabbix
• zabbix_keys_to_send.txt – the input file used by DO_Zabbix.cmd. This version is the one I run when the primary node is active. IP addresses would need to be changed for this to run on a passive node.

————————————

A few additional notes:

Because Neverfail  is continually pushing the perf data to the registry it does happen on occasion that the script will catch spuriously large or odd values for some counters.

If I were to use the zabbix_agent on my Neverfail nodes it is possible to include all this same monitoring within the agents configuration so that the agent pushes the data rather than using zabbix_sender via a scheduled task. However that’s a post for some other time…
-crt