So in the previous post I put together a simple script for getting the data out of a specified registry entry that handled the REG DWORD BIG ENDIAN data type.  In this one I’ll go over the general process of getting the registry based perf data into Zabbix and setting up alerting based on it.

Setting up Zabbix

I won’t cover the actual installation of Zabbix here, but before we can put data into Zabbix we need to add the counters/items that I will be populating in the future. The first thing I need to do is determine exactly what those counters are and which of the nodes they need to come from.

Because I have multiple Neverfail clusters in my environment I will create a template in Zabbix that has all the necessary counters associated with it that I can then apply to the hosts rather than adding them manually to each host.  Since a host can have multiple templates assigned to it I’ll also include a new “application” called Neverfail to help with separating Neverfail counters from any other counters that might be associated with a host (ex: Exchange counters).

To help with some of the drudgery associated with manually creating all the items, I’ve provided  a version of the template that can simply be imported into Zabbix. The template includes all of the counters from above as well as a couple of basic triggers for alerting.

Here are a couple of short videos that walk through manually creating a template, and importing the one I’ve provided.

Creating a template

Importing the Neverfail Template into Zabbix

Sharp eyes might notice that I’m capturing  bothUnsafeUpdateQueueSize and safeUpdateQueueSize twice.  In doing so these values are being treated differently. The first is a simple measurement of how much data is in the queue.

About Zabbix_sender

Now turning our attention to how we get the info into Zabbix let’s look at Zabbix_sender.  It’s available a pre-compiled binary for Windows from Zabbix’s website. Getting it ready is as simple as unzipping the download and putting the executable somewhere. By running zabbix_sender -h we can see it can take a number of options.

The ones I use  are -s, -z, -k and -o.  So a typical command line for me would look something like:

Breaking down the command line:

• zabbix.crtcorp.com is the Zabbix server we’re sending this data to
• neverfail01 is the Neverfail node we’re sending information about
• the key for the Zabbix item (i.e. counter) we want the information associated with is nf_cluster[file_sync_status];
• the value we want in the key is  ”/Synchronized“

In the example the value we’re putting into Zabbix is a string rather than a numerical value. Here’s an example with a numeric value being put into Zabbix:

C:\temp\Zabbix_sender -z zabbix.crtcorp.com -s neverfail01 -k”nf_cluster[throughput]” -o “103453″

Here we’re specifying the item with key nf_cluster[throughput] and giving it a value of 103453.

Adding Zabbix_Sender

Now what I needed to do is to combine the script I wrote earlier with zabbix_sender to actually put the registry data into Zabbix. So  I added a new function to the GetRegValue.vbs script to execute the actual zabbix_send. It is pretty straightforward it builds a formulaic command line and then executes it. You’ll notice there is no error checking.

The next step is to modify the main body of the original GetRegValue script to turn it into a function. I then changed the WScript.Echos so that we were returning the registry value rather than simply writing it to the console.  (WScript.Echo HexToDec(HexValue) -> GetRegValue = HexToDec(HexValue) , Wscript.Echo strValue -> GetRegValue=strValue, and so on).  At the end we have this script which is good for reading one specified registry value and then inserting it into Zabbix.

Since there are a number of values we want to put into Zabbix we need to think about how to approach this given that the script only handles one value at a time.  What I settled on was a a batch file that used a for loop to go through a file with a list of registry based perf counters related to Neverfail.  The script as it now stands needs three arguments passed to it. It needs the ZabbixKey, the registry key path, and the registry value .  For values I want to get from the passive node the registry path needs to include the private IP address of the passive node (ex: \\10.0.0.2\HKLM\Software\Neverfail\R2\Performance) so that reg.exe knows where to go get them from.  The script can then query the registry using the path and value combination to get the data which it can then send to Zabbix using the key specified on the command line.  So having the list of registry values from the part 1 post I’m able to put together my file.

Because  I need to specify a delimiter to the for command and I use commas ‘,’ in the Zabbix keys that I’ve defined, I need to use something else as a delimiter for my input file, so I’ve settled on using a pipe symbol as shown below.

While my batch file  is about 35 lines, it really boils down to one line which does all the real work:

With the environment variables expanded it would look more like;

This for loop reads in each line of  the text file zbx_keys_to_send.txt and using the pipe symbol as a delimiter reads in the first three tokens/strings of each line and call the SENDVALUES.vbs script with the three tokens/strings as arguments.  The script and input file worked fine when I ran them on the primary node, but not so well when I ran them while the secondary was active. After some troubleshooting I realized  that one thing I didn’t think through at first wat that I actually need two lists/input files. Since the private IP address I want to use to get data from the passive node’s registry will change depending on which node is active I’ll need one list for when the script is sending from the primary node (10.0.0.1)  and another for when the secondary (10.0.0.2) is active. The lists should essentially be identical with the only difference being the IP address specifed for the passive node.

Now all that is left to do is copy the batch file, the vbscript file and the approapirate inputer to each node in the cluster. Prior to setting up the scehiled task I like to manually run the batch file a few time to make sure  the data is getting populated into Zabbix. To do this I need to use a local account that exists on both nodes (in my case I use the local Administrator account). This is so that the reg.exe util can seamlessly get values from the passive node (assuming the account has the same password on both nodes).

A little troubleshooting hint.

When running the script manually I can see each time the VBScript file calls zabbix_sender and whether or not that submission was successful. Running zabbix_sender and mistyping the key was not an common issue when I was putting this together. Fortunately zabbix_sender lets me know what happened when I attempted to submit data.  As an example, below is the output I get when trying to submit a value for the nf_q[safe,size] key, if I mistype the key as nfq[safe,size]

I can see that it reports that I have 1 failed item, and no Processed items. When I run it without any typos (intentional or otherwise) I get:

Now I can see that I had 1 item processed successfully and no Failed ones.

Setting up Alerting

If you import the template I’ve provided it should have also created four triggers that can be used to generate actions within Zabbix.

These triggers are based on situations I’ve run into in my environment that I want to be aware of.  The first is when the size of either the Safe or Unsafe queue has been above 2GB for over an hour. Neverfail was great at letting me know the queue was full and it was going to stop replicating but not so much on the warning me it was happening front.  I generally wanted to be aware well before we got to that state where it stopped replicating and these triggers are a way of warning me something is going on.  The second situation is when data to be replicated has been sitting in one of the queue’s for more than a specified amount of time.  This is similar to watching the queue get beyond a certain size as the first two triggers do but is helpful in situations where there isn’t a whole of data changing on the active node(i.e. over weekends).

It is of course  possible to change these and set them to what fits for your environment and even to add other triggers. In later versions of this monitoring I’ve added some other counters/keys related to the task state using the nfcmd.exe command line tool. This allows me to see when a server is doing a full system check or even the dreaded “internal system task” as well as how much progress it’s made.  Some example screenshots are included below.

Sample Data for one cluster

Cluster Overview

Sample Graph of the Safe Queue size

Enhanced view

The three files I use are included here:

• DO_Zabbix.cmd – The batch file that reads the input file with reg values & zabbix keys and calls SendRegValue.vbs
• SendRegValue.vbs – The vbscript file that actually reads the registry entry and does any necessary conversions to send the value to Zabbix
• zabbix_keys_to_send.txt – the input file used by DO_Zabbix.cmd. This version is the one I run when the primary node is active. IP addresses would need to be changed for this to run on a passive node.

————————————

A few additional notes:

Because Neverfail  is continually pushing the perf data to the registry it does happen on occasion that the script will catch spuriously large or odd values for some counters.

If I were to use the zabbix_agent on my Neverfail nodes it is possible to include all this same monitoring within the agents configuration so that the agent pushes the data rather than using zabbix_sender via a scheduled task. However that’s a post for some other time…
-crt

.

In order to actually be able to implement this I first had to go configure the appropriate networking configuration on each of the ESXi hosts.

First I needed to create a virtual switch utilizing the NIC attached to the crossover cable. This is done by going to the “Configuration” tab for the ESX host within the vSphere Client.

Clicking the “Add Networking” option will walk one through the wizard to configure the new switch. I started by choosing “Virtual Machine” on the Connection Type Screen.

On the next page I choose to create a new virtual switch and pick the appropriate physical NIC that will be used to communicate with the other host. (If I had the capacity to put all the VMs one one host I could create the vSwitch without having to specify a network adapter)

Then I created an initial Port Group and specified a VLAN ID for it. In this case for the Remote Site (192.168.3.X/24) I’m specifying VLAN ID 23.

Once completed the new virtual switch should look similar to the one shown below.

Now that the vSwitch has been created, I can add port groups for the other networks: DMZ (192.168.254.X), Internet (10.0.0.X) and HQ(192.168.2.X). Each one of these should have a unique VLAN ID associated with it which is also used when these port groups get created on the second host.

In addition I need to add port groups for the network between the DMZ and Internet router (192.168.5.X), as well as the one between the internal router and the DMZ (192.168.4.X). Because I’ll put all three routers on the same host, these last two port groups need only exist on that one host.

When all is said and done there should be 6 port groups defined on the vSwitch on the host where I’ll put the routers and 4 port groups on the other host. Notice that the VLAN IDs for each network match up across the two switches.

I initially tried this setup using an alpha version 6 of Vyatta but the routers felt a little slow to me (though I have no empirical evidence to back that up) so I went back to using version 5 LiveCDs for the routers as discussed in the earlier entry.

Building the internal router

I started by creating a custom VM for the internal or home router with 4 network adapters configured. I used the custom configuration because this VM won’t actually have a virtual hard disk. The four network adapters will be attached to my home LAN (lostroncos_01), the HQ Site Network, the Remote Site Network and the intranet to DMZ network. (Note: This post has a more detailed description of the process I used to create Vyatta routers. )

After powering on the router and opening the VM console I like to enable the NIC on my home network and ssh so I can do the rest of the configuration via an ssh client such as Putty. Once the VM has initially been powered on, I can go back and look at the settings and determine the MAC address of the NIC attached to my home network (lostroncos_01) is 00:0c:29:76:e2:07. Upon logging into the VM console I can go into configure mode by entering the command “configure” and then get a list of the interfaces the router knows about by entering “show interfaces”.

The one I’m interested in initially configuring is eth0. I can tell because the MAC matches the one I took note of earlier. To configure it to use 192.168.1.254 as it’s address and enable ssh I can do the following:

Let’s look at these commands in a little more detail.

The set interfaces command can take a wide variety of arguments. Here I’m specifying that I want to work on an ethernet interface. Other interface types include adsl, bonding, bridge, loopback, multilink, openvpn, serial, tunnel and wirelessmodem.

Even after specifying the type as ethernet and the particular interface (eth0) I can still use one of several subcommands/options.

Because I’m using VMs and a fairly generic setup I really only need to worry about the address option. So I specify the address and netmask notation by specifying the number of bits to use for the netmask (24).

The next command “set service ssh” simply enables the SSH server.

Now I’ve configured the interface and enabled SSH, but the settings haven’t been made active. That’s what the “commit” command does.

Executing the “show interfaces” command again I can see that the IP(v4) address is set for eth0 but not any of the other interfaces. Now I can use my ssh client to connect to the VM to finish the configuration. (One can also use the VMware console to perform all of the configuration steps, I just prefer an external SSH client).

After making sure the MAC addresses and networks match up the way I think they should, I can set the address for each of the other interfaces and commit the changes. Then I can exit the configuration mode by typing “exit” and save the configuration to the virtual floppy using the “init-floppy” command. This then ensures that my configuration will survive reboots of the router.

One of the other things than can be helpful to do is to use the “description” option with set interfacse to provide a little more information when logged into the router.

After doing this for each interface and then committing the changes I can see the descriptions when doing show interfaces from either the configure mode

Or from the actual router shell

Since I happen to  already have VMs on both the HQ and Remote networks I can verify basic connectivity from the router by pinging those VMs as well as my PC.

The next trick is to validate that I can ping the HQ and Remote subnets from my physical PC. Since both of those subnets are directly connected to the router I don’t need to add any static routes on the router. However I do need to add a route on my PC to reach those subnets. Otherwise it’ll try to use the default gateway out to my ISP (as shown below) which doesn’t know anything about this lab environment.

To reach the 192.168.[2,3,4] subnets I want to use the interface on rtr-home that’s attached to my home network (192.168.1.254). So using the “route” command from a command prompt on my Windows7 machine I can add the route to 192.168.4.X.

The route command in this case takes a few arguments…

#1 add tells route I’m adding a new one

#2 Next I specify the destination, and since ultimately I’m trying to get to a subnet we’ll specify 192.168.4.0.

#3 & #4 MASK says the next argument is the netmask relative to the destination

#5 is the gateway for this destination, in this case the interface on my virtual router that’s attached to the home network.

Once that’s done I can then try to ping & tracert the 192.168.4.2 interface on the router again.

Executing the “route print” command from the Command Prompt  I can see the entry for the 192.168.4.X network as I would expect.

Now that this particular routing entry appears to be working properly I can re-run the route command with the -p option so that it’s persistent across reboots of my home PC.

Next I need to add routes for the 192.168.2.X and 192.168.3.X networks to the PC as well.

At this point, I can try to do a traceroute from my PC to the VM on the HQ Site network that I was able to ping earlier from the router.

Because I’m occasionally paranoid, the next step for me was to connect to the console of the VMs on the HQ (shown below) and Remote (not shown) subnets and verify that I can ping machines on my home LAN across the internal router, rtr-home.

At this point my internal Vyatta router is routing traffic between subnets it is directly connected to (Home LAN, the HQ Site and Remote Site). Once the DMZ and Internet routers are set up some changes will need to be made on the internal router to get traffic to those non-connected subnets. I’ll go through that process in a subsequent post.

-crt

I did discover some snippets and information that helped point me in the right direction, but no simple complete “here run this” kind of examples. I initially came across an article by Thomas Rabaix on using SOAP PHP and NTLM authentication (http://rabaix.net/en/articles/2008/03/13/using-soap-php-with-ntlm-authentication ). This code used cURL to help handle NTLM authentication to an IIS server. That then led me to an article by Erik Cederstrand http://www.howtoforge.com/talking-soap-with-exchange that built on Thomas’ work. Between them these two extend the PHP SOAPClient and then override some of the methods to use curl to handle the NTLM authentication that EWS uses by default. Using these examples as a starting point and some other bits of info I’ve come across I’ve been able to put together a couple of scripts that will generate an ATOM based feed using a user’s calendar. What I’ve managed to cobble together works but is not something I’d describe as robust. The solution is briefly described below. I’m hoping to follow up with a few other posts that go into a bit more detail about how all the pieces work. [the code is available from here or from Google Code ]

The scripts in action

In my example Exchange environment I’ve created a user called (imaginatively enough) ctronco. Opening up Outlook for the user I can see the following events on this weeks calendar.

Turning now to the Ubuntu box where I’ve installed the PHP scripts. Firing up a browser and pointing it to the correct URL (http://192.168.1.175/ewscalendarfeed/getfeed/ctronco) I get the following results in Firefox, Internet Explorer and Opera (Chrome doesn’t appear to like feeds so I haven’t included it).

Output in Firefox	Output in Internet Explorer	Output in Opera

Clicking on the link of any of the individual entries will return a web a page with the same information about the individual appointment as well:

How it works (at a high level)?

Rather than keeping each user’s credentials around I decided to use a non-privileged account. This means that that account has to have Reviewer permissions for every calendar that will have a feed generated for it. Adding permissions for the account is a relatively straightforward proposition if you’re using Outlook. It can also be done via EWS for non-Outlook folks, I just haven’t gotten that piece done yet.

The configuration file for the scripts has a list of known calendar “names” and the identifying information to retrieve the calendar from Exchange. Each accessible calendar has to have a unique name as far as the scripts are concerned. This is because each user can have more than one calendar available for generating a feed and I wanted to be able to support showing more than one per user.

The naming convention you use will be up to you. Using the four calendars shown above in my test user’s mailbox either of the following example schemes would work for creating unique name for each of the calendars.

In addition to the unique name the configuration file also needs to be given the unique Folder Id and Changekey values from EWS to retrieve them. These values aren’t easily accessible via Outlook (as far as I can tell)so I’ve included a web form (list_calendars.html) which will prompt for a user’s credentials and then list the calendars in their mailbox as well as the associated Folder Id and ChangeKey. Example output is shown:

If you install the package php5-cli, list_calendars.php (the script called by the html file) can also be run from a shell on the Ubuntu box by passing the username and password as parameters.

Setting things up

Exchange

The first thing I had to do was set up Windows 2008 and Exchange 2007 in my home lab (at the time I had hardware that wouldn’t support 64-bit VMs so I couldn’t do Exchange 2010). I set up a single machine as both an AD Domain Controller and an Exchange server with the HUB, CAS, and Mailbox roles on it. Based on my initial testing this will also work just fine with Exchange 2010.

Ubuntu

For my test PHP environment I set up another VM running Ubuntu 9.10 Server with the LAMP option chosen at installation.

Once the OS install was complete I also had to install the libcurl3 and php5-curl packages.

The initial version of the source code is available here as a zip file. In addition I’ve created a project on Google Code where the most recent versions can be retrieved as I work on it further.

To install it simply unzip the tar file in a location where apache (or the web server of your choice) can read the files. It will uninstall into a directory called ews-cal-rss. You’re welcome to change this to whatever you desire.

In the example I’ve also changed the ownership of the directory so that it’s owned by the same login the web server runs under.

In the directory where the extracted files are you’ll find cfg_options.php. This contains almost all the configurable values. The following values need to be defined:

• $cfg_option['user'] – Login Id of the non-privileged account that will used to read all the calendars
• $cfg_option[''] -
• $cfg_option['authmethod'] – the scripts support both Basic and NTLM authentication when talking to EWS. NTLM required installation of curl
• $cfg_option['wsdl'] – path to the appropriate Exchange Web Services WSDL file.
• $cfg_option['installpath'] – full path to the scripts
• $cfg_option['urlpath'] – URL for the scripts. If the script URL is http://host/ews-cal-rss/getfeed.php this would be “/ews-cal-rss”
• You also need to populate the variable $PFIDs using the list_calendars.html file as shown above.

In addition you need to modify the appropriate WSDL file for your environment. WSDL files are included for both Exchange 2007 and 2010. They are in <installdir>/e2k7_wsdl or <installdir>/e2kx_wsdl respectively. The services.wsdl needs to be modified to point to your Exchange server. At the end of the services.wsdl file you’ll find the following:

You need to modify the soap:address entry to point to your Exchange server. In my case that would be

A few miscellaneous notes:

• It is also possible to use calendars in a public folder to generate the feeds. However listing them is slightly more involved than it is for a users calendars.

• Re: the use of NTLM. It is also possible to modify the Exchange server’s web server configuration to enable the use of ‘Basic’ authentication in addition to NTLM. Since by default communication with the Exchange server takes place over HTTPS this may or may not be acceptable in your environment. Using basic authentication gets you out of having to install the cURL bits, but may or may not be acceptable from a security perspective depending on your environment.
• This is a test to see if this works better than the 32 bit version of IE does in full screen visual mode with wordpress….

Thwe original idea behind generating the report was to have the info come to me rather than logging into multiple servers and firing up the console multiple times (what can I say I’m lazy).

The report in the first version of the script was straighforward text. Recently I’ve been looking into and thinking about different ways to present the information in the report so I could just sort of glance at it and get the status. The disk related portion of the report wasn’t initially where I was focusing my attention. I was more interested in being able to get a quick idea of where we stood with the # of Recovery Points we were expecting to have.  An example of  one of the simple reports is below. From this we can see that we’re in pretty good shape with 100% valid RPs spanning about 24 days.

Starting Script at 04/30/2009 23:20:12

Replay Service is running

Server mailserver.company.com snapshots are being stored on R: and currently using 818.54GB. This is 99.98% of the used space(818.68GB) on the volume which is 1,360.22GB

The drive currently has 39.81% free space (e.g. 541.54GB)

Number of reported Recovery Points is 395 of these 395 are valid, and 0 are invalid (100.00%).
The valid RPs span 23.98 days

The most recent valid RP was taken 1 Minutes ago

The issue becomes less clear when invalid RPs occur for whatever reason. If I have 395 RPs and only 250 of the are valid is that a good or bad state? It’s not immediately clear but one can log in to the Replay server and get a better idea of how things stand. It might be the case where there was network issue during the day and instead of 96 RPs  (that’s an RP every 15 minutes * 24 hrs) for each of the last three days we’ve only gotten 40 RPs each of those days which while less than ideal might still be an okay state. Or it could be that there are several days for which we don’t have RPs.

I was trying to think of a way to visualize this information. Because of the retention schedule some days we’d expect a large number of RPs (~90) and some other days we’d expect to have just one.  I looked into the possibility of using sparklines even going so far as to download a C# based version of  a PHP based sparkline web service from Joe Gregorio.

I tried several different iterations of the script using sparklines trying to use the data I had in different ways (ex: use percentages of expected RPs, diffs between expected and actaul) but wasn’t able to find a good way to represent the state using those. In digging around I came across the Google Chart API and that looked at bar different ways of using bar graphs to represent the info I wanted.  Using either side-by-side bar graphs

overlapping ones with green and red where a lot of red would be a bad thing.

Either of these would have been an improvement over what I was getting in the text based report.  While trying to refine the overlaid version I came across an example in the Google documentation on line styles of  this graph:

Chart Data Line example from Google Chart API

This caught my eye as a possible solution to my problem about how to present this data because of the ability to show both sets of data overlaid on each other while still managing to keep both of them visible.

In my particular scenario the retention schedule is:

• RPs every 15  minutes which are kept for 4 days
• These roll up to hourly RPs which are kept for 5 days
• Hourly’s roll up to dailies which are kept for ~25 days

Our goal is to keep about 30 consecutive days worth of RPs on hand. When plotting out the # of expected RPs per day we get a graph that looks like the one below.

As one can see the number of Recovery Points per day decreases over time. When adding the line showing the number of actual RPs it can be hard to tell what the status is for the days where there’s only one RP per day. If things are going well the green bars will be obscured by the red line.

In the rare instance where we might be missing a few daily RPs the green bars do become somewhat visible as shown in the blue box below.

I experimented with a couple of different ways to try to make this more obvious including altering the width and height of the chart to make it more obvious. (see below). Using the Google Chart API one is limited to an image with 300000 pixels (500×600)

But for me persoanlly making the chart a lot bigger like  this seemed like it didn’t really add all that much to being able to see what was going on.  So I stuck with 600×250 for the graph.

It should also be noted that in the case where you aren’t taking snapshots every 15 minutes but evey 30 minutes or maybe even every hour it becomes easier to see missed daily RPs.  Here’s an example

After going through all of this with the RPs I almost as an afterhtought went back and added the logic to graph the disk usage data as well. It shows the size of the Replay archive data, the free space and other used space on the drive by generating something like this:

Here’s a “real-life”  example of the whole report.

The script is available here. If it’s of any use to you please drop me a line and let me know.

To use it rename it to something like ReplayReport.ps1. You’ll need to modify the variables at the beginning of the file:

• $ReportRecipients – array of recipient email addresses.
• $MailServer – The SMTP server to use to send the report out
• $ReportSender -Address the email should appear to come from.
• $replay_exe – Path to the Replayc.exe file. May differ on x64 vs x86  servers.
• $ExpectedRPCount – array containing the number of expected RPs for the last X days. Used to generate the graph of expected vs present RPs. (See Note below)

The script doesn’t take any arguments to run. I run it via a scheduled task on the replay server.

***************

In reference to $ExpectedRPCount the time of day that the report is run will affect how the first data point on the graph appears. RPs are tracked by the date that they were taken. If you run the script just before midnight there will obviously be a lot more RPs for “today” than if you run it at 1 am.

I also finally found the URL for the downloads to the other versions of the VI-perl stuff as well at : http://communities.vmware.com/community/developer/vsphere_sdk_perl

This is the first of a couple of posts on how I’ve cobbled together some basic monitoring of Neverfail’s  Neverfail Heartbeat H/A software which is also now the basis for VMWare’s vCenter Server Heartbeat. Since Neverfail seems to consider their command lines privileged information I will only cover how to do some simple monitoring using the registry. When starting on this effort internally I was only interested initially in figuring out a quick and simple way to get the info I needed and not so much on the how to get it into something part.

I’ve been working with another team where I work to look at Zabbix as an alternative for some of the monitoring we do in our environment. We use Microsoft Operations Manager 2005 (MOM) but haven’t fully cut over from out previous monitoring solution. I had looked at Zabbix earlier as a potential solution for monitoring a bunch of VMware ESX boxes but another team ended up getting tasked with that particular duty. So I had had some experience with Zabbix but hadn’t done too much with it since.

One of the things that’d been rattling around in my brain is using the capabilities of using the zabbix_sender feature/client to monitor some of other components/things we can’t easily get into MOM.  Zabbix_Sender is a utility that is available for use with Zabbix that allows one to “send” information to Zabbix. In my case it was appealing because we’re already running two different monitoring agents on the Exchange servers where we have Neverfail installed.  Since I only wanted to use Zabbix to monitor a small set of data related specifically to Neverfail zabbix_sender lets me do that without having to run the fullblown zabbix_agent as a service on the boxes.

Getting the Data

Neverfail (at least the versions we have installed) doesn’t obviously expose performance data. However if you look in the registry on each Neverfail server you will find some registry values (see HKLM\Software\Neverfail\R2\Performance) that get updated on a regular and frequent basis that correspond to data presented in the Neverfail GUI . Because of the way Neverfail works some of this data (Unsafe Queue info) is available on the Active node and some of it (Safe Queue info) is in the registry on the Passive node. This presents a couple of issues when trying to put together the solution (at least in my environment).

The first of these is trying to find a single consistent way to get the data out of the registry, especially since all the counters involved are of the REG_DWORD_BIG_ENDIAN variety (you can see a previous entry related to BIG_ENDIAN here).  I ended up settling on using the Reg.exe util available in Windows.  This utility let’s you manipulate the registry locally and remotely. While it doesn’t necessarily deal happily with REG_DWORD_BIG_ENDIAN (RDBE) entries in the registry it is able to extract the data which we can then manipulate to get the correct values.

As an example if I have the following two values in the registry as shown by RegEdit

When I run reg.exe I get the following output…

So while Dword_example and DWORD_BE_Example nominally have the same value reg.exe doesn’t get the data out correctly for the latter. However as I said earlier once we have the data out we can actually do some magic to get the right value.

We can also use reg.exe to get values on a remote machine (i.e. our Passive Neverfail node) by pre-pending the host info to the query registry path. So in this case to reach the passive secondary node over the private channel at 10.0.0.2 I can do something like  reg.exe Query \\10.0.0.2\CRT_CORP\Performance. Testing this out leads us to  a second issue. Getting an  ”Acces is denied” error.

Since my passive Neverfail node is essentially off-net but still thinks the network cables is live I can’t use a domain based account to run the reg.exe command because it can’t contact a domain controller to authenticate my domain account. However if I use the local Administrator account which has a common password on both nodes I can get this work just fine. (It may be possible to use an account other than the local Administrator but in my case where I also run some Neverfail command lines I need an account that’s authorized in Neverfail)

Given this info I was able to put together a vbscript that takes two arguments: a reg path and a value name;  and it returns the data value to the console converting REG_DWORD and REG_DWORD_BIG_ENDIAN to the correct decimal value. Using this script it’s then possible to get  any of the counters we’re interested in on either the active or passive node.  So based on the example above where I ran reg.exe hklm\software\CRT_CORP\Performance /s we can run the script for each of the values and see that we do in fact get the right decimal value for each one.

So now the trick is to figure out which of the registry based perf values we want to use and which host we need to draw them from.  Each of the Neverfail nodes has the same set of values present even though they’re not all populated the same way. That is to say that the counters related to the Safe Queue are not updated on the Active node since the Safe Queue exists on the passive node. And the converse is true with regard to the UnsafeQueue counters.  As I was mostly interested in alerting related to an issue we have occur occassionally I really wanted to get the SafeQueue and UnsafeQueue related counters (OldestSafeUpdateQueueEntry, SafeUpdateQueueSize etc). But since the other counters are also equally easy to get I decided I to include several more.  The image below shows the available values.

So now that I have a simple way of getting the information I want I can focus on how to get it into whatever system I want to monitor with whether it’s Zabbix (now) or Systems Center Operations Manager 2007 (later).  In the next article(s) I’ll talk about setting up the Zabbix part of this monitoring.

Acknowledgement: The hex to decimal routine in the GetRegValue.vbs script is lifted directly from http://www.sonofsofaman.com/hobbies/code/hextodec.asp Thanks to Joel for keeping me from having to reinvent the wheel. -crt

Addendum: While traipsing through the registry in figuring this stuff out I also discovered that there’s a bunch of configuration information stored in a whole different key under HKLM\Software\Javasoft\Prefs\neverfail\current\* It’s also possible to watch a few entries here to help monitor the  file and registry synchronization status even though it’s not as granular/descriptive/timely as can be obtained by using the command line.

The two items I’ve found that might be of interest are the /Registry/State/Manager\/Status Key and the /Value entry

and  the /New/File/State/Mgr\/Synchronization/Status key and /Tag entry

In poking around the install directory I came across the Replayc.exe command. Replayc is a command line utilty that offers information about the Replay server and a way to manually mount and dismount Recovery Points (RPs). After playing with it a little and being the very lazy person that I am  I decided to write a Powershell script to help give me a high level status overview of my servers.  The script runs on each server at about the same time (relative to me here in Oregon) every day and emails me the output. So instead of having to muck around in the console Ionly have to spend a few seconds each to make sure everything’s running properly.

The script is available here and needs to be renamed appropriately.

When the script runs the email (HTML formatted)  I get is like the one below.  It tells me a number of things:

• The status of the Replay Server (running/not running)
• The name of the server that’s being protected
• How much disk space is available and being used for RPs for that protected server
• The size of the disk where those RPs are being stored
• The # of valid and invalid RPs
• The timespan between first and last valid RP
• Last time an RP occurred.

Example Email:

Starting Script at 04/30/2009 23:20:12

Replay Service is running

Server mailserver.company.com snapshots are being stored on R: and currently using 818.54GB. This is 99.98% of the used space(818.68GB) on the volume which is 1,360.22GB

The drive currently has 39.81% free space (e.g. 541.54GB)

Number of reported Recovery Points is 395 of these 395 are valid, and 0 are invalid (100.00%).
The valid RPs span 23.98 days

The most recent valid RP was taken 1 Minutes ago

So what it it turns out I was getting from the GetValue method was a sequence of 4 bytes as an array. This made me wonder what the heck was going on so I fired up RegEdit to take a look. The contents of the Data column looked okay but the Type was different, so I expanded the column to see what was different and discovered REG_DWORD_BIG_ENDIAN as a type. This was one I don’t recall having seen before.

TechNet has the following to say about DWORD and it’s brethren:

REG_DWORD
A 32-bit (4-byte) number. Boolean (”True” or “False”) values and many entries for device drivers and services use this data type. REG_DWORD data can be displayed and entered in hexadecimal or decimal format in the registry editor Regedit.exe. For an example, see the ActivityLogFlag entry.

REG_DWORD_BIG_ENDIAN
Same as REG_DWORD. A 32-bit number in which the most significant byte is displayed as the leftmost (or high-order) byte. This is the most common format for storing numbers in computers that are running Windows Server 2003.

REG_DWORD_LITTLE_ENDIAN
A 32-bit number in which the most significant byte is displayed as the rightmost (or low-order) byte. This is opposite of the order in which bytes are stored in the REG_DWORD and REG_DWORD_BIG_ENDIAN data types.

If you’re not paying attention it could be easy to miss the difference when using RegEedit since they appear almost identical to DWORD values. The only obvious difference is the “Type” field.

So in the example above GetValue returns different values for “PerfCtr2″ and “PerfCtr2 DWORD” which are nominally the same value (at least according to RegEdit).

To help me figure out how to get the info I was looking for I put together a test and created a couple of dummy registry keys with each of the types of reg keys and some examples.

If we try to see what Powershell tells us about each of these keys we see that for our BIG_ENDIAN friend GetValueKind returns “unknown.”

GetValue converts each byte to a decimal value. Our key REG_DWORD_BIG_ENDIAN (0xa1b2c3d4) can be expressed as 4 bytes “a1″ “b2″ “c3″ “d4″ which when converted become the values “161″ “178″ “195″ “212″. While this is mildly useful it doesn’t help us easily get the value we want 2712847316. While it is possible to get the right value by doing some math [ (byte1 * 256^3) + (byte2 * 256^2) + (byte3 * 256) + byte4 ] I thought my resulting attempts to write a snippet to do this were ugly since it doesn’t appear Powershell has any easy way to do exponentiation.

The third method was to write out the formula a little more explicitly so that (byte1 * 256^3) + (byte2 * 256^2) + (byte3 * 256^1) + (byte4 *256^0) becomes (byte1 * 16777216) + (byte2 * 65536) + (byte3 * 256) + byte4

I was curious though as to why it was we could get the individual bytes converted but there wasn’t (to me) an obvious way to do the whole value. I came across a mention of the Convert Class on MSDN which does make it possible. Convert class in the .Net framework. After playing around some I was able to come up with a different way using the Convert Class that to me seems a little cleaner. The Convert class has several overloaded methods. One version of the ToInt64 method converts a string version of a number into 64-bit signed integer. Either this or ToUInt32 will work for our purposes here. ToInt32 won’t work because of the value may incorrectly (for our purposed) return a negative (i.e. signed) integer. The call to the ToInt64 method requires an argument that specifies the base of the number the string represents (in this case hex= base 16). So in theory that would work if we were able to represent the bytes as a hex string.. i.e. 0xA1B2C3D4. That’s when I came across a VBScript to Powershell page describing converting numbers from decimal to hex that helped fill in the missing piece. Putting these two things together I was able to come up with a function to use in my script which seemed to work.

The operative word here was “seemed”. I noticed in testing that there were instances where I got a very wrong answer. As an example if the Registry value were (0xA102C4D4, or 2701312980 decimal) the function would return. 169001940 decimal as the value…I realized that in building $tmpString if the value of a particular byte was less than 16 it’d spit out a single character 0xC rather than 0×0C. This was fine if I was interested in the value of a single byte but when concatenating the values together makes for a big difference in the resulting value as 0xA102C3D4 would become 0xA12C3D4. The resolution for this was to change the format string to pad the value with a leading 0 if necessary. So $tmpString += “{0:X}” -f $byte became $tmpString += “{0:X2}” -f $byte.
So the final function I ended up using looks like this: