So in the previous post I put together a simple script for getting the data out of a specified registry entry that handled the REG DWORD BIG ENDIAN data type.  In this one I’ll go over the general process of getting the registry based perf data into Zabbix and setting up alerting based on it.

Setting up Zabbix

I won’t cover the actual installation of Zabbix here, but before we can put data into Zabbix we need to add the counters/items that I will be populating in the future. The first thing I need to do is determine exactly what those counters are and which of the nodes they need to come from.

Because I have multiple Neverfail clusters in my environment I will create a template in Zabbix that has all the necessary counters associated with it that I can then apply to the hosts rather than adding them manually to each host.  Since a host can have multiple templates assigned to it I’ll also include a new “application” called Neverfail to help with separating Neverfail counters from any other counters that might be associated with a host (ex: Exchange counters).

To help with some of the drudgery associated with manually creating all the items, I’ve provided  a version of the template that can simply be imported into Zabbix. The template includes all of the counters from above as well as a couple of basic triggers for alerting.

Here are a couple of short videos that walk through manually creating a template, and importing the one I’ve provided.

Creating a template

Importing the Neverfail Template into Zabbix

Sharp eyes might notice that I’m capturing  bothUnsafeUpdateQueueSize and safeUpdateQueueSize twice.  In doing so these values are being treated differently. The first is a simple measurement of how much data is in the queue.

About Zabbix_sender

Now turning our attention to how we get the info into Zabbix let’s look at Zabbix_sender.  It’s available a pre-compiled binary for Windows from Zabbix’s website. Getting it ready is as simple as unzipping the download and putting the executable somewhere. By running zabbix_sender -h we can see it can take a number of options.

The ones I use  are -s, -z, -k and -o.  So a typical command line for me would look something like:

Breaking down the command line:

• zabbix.crtcorp.com is the Zabbix server we’re sending this data to
• neverfail01 is the Neverfail node we’re sending information about
• the key for the Zabbix item (i.e. counter) we want the information associated with is nf_cluster[file_sync_status];
• the value we want in the key is  ”/Synchronized“

In the example the value we’re putting into Zabbix is a string rather than a numerical value. Here’s an example with a numeric value being put into Zabbix:

C:\temp\Zabbix_sender -z zabbix.crtcorp.com -s neverfail01 -k”nf_cluster[throughput]” -o “103453″

Here we’re specifying the item with key nf_cluster[throughput] and giving it a value of 103453.

Adding Zabbix_Sender

Now what I needed to do is to combine the script I wrote earlier with zabbix_sender to actually put the registry data into Zabbix. So  I added a new function to the GetRegValue.vbs script to execute the actual zabbix_send. It is pretty straightforward it builds a formulaic command line and then executes it. You’ll notice there is no error checking.

The next step is to modify the main body of the original GetRegValue script to turn it into a function. I then changed the WScript.Echos so that we were returning the registry value rather than simply writing it to the console.  (WScript.Echo HexToDec(HexValue) -> GetRegValue = HexToDec(HexValue) , Wscript.Echo strValue -> GetRegValue=strValue, and so on).  At the end we have this script which is good for reading one specified registry value and then inserting it into Zabbix.

Since there are a number of values we want to put into Zabbix we need to think about how to approach this given that the script only handles one value at a time.  What I settled on was a a batch file that used a for loop to go through a file with a list of registry based perf counters related to Neverfail.  The script as it now stands needs three arguments passed to it. It needs the ZabbixKey, the registry key path, and the registry value .  For values I want to get from the passive node the registry path needs to include the private IP address of the passive node (ex: \\10.0.0.2\HKLM\Software\Neverfail\R2\Performance) so that reg.exe knows where to go get them from.  The script can then query the registry using the path and value combination to get the data which it can then send to Zabbix using the key specified on the command line.  So having the list of registry values from the part 1 post I’m able to put together my file.

Because  I need to specify a delimiter to the for command and I use commas ‘,’ in the Zabbix keys that I’ve defined, I need to use something else as a delimiter for my input file, so I’ve settled on using a pipe symbol as shown below.

While my batch file  is about 35 lines, it really boils down to one line which does all the real work:

With the environment variables expanded it would look more like;

This for loop reads in each line of  the text file zbx_keys_to_send.txt and using the pipe symbol as a delimiter reads in the first three tokens/strings of each line and call the SENDVALUES.vbs script with the three tokens/strings as arguments.  The script and input file worked fine when I ran them on the primary node, but not so well when I ran them while the secondary was active. After some troubleshooting I realized  that one thing I didn’t think through at first wat that I actually need two lists/input files. Since the private IP address I want to use to get data from the passive node’s registry will change depending on which node is active I’ll need one list for when the script is sending from the primary node (10.0.0.1)  and another for when the secondary (10.0.0.2) is active. The lists should essentially be identical with the only difference being the IP address specifed for the passive node.

Now all that is left to do is copy the batch file, the vbscript file and the approapirate inputer to each node in the cluster. Prior to setting up the scehiled task I like to manually run the batch file a few time to make sure  the data is getting populated into Zabbix. To do this I need to use a local account that exists on both nodes (in my case I use the local Administrator account). This is so that the reg.exe util can seamlessly get values from the passive node (assuming the account has the same password on both nodes).

A little troubleshooting hint.

When running the script manually I can see each time the VBScript file calls zabbix_sender and whether or not that submission was successful. Running zabbix_sender and mistyping the key was not an common issue when I was putting this together. Fortunately zabbix_sender lets me know what happened when I attempted to submit data.  As an example, below is the output I get when trying to submit a value for the nf_q[safe,size] key, if I mistype the key as nfq[safe,size]

I can see that it reports that I have 1 failed item, and no Processed items. When I run it without any typos (intentional or otherwise) I get:

Now I can see that I had 1 item processed successfully and no Failed ones.

Setting up Alerting

If you import the template I’ve provided it should have also created four triggers that can be used to generate actions within Zabbix.

These triggers are based on situations I’ve run into in my environment that I want to be aware of.  The first is when the size of either the Safe or Unsafe queue has been above 2GB for over an hour. Neverfail was great at letting me know the queue was full and it was going to stop replicating but not so much on the warning me it was happening front.  I generally wanted to be aware well before we got to that state where it stopped replicating and these triggers are a way of warning me something is going on.  The second situation is when data to be replicated has been sitting in one of the queue’s for more than a specified amount of time.  This is similar to watching the queue get beyond a certain size as the first two triggers do but is helpful in situations where there isn’t a whole of data changing on the active node(i.e. over weekends).

It is of course  possible to change these and set them to what fits for your environment and even to add other triggers. In later versions of this monitoring I’ve added some other counters/keys related to the task state using the nfcmd.exe command line tool. This allows me to see when a server is doing a full system check or even the dreaded “internal system task” as well as how much progress it’s made.  Some example screenshots are included below.

Sample Data for one cluster

Cluster Overview

Sample Graph of the Safe Queue size

Enhanced view

The three files I use are included here:

• DO_Zabbix.cmd – The batch file that reads the input file with reg values & zabbix keys and calls SendRegValue.vbs
• SendRegValue.vbs – The vbscript file that actually reads the registry entry and does any necessary conversions to send the value to Zabbix
• zabbix_keys_to_send.txt – the input file used by DO_Zabbix.cmd. This version is the one I run when the primary node is active. IP addresses would need to be changed for this to run on a passive node.

————————————

A few additional notes:

Because Neverfail  is continually pushing the perf data to the registry it does happen on occasion that the script will catch spuriously large or odd values for some counters.

If I were to use the zabbix_agent on my Neverfail nodes it is possible to include all this same monitoring within the agents configuration so that the agent pushes the data rather than using zabbix_sender via a scheduled task. However that’s a post for some other time…
-crt

Thwe original idea behind generating the report was to have the info come to me rather than logging into multiple servers and firing up the console multiple times (what can I say I’m lazy).

The report in the first version of the script was straighforward text. Recently I’ve been looking into and thinking about different ways to present the information in the report so I could just sort of glance at it and get the status. The disk related portion of the report wasn’t initially where I was focusing my attention. I was more interested in being able to get a quick idea of where we stood with the # of Recovery Points we were expecting to have.  An example of  one of the simple reports is below. From this we can see that we’re in pretty good shape with 100% valid RPs spanning about 24 days.

Starting Script at 04/30/2009 23:20:12

Replay Service is running

Server mailserver.company.com snapshots are being stored on R: and currently using 818.54GB. This is 99.98% of the used space(818.68GB) on the volume which is 1,360.22GB

The drive currently has 39.81% free space (e.g. 541.54GB)

Number of reported Recovery Points is 395 of these 395 are valid, and 0 are invalid (100.00%).
The valid RPs span 23.98 days

The most recent valid RP was taken 1 Minutes ago

The issue becomes less clear when invalid RPs occur for whatever reason. If I have 395 RPs and only 250 of the are valid is that a good or bad state? It’s not immediately clear but one can log in to the Replay server and get a better idea of how things stand. It might be the case where there was network issue during the day and instead of 96 RPs  (that’s an RP every 15 minutes * 24 hrs) for each of the last three days we’ve only gotten 40 RPs each of those days which while less than ideal might still be an okay state. Or it could be that there are several days for which we don’t have RPs.

I was trying to think of a way to visualize this information. Because of the retention schedule some days we’d expect a large number of RPs (~90) and some other days we’d expect to have just one.  I looked into the possibility of using sparklines even going so far as to download a C# based version of  a PHP based sparkline web service from Joe Gregorio.

I tried several different iterations of the script using sparklines trying to use the data I had in different ways (ex: use percentages of expected RPs, diffs between expected and actaul) but wasn’t able to find a good way to represent the state using those. In digging around I came across the Google Chart API and that looked at bar different ways of using bar graphs to represent the info I wanted.  Using either side-by-side bar graphs

overlapping ones with green and red where a lot of red would be a bad thing.

Either of these would have been an improvement over what I was getting in the text based report.  While trying to refine the overlaid version I came across an example in the Google documentation on line styles of  this graph:

Chart Data Line example from Google Chart API

This caught my eye as a possible solution to my problem about how to present this data because of the ability to show both sets of data overlaid on each other while still managing to keep both of them visible.

In my particular scenario the retention schedule is:

• RPs every 15  minutes which are kept for 4 days
• These roll up to hourly RPs which are kept for 5 days
• Hourly’s roll up to dailies which are kept for ~25 days

Our goal is to keep about 30 consecutive days worth of RPs on hand. When plotting out the # of expected RPs per day we get a graph that looks like the one below.

As one can see the number of Recovery Points per day decreases over time. When adding the line showing the number of actual RPs it can be hard to tell what the status is for the days where there’s only one RP per day. If things are going well the green bars will be obscured by the red line.

In the rare instance where we might be missing a few daily RPs the green bars do become somewhat visible as shown in the blue box below.

I experimented with a couple of different ways to try to make this more obvious including altering the width and height of the chart to make it more obvious. (see below). Using the Google Chart API one is limited to an image with 300000 pixels (500×600)

But for me persoanlly making the chart a lot bigger like  this seemed like it didn’t really add all that much to being able to see what was going on.  So I stuck with 600×250 for the graph.

It should also be noted that in the case where you aren’t taking snapshots every 15 minutes but evey 30 minutes or maybe even every hour it becomes easier to see missed daily RPs.  Here’s an example

After going through all of this with the RPs I almost as an afterhtought went back and added the logic to graph the disk usage data as well. It shows the size of the Replay archive data, the free space and other used space on the drive by generating something like this:

Here’s a “real-life”  example of the whole report.

The script is available here. If it’s of any use to you please drop me a line and let me know.

To use it rename it to something like ReplayReport.ps1. You’ll need to modify the variables at the beginning of the file:

• $ReportRecipients – array of recipient email addresses.
• $MailServer – The SMTP server to use to send the report out
• $ReportSender -Address the email should appear to come from.
• $replay_exe – Path to the Replayc.exe file. May differ on x64 vs x86  servers.
• $ExpectedRPCount – array containing the number of expected RPs for the last X days. Used to generate the graph of expected vs present RPs. (See Note below)

The script doesn’t take any arguments to run. I run it via a scheduled task on the replay server.

***************

In reference to $ExpectedRPCount the time of day that the report is run will affect how the first data point on the graph appears. RPs are tracked by the date that they were taken. If you run the script just before midnight there will obviously be a lot more RPs for “today” than if you run it at 1 am.

In poking around the install directory I came across the Replayc.exe command. Replayc is a command line utilty that offers information about the Replay server and a way to manually mount and dismount Recovery Points (RPs). After playing with it a little and being the very lazy person that I am  I decided to write a Powershell script to help give me a high level status overview of my servers.  The script runs on each server at about the same time (relative to me here in Oregon) every day and emails me the output. So instead of having to muck around in the console Ionly have to spend a few seconds each to make sure everything’s running properly.

The script is available here and needs to be renamed appropriately.

When the script runs the email (HTML formatted)  I get is like the one below.  It tells me a number of things:

• The status of the Replay Server (running/not running)
• The name of the server that’s being protected
• How much disk space is available and being used for RPs for that protected server
• The size of the disk where those RPs are being stored
• The # of valid and invalid RPs
• The timespan between first and last valid RP
• Last time an RP occurred.

Example Email:

Starting Script at 04/30/2009 23:20:12

Replay Service is running

Server mailserver.company.com snapshots are being stored on R: and currently using 818.54GB. This is 99.98% of the used space(818.68GB) on the volume which is 1,360.22GB

The drive currently has 39.81% free space (e.g. 541.54GB)

Number of reported Recovery Points is 395 of these 395 are valid, and 0 are invalid (100.00%).
The valid RPs span 23.98 days

The most recent valid RP was taken 1 Minutes ago

I installed 3i on a new PE 2900 today to take a look at this again. I had previously pulled one of the disks in the server so that I could be certain something was “wrong” so I had something to test against. Below is the “Health Status” as shown via the VI client. As you can see “Storage” shows up as being in a warning state since RAID 6 Virtual Disk shows as being in a “Warning” state. It’s worth noting that since I pulled a hard drive Physical Disk 7 does not show in the list of items under Storage. I’m assuming that if the drive was actually bad it’d show up as failed. But I don’t know that I want to damage a perfectly good drive to find out.

You can also see this same info via the “Browse Objects managed by this host” option on the local web page for the host. (http://<esx_ip_address>/mob).

As you browse through the managed objects you’ll notice each screen has a Properties and a Methods section. The Properties section will have three columns Name, Type and Value. The path to get to the storage info object is presented using a Name (Value) format. [ex: on the screen below to represent the value "content" would be written as content(content)] If you don’t want to navigate manually you can try the following URL:

https://<esx_server_ip>/mob/?moid=healthStatusSystem&doPath=runtime%2ehardwareStatusInfo%2estorageStatusInfo

Home MOB Page -> Content(content) -> rootFolder(ha-folder-root) -> childEntity(ha-datacenter) -> hostFolder (ha-folder-host) -> Child-entity(ha-compute-res) -> Host (ha-host) -> configManager(configManager) -> healthStatusSystem(healthStatusSystem) -> Runtime(runtime) ->Hardwarestatusinfo (hardwareStatusInfo) -> Storagestatusinfo(storageStatusInfo)

Once there you’ll (hopefully) see that we have an array of HostStorageElementInfo objects. Here we’re interested mainly in the name and status->key values.

All of this info should be accessible via the VI SDK. I had earlier tried to download the VI Perl Toolkit for both Windows and Linux but was having issues with the Windows version. I also has some issues with the Linux version but that was solved by downloading and building some modules that aren’t apparently part of the standard perl install. Rather than mucking with that too long, I downloaded and installed the VI Perl Toolkit Virtual Appliance. This virtual appliance is unfortunately in Open Virtual Machine Format (OVF) which meant I had to go download another tool from the VMware website to convert it for use with ESX. Once I got it up and running I was able to run some of the sample scripts to get a feel for what they did and how they worked.

In a little while I was able to take one of them, datacenterlisting.pl and modify it to show me the same storage info I was able to see via the VI client. This is where the ability to use the manage objects browser came in handy since that provided the info I needed to be able to modify the script to get what I wanted. This first version doesn’t do anything fancy, it simply lists each of the storage elements and their status as Red, Yellow, or Green. If we have to run a script to regularly monitor the servers we’re deploying we’ll probably use already existing internal Nagios server to monitor these ESX boxes. That will require a slightly less verbose version of the script that only really outputs info when there’s an issue.

Here’s the simple version of the script. There’s a sample below of the command line. But you need to specify a user, password and ipaddress/hostname for the server you’re checking on. You should also be able to point this script at Virtual Center but I don’t know what it will do if it encounters 3.5 hosts vs 3i hosts. I added a dummy user for nagios on the host with readonly permissions since that’s our ultimate target.

 check_3i_storage_simple –password <passwd> –username <user> –server <ip addr/hostname> –datacenter ha-datacenter

the setting ‘ha-datacenter’ is the value at the root of the managed object hierarchy that we use to find the host info. (This is true even on a standalone host).

visdk@vaos:~$ ./check_3i_storage_simple –password badpassword –username root –server 11.22.33.44 –datacenter ha-datacenter

Datacenter = ha-datacenter
Hosts found:
1: w35d154.company.com
Boot time 2008-02-21T12:36:34.19314Z
Controller 0 (PERC 6/i Integrated) Has status of Green
Battery of Controller 0 Has status of Green
Physical Disk 0/E32 of Controller 0 Has status of Green
Physical Disk 1/E32 of Controller 0 Has status of Green
Physical Disk 2/E32 of Controller 0 Has status of Green
Physical Disk 3/E32 of Controller 0 Has status of Green
Physical Disk 4/E32 of Controller 0 Has status of Green
Physical Disk 5/E32 of Controller 0 Has status of Green
Physical Disk 6/E32 of Controller 0 Has status of Green
Physical Disk 8/E32 of Controller 0 Has status of Green
Physical Disk 9/E32 of Controller 0 Has status of Green
RAID 6 Virtual Disk 0 of Controller 0 Has status of Yellow
Port 0 of Controller 0 Has status of Green
Port 1 of Controller 0 Has status of Green
Port 2 of Controller 0 Has status of Green
Port 3 of Controller 0 Has status of Green
Port 4 of Controller 0 Has status of Green
Port 5 of Controller 0 Has status of Green
Port 6 of Controller 0 Has status of Green
Port 7 of Controller 0 Has status of Green

visdk@vaos:~$

Reference stuff:

The VI-Perl toolkit in its various incarnations as well as the VMware Infrastructure SDK are available from VMware at http://www.vmware.com/support/pubs/sdk_pubs.html To actually download components you have to have an account on their site.

The original script that I started with is in the samples/discovery folder and called datacenterlisting.pl.

We’ll walk through the steps to set up the server for monitoring via IPMI using the DRAC. It is supposedly possible to do this without the DRAC, but I haven’t had a reason to try to do that yet.  

1. Log in the DRAC (ex: https://<RAC IP Address>/ )
2. Once logged in (see below) choose the “Remote Access” option
    
3. Choose the “Configuration” tab
    
4. This should take you to the “Network Configuration” page.
5. Scroll down to the IPMI LAN Settings section and ensure the “Enable IPMI over LAN” option is checked. If it is not, check it and click the “Apply Changes” button at the bottom of the page.

   You can also change the Privilege Level Limit setting (i.e. restricting the kinds of actions that can be executed. Options are Administrator, Operator and User) I’ve left this one at Administrator because I will be limiting the user account we set up. However you also need to be aware that ipmitool by default wants to connect with Admin privileges but we can override this on the command line.

   You can also set the encryption key but we won’t be using this with ipmitool because there appears to be an issue with entering it in an easy fashion.

6. Back up at the top (the “Configuration” tab) choose “Users”
7. Click on the number representing one of the “Disabled” user accounts. We’ll pick #4.

   Check the box to enable this user. Enter the user name and password. Under the User Privileges sections set the “Maximum LAN User Privilege Granted” to “User”. Under the DRAC settings configure as you wish if you want this account to be able to manage the DRAC or just leave it empty. The completed form looks like:

   

8. Click the “Apply Changes” button at the bottom of the screen.
   That’s it. Our user is now set up for IPMI access. Using ipmitool will follow in another post.

When installed 3i can show the status of the hardware it’s running on. The shot below is from a new Dell PE 2950. However some of the HP blades I’ve seen with 3i installed on them have limited information. My assumption is that this requires integration between Vmware and the hardware vendor. Since the 2950 is the first (and so far only) system that is certified to run 3i I expect this to show up on the HP blades at some point in the future.

Unfortunately what it doesn’t allow us to do is to alert when a component has issues (i.e. non “Normal” status). In various discussions about the issue wed talked about looking at IPMI and what it can tell us.

I’ve set up Nagios on a VM running Ubuntu and installed ipmi-tool to poke around. I’ll follow up with a couple of posts on setting up the DRAC card in the dell to allow access via IPMI, as well as what it can/does show us.