As discussed earlier you could use IPMI just as easily to get the sensor data as the VMware APIs, but I added it just for consistency’s sake rather than having to manage two interfaces and sets of ids and passwords. Since the only servers I have access to at the moment are new Dell Poweredge’s right now I can only assume that these will also work with other manufacturers servers as they become supported.

As a reminder you need to install the appropriate VMware pieces for these scripts to run. Namely the VMware VI-Perl toolkit.

Script arguments are the same for all four of these scripts :

check_3i_storage –password <passwd> –username <user> –server <ip addr/hostname>

check_3i_cpu –password <passwd> –username <user> –server <ip addr/hostname>

check_3i_memory –password <passwd> –username <user> –server <ip addr/hostname>

check_3i_sensors –password <passwd> –username <user> –server <ip addr/hostname>

I’ve also included the updated service and command definitions I’ve used in my test Nagios environment for reference.

define service{
use                             generic-service
host_name                       svr-esx-test-01.company.com
service_description             ESX 3i Storage Status
check_command                   check_3i_storage!root!mypassword!
}

define command{
command_name    check_3i_storage
command_line    $USER1$/check_3i_storage --server $HOSTADDRESS$ --username $ARG1$ --password $ARG2$
}

I installed 3i on a new PE 2900 today to take a look at this again. I had previously pulled one of the disks in the server so that I could be certain something was “wrong” so I had something to test against. Below is the “Health Status” as shown via the VI client. As you can see “Storage” shows up as being in a warning state since RAID 6 Virtual Disk shows as being in a “Warning” state. It’s worth noting that since I pulled a hard drive Physical Disk 7 does not show in the list of items under Storage. I’m assuming that if the drive was actually bad it’d show up as failed. But I don’t know that I want to damage a perfectly good drive to find out.

You can also see this same info via the “Browse Objects managed by this host” option on the local web page for the host. (http://<esx_ip_address>/mob).

As you browse through the managed objects you’ll notice each screen has a Properties and a Methods section. The Properties section will have three columns Name, Type and Value. The path to get to the storage info object is presented using a Name (Value) format. [ex: on the screen below to represent the value "content" would be written as content(content)] If you don’t want to navigate manually you can try the following URL:

https://<esx_server_ip>/mob/?moid=healthStatusSystem&doPath=runtime%2ehardwareStatusInfo%2estorageStatusInfo

Home MOB Page -> Content(content) -> rootFolder(ha-folder-root) -> childEntity(ha-datacenter) -> hostFolder (ha-folder-host) -> Child-entity(ha-compute-res) -> Host (ha-host) -> configManager(configManager) -> healthStatusSystem(healthStatusSystem) -> Runtime(runtime) ->Hardwarestatusinfo (hardwareStatusInfo) -> Storagestatusinfo(storageStatusInfo)

Once there you’ll (hopefully) see that we have an array of HostStorageElementInfo objects. Here we’re interested mainly in the name and status->key values.

All of this info should be accessible via the VI SDK. I had earlier tried to download the VI Perl Toolkit for both Windows and Linux but was having issues with the Windows version. I also has some issues with the Linux version but that was solved by downloading and building some modules that aren’t apparently part of the standard perl install. Rather than mucking with that too long, I downloaded and installed the VI Perl Toolkit Virtual Appliance. This virtual appliance is unfortunately in Open Virtual Machine Format (OVF) which meant I had to go download another tool from the VMware website to convert it for use with ESX. Once I got it up and running I was able to run some of the sample scripts to get a feel for what they did and how they worked.

In a little while I was able to take one of them, datacenterlisting.pl and modify it to show me the same storage info I was able to see via the VI client. This is where the ability to use the manage objects browser came in handy since that provided the info I needed to be able to modify the script to get what I wanted. This first version doesn’t do anything fancy, it simply lists each of the storage elements and their status as Red, Yellow, or Green. If we have to run a script to regularly monitor the servers we’re deploying we’ll probably use already existing internal Nagios server to monitor these ESX boxes. That will require a slightly less verbose version of the script that only really outputs info when there’s an issue.

Here’s the simple version of the script. There’s a sample below of the command line. But you need to specify a user, password and ipaddress/hostname for the server you’re checking on. You should also be able to point this script at Virtual Center but I don’t know what it will do if it encounters 3.5 hosts vs 3i hosts. I added a dummy user for nagios on the host with readonly permissions since that’s our ultimate target.

 check_3i_storage_simple –password <passwd> –username <user> –server <ip addr/hostname> –datacenter ha-datacenter

the setting ‘ha-datacenter’ is the value at the root of the managed object hierarchy that we use to find the host info. (This is true even on a standalone host).

visdk@vaos:~$ ./check_3i_storage_simple –password badpassword –username root –server 11.22.33.44 –datacenter ha-datacenter

Datacenter = ha-datacenter
Hosts found:
1: w35d154.company.com
Boot time 2008-02-21T12:36:34.19314Z
Controller 0 (PERC 6/i Integrated) Has status of Green
Battery of Controller 0 Has status of Green
Physical Disk 0/E32 of Controller 0 Has status of Green
Physical Disk 1/E32 of Controller 0 Has status of Green
Physical Disk 2/E32 of Controller 0 Has status of Green
Physical Disk 3/E32 of Controller 0 Has status of Green
Physical Disk 4/E32 of Controller 0 Has status of Green
Physical Disk 5/E32 of Controller 0 Has status of Green
Physical Disk 6/E32 of Controller 0 Has status of Green
Physical Disk 8/E32 of Controller 0 Has status of Green
Physical Disk 9/E32 of Controller 0 Has status of Green
RAID 6 Virtual Disk 0 of Controller 0 Has status of Yellow
Port 0 of Controller 0 Has status of Green
Port 1 of Controller 0 Has status of Green
Port 2 of Controller 0 Has status of Green
Port 3 of Controller 0 Has status of Green
Port 4 of Controller 0 Has status of Green
Port 5 of Controller 0 Has status of Green
Port 6 of Controller 0 Has status of Green
Port 7 of Controller 0 Has status of Green

visdk@vaos:~$

Reference stuff:

The VI-Perl toolkit in its various incarnations as well as the VMware Infrastructure SDK are available from VMware at http://www.vmware.com/support/pubs/sdk_pubs.html To actually download components you have to have an account on their site.

The original script that I started with is in the samples/discovery folder and called datacenterlisting.pl.

We’ll walk through the steps to set up the server for monitoring via IPMI using the DRAC. It is supposedly possible to do this without the DRAC, but I haven’t had a reason to try to do that yet.  

1. Log in the DRAC (ex: https://<RAC IP Address>/ )
2. Once logged in (see below) choose the “Remote Access” option
    
3. Choose the “Configuration” tab
    
4. This should take you to the “Network Configuration” page.
5. Scroll down to the IPMI LAN Settings section and ensure the “Enable IPMI over LAN” option is checked. If it is not, check it and click the “Apply Changes” button at the bottom of the page.

   You can also change the Privilege Level Limit setting (i.e. restricting the kinds of actions that can be executed. Options are Administrator, Operator and User) I’ve left this one at Administrator because I will be limiting the user account we set up. However you also need to be aware that ipmitool by default wants to connect with Admin privileges but we can override this on the command line.

   You can also set the encryption key but we won’t be using this with ipmitool because there appears to be an issue with entering it in an easy fashion.

6. Back up at the top (the “Configuration” tab) choose “Users”
7. Click on the number representing one of the “Disabled” user accounts. We’ll pick #4.

   Check the box to enable this user. Enter the user name and password. Under the User Privileges sections set the “Maximum LAN User Privilege Granted” to “User”. Under the DRAC settings configure as you wish if you want this account to be able to manage the DRAC or just leave it empty. The completed form looks like:

   

8. Click the “Apply Changes” button at the bottom of the screen.
   That’s it. Our user is now set up for IPMI access. Using ipmitool will follow in another post.

When installed 3i can show the status of the hardware it’s running on. The shot below is from a new Dell PE 2950. However some of the HP blades I’ve seen with 3i installed on them have limited information. My assumption is that this requires integration between Vmware and the hardware vendor. Since the 2950 is the first (and so far only) system that is certified to run 3i I expect this to show up on the HP blades at some point in the future.

Unfortunately what it doesn’t allow us to do is to alert when a component has issues (i.e. non “Normal” status). In various discussions about the issue wed talked about looking at IPMI and what it can tell us.

I’ve set up Nagios on a VM running Ubuntu and installed ipmi-tool to poke around. I’ll follow up with a couple of posts on setting up the DRAC card in the dell to allow access via IPMI, as well as what it can/does show us.