{"id":2995,"date":"2011-07-08T15:30:00","date_gmt":"2011-07-08T13:30:00","guid":{"rendered":"https:\/\/ingmarverheij.com\/2011\/07\/monitor-hp-proliant-server-health-on-citrix-xenserver-with-nagios\/"},"modified":"2011-07-22T07:40:06","modified_gmt":"2011-07-22T05:40:06","slug":"monitor-hp-proliant-server-health-on-citrix-xenserver-with-nagios","status":"publish","type":"post","link":"https:\/\/ingmarverheij.com\/en\/monitor-hp-proliant-server-health-on-citrix-xenserver-with-nagios\/","title":{"rendered":"Monitor “HP Proliant Server health on “Citrix XenServer” with Nagios"},"content":{"rendered":"

\"Nagios<\/p>\n

Nagios is a powerful monitoring system that enables organizations to identify and resolve IT infrastructure problems before they affect critical business processes.<\/p>\n

After experiencing problems with a HP Proliant DL380G6 that unexpectedly restarts<\/a>, caused by a Automated Server Recovery (ASR), monitoring the status of the Citrix XenServers running on HP Proliant Servers is required in Nagios<\/a>.<\/p>\n

Nagios is a flexible solution that can be expanded with plugins. Plugins can be found at Nagios Exchange<\/a>, this is where I found the check check_hpasm<\/a> plugin (direct link<\/a>). Unfortunately this plugin does not check the ASR status.<\/p>\n

In this article I will describe how I\u2019ve configured Groundwork<\/a> (using Nagios) to monitor the health of HP Proliant Servers and expanded the check_hpasm plugin to check for ASR health.<\/p>\n

<\/p>\n

<\/h4>\n

check_hpasm operation modes<\/h4>\n

The check_hpasm plugin can operate in two modes: local and remote.<\/p>\n

\"check_hpasm<\/a>In local mode the check_hpasm plugin is installed on the Citrix XenServer accompanied with a Nagios plugin. The Nagios plugin (nrpe) queries the check_hpasm plugin. This requires at least two additional agents to be installed on the Citrix XenServer which is not recommended.<\/p>\n

The remote mode uses SNMP to query the HP System Health Monitor. This prevents installing plugins on the Citrix XenServer but requires SNMP to be configured and accessible from the Nagios server.<\/p>\n

For this setup I\u2019ve configured the remote mode using SNMP (and preventing installing plugins on the Citrix XenServers).<\/p>\n

 <\/p>\n

Prerequisite \u2013 HP System Health Monitor (hpasmd)<\/h4>\n

On the Citrix XenServer the \u201cHP Proliant Support Pack\u201d and \u201cHP Health Application and Insight Management Agent\u201d needs to be installed. These can be downloaded from hp.com<\/a> under \u201cSupport & Drivers\u201d. Or you can download the \u201cHP SNMP Agent for Citrix XenServer 5.x\u201d here<\/a>.<\/p>\n

Before continuing to installing the check_hpasm plugin you need to make sure the hpasm agent (daemon) is accessible via SNMP from the Nagios server.<\/p>\n

# snmpwalk -c public \u2013v1 <IP-address of your XenServer> 1.3.6.1.4.1.232\r\n\r\nSNMPv2-SMI::enterprises.232.1.1.1.0 = INTEGER: 1\r\nSNMPv2-SMI::enterprises.232.1.1.2.0 = INTEGER: 23\r\nSNMPv2-SMI::enterprises.232.1.1.3.0 = INTEGER: 2\r\nSNMPv2-SMI::enterprises.232.1.2.1.4.1.0 = INTEGER: 30\r\nSNMPv2-SMI::enterprises.232.1.2.1.4.2.1.1.1 = INTEGER: 1\r\nSNMPv2-SMI::enterprises.232.1.2.1.4.2.1.2.1 = STRING: \"Compaq Standard Equipment Agent for Linux\"<\/pre>\n
<\/h4>\n
Installing check_hpasm plugin<\/h4>\n
Since where using the remote operation mode of the check_hpasm plugin the plugin needs to be installed on the Nagios (or Groundwork) server.<\/p>\n
The check_hpasm plugin can be downloaded from Console Labs<\/a>. Either download the files from the Nagios server or upload them, for instance using WinSCP<\/a>.<\/p>\n
Connect via a Secure Shell<\/a> (SSH) to the Nagios server, for instance with PuTTY<\/a>, and change the directory to the location where you placed the tarball<\/a>\u00a0 (.tar.gz).<\/p>\n
Unpack the tarball<\/p>\n
# tar xzvf check_hpasm-4.1.2.tar.gz<\/pre>\n
Configure the plugin with the following options<\/p>\n\n\n\n\n\n\n\n\n\n\n
Option<\/strong><\/td>\nExplanation<\/strong><\/td>\n<\/tr>\n
–disable-hwinfo<\/td>\nIf you don\u2019t want to see type, serial number and biosrelease in the output, you can switch this off by using this option.<\/td>\n<\/tr>\n
–enable-hpacucli<\/td>\nActivate checking of RAID controllers.<\/td>\n<\/tr>\n
–enable-perfdata<\/td>\nAdd performance data to it\u2019s output by default.<\/td>\n<\/tr>\n
–with-degrees<\/td>\nTemperatures values displayed in celcius instead of fahrenheit.<\/td>\n<\/tr>\n
–with-nagios-user<\/td>\nDefines the Nagios user<\/td>\n<\/tr>\n
–with-nagios-group<\/td>\nDefines the Nagios group<\/td>\n<\/tr>\n
–with-noinst-level<\/td>\nDefines the exit code if no hpasm rpm was installed.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
 <\/p>\n
.\/configure --prefix=\/opt\/plugins\/custom\/hp-insight --with-nagios-user=monitor --with-nagios-group=users \u2013enable-hpacucli --enable-perfdata \u2013with-degrees<\/pre>\n
Compile<\/p>\n
# make\r\n...\r\n# make install\r\n...<\/pre>\n
And finally test the plugin<\/p>\n
# \/opt\/plugins\/custom\/hp-insight\/libexec\/check_hpasm -H 10.0.0.1 -C public\r\n\r\nOK - System: 'proliant dl360 g3', S\/N: '7J31LMW6N01D', ROM: 'P31 01\/28\/2004', hardware working fine, da: 1 logical drives, 1 physical drives | fan_1=50% fan_2=50% temp_1_cpu=16;50;50 temp_2_cpu=15;65;65 temp_3_ioBoard=21;56;56 temp_4_cpu=20;65;65<\/pre>\n
You now have configured the check_hpasm plugin and the hardware monitoring running.<\/p>\n
 <\/p>\n
Altering check_hpasm to incorporate ASR<\/h4>\n
The check_hpasm plugin checks the health of Processors, Power supplies, Memory modules, Fans, CPU- and board-temperatures and Raids. Unfortunately the status of ASR isn\u2019t part of the health check.<\/p>\n
<\/h5>\n
Determine ASR status<\/h5>\n
I\u2019ve downloaded the cpqhlth.mib MIB (part of the HP Insight Management MIB Kit<\/a>) and ran a query via SNMP on one of the Citrix XenServers via iReasoning MIB Browser<\/a> (Free and really useful). If found that there is an OID<\/a> that specifies the overall condition of the ASR feature (1.3.6.1.4.1.232.6.2.5.17.0 or .iso.dod.internet.private.enterprise.compaq.cpqHealth.cpqHeComponent.cpqHeAsr.cpqHeAsrCondition.0).<\/p>\n
\"SNMP<\/a><\/p>\n
I\u2019ve checked the status of the VH04 (one of the servers that unexpectedly restarts<\/a>) and saw that the status reported was degraded (3). A check on the VH01, which didn\u2019t suffer problems with ASR showed me a different result : ok (2), as expected.<\/p>\n
\"SNMP<\/a><\/p>\n
<\/h5>\n
Editing script<\/h5>\n
I\u2019ve downloaded the check_hpasm script via WinSCP (which can be found in \/opt\/plugins\/custom\/hp-insight\/libexec) and it in Perl Express<\/a> (A Free Perl IDE\/Editor for Windows).<\/p>\n
I noticed there was a procedure called \u2018overall_check\u2019 where I could incorporate a simple check for the ASR status. I\u2019ve altered the script on a few places to incorporate the check via the OID I found in the MIB (still following me?).<\/p>\n
sub <\/span>overall_init<\/h6>\n
sub overall_init {\r\n...\r\n  my $cpqHeAsrCondition = '1.3.6.1.4.1.232.6.2.5.17.0';\r\n  my $cpqHeAsrConditionValue = {\r\n    1 => 'other',\r\n    2 => 'ok',\r\n    3 => 'degraded',\r\n    4 => 'failed',\r\n  };\r\n...\r\n  $self->{asrstatus} = lc SNMP::Utils::get_object_value(\r\n      $snmpwalk, $cpqHeAsrCondition,\r\n      $cpqHeThermalSystemFanStatusValue);\r\n...<\/pre>\n
<\/h6>\n
sub <\/span>overall_check<\/h6>\n
sub overall_check {\r\n...\r\n  if ($self->{asrstatus}) {\r\n    if ($self->{asrstatus} eq 'degraded') {\r\n      $result = 1;\r\n      $self->add_message(WARNING,\r\n          sprintf 'ASR overall status is %s', $self->{asrstatus});\r\n    } elsif ($self->{asrstatus} eq 'failed') {\r\n      $result = 2;\r\n      $self->add_message(CRITICAL,\r\n          sprintf 'ASR overall status is %s', $self->{asrstatus});\r\n    }\r\n  } else {\r\n    $self->add_info('This system does not have ASR.');\r\n  }\r\n...<\/pre>\n
<\/h6>\n
sub <\/span>collect<\/h6>\n
sub collect {\r\n...\r\n      my $cpqHeAsr     = \"1.3.6.1.4.1.232.6.2.5\";\r\n...\r\n      # Walk for ASR\r\n      $tic = time;\r\n      my $response2a = $session->get_table(\r\n          -maxrepetitions => 1,\r\n          -baseoid => $cpqHeAsr);\r\n      if (scalar (keys %{$response2a}) == 0) {\r\n        $self->trace(2, sprintf \"maxrepetitions failed. fallback\");\r\n        $response2a = $session->get_table(\r\n            -baseoid => $cpqHeAsr);\r\n      }\r\n      $tac = time;\r\n      $self->trace(2, sprintf \"%03d seconds for walk $cpqHeAsr (%d oids)\",\r\n          $tac - $tic, scalar(keys %{$response2a}))\r\n...\r\n      map { $response->{$_} = $response2a->{$_} } keys %{$response2a};\r\n...<\/pre>\n
The end result would look like this:<\/p>\n
\"check_hpasm<\/a>\"check_hpasm<\/a>\"check_hpasm<\/a>\"check_hpasm<\/a>\"check_hpasm<\/a><\/p>\n
 <\/p>\n
 <\/p>\n
 <\/p>\n
 <\/p>\n
The script is uploaded again and tested from the Nagios server to determine the health of the VH04 server.<\/p>\n
\"check_hpasm<\/a><\/p>\n
The result is as expected, the check_hpasm gives a warning about a degraded ASR on the VH04.<\/p>\n
 <\/p>\n
Groundwork \/ Nagios<\/h4>\n
Now the plugin has to be incorporated in the Nagios (or in my case Groundwork which uses Nagios) configuration. This can be done by editing the configuration files or via the GUI.<\/p>\n
<\/h5>\n
Edit configuration files<\/h5>\n
The command<\/strong> \u2018check_hpasm\u2019 is stored in the checkcommands.cfg file. This file is located in the \u2018?\/nagios\/etc\/ directory for Nagios <\/strong>and in the ‘\/usr\/local\/groundwork\/core\/monarch\/workspace\/\u2019 directory for Groundwork<\/strong>.<\/p>\n
checkcommands.cfg<\/h6>\n
# command 'check_hpasm'\r\ndefine command{\r\n    command_name                   check_hpasm\r\n    command_line                   \/opt\/plugins\/custom\/hp-insight\/libexec\/check_hpasm -H $HOSTADDRESS$ -C $ARG1$\r\n    }<\/pre>\n
<\/h5>\n
GUI<\/h5>\n
Since I\u2019m using Groundwork I prefer to add the command and the service via the GUI.<\/p>\n
Command \u2018check_hpasm\u2019<\/span><\/p>\n
    \n
  1. Open the \u2018Configuration\u2019 tab<\/li>\n
  2. Click on the \u2018Commands\u2019 tab<\/li>\n
  3. On the left pane click \u2018Commands > New\u2019<\/li>\n
  4. Enter the details of the command\n