Riccardo Riva

At some customer sites I have to shutdown all Virtual Machines and all ESX hosts when a power failure occurs, but I didn’t find a place in which collect all informations needed to let all works.
Finally I collect all the information needed and adjust some script find all over the internet and the following is the result.

I’ve tested it on ESX Server 3.5 Update 4 with APC PowerChute Agent v2.2.3 and APC SMART-UPS 5000i UPS

The goal was to :
If a power failure occurs then wait some minutes to check if power come back, and if not :
- try to shutdown all virtual machines running on esx host with a soft shutdown command
- if soft shutdown above fail, shut them down with a hard shutdon command
- wait for virtual machine to shutdown (300 sec).
- if virtual machine is already running then do hard power off
- shutdown esx host

This post assume you have an ESX 3 server running.
This post also assume you have an APC UPS with a network management card that is correctly configured and reachable from the esx host.

The first step is to install and configure APC Network shutdown on your esx host.
To do so :

Configure VMware ESX firewall.
Logon to your service console and add some firewall rules to let the esx host able to talk with UPS management card by execute the following commands :

esxcfg-firewall -o 80,tcp,out,”APC 80/tcp out”
esxcfg-firewall -o 3052,tcp,in,”APC 3052/tcp in”
esxcfg-firewall -o 3052,udp,out,”APC 3052/udp out”
esxcfg-firewall -o 3052,udp,in,”APC 3052/udp in”

Add another firewall rules to let your esx host send a mail message when the script start by execute the following command :

esxcfg-firewall -o 25,tcp,out,”SMTP 25/tcp out”

Sending mail from ESX service console is not built-in with VMware, but you should follow this posts to let it possible :

http://www.yellow-bricks.com/2008/01/23/howto-sending-html-email-from-the-service-console/

In this posts I’ll use that method, so if you don’t want to customize the script, please use the information on Yellow-Bricks, and copy the smtp_send.pl file in /usr/local/bin.

Install the APC Network Shutdown agent on your ESX host.
Download the “PowerChute Network Shutdown for VMWare” from the APC website (http://www.apc.com) and run the following command :

rpm -Uhv pcns-2.2.3-100.i386.rpm

Configure the APC Network Shutdown agent.
You have to configure your agent to let it talk with the APC network interface card of the UPS, for doing so you have to run the APC agent configuration script :

/usr/local/bin/PowerChute/PCNSConfig.sh

After the configuration you should manage your ESX Agent trough a wen interface pointing your browser to :
http://esxhost:3052

It’s very important that you check the event log searching messages from the agent and you should find a message showing that connections between esx and ups was established.

At this point you must configure your esx agent on how action to do when a power failure occurs.
One very important thing to note is to untick the flag from “Turn off the UPS after the shutdown finishes” because otherwise, at the end of the shutdown process, the agent will send a shutdown command even to the UPS and if you have other server connected to that UPS it will be shutted down immediately.
Another important thing to do is to tell what esx will do when a the ups send a shutdown command for a too low battery charge.
You have to configure your esx server to run a the following bash script who :
- try to shutdown all virtual machines running on esx host with a soft shutdown command
- if soft shutdown above fail, shut them down with a hard shutdon command
- wait for virtual machine to shutdown (300 sec).
- if virtual machine is already running then do hard power off
- shutdown esx host

for doing so, create the /usr/local/bin/esxshutdown.sh with the following content :

#!/bin/sh
#
# VM shutdown script for VMware Server ESX 3x for use with APC PowerChute agents
#

# set the paths that the vmware tools need
PATH=”/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin”
LOGFILE=/tmp/esxshutdown.log
ADMINMAIL=yourmail@yourdomain.ext
RELAYHOST=your.smtp.relay.host

# Set the Internal Word Separator used for word splitting
OLDIFS=$IFS
IFS=$’\n’

# How long to wait for VMs to shut down, the value is in seconds.
VM_SHUTDOWN_DELAY=120

# Delete the old log file
rm -rf $LOGFILE >/dev/null 2>&1;

# Write in the log file the start date
printf “#######################################\n” >$LOGFILE;
printf “%s `DATE` VM and ESX host shutdown Script started \n” >>$LOGFILE;
printf “%s `DATE` All detailed operations will be logged to $LOGFILE \n” >>$LOGFILE;
printf “#######################################\n” >>$LOGFILE;

# Notify System Administrator for script startup
/usr/local/bin/smtp_send.pl -t $ADMINMAIL -s “VM and ESX host shutdown Script started” -f `uname -n` -m “`cat $LOGFILE`” -r $RELAYHOST

printf “#######################################\n” >>$LOGFILE;
printf “%s `DATE` Soft Shutdown for VM with state ON started \n” >>$LOGFILE;
printf “#######################################\n” >>$LOGFILE;

count_vm_on=0
for vm in `vmware-cmd -l` ; do
for VMstate in `vmware-cmd “$vm” getstate -q` ; do
if [ $VMstate = "on" ] ; then
vmware-cmd “$vm” stop trysoft > /dev/null
vmwarecmd_exitcode=$(expr $?)
if [ $vmwarecmd_exitcode -ne 0 ] ; then
vmware-cmd “$vm” stop hard
fi
count_vm_on=$(expr $count_vm_on + 1)
sleep 1
fi
done
done

printf “#######################################\n” >>$LOGFILE;
printf “%s `DATE` ” Nice Shutdown executed on $count_vm_on Virtual Machines \n” >>$LOGFILE;
printf “#######################################\n” >>$LOGFILE;

# Wait a few seconds to let VM shut down
sleep 30
for (( second=0; second<$VM_SHUTDOWN_DELAY; second=second+5 )); do
if [ $count_vm_on -eq 0 ] ; then
break
fi
count_vm_on=0
sleep 5
for vm in `vmware-cmd -l` ; do
for VMstate in `vmware-cmd “$vm” getstate -q` ; do
if [ $VMstate = "on" ] ; then
count_vm_on=$(expr $count_vm_on + 1)
fi
done
done
done

for vm in `vmware-cmd -l` ; do
for VMstate in `vmware-cmd “$vm” getstate -q` ; do
if [ $VMstate = "on" ] ; then
vmware-cmd “$vm” stop hard
sleep 1
fi
done
done

printf “#######################################\n” >>$LOGFILE;
printf “%s `DATE` ” Virtual Machines shutdown completed \n” >>$LOGFILE;
printf “#######################################\n” >>$LOGFILE;

printf “#######################################\n” >>$LOGFILE;
printf “%s `DATE` ” Put ESX host in maintenance mode to avoid VMotion \n” >>$LOGFILE;
printf “#######################################\n” >>$LOGFILE;

vimsh -n -e /hostsvc/maintenance_mode_enter

# Wait a few seconds to let esx complete maintenance mode procedure
sleep 30

printf “#######################################\n” >>$LOGFILE;
printf “%s `DATE` ” Shutdown ESX host \n” >>$LOGFILE;
printf “#######################################\n” >>$LOGFILE;

shutdown -h now
poweroff

printf “#######################################\n” >>$LOGFILE;
printf “%s `DATE` ” VM and ESX host shutdown Script completed \n” >>$LOGFILE;
printf “#######################################\n” >>$LOGFILE;

Assign to it executable permission and correct ownership so it can’t be executed from anyone than root with the following commands :

chmod 700 /usr/local/bin/esxshutdown.sh
chown root:root /usr/local/bin/esxshutdown.sh

Configure your agent to execute /usr/local/bin/esxshutdown.sh in case of persistent power loss.

For any needs you should find the script above here : esxshutdown.sh

And you should download the smtp_send.pl script from here : smtp_send.pl

Hope this help

Bye
Riccardo

Bookmark It

One Response to “How-To shutdown VM and ESX with APC Network Shutdown”