Scheduled monitoring of ESXi hosts
Posted by: hbr in vscsiStats, troubleshooting, script, esx on
Sep 25, 2012
In my post on sizing up the storage for block size usage, I wrote a script to get the data into a manageable form. This script runs fine from the command line but you probably want to have some sort of trend to analyze.
In that case, you want to go and schedule running the script. Now the ESX host has a scheduler called ‘cron’. In full versions you would simply add a script to /etc/cron.hourly and it would pick it up on the next run. In ESXi this is slightly different. To schedule a command you need to add it to the root crontab which resides in
/var/spool/cron/crontabs/root
So we want to add the next 2 lines to the crontab, one for esxtop and one for vScsiStats:
0 7-9,13,16 * * 1-5 esxtop -b -a -d 10 -n 360 | gzip > /vmfs/volumes/local-storage/esxtop-$(date +%F-%T).csv.gz
0 7-9,13,16 * * 1-5 /vmfs/volumes/local-storage/vstats.py 60 59 > /vmfs/volumes/local-storage/vstats-$(date +%F-%T).csv
This of course assumes you put the vstats script from http://virtuall.eu/blog/hbr/ita-s-sample-time-with-vscsistats into a file called vstats.py in the /vmfs/volumes/local-storage directory and made it executable.
To clarify, the numbers before the command mean:
- 0 = run at 0 minutes of the hour
- 7-9,13,16 = run at hours 7 to 9, 13 and 16 of the day (so 5 times a day)
- * = run at all days of the month
- * = run at all months of the year
- 1-5 = run at day 1 to 5 (monday – friday) of the week
Since both commands run for a full hour, this gives a nice trending dataset to analyze.
One side note though. The vstats script stops the vscsiStats collection after the last sample. So if the last sample overlaps the next run, it will stop the next run’s collection. Therefor we only collect 59 samples instead of 60.
Next thing to do is to save these settings. ESXi forgets changes over a reboot unless you perform some trickery to make it save your files. Since our scripts are safely stored on a local vmfs, all we need to do is add the cron lines to the crontab at every reboot. We do this from the /etc/rc.local file by adding the next lines:
cat << EOC >> /var/spool/cron/crontabs/root
0 7-9,13,16 * * 1-5 esxtop -b -a -d 10 -n 360 | gzip > /vmfs/volumes/local-storage/esxtop-\$(date +%F-%T).csv.gz
0 7-9,13,16 * * 1-5 /vmfs/volumes/local-storage/vstats.py 60 59 > /vmfs/volumes/local-storage/vstats-\$(date +%F-%T).csv
EOC
kill $(cat /var/run/crond.pid)
/bin/busybox crond
Now we force a backup to save the rc.local file to the ESXi state with ‘/sbin/auto-backup.sh’ and we are set. You can either reboot the host or run /etc/rc.local to get the schedule invoked.
