Tutorial
So many people asking how to keep track on the OS memory usage by process, as you know a bit about Physical Memory, Virtual Memory (VSZ), Resident Set Size (RSS), Shared Memory, Paging Memory…etc, counting RSS to determine memory usage is very common way and easier to understand for dummies like me, although it’s not a precise figure while you’re ignoring shared memory, fs cache..etc.
But anyway this tutorial is starting point to help you monitor the process RSS usage and keep tracking the usage that allows to identify peak/min.
Alright, you need to create one database or table, and one script and one cron job.
Step 1 – Create a database for storing stat data
CREATE DATABASE sysstat DEFAULT CHARACTER SET latin1;
GRANT INSERT, UPDATE, DELETE, SELECT on sysstat.* to sysstat@'localhost' identified by 'passw0rd';
USE sysstat;
CREATE TABLE 'memory' ( 'id' int(11) NOT NULL auto_increment, 'date' datetime NOT NULL, 'memory' int(20) NOT NULL, 'process' varchar(300) NOT NULL, PRIMARY KEY ('id')) ENGINE=MyISAM AUTO_INCREMENT=21794 DEFAULT CHARSET=latin1;
Step 2 – Script
Usage:
To show all memory usage by process
./memory_stat.sh usage
[root@vm1 sbin]# ./memory_stat.sh usage
2011-05-01 01:55:39 16576 /usr/lib/courier-imap/bin/couriertls
2011-05-01 01:55:39 35576 /usr/lib/courier-imap/bin/imapd
2011-05-01 01:55:39 38832 /usr/sbin/httpd.worker
2011-05-01 01:55:39 73100 MailScanner:
2011-05-01 01:55:39 125632 clamd
2011-05-01 01:55:39 208964 /usr/libexec/mysqld
2011-05-01 01:55:39 429720 /usr/bin/php-cgi
To record into database
./memory_stat.sh record
To retrieve the usage report
./memory_stat.sh report {process} {date - optional}
[root@vm1 sbin]# ./memory_stat.sh report php "2011-05-01 01"
php usage report on vm1.dreamerworks.net
date memory
2011-05-01 00:00:01 471588
2011-05-01 00:05:02 468772
2011-05-01 00:10:02 470788
2011-05-01 00:15:01 471392
2011-05-01 00:20:01 472140
2011-05-01 00:25:01 469016
2011-05-01 00:30:02 469016
2011-05-01 00:35:01 469016
2011-05-01 00:40:01 472376
To housekeep the database
./memory_stat.sh housekeep
memory_stat.sh
#!/bin/bash
process=`ps aux|awk '{print $11}'|sort|uniq|grep -v -e grep -e awk -e ps -e sort -e uniq`
date=`date +%Y-%m-%d\ %H:%M:%S`
database="sysstat"
dbuser="sysstat"
password="passw0rd"
housekeepday="30"
usage(){
for PS in $process; do
echo "$date `ps aux | grep -- $PS |grep -v grep | awk '{sum +=$6}; END {print sum}'` $PS"
done
}
record() {
usage > /tmp/usage.log
mysql -u$dbuser -p$password <<EOF
use $database;
`while read date time memory process;
do
echo "insert into memory (date,memory,process) values ('$date $time','$memory','$process');"
done < /tmp/usage.log
`
EOF
rm -f /tmp/usage.log
}
housekeep() {
mysql -u$dbuser -p$password <<EOF
use $database;
delete from memory where date < CURRENT_DATE - $housekeepday;
EOF
}
report() {
if [ x$1 = "x" ];then
echo "Usage: $0 [process]"
exit
fi
process=$1
datetime_arg="AND date like '$2%'"
echo "$process usage report on `hostname`"
mysql -u$dbuser -p$password <<EOF
use $database;
`
echo "select date, memory from memory where process like '%$1%' $datetime_arg order by date";
`
EOF
}
case $1 in
usage)
usage | sort -k 3n
;;
housekeep)
housekeep
;;
record)
record
;;
report)
report $2 $3
;;
*)
echo "Usage: $1 (usage|record|housekeep|report [process] [date])"
;;
esac
Step 3 – Cron Job
crontab -e
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/local/sbin/memory_stat.sh record
That’s it
At least, you may tell a bit more about which process is huger one and then you’ll able to narrow down for investigation.
