3 Step to keep track OS memory usage by process

Tutorial

So many people asking how to keep track on the OS memory usage by process, as you know a bit about Physical Memory, Virtual Memory (VSZ), Resident Set Size (RSS), Shared Memory, Paging Memory…etc, counting RSS to determine memory usage is very common way and easier to understand for dummies like me, although it’s not a precise figure while you’re ignoring shared memory, fs cache..etc.
But anyway this tutorial is starting point to help you monitor the process RSS usage and keep tracking the usage that allows to identify peak/min.

Alright, you need to create one database or table, and one script and one cron job.

Step 1 – Create a database for storing stat data

CREATE DATABASE sysstat DEFAULT CHARACTER SET latin1;
GRANT INSERT, UPDATE, DELETE, SELECT on sysstat.* to sysstat@'localhost' identified by 'passw0rd';
USE sysstat;
CREATE TABLE 'memory' (  'id' int(11) NOT NULL auto_increment,  'date' datetime NOT NULL,  'memory' int(20) NOT NULL,  'process' varchar(300) NOT NULL,  PRIMARY KEY  ('id')) ENGINE=MyISAM AUTO_INCREMENT=21794 DEFAULT CHARSET=latin1;

Step 2 – Script

Usage:

To show all memory usage by process

./memory_stat.sh usage
[root@vm1 sbin]# ./memory_stat.sh usage
2011-05-01 01:55:39 16576 /usr/lib/courier-imap/bin/couriertls
2011-05-01 01:55:39 35576 /usr/lib/courier-imap/bin/imapd
2011-05-01 01:55:39 38832 /usr/sbin/httpd.worker
2011-05-01 01:55:39 73100 MailScanner:
2011-05-01 01:55:39 125632 clamd
2011-05-01 01:55:39 208964 /usr/libexec/mysqld
2011-05-01 01:55:39 429720 /usr/bin/php-cgi

To record into database

./memory_stat.sh record

To retrieve the usage report

./memory_stat.sh report {process} {date - optional} 
[root@vm1 sbin]# ./memory_stat.sh report php "2011-05-01 01"
php usage report on vm1.dreamerworks.net
date	memory
2011-05-01 00:00:01	471588
2011-05-01 00:05:02	468772
2011-05-01 00:10:02	470788
2011-05-01 00:15:01	471392
2011-05-01 00:20:01	472140
2011-05-01 00:25:01	469016
2011-05-01 00:30:02	469016
2011-05-01 00:35:01	469016
2011-05-01 00:40:01	472376

To housekeep the database

./memory_stat.sh housekeep

memory_stat.sh

#!/bin/bash

process=`ps aux|awk '{print $11}'|sort|uniq|grep -v -e grep -e awk -e ps -e sort -e uniq`
date=`date +%Y-%m-%d\ %H:%M:%S`
database="sysstat"
dbuser="sysstat"
password="passw0rd"
housekeepday="30"

usage(){
for PS in $process; do

echo "$date `ps aux | grep -- $PS |grep -v grep | awk '{sum +=$6}; END {print sum}'` $PS"

done
}


record() {
usage > /tmp/usage.log
mysql -u$dbuser -p$password <<EOF
use $database;
`while read date time memory process;
do
echo "insert into memory (date,memory,process) values ('$date $time','$memory','$process');"
done < /tmp/usage.log
`
EOF

rm -f /tmp/usage.log

}

housekeep() {
mysql -u$dbuser -p$password <<EOF
use $database;
delete from memory where date < CURRENT_DATE - $housekeepday;
EOF
}


report() {
if [ x$1 = "x" ];then
        echo "Usage: $0 [process]"
        exit
fi
process=$1
datetime_arg="AND date like '$2%'"
echo "$process usage report on `hostname`"
mysql -u$dbuser -p$password <<EOF
use $database;
`
echo "select date, memory from memory where process like '%$1%' $datetime_arg order by date";
`
EOF

}

case $1 in
	usage)
	usage | sort -k 3n
	;;
        housekeep)
        housekeep
        ;;
        record)
        record
        ;;
	report)
	report $2 $3
	;;
        *)
        echo "Usage: $1 (usage|record|housekeep|report [process] [date])"
        ;;
esac

Step 3 – Cron Job

crontab -e
0,5,10,15,20,25,30,35,40,45,50,55 * * * *       /usr/local/sbin/memory_stat.sh record

That’s it

At least, you may tell a bit more about which process is huger one and then you’ll able to narrow down for investigation.

Ubuntu 10.10 Maverick with HP Elitebook 2540p – Updated

Background

After thousand time try and errors with linux distribution, opensuse, fedora, ubtuntu… etc, finally get it work perfectly on ubtuntu 10.10 Alpha 2 installation.

There is some changes since 10.04, /etc/X11/xorg.conf is no more present after installation, Xorg now communicate with dbus to get best configure for your hardware.

You may experience blank screen, screen flicking, screen can’t turn on after system suspend, Xorg hang for few mins randomly. So here is the workaround for me to install ubuntu on this laptop.

Installation

1. Grab the Maverick CD image, ( Alpha 2 for my case)
2. During CD boot, hit any button to interrupt autoboot
3. Press [F6] to append boot parameters
Continue reading “Ubuntu 10.10 Maverick with HP Elitebook 2540p – Updated”

Pacemaker non-clones resource restart when clones stop/start on other nodes

Scenario

Node alpha holds resource groups rg_vg01.

Node beta holds the corresponding resources as same as alpha

Both nodes hold instances of the CLVM DLM clone.

As beta is switched into standby mode, the clone
instance on beta correctly transition to the Stopped state.
However, the rg_vg01 resource groups are
restarted in-place on alpha needlessly.

Solution

It was caused by resource-stickness score INFINITY order was defined between clone and standalone resource.
eg:

order order-clvm-vg inf: clvm-dlm-clone rg_vg01

To solve that, you should change the score to 0 between standalone and clones.
like:

order order-clvm-vg 0: clvm-dlm-clone rg_vg01

A related bug fix.
http://hg.clusterlabs.org/pacemaker/stable-1.0/rev/f17e29306fa1

howto create GFS on DRBD network disk mirroring

Introduction

This howto will cover how to create a DRBD disk set with GFS concurrent disk access setup.

Why DRBD?

DRBD disk mirroring benefit as a shared SAN disk to archive higher service resilience with low cost hardware component. Alternately, it may be configured for backup purpose with it’s flexibility.

Prerequisite

1. RedHat Cluster Suit and GFS packages

cman-2.0.115-1.el5
gfs2-utils-0.1.62-1.el5
gfs-utils-0.1.20-1.el5
kmod-gfs-0.1.34-2.el5

2. drbd83 package

yum install drbd83

3. Create a partition on both node with using same size, we use [sda5] for this show case.
Continue reading “howto create GFS on DRBD network disk mirroring”

Openldap recovery howto

If you are experiencing LDAP errors or startup without error but no ldap port (389|636) is listening for service and you have tried restarting the LDAP server by running /etc/init.d/ldap restart, then you should try running the following recovery procedure:

Typical corruption

1. Stop the LDAP server:

/etc/init.d/ldap stop

2. Run the daemon manually with debug flag

/usr/sbin/slapd -u ldap -h ldap://127.0.0.1:389/ -d 256

If the database is corrupted, it may stop at database initiation.
Continue reading “Openldap recovery howto”

howto reinstall GRUB in rescue mode while using lvm / mdadm

This article is how to activate LVM and mdadm software raid in rescue and reinstall GRUB boot loader, since the boot CD will not do this job for you. By following the steps, you may able to do system recovery or disk operation.

RedHat 5.x

Activate the RAID

mkdir /etc/mdadm
mdadm --examine --scan > /etc/mdadm/mdadm.conf
mdadm -A --scan

Continue reading “howto reinstall GRUB in rescue mode while using lvm / mdadm”

Enlarge linux software mirror raid 1 howto

Scenario

A linux box running on a software raid 1 mirror harddisk (120GB), recently the server is running out of disk space. A new pair of harddisk with larger capacity (500GB) is going to replace the existing pair.

Current disk layout

100MB md0:/dev/sda1, /dev/sda2
111GB md1:/dev/sdb1, /dev/sdb2

/boot /dev/md0
VolGroup00 /dev/md1
– LogVol00 swap
– LogVol01 /

Synchronize the data to new harddisk

  1. Shutdown the box
  2. remove sdb from the box
  3. replace with the new 500GB disk

Continue reading “Enlarge linux software mirror raid 1 howto”

Add a cron job in a command line

As you familiar with UNIX cron with crontab -e when you going to add a new schedule cron job.. do you ever tried to add a cron job up to 100 machine?

Simply to use pipe-in with crontab -l to regenerate the crontab.

Here you are

(crontab -l; echo "0 0 * * * /path/to/myscript.sh") | crontab -

Easy?!