3 Step to keep track OS memory usage by process

Tutorial

So many people asking how to keep track on the OS memory usage by process, as you know a bit about Physical Memory, Virtual Memory (VSZ), Resident Set Size (RSS), Shared Memory, Paging Memory…etc, counting RSS to determine memory usage is very common way and easier to understand for dummies like me, although it’s not a precise figure while you’re ignoring shared memory, fs cache..etc.
But anyway this tutorial is starting point to help you monitor the process RSS usage and keep tracking the usage that allows to identify peak/min.

Alright, you need to create one database or table, and one script and one cron job.

Step 1 – Create a database for storing stat data

CREATE DATABASE sysstat DEFAULT CHARACTER SET latin1;
GRANT INSERT, UPDATE, DELETE, SELECT on sysstat.* to sysstat@'localhost' identified by 'passw0rd';
USE sysstat;
CREATE TABLE 'memory' (  'id' int(11) NOT NULL auto_increment,  'date' datetime NOT NULL,  'memory' int(20) NOT NULL,  'process' varchar(300) NOT NULL,  PRIMARY KEY  ('id')) ENGINE=MyISAM AUTO_INCREMENT=21794 DEFAULT CHARSET=latin1;

Step 2 – Script

Usage:

To show all memory usage by process

./memory_stat.sh usage
[root@vm1 sbin]# ./memory_stat.sh usage
2011-05-01 01:55:39 16576 /usr/lib/courier-imap/bin/couriertls
2011-05-01 01:55:39 35576 /usr/lib/courier-imap/bin/imapd
2011-05-01 01:55:39 38832 /usr/sbin/httpd.worker
2011-05-01 01:55:39 73100 MailScanner:
2011-05-01 01:55:39 125632 clamd
2011-05-01 01:55:39 208964 /usr/libexec/mysqld
2011-05-01 01:55:39 429720 /usr/bin/php-cgi

To record into database

./memory_stat.sh record

To retrieve the usage report

./memory_stat.sh report {process} {date - optional} 
[root@vm1 sbin]# ./memory_stat.sh report php "2011-05-01 01"
php usage report on vm1.dreamerworks.net
date	memory
2011-05-01 00:00:01	471588
2011-05-01 00:05:02	468772
2011-05-01 00:10:02	470788
2011-05-01 00:15:01	471392
2011-05-01 00:20:01	472140
2011-05-01 00:25:01	469016
2011-05-01 00:30:02	469016
2011-05-01 00:35:01	469016
2011-05-01 00:40:01	472376

To housekeep the database

./memory_stat.sh housekeep

memory_stat.sh

#!/bin/bash

process=`ps aux|awk '{print $11}'|sort|uniq|grep -v -e grep -e awk -e ps -e sort -e uniq`
date=`date +%Y-%m-%d\ %H:%M:%S`
database="sysstat"
dbuser="sysstat"
password="passw0rd"
housekeepday="30"

usage(){
for PS in $process; do

echo "$date `ps aux | grep -- $PS |grep -v grep | awk '{sum +=$6}; END {print sum}'` $PS"

done
}


record() {
usage > /tmp/usage.log
mysql -u$dbuser -p$password <<EOF
use $database;
`while read date time memory process;
do
echo "insert into memory (date,memory,process) values ('$date $time','$memory','$process');"
done < /tmp/usage.log
`
EOF

rm -f /tmp/usage.log

}

housekeep() {
mysql -u$dbuser -p$password <<EOF
use $database;
delete from memory where date < CURRENT_DATE - $housekeepday;
EOF
}


report() {
if [ x$1 = "x" ];then
        echo "Usage: $0 [process]"
        exit
fi
process=$1
datetime_arg="AND date like '$2%'"
echo "$process usage report on `hostname`"
mysql -u$dbuser -p$password <<EOF
use $database;
`
echo "select date, memory from memory where process like '%$1%' $datetime_arg order by date";
`
EOF

}

case $1 in
	usage)
	usage | sort -k 3n
	;;
        housekeep)
        housekeep
        ;;
        record)
        record
        ;;
	report)
	report $2 $3
	;;
        *)
        echo "Usage: $1 (usage|record|housekeep|report [process] [date])"
        ;;
esac

Step 3 – Cron Job

crontab -e
0,5,10,15,20,25,30,35,40,45,50,55 * * * *       /usr/local/sbin/memory_stat.sh record

That’s it

At least, you may tell a bit more about which process is huger one and then you’ll able to narrow down for investigation.

Convert XML to CSV using xml2

What you need

  • Red Hat Linux 4, 5
  • gcc
  • libxml2
  • libxml2-devel
  • xml2

Compiling xml2

1. Extract xml2 tar ball

 tar zxvf ./xml2-0.4.tar.gz

2. Modify compiling option in “configure” file

search line between 3428-3470, replace “libxml” with “libxml-2.0”

for example:

From


    { (echo "$as_me:$LINENO: \$PKG_CONFIG --exists --print-errors \"libxml\"") >&5
  ($PKG_CONFIG --exists --print-errors "libxml") 2>&5
  ac_status=$?
  echo "$as_me:$LINENO: \$? = $ac_status" >&5
  (exit $ac_status); }; then
  pkg_cv_XML_CFLAGS=`$PKG_CONFIG --cflags "libxml" 2>/dev/null`

To

    { (echo "$as_me:$LINENO: \$PKG_CONFIG --exists --print-errors \"libxml-2.0\"") >&5
  ($PKG_CONFIG --exists --print-errors "libxml-2.0") 2>&5
  ac_status=$?
  echo "$as_me:$LINENO: \$? = $ac_status" >&5
  (exit $ac_status); }; then
  pkg_cv_XML_CFLAGS=`$PKG_CONFIG --cflags "libxml-2.0" 2>/dev/null`

3. Run configure

 ./configure 

4. Modify the “Makefile”

Change the “XML_CFALGS” var to
Continue reading “Convert XML to CSV using xml2”

Pacemaker non-clones resource restart when clones stop/start on other nodes

Scenario

Node alpha holds resource groups rg_vg01.

Node beta holds the corresponding resources as same as alpha

Both nodes hold instances of the CLVM DLM clone.

As beta is switched into standby mode, the clone
instance on beta correctly transition to the Stopped state.
However, the rg_vg01 resource groups are
restarted in-place on alpha needlessly.

Solution

It was caused by resource-stickness score INFINITY order was defined between clone and standalone resource.
eg:

order order-clvm-vg inf: clvm-dlm-clone rg_vg01

To solve that, you should change the score to 0 between standalone and clones.
like:

order order-clvm-vg 0: clvm-dlm-clone rg_vg01

A related bug fix.
http://hg.clusterlabs.org/pacemaker/stable-1.0/rev/f17e29306fa1

Openldap recovery howto

If you are experiencing LDAP errors or startup without error but no ldap port (389|636) is listening for service and you have tried restarting the LDAP server by running /etc/init.d/ldap restart, then you should try running the following recovery procedure:

Typical corruption

1. Stop the LDAP server:

/etc/init.d/ldap stop

2. Run the daemon manually with debug flag

/usr/sbin/slapd -u ldap -h ldap://127.0.0.1:389/ -d 256

If the database is corrupted, it may stop at database initiation.
Continue reading “Openldap recovery howto”

Apache Web Load Balancing howto

Setup Overview

A apache load balancer www.mydomain.com setup for redirect incoming request to two underlying web server www1.mydomain.com, www2.mydomain.com.

Hosts

Host#1 www1.mydomain.com: 192.168.1.1
Host#2 www2.mydomain.com: 192.168.1.2
The apache load balancer instance www.mydomain.com running on RedHat Cluster floating IP 192.168.1.3 or you can activate it on either one server using IP alias without using Cluster Suit.

Load balancer setup www.mydomain.com

Parameters Value Description
ProxyPass balancer://<NAME>/ Define your cluster name
stickysession BALANCEID | PHPSESSIONID | JSESSIONID Define your preferred session sticky method,
BALANCEID likely by source IP

PHP|JSESSIONID require to add extra session header on the node member web server

nofailover On | Off If your web servers do not support session replication, turn this flag on in order to NOT failovering the current session to other member while the node is failed.
route node name
lbmethod byrequests | bytraffic | bybusyness
ProxyPreserveHost On | Off While application using mod_rewrite for friendly URL and it rely on the REQUEST_URI , set this On to preserve the URL request header to www.mydomain.com

Continue reading “Apache Web Load Balancing howto”

howto reinstall GRUB in rescue mode while using lvm / mdadm

This article is how to activate LVM and mdadm software raid in rescue and reinstall GRUB boot loader, since the boot CD will not do this job for you. By following the steps, you may able to do system recovery or disk operation.

RedHat 5.x

Activate the RAID

mkdir /etc/mdadm
mdadm --examine --scan > /etc/mdadm/mdadm.conf
mdadm -A --scan

Continue reading “howto reinstall GRUB in rescue mode while using lvm / mdadm”

Enlarge linux software mirror raid 1 howto

Scenario

A linux box running on a software raid 1 mirror harddisk (120GB), recently the server is running out of disk space. A new pair of harddisk with larger capacity (500GB) is going to replace the existing pair.

Current disk layout

100MB md0:/dev/sda1, /dev/sda2
111GB md1:/dev/sdb1, /dev/sdb2

/boot /dev/md0
VolGroup00 /dev/md1
– LogVol00 swap
– LogVol01 /

Synchronize the data to new harddisk

  1. Shutdown the box
  2. remove sdb from the box
  3. replace with the new 500GB disk

Continue reading “Enlarge linux software mirror raid 1 howto”