3 Step to keep track OS memory usage by process
Tutorial
So many people asking how to keep track on the OS memory usage by process, as you know a bit about Physical Memory, Virtual Memory (VSZ), Resident Set Size (RSS), Shared Memory, Paging Memory…etc, counting RSS to determine memory usage is very common way and easier to understand for dummies like me, although it’s not a precise figure while you’re ignoring shared memory, fs cache..etc.
But anyway this tutorial is starting point to help you monitor the process RSS usage and keep tracking the usage that allows to identify peak/min.
Alright, you need to create one database or table, and one script and one cron job.
Step 1 – Create a database for storing stat data
CREATE DATABASE sysstat DEFAULT CHARACTER SET latin1;
GRANT INSERT, UPDATE, DELETE, SELECT on sysstat.* to sysstat@'localhost' identified by 'passw0rd';
USE sysstat;
CREATE TABLE 'memory' ( 'id' int(11) NOT NULL auto_increment, 'date' datetime NOT NULL, 'memory' int(20) NOT NULL, 'process' varchar(300) NOT NULL, PRIMARY KEY ('id')) ENGINE=MyISAM AUTO_INCREMENT=21794 DEFAULT CHARSET=latin1;
Step 2 – Script
Usage:
To show all memory usage by process
./memory_stat.sh usage
[root@vm1 sbin]# ./memory_stat.sh usage 2011-05-01 01:55:39 16576 /usr/lib/courier-imap/bin/couriertls 2011-05-01 01:55:39 35576 /usr/lib/courier-imap/bin/imapd 2011-05-01 01:55:39 38832 /usr/sbin/httpd.worker 2011-05-01 01:55:39 73100 MailScanner: 2011-05-01 01:55:39 125632 clamd 2011-05-01 01:55:39 208964 /usr/libexec/mysqld 2011-05-01 01:55:39 429720 /usr/bin/php-cgi
To record into database
./memory_stat.sh record
To retrieve the usage report
./memory_stat.sh report {process} {date - optional}
[root@vm1 sbin]# ./memory_stat.sh report php "2011-05-01 01" php usage report on vm1.dreamerworks.net date memory 2011-05-01 00:00:01 471588 2011-05-01 00:05:02 468772 2011-05-01 00:10:02 470788 2011-05-01 00:15:01 471392 2011-05-01 00:20:01 472140 2011-05-01 00:25:01 469016 2011-05-01 00:30:02 469016 2011-05-01 00:35:01 469016 2011-05-01 00:40:01 472376
To housekeep the database
./memory_stat.sh housekeep
memory_stat.sh
#!/bin/bash
process=`ps aux|awk '{print $11}'|sort|uniq|grep -v -e grep -e awk -e ps -e sort -e uniq`
date=`date +%Y-%m-%d\ %H:%M:%S`
database="sysstat"
dbuser="sysstat"
password="passw0rd"
housekeepday="30"
usage(){
for PS in $process; do
echo "$date `ps aux | grep -- $PS |grep -v grep | awk '{sum +=$6}; END {print sum}'` $PS"
done
}
record() {
usage > /tmp/usage.log
mysql -u$dbuser -p$password <<EOF
use $database;
`while read date time memory process;
do
echo "insert into memory (date,memory,process) values ('$date $time','$memory','$process');"
done < /tmp/usage.log
`
EOF
rm -f /tmp/usage.log
}
housekeep() {
mysql -u$dbuser -p$password <<EOF
use $database;
delete from memory where date < CURRENT_DATE - $housekeepday;
EOF
}
report() {
if [ x$1 = "x" ];then
echo "Usage: $0 [process]"
exit
fi
process=$1
datetime_arg="AND date like '$2%'"
echo "$process usage report on `hostname`"
mysql -u$dbuser -p$password <<EOF
use $database;
`
echo "select date, memory from memory where process like '%$1%' $datetime_arg order by date";
`
EOF
}
case $1 in
usage)
usage | sort -k 3n
;;
housekeep)
housekeep
;;
record)
record
;;
report)
report $2 $3
;;
*)
echo "Usage: $1 (usage|record|housekeep|report [process] [date])"
;;
esac
Step 3 – Cron Job
crontab -e 0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/local/sbin/memory_stat.sh record
That’s it
At least, you may tell a bit more about which process is huger one and then you’ll able to narrow down for investigation.
ediulia and 9 others are discussing. 
Anthony_A, and 
rwheindl are discussing. 
levin 6:29 pm on September 15, 2010 Permalink |
Please note that, VirtualBox 3.2.8 is incompatible on 2.6.36-rc4-maverick kernel
Travis 7:28 am on September 18, 2010 Permalink |
I’m having some issues with flickering on my laptop monitor ( elitebook 2540p ) but not on external monitor. Have you seen / resolved this?
levin 12:41 am on September 19, 2010 Permalink |
Hi Travis, you need to update a beta kernel to fix the screen flickering problem, here to download the upstream kernel packages. http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.36-rc4-maverick/
levin 12:42 am on September 19, 2010 Permalink |
also update xorg-intel driver from “Synaptic Package” as well.
Travis 3:23 am on September 23, 2010 Permalink |
My Kernel: 2.6.36-020636rc5-generic #201009211328 SMP Tue Sep 21 13:31:25 UTC 2010 x86_64 GNU/Linux
I updated the intel driver as well. Seems to be better than initial install, but still seeing the flicker at times.
It’s odd, with Fedora 13 I get no flicker, but can’t dock it with external monitor, but with Ubuntu I can dock etc.. but get the flicker. Frustrating.
Thanks
-Travis
levin 5:51 pm on September 23, 2010 Permalink
That’s odd, but I’m not sure rc5 is good version or not, since i believe 36.rc4 already fixed the flicker issue.
I just tested with internal display only. So maybe not apply for your case, I’ll try that with a external monitor as I can.
Can you produce the following result, see if it is consistently flickering
For No Docking
1. Boot up with internal display
2. Boot up with external display
3. Bootup with internal display and switch over to external display when reached X login and switch it back to internal display
Repeat 2,3 with docking attached.
Thanks
lets compare with my working version.
libdrm-intel1 2.4.21-1ubuntu2
xorg 1:7.5+6ubuntu2
xserver-xorg-video-intel 2:2.12.0-1ubuntu4
linux-generic 2.6.36-020636rc4.201009130905
linux-headers-2.6.36-020636rc4.201009130905
linux-headers-2.6.36-020636rc4.201009130905-generic 2.6.36-020636rc4.201009130905
linux-image 2.6.36-020636rc4.201009130905
linux-image-2.6.36-020636rc4.201009130905-generic 2.6.36-020636rc4.201009130905
foo 5:46 am on October 29, 2010 Permalink
tried rc4 and rc7 with the same results as Travis. Switching from 60 to 40Hz update freq finally solved the flickering problem.
Travis 7:34 am on September 18, 2010 Permalink |
Great info by the way.. Thanks
Ares 10:46 am on October 4, 2010 Permalink |
You are the man! Honestly I’ve had been trying to make it work for my girlfriends laptop for a while now, and nothing had worked. Thanks!
Justin 12:45 am on October 21, 2010 Permalink |
I tried the rc4 kernel and the rc7 kernel (both i386) but my video is still flashing every few seconds. Do you know if there is a patch to apply or a way to track down why this is happening? I checked the package manager but it appears I already have intel-xorg-video-intel package installed.
Also, the multitouch works great for the track pad but how can I set those settings to automatically apply when the computer turns on? A startup script just seems like the wrong way to do it.
Thanks
levin 12:24 pm on October 30, 2010 Permalink |
put these line in /etc/X11/xorg.conf for auto-enable multitouch
Section “InputDevice”
Identifier “Synaptics Touchpad”
Driver “synaptics”
Option “SendCoreEvents” “true”
Option “Device” “/dev/psaux”
Option “Protocol” “auto-dev”
Option “SHMConfig” “on”
Option “VertTwoFingerScroll” “1″
Option “HorizTwoFingerScroll” “1″
Option “EmulateTwoFingerMinW” “5″
Option “EmulateTwoFingerMinZ” “50″
EndSection
Haggan 10:05 pm on October 29, 2010 Permalink |
Iam testing 40Hz now seams to be flicker free with rc8 of the 36 kernel. What about the smart card reader?
Any one know how to get it working? Also has any one working N speed with the wifi?
Travis 6:19 am on November 19, 2010 Permalink |
Latest Kernel seems to kill the display on the HP 2540. Docked, or not. The external display when docked works fine. Happened on both of the 2.6.37 releases. Anyone else?
Fredrik 3:29 pm on November 19, 2010 Permalink |
Travis, the same thing happens to me: I get a black (internal) screen with the 2.6.37rc’s.
Haggan 7:36 pm on December 3, 2010 Permalink |
40Hz dont work for me with 2.6.36 rc8 or the 2.6.35-23 kernel. I have tried 2.6.37 I also got blanck screen.
About the smart card HP suggest the following. I cant get it workin I guess it because I am running x86_64
Note: HP does not take any responsibility in doing the same. This would be at the customers own risk.
Step 1:
Run the command – lspcmci -v and get:
Product Name:
Identification: manf_id:
Check for: prod_id(1):
Step 2:
Goto – http://www.scmmicro.com/support/pc-security-support/downloads.html
Download the driver – SCR241 – Linux 32-bit (2.4.x) Driver
Extract the files, and find the file scr241_main.c
Search for the lines containing
PCMCIA_DEVICE_PROD_ID1(“SCR243 PCMCIA”,0x2054e8de),
PCMCIA_DEVICE_PROD_ID1(“SCR24x PCMCIA”,0x54a33665),
and add the line:
PCMCIA_DEVICE_PROD_ID1(“HP”, 0x53cb94f9),
Note: “HP”, 0x53cb94f9 is the prod_id(1) what I have got in my unit – so this has to be changed with what the customer gets on his unit when he runs the lspcmci -v command
Joao 8:08 pm on December 28, 2010 Permalink |
I tried everything. I updated the kernel, verified that I have the newest driver, changed the frequency to 40Hz, updated the bios, disabled the light sensor in the bios.
Nothing helped. Why can I not get rid of the problem?
levin 9:50 pm on December 28, 2010 Permalink |
The flickering issues seems broken in upsteam release, so far as i knew the lastest working kernel is http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.36-rc4-maverick/
Fredrik 5:57 pm on January 19, 2011 Permalink |
The flickering seems to be solved by the latest daily kernel:
http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/2011-01-18-natty/
However I still have some troubles with plugging/unplugging external displays.
levin 6:16 pm on January 19, 2011 Permalink |
actually, the external display problem seems never been fixed… btw thanks for your update! cheers
ediulia 8:19 pm on January 19, 2011 Permalink |
i understand that there is no option at this time to install ubuntu 10.10 or 10.04 on 2540p without to spend at list a week for fix problems after installation. i will wait for 11.04. may be next version will solve screen problems without create new bugs…
david 11:05 pm on February 4, 2011 Permalink |
This kinda works for me, The flickering disappeared..
I installed the kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/current/
But now the openafs kernel module wont compile..
levin 7:42 pm on April 29, 2011 Permalink |
Just a quick update, 11.04 work flawlessly with HP 2540p. Everything work as expected!