CPU Usage Raised to 100% Because
of dbresp.pl
Recently, I got a call from my friend claiming performance
degredation on one of the production databases. When connecting to
SQL*Plus or RMAN, I noted a delay, so I ran the TOP command and
checked the processes running on the system. When running ps – ef
command, I saw hundreds of perl executables running on the system at
that time:
1 oracle 15560 1 3
Jan11 ? 05:50:07 /opt/oracle/product/10.2/db_1/perl/bin/perl
/opt/oracle/product/10.2/db_1/sysman/admin/scripts/db/dbresp.pl
2 oracle
16309 1 3 Jan11 ? 05:44:53
/opt/oracle/product/10.2/db_1/perl/bin/perl
/opt/oracle/product/10.2/db_1/sysman/admin/scripts/db/dbresp.pl
3 .....
4
.....
As the dbresp.pl file locates under sysman folder, I concluded
that it has some relation with EM, so I checked the EM trace file:
01 tail -50 emagent.trc | more
02
03
2011-01-11 08:51:37 Thread-4096777120 ERROR fetchlets.oslinetok:
Metric execution timed out in 600 seconds
04 2011-01-11 08:51:37
Thread-4096777120 ERROR command: failed to kill process 24963
running perl: (errno=3: No such process)
05 2011-01-11 08:51:37
Thread-4096777120 ERROR engine: [oracle_database,prod_db,Response] :
nmeegd_GetMetricData failed : Metric execution timed out in 600
seconds
06 2011-01-11 09:06:37 Thread-4113513376 ERROR
fetchlets.oslinetok: Metric execution timed out in 600 seconds
07 2011-01-11 09:06:37 Thread-4113513376 ERROR command: failed to
kill process 25393 running perl: (errno=3: No such process)
08
2011-01-11 09:06:37 Thread-4113513376 ERROR engine:
[oracle_database,prod_db,Response] : nmeegd_GetMetricData failed :
Metric execution timed out in 600 seconds
09 2011-01-11 09:21:37
Thread-4096777120 ERROR fetchlets.oslinetok: Metric execution timed
out in 600 seconds
10 2011-01-11 09:21:37 Thread-4096777120
ERROR command: failed to kill process 26068 running perl: (errno=3:
No such process)
11 2011-01-11 09:21:37 Thread-4096777120 ERROR
engine: [oracle_database,prod_db,Response] : nmeegd_GetMetricData
failed : Metric execution timed out in 600 seconds
12 2011-01-11
09:36:37 Thread-4099926944 ERROR fetchlets.oslinetok: Metric
execution timed out in 600 seconds
Woah!! Interesting output. I then decided to check metalink
and found the following note:
Server Has 100% Of Cpu Because Of Dbresp.pl [ID
764140.1]
Unfortunately, the note advised me to refer to the following
metalink note as a solution: “ Ext/Mod Problem Performance Agent
High CPU Consumption Gen”. This note advises a change to the
alert.log file name to solve the issue. It wasn’t a real solution,
so I decided to take down the EM and kill all processes using the
following:
emctl stop dbconsole
I then called the following command and got the list of all
dbresp.pl processes and got the script which kills them all:
1 ps -ef | grep dbresp.pl | awk {'print "kill -9
" $2'} > kill.sh
2
3 more kill.sh
4 kill -9 23989
5 kill -9 24569
6 kill -9 25145
7 kill -9 25723
8 .....
9 .....
Next, I made it executable and run :
1 oracle@host:~> chmod 755 kill.sh
2
oracle@host:~> ./kill.sh
3 oracle@host:~>
4 oracle@host:~>
ps -ef | grep dbresp
5 oracle 32454 29520 0 10:48 pts/0
00:00:00 grep dbresp
After killing all unnecessary processes, CPU usage went down.
To deal with this bug, you can check the count of dbresp.pl
files, take down the EM, kill all processes and start it again
using any cron job.
Also, a savvy Linux guru noted:
You can use ps -ef | grep dbresp.pl | awk
{‘print “kill -9 ” $2′} |bash (or other shell) to do the same
job as kill.sh.
|
|
 |
|
Get the Complete
Oracle Tuning Details
The landmark book
"Oracle
Tuning: The Definitive Reference Second Edition" has been
updated with over 1,150 pages of expert performance tuning tips.
It's packed with scripts and tools to hypercharge Oracle 11g
performance and you can
buy it for 40% off directly from the publisher.
|
|
|
Download your Oracle scripts now:
www.oracle-script.com
The
definitive Oracle Script collection for every Oracle professional DBA
|
|
| |
| |
Copyright ©
1996 -2011 by Burleson Enterprises. All rights reserved.
Oracle® is the registered trademark of Oracle
Corporation. SQL Server® is the registered trademark of Microsoft
Corporation.
Many of the designations used by computer vendors to distinguish their
products are claimed as Trademarks
|
|
