If you have an Oracle E-Business Suite 11.5.10 database running on Oracle 10gR2 (10.2.0.2) / Solaris x86-64 platform, make sure you have the following two Oracle patches to avoid concurrency issues and intermittent Oracle shadow process crashes.
Oracle patches
1) 4770693 BUG: Intel Solaris: Unnecessary latch sleeps by default
2) 5666714 BUG: ORA-7445 ON DELETE
Symptoms
1) If the top 5 database timed events look something similar to the following in AWR, it is very likely that the database is running into the bug 4770693 Intel Solaris: Unnecessary latch sleeps by default.
Top 5 Timed Events
| Event | Waits | Time(s) | Avg Wait(ms) | % Total Call Time | Wait Class |
|---|---|---|---|---|---|
| latch: cache buffers chains | 94,301 | 169,403 | 1,796 | 86.1 | Concurrency |
| CPU time | 5,478 | 2.8 | |||
| wait list latch free | 247,466 | 4,756 | 19 | 2.4 | Other |
| buffer busy waits | 14,928 | 1,382 | 93 | .7 | Concurrency |
| db file sequential read | 98,750 | 552 | 6 | .3 | User I/O |
Apply Oracle server patch 4770693 to get rid of the concurrency issue(s). Note that the fix will be part of 10.2.0.3.
2) If the application becomes unstable and if you notice core dumps in
cdump directory, have a look at the corresponding stack traces generated in udump directory. If the call stack looks similar to the following stack, apply Oracle server patch 5666714 to overcome this problem.Alert log will have the following errors:
Errors in file /opt/oracle/admin/VIS/udump/vis_ora_1040.trc:
ORA-07445: exception encountered: core dump [SIGSEGV] [Address not mapped to object] [2168] [] [] []
Fri Jun 15 01:30:38 2007
Errors in file /opt/oracle/admin/VIS/udump/vis_ora_1040.trc:
ORA-07445: exception encountered: core dump [SIGSEGV] [Address not mapped to object] [9] [] [] []
ORA-07445: exception encountered: core dump [SIGSEGV] [Address not mapped to object] [2168] [] [] []
Fri Jun 15 01:30:38 2007
Errors in file /opt/oracle/admin/VIS/udump/vis_ora_1040.trc:
ORA-07445: exception encountered: core dump [SIGSEGV] [Address not mapped to object] [9] [] [] []
ORA-07445: exception encountered: core dump [SIGSEGV] [Address not mapped to object] [9] [] [] []
ORA-07445: exception encountered: core dump [SIGSEGV] [Address not mapped to object] [2168] [] [] []
% more vis_ora_1040.trc
...
...
*** 2007-06-15 01:30:38.403
*** SERVICE NAME:(VIS) 2007-06-15 01:30:38.402
*** SESSION ID:(1111.4) 2007-06-15 01:30:38.402
Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x878
*** 2007-06-15 01:30:38.403
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [SIGSEGV] [Address not mapped to object] [2168] [] [] []
Current SQL statement for this session:
INSERT /*+ IDX(0) */ INTO "INV"."MLOG$_MTL_SUPPLY" (dmltype$$,old_new$$,snaptime$$,change_vector$$,m_row$$) VALUES (:d,:o,to_
date('4000-01-01:00:00:00','YYYY-MM-DD:HH24:MI:SS'),:c,:m)
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst()+23 ? 0000000000000001 00177A9EC 000000000 0061D0A60
000000000
ksedmp()+636 ? 0000000000000001 001779481 000000000 00000000B
000000000
ssexhd()+729 ? 0000000000000001 000E753CE 000000000 0061D0B90
000000000
_sigsetjmp()+25 ? 0000000000000001 0FDDCB7E6 0FFFFFD7F 0061D0B50
000000000
call_user_handler() ? 0000000000000001 0FDDC0BA2 0FFFFFD7F 0061D0EF0
+589 000000000
sigacthandler()+163 ? 0000000000000001 0FDDC0D88 0FFFFFD7F 000000002
000000000
kglsim_pin_simhp()+ ? 0000000000000001 0FFFFFFFF 0FFFFFFFF 00000000B
173 000000000
kxsGetRuntimeLock() ? 0000000000000001 001EBF830 000000000 005E5D868
+683 000000000
kksfbc()+7361 ? 0000000000000001 001FB60A6 000000000 005E5D868
000000000
opiexe()+1691 ? 0000000000000001 0029045D0 000000000 0FFDF9250
0FFFFFD7F
opiall0()+1316 ? 0000000000000001 0028E9FB9 000000000 000000001
000000000
opikpr()+536 ? 0000000000000001 00290B2DD 000000000 0000000B7
000000000
opiodr()+1087 ? 0000000000000001 000E7BE1C 000000000 000000001
000000000
rpidrus()+217 ? 0000000000000001 000E8058E 000000000 0FFDFA6B8
0FFFFFD7F
skgmstack()+163 ? 0000000000000001 003F611D0 000000000 005E5D868
000000000
rpidru()+129 ? 0000000000000001 000E808A6 000000000 005E6FAD0
000000000
rpiswu2()+431 ? 0000000000000001 000E7FD8C 000000000 0FFDFB278
0FFFFFD7F
kprball()+1189 ? 0000000000000001 000E86E6A 000000000 0FFDFB278
0FFFFFD7F
kntxslt()+3150 ? 0000000000000001 0030601F3 000000000 005F7C538
000000000
kntxit()+998 ? 0000000000000001 003058EBB 000000000 005F7C538
000000000
0000000001E4866E ? 0000000000000001 001E4864B 000000000 000000000
000000000
delrow()+9170 ? 0000000000000001 0032020B7 000000000 000000002
000000000
qerdlFetch()+640 ? 0000000000000001 0033545F5 000000000 0EF38B020
000000003
delexe()+909 ? 0000000000000001 0032034EA 000000000 005E6FC50
000000000
opiexe()+9267 ? 0000000000000001 002906368 000000000 000000001
000000000
opiodr()+1087 ? 0000000000000001 000E7BE1C 000000000 0FFDFCD10
0FFFFFD7F
ttcpip()+1168 ? 0000000000000001 003D031AD 000000000 0FFDFEDF4
0FFFFFD7F
opitsk()+1212 ? 0000000000000001 000E77C41 000000000 000E7BA00
000000000
opiino()+931 ? 0000000000000001 000E7B0D8 000000000 005E5B8F0
000000000
opiodr()+1087 ? 0000000000000001 000E7BE1C 000000000 000000000
000000000
opidrv()+748 ? 0000000000000001 000E76A11 000000000 0FFDFF6D8
0FFFFFD7F
sou2o()+86 ? 0000000000000001 000E73E6B 000000000 000000000
000000000
opimai_real()+127 ? 0000000000000001 000E3A7C4 000000000 000000000
000000000
main()+95 ? 0000000000000001 000E3A694 000000000 000000000
000000000
0000000000E3A4D7 ? 0000000000000001 000E3A4DC 000000000 000000000
000000000
--------------------- Binary Stack Dump ---------------------
========== FRAME [1] (ksedst()+23 -> 0000000000000001) ==========
Dump of memory from 0x00000000061D0910 to 0x00000000061D0920
0061D0910 061D0920 00000000 0177A9EC 00000000 [ .........w.....]
========== FRAME [2] (ksedmp()+636 -> 0000000000000001) ==========
Dump of memory from 0x00000000061D0920 to 0x00000000061D0A60
0061D0920 061D0A60 00000000 01779481 00000000 [`.........w.....]
0061D0930 0000000B 00000000 061D0EF0 00000000 [................]
0061D0940 05E5B96C 00000000 05E5C930 00000000 [l.......0.......]
0061D0950 05E5C930 00000000 FE0D2000 FFFFFD7F [0........ ......]
0061D0960 061D0A40 00000000 00000000 00000000 [@...............]
0061D0970 00000000 00000000 00000000 00000000 [................]
...
...
After installing the Oracle database patches, check the installed patches by running
opatch lsinventory on your database server.% opatch lsinventory_______________
Invoking OPatch 10.2.0.1.0
Oracle interim Patch Installer version 10.2.0.1.0
Copyright (c) 2005, Oracle Corporation. All rights reserved..
Oracle Home : /oracle/product/10.1.0
Central Inventory : /export/home/oracle/oraInventory
from : /oracle/product/10.1.0/oraInst.loc
OPatch version : 10.2.0.1.0
OUI version : 10.2.0.1.0
OUI location : /oracle/product/10.1.0/oui
Log file location : /oracle/product/10.1.0/cfgtoollogs/opatch/opatch-2007_Aug_10_21-56-03-PDT_Fri.log
Lsinventory Output file location : /oracle/product/10.1.0/cfgtoollogs/opatch/lsinv/lsinventory-2007_Aug_10_21-56-03-PDT_Fri.txt
--------------------------------------------------------------------------------
Installed Top-level Products (3):
Oracle Database 10g 10.2.0.1.0
Oracle Database 10g Products 10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 1 10.2.0.2.0
There are 3 products installed in this Oracle Home.
Interim patches (2) :
Patch 4770693 : applied on Thu Aug 02 15:27:23 PDT 2007
Created on 12 Jul 2006, 11:52:39 hrs US/Pacific
Bugs fixed:
4770693
Patch 5666714 : applied on Fri Jul 20 10:21:33 PDT 2007
Created on 29 Nov 2006, 04:52:58 hrs US/Pacific
Bugs fixed:
5666714
--------------------------------------------------------------------------------
OPatch succeeded.
Technorati Tags:
Oracle | 
Database | 
E-Business Suite | 
Oracle Applications | 
Solaris
No comments:
Post a Comment