Mandalika's scratchpad | [ Work blog @Oracle | My Music Compositions ] |
Old Posts: 09.04 10.04 11.04 12.04 01.05 02.05 03.05 04.05 05.05 06.05 07.05 08.05 09.05 10.05 11.05 12.05 01.06 02.06 03.06 04.06 05.06 06.06 07.06 08.06 09.06 10.06 11.06 12.06 01.07 02.07 03.07 04.07 05.07 06.07 08.07 09.07 10.07 11.07 12.07 01.08 02.08 03.08 04.08 05.08 06.08 07.08 08.08 09.08 10.08 11.08 12.08 01.09 02.09 03.09 04.09 05.09 06.09 07.09 08.09 09.09 10.09 11.09 12.09 01.10 02.10 03.10 04.10 05.10 06.10 07.10 08.10 09.10 10.10 11.10 12.10 01.11 02.11 03.11 04.11 05.11 07.11 08.11 09.11 10.11 11.11 12.11 01.12 02.12 03.12 04.12 05.12 06.12 07.12 08.12 09.12 10.12 11.12 12.12 01.13 02.13 03.13 04.13 05.13 06.13 07.13 08.13 09.13 10.13 11.13 12.13 01.14 02.14 03.14 04.14 05.14 06.14 07.14 09.14 10.14 11.14 12.14 01.15 02.15 03.15 04.15 06.15 09.15 12.15 01.16 03.16 04.16 05.16 06.16 07.16 08.16 09.16 12.16 01.17 02.17 03.17 04.17 06.17 07.17 08.17 09.17 10.17 12.17 01.18 02.18 03.18 04.18 05.18 06.18 07.18 08.18 09.18 11.18 12.18 01.19 02.19 05.19 06.19 08.19 10.19 11.19 05.20 10.20 11.20 12.20 09.21 11.21 12.22
Some Context
Oracle database was hosted on ZFS Storage Appliance (NAS). The database files are accessible from the database server node via NFS mounted filesystems. Solaris 10 is the operating system on DB node.
Someone forgets to shutdown the database instance and unmount the remote filesystems before rebooting the database server node. After the system boots up, Oracle RDBMS fails to bring up the database due to locked-out data files.
eg.,
SQL> startup ORACLE instance started. Total System Global Area 1.7108E+10 bytes Fixed Size 2165208 bytes Variable Size 9965671976 bytes Database Buffers 6845104128 bytes Redo Buffers 295329792 bytes Database mounted. ORA-01157: cannot identify/lock data file 1 - see DBWR trace file ORA-01110: data file 1: '/orclvol4/entDB/system01.dbf' ====================== Extract from alert log: ====================== ... ALTER DATABASE OPEN Fri Aug 05 21:30:54 2011 Errors in file /oracle112/diag/rdbms/entdb/entDB/trace/entDB_dbw0_7235.trc: ORA-01157: cannot identify/lock data file 1 - see DBWR trace file ORA-01110: data file 1: '/orclvol4/entDB/system01.dbf' ORA-27086: unable to lock file - already in use SVR4 Error: 11: Resource temporarily unavailable Additional information: 8 Additional information: 21364 Errors in file /oracle112/diag/rdbms/entdb/entDB/trace/entDB_dbw0_7235.trc: ORA-01157: cannot identify/lock data file 2 - see DBWR trace file ORA-01110: data file 2: '/orclvol4/entDB/sysaux01.dbf' ORA-27086: unable to lock file - already in use SVR4 Error: 11: Resource temporarily unavailable Additional information: 8 Additional information: 21364 ...
Reason for the lock failure:
Because of the sudden ungraceful shutdown of the database, file locks on data files were not released by the NFS server (ZFS SA in this case). NFS server held on to the file locks even after the NFS client (DB server node in this example) was restarted. Due to this, Oracle RDBMS is not able to lock those data files residing on NFS server (ZFS SA). As a result, database instance was failed to start up in exclusive mode.
Workaround
Manually clear the NFS locks as outlined below.
On NFS Client (database server node):
Execute: clear_locks -s <nfs_server_host>
eg.,
# clear_locks -s sup16 Clearing locks held for NFS client ipsedb1 on server sup16 clear of locks held for ipsedb1 on sup16 returned success
On NFS Server (ZFS SA):
(this step may not be necessary but wouldn't hurt to perform)
clear_locks <nfs_client_host>
eg.,
sup16# clear_locks 10.129.207.93 Clearing locks held for NFS client 10.129.207.93 on server sup16 clear of locks held for 10.129.207.93 on sup16 returned success
Again back on NFS Client (database server node):
# svcadm -v disable nfs/client # svcadm -v enable nfs/client
Also see:
Listing file locks on Solaris 10
Siebel server architecture supports spawning multiple application object manager processes. The Siebel Connection Broker, SCBroker, tries to balance the load (incoming requests) across different object manager processes running in a single Siebel server.
Least Loaded or Round Robin?
By default, SCBroker forwards the incoming request to any object manager process that is least loaded - meaning the process with the least number of running tasks. In Siebel terminology, this behavior is referred as "least-loaded" or "LL" connection forwarding algorithm. While the default LL algorithm provides the optimal behavior in the best case scenarios, it may lead to serious availability problems if one of several object manager prcesses running in a Siebel server stops responding in a timely fashion [for some reason]. Such an object manager may still accept requests though it may timeout. At some point, the unresponsive/hung or erroneous object manager will have the least number of tasks that may prompt SCBroker component to forward new incoming requests to that object manager process - which in turn leads to a stalemate. To avoid such situations, it is recommended to configure "round-robin" or "RR" algorithm in SCBroker component. When round-robin algorithm is configured, SCBroker ignores the number of running tasks per object manager process and routes all requests to all object managers in a round robin fashion.
While both algorithms have their strengths and weaknesses, customers must weigh both options and choose the one that fits best in their deployment.
eg.,
Find the current load balancing algorithm:
srvrmgr> list advanced param ConnForwardAlgorithm for comp SCBroker show PA_ALIAS, PA_VALUE, PA_NAME PA_ALIAS PA_VALUE PA_NAME -------------------- -------- ----------------------------------------- ConnForwardAlgorithm LL Connection Forward algorithm for SCBroker
Configure SCBroker to use round-robin algorithm:
srvrmgr> change param ConnForwardAlgorithm=RR for comp SCBroker server SERVER_NAME Command completed successfully. srvrmgr> list advanced param ConnForwardAlgorithm for comp SCBroker show PA_ALIAS, PA_VALUE, PA_NAME PA_ALIAS PA_VALUE PA_NAME -------------------- -------- ----------------------------------------- ConnForwardAlgorithm RR Connection Forward algorithm for SCBroker
Other SCBroker parameters of interest: ConnForwardTimeout
and ConnRequestTimeout
Labels: oracle siebel CRM SCBroker load+balancing
2004-2019 |