Mandalika's scratchpad | [ Work blog @Oracle | My Music Compositions ] |
Old Posts: 09.04 10.04 11.04 12.04 01.05 02.05 03.05 04.05 05.05 06.05 07.05 08.05 09.05 10.05 11.05 12.05 01.06 02.06 03.06 04.06 05.06 06.06 07.06 08.06 09.06 10.06 11.06 12.06 01.07 02.07 03.07 04.07 05.07 06.07 08.07 09.07 10.07 11.07 12.07 01.08 02.08 03.08 04.08 05.08 06.08 07.08 08.08 09.08 10.08 11.08 12.08 01.09 02.09 03.09 04.09 05.09 06.09 07.09 08.09 09.09 10.09 11.09 12.09 01.10 02.10 03.10 04.10 05.10 06.10 07.10 08.10 09.10 10.10 11.10 12.10 01.11 02.11 03.11 04.11 05.11 07.11 08.11 09.11 10.11 11.11 12.11 01.12 02.12 03.12 04.12 05.12 06.12 07.12 08.12 09.12 10.12 11.12 12.12 01.13 02.13 03.13 04.13 05.13 06.13 07.13 08.13 09.13 10.13 11.13 12.13 01.14 02.14 03.14 04.14 05.14 06.14 07.14 09.14 10.14 11.14 12.14 01.15 02.15 03.15 04.15 06.15 09.15 12.15 01.16 03.16 04.16 05.16 06.16 07.16 08.16 09.16 12.16 01.17 02.17 03.17 04.17 06.17 07.17 08.17 09.17 10.17 12.17 01.18 02.18 03.18 04.18 05.18 06.18 07.18 08.18 09.18 11.18 12.18 01.19 02.19 05.19 06.19 08.19 10.19 11.19 05.20 10.20 11.20 12.20 09.21 11.21 12.22
Some Context
Oracle database was hosted on ZFS Storage Appliance (NAS). The database files are accessible from the database server node via NFS mounted filesystems. Solaris 10 is the operating system on DB node.
Someone forgets to shutdown the database instance and unmount the remote filesystems before rebooting the database server node. After the system boots up, Oracle RDBMS fails to bring up the database due to locked-out data files.
eg.,
SQL> startup ORACLE instance started. Total System Global Area 1.7108E+10 bytes Fixed Size 2165208 bytes Variable Size 9965671976 bytes Database Buffers 6845104128 bytes Redo Buffers 295329792 bytes Database mounted. ORA-01157: cannot identify/lock data file 1 - see DBWR trace file ORA-01110: data file 1: '/orclvol4/entDB/system01.dbf' ====================== Extract from alert log: ====================== ... ALTER DATABASE OPEN Fri Aug 05 21:30:54 2011 Errors in file /oracle112/diag/rdbms/entdb/entDB/trace/entDB_dbw0_7235.trc: ORA-01157: cannot identify/lock data file 1 - see DBWR trace file ORA-01110: data file 1: '/orclvol4/entDB/system01.dbf' ORA-27086: unable to lock file - already in use SVR4 Error: 11: Resource temporarily unavailable Additional information: 8 Additional information: 21364 Errors in file /oracle112/diag/rdbms/entdb/entDB/trace/entDB_dbw0_7235.trc: ORA-01157: cannot identify/lock data file 2 - see DBWR trace file ORA-01110: data file 2: '/orclvol4/entDB/sysaux01.dbf' ORA-27086: unable to lock file - already in use SVR4 Error: 11: Resource temporarily unavailable Additional information: 8 Additional information: 21364 ...
Reason for the lock failure:
Because of the sudden ungraceful shutdown of the database, file locks on data files were not released by the NFS server (ZFS SA in this case). NFS server held on to the file locks even after the NFS client (DB server node in this example) was restarted. Due to this, Oracle RDBMS is not able to lock those data files residing on NFS server (ZFS SA). As a result, database instance was failed to start up in exclusive mode.
Workaround
Manually clear the NFS locks as outlined below.
On NFS Client (database server node):
Execute: clear_locks -s <nfs_server_host>
eg.,
# clear_locks -s sup16 Clearing locks held for NFS client ipsedb1 on server sup16 clear of locks held for ipsedb1 on sup16 returned success
On NFS Server (ZFS SA):
(this step may not be necessary but wouldn't hurt to perform)
clear_locks <nfs_client_host>
eg.,
sup16# clear_locks 10.129.207.93 Clearing locks held for NFS client 10.129.207.93 on server sup16 clear of locks held for 10.129.207.93 on sup16 returned success
Again back on NFS Client (database server node):
# svcadm -v disable nfs/client # svcadm -v enable nfs/client
Also see:
Listing file locks on Solaris 10
2004-2019 |