Mandalika's scratchpad [ Work blog @Oracle | Stock Market Notes | My Music Compositions ]

Old Posts: 09.04  10.04  11.04  12.04  01.05  02.05  03.05  04.05  05.05  06.05  07.05  08.05  09.05  10.05  11.05  12.05  01.06  02.06  03.06  04.06  05.06  06.06  07.06  08.06  09.06  10.06  11.06  12.06  01.07  02.07  03.07  04.07  05.07  06.07  08.07  09.07  10.07  11.07  12.07  01.08  02.08  03.08  04.08  05.08  06.08  07.08  08.08  09.08  10.08  11.08  12.08  01.09  02.09  03.09  04.09  05.09  06.09  07.09  08.09  09.09  10.09  11.09  12.09  01.10  02.10  03.10  04.10  05.10  06.10  07.10  08.10  09.10  10.10  11.10  12.10  01.11  02.11  03.11  04.11  05.11  07.11  08.11  09.11  10.11  11.11  12.11  01.12  02.12  03.12  04.12  05.12  06.12  07.12  08.12  09.12  10.12  11.12  12.12  01.13  02.13  03.13  04.13  05.13  06.13  07.13  08.13  09.13  10.13  11.13  12.13  01.14  02.14  03.14  04.14  05.14  06.14 


Monday, September 04, 2006
 
Solaris 10/Oracle: Fixing ORA-27102: out of memory Error

Symptom:

As part of a database tuning effort you increase the SGA/PGA sizes; and Oracle greets with an ORA-27102: out of memory error message. The system had enough free memory to serve the needs of Oracle.
SQL> startup
ORA-27102: out of memory
SVR4 Error: 22: Invalid argument

Diagnosis
$ oerr ORA 27102
27102, 00000, "out of memory"
// *Cause: Out of memory
// *Action: Consult the trace file for details

Not so helpful. Let's look the alert log for some clues.
% tail -2 alert.log
WARNING: EINVAL creating segment of size 0x000000028a006000
fix shm parameters in /etc/system or equivalent

Oracle is trying to create a 10G shared memory segment (depends on SGA/PGA sizes), but operating system (Solaris in this example) responded with an invalid argument (EINVAL) error message. There is a little hint about setting shm parameters in /etc/system.

Prior to Solaris 10, shmsys:shminfo_shmmax parameter has to be set in /etc/system with maximum memory segment value that can be created. 8M is the default value on Solaris 9 and prior versions; where as 1/4th of the physical memory is the default on Solaris 10 and later. On a Solaris 10 (or later) system, it can be verified as shown below:
% prtconf | grep Mem
Memory size: 32760 Megabytes

% id -p
uid=59008(oracle) gid=10001(dba) projid=3(default)

% prctl -n project.max-shm-memory -i project 3
project: 3: default
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
        privileged      7.84GB      -   deny                                 -
        system          16.0EB    max   deny                                 -

Now it is clear that the system is using the default value of 8G in this scenario, where as the application (Oracle) is trying to create a memory segment (10G) larger than 8G. Hence the failure.

So, the solution is to configure the system with a value large enough for the shared segment being created, so Oracle succeeds in starting up the database instance.

On Solaris 9 and prior releases, it can be done by adding the following line to /etc/system, followed by a reboot for the system to pick up the new value.

set shminfo_shmmax = 0x000000028a006000

However shminfo_shmmax parameter was obsoleted with the release of Solaris 10; and Sun doesn't recommend setting this parameter in /etc/system even though it works as expected.

On Solaris 10 and later, this value can be changed dynamically on a per project basis with the help of resource control facilities . This is how we do it on Solaris 10 and later:
% prctl -n project.max-shm-memory -r -v 10G -i project 3

% prctl -n project.max-shm-memory -i project 3
project: 3: default
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
        privileged      10.0GB      -   deny                                 -
        system          16.0EB    max   deny                                 -

Note that changes made with the prctl command on a running system are temporary, and will be lost when the system is rebooted. To make the changes permanent, create a project with projadd command and associate it with the user account as shown below:
% projadd -p 102  -c 'eBS benchmark' -U oracle -G dba  -K 'project.max-shm-memory=(privileged,10G,deny)' OASB
% usermod -K project=OASB oracle

Finally make sure the project is created with projects -l or cat /etc/project commands.
% projects -l
...
...
OASB
        projid : 102
        comment: "eBS benchmark"
        users  : oracle
        groups : dba
        attribs: project.max-shm-memory=(privileged,10737418240,deny)

% cat /etc/project
...
...
OASB:102:eBS benchmark:oracle:dba:project.max-shm-memory=(privileged,10737418240,deny)

With these changes, Oracle would start the database up normally.
SQL> startup
ORACLE instance started.

Total System Global Area 1.0905E+10 bytes
Fixed Size                  1316080 bytes
Variable Size            4429966096 bytes
Database Buffers         6442450944 bytes
Redo Buffers               31457280 bytes
Database mounted.
Database opened.

Related information:

  1. What's New in Solaris System Tuning in the Solaris 10 Release?
  2. Resource Controls (overview)
  3. System Setup Recommendations for Solaris 8 and Solaris 9
  4. Man page of prctl(1)
  5. Man page of projadd


Addendum : Oracle RAC settings

Anonymous Bob suggested the following settings for Oracle RAC in the form of a comment for the benefit of others who run into similar issue(s) when running Oracle RAC. I'm pasting the comment as is (Disclaimer: I have not verified these settings):

Thanks for a great explanation, I would like to add one comment that will help those with an Oracle RAC installation. Modifying the default project covers oracle processes great and is all that is needed for a single instance DB. In RAC however, the CRS process starts the DB and it is a root owned process and root does not use the default project. To fix ORA-27102 issue for RAC I added the following lines to an init script that runs before the init.crs script fires.

# Recommended Oracle RAC system params
ndd -set /dev/udp udp_xmit_hiwat 65536
ndd -set /dev/udp udp_recv_hiwat 65536

# For root processes like crsd
prctl -n project.max-shm-memory -r -v 8G -i project system
prctl -n project.max-shm-ids -r -v 512 -i project system

# For oracle processes like sqlplus
prctl -n project.max-shm-memory -r -v 8G -i project default
prctl -n project.max-shm-ids -r -v 512 -i project default

So simple yet it took me a week working with Oracle and SUN to come up with that answer...Hope that helps someone out.

Bob
# posted by Blogger Bob : 6:48 AM, April 25, 2008

_____________
Technorati tags:
| | |


Comments:
Thank you for your clear and helpful explanation and instructions!!
 
Best explinatino of this Ive seen so far. Thanks for the effort.
 
Thank you very much, if only everyones instructions on the internet were as clear as yours. The world would be a better place :P
 
Rocking!!!
 
Excellent! Thank you very much.
 
Thanks for a great explanation, I would like to add one comment that will help those with an Oracle RAC installation. Modifying the default project covers oracle processes great and is all that is needed for a single instance DB. In RAC however, the CRS process starts the DB and it is a root owned process and root does not use the default project. To fix ORA-27102 issue for RAC I added the following lines to an init script that runs before the init.crs script fires.

# Recommended Oracle RAC system params
ndd -set /dev/udp udp_xmit_hiwat 65536
ndd -set /dev/udp udp_recv_hiwat 65536
# For root processes like crsd
prctl -n project.max-shm-memory -r -v 8G -i project system
prctl -n project.max-shm-ids -r -v 512 -i project system
# For oracle processes like sqlplus
prctl -n project.max-shm-memory -r -v 8G -i project default
prctl -n project.max-shm-ids -r -v 512 -i project default

So simple yet it took me a week working with Oracle and SUN to come up with that answer...Hope that helps someone out.

Bob
 
Wonderful solution.. it worked for me.. great.. Ananth - UK
 
helpful blog!
 
Handy!
 
Thanks ... Great and simple explanation ..please keep up the good work. Requesting more info from you.

Al
 
Many Thanks to Giri and Bob for their clear instructions.

Abdulbari - KSA
 
Thank you for this helpful explanation.
May be you need to set default project to user oracle.
This command can do that:
usermod -K project=OABS oracle

otherwise, oracle still use project 3 (default)
 
Thank you very much.
 
Thank you!! You just saved my job! It's amazing that you wrote this two years ago, and it's become very popular of late.
 
I followed the instructions but still receive the same error message even on a startup nomount. Any ideas?
 
Awesome blog, thank you very much
 
Thank you for the excellent info.
All the best,
Ashraf N.
 
Thank you so much, its worked for us.
 
Thank you. Well documented.
 
Thank you good info, really helpfull
 
Hey thnaks a lot for detailed explaination:)

This worked for us...

Gurudutt
 
Cool blog as for me. I'd like to read something more concerning this theme. Thank you for giving that material.
Sexy Lady
Girls for companionship in London
 
"Hi guys, I really enjoyed this information about "Solaris 10/Oracle: Fixing ORA-27102: out of memory Error" is very interesting, I would like get more updates about this"
 
I would never find a better place to read as good comments as this site, never seen before, easy to find easy to understand, and it have serious comments not sick jokes as others, thanks and please keep like this. I would be pretty pleased if you as I do go to my links and maybe post a comment about what you think of mine. Great investment opportunity at Costa Rica
 
Hello I want to congratulate to them by its site of the Web of the excellent looks like entertained and very good very to me it elaborated. I invite them to that they explore a little on my Web site. Lots in Costa Rica Costa Rica Cheap Land for Sale
 
I think this blog is pretty cool,it has a lot of good and interesting content
 
Thanks for sharing .Keep posting articles like this.A good example of content presentation.A piece of information from you every now and then is really great.Great article post.I appreciate your writing skills.
 
Thank you very much.!! Good Work...
 
Hello .. firstly I would like to send greetings to all readers. After this, I recognize the content so interesting about this article. For me personally I liked all the information. I would like to know of cases like this more often. In my personal experience I might mention a book called Generic Viagra in this book that I mentioned have very interesting topics, and also you have much to do with the main theme of this article.
 
I have decided that this stuff is the most important tool that I have to masterize in order to reach a successful life. The fact of matter is that I have always loved every single thing which is somehow related to computers and stuff.
By the way, I have found pretty interesting stuff in this column. In short, I realized this is certainly a great piece of work.
 
Thanks a lot. It was really helpful.
 
Hey there,
Really nice job, There are many people searching about that now they will find enough sources by your tips.
Also looking forward for more tips about that
 
Time is cash as the previous saying goes, and by reading this publish, I realized that I saved myself a number of valuable time, which would have been otherwise spent on reading low consistency information throughout the almighty web. Thank you for the straight to the purpose, invaluable input!
 
Nice to be visiting your blog again, it has been months for me. Well this article that I've been waited for so long. I need this article to complete my assignment in the college, and it has same topic with your article. Thanks, great share.
 
Nice work, I would like to read your blog every day Thanks
 
Great tips, I would like to join your blog anyway
 
Really great post, Thank you for sharing This knowledge.
 
Really great post, Thank you for sharing This knowledge.Thank you very much for sharing this knowledge.this graph really conveyed the part which i was looking for.
 
Mandalika, you are the best! The solution provided still works even after more than 6 years. I was on a critical time path to upgrade oracle and your solution has saved me bunch of time. Credits to you!
 
thanks a lot for the very useful post...really save my day! :)
 
Thanks lot for the clear info given to solve that issue...
 
It really helped me . Thanks a lot.
 
Post a Comment

Links to this post:

Create a Link



<< Home


2004-2014 

This page is powered by Blogger. Isn't yours?