Mandalika's scratchpad [ Work blog @Oracle | My Music Compositions ]

Old Posts: 09.04  10.04  11.04  12.04  01.05  02.05  03.05  04.05  05.05  06.05  07.05  08.05  09.05  10.05  11.05  12.05  01.06  02.06  03.06  04.06  05.06  06.06  07.06  08.06  09.06  10.06  11.06  12.06  01.07  02.07  03.07  04.07  05.07  06.07  08.07  09.07  10.07  11.07  12.07  01.08  02.08  03.08  04.08  05.08  06.08  07.08  08.08  09.08  10.08  11.08  12.08  01.09  02.09  03.09  04.09  05.09  06.09  07.09  08.09  09.09  10.09  11.09  12.09  01.10  02.10  03.10  04.10  05.10  06.10  07.10  08.10  09.10  10.10  11.10  12.10  01.11  02.11  03.11  04.11  05.11  07.11  08.11  09.11  10.11  11.11  12.11  01.12  02.12  03.12  04.12  05.12  06.12  07.12  08.12  09.12  10.12  11.12  12.12  01.13  02.13  03.13  04.13  05.13  06.13  07.13  08.13  09.13  10.13  11.13  12.13  01.14  02.14  03.14  04.14  05.14  06.14  07.14  09.14  10.14  11.14  12.14  01.15  02.15  03.15  04.15  06.15  09.15  12.15  01.16  03.16  04.16  05.16  06.16  07.16  08.16  09.16  12.16  01.17  02.17  03.17  04.17  06.17  07.17  08.17  09.17  10.17  12.17  01.18  02.18  03.18  04.18  05.18  06.18  07.18  08.18  09.18  11.18  12.18  01.19  02.19  05.19  06.19  08.19  10.19  11.19  05.20  10.20  11.20  12.20  09.21  11.21  12.22 


Wednesday, February 02, 2005
 
Siebel on Sun Solaris: Know the process resource limits

Symptom:
On Solaris OS, none of the Object Managers start up with total MaxTasks > 9200 thereby limiting the maximum number of users to 9000 (assuming one task per user and some overhead tasks for other server components) per single instance of Siebel server

Resolution:
On Solaris, for any user process, the maximum data segment or heap will have a soft/hard limit of 2G, by default. Similarly, stack will have a soft limit of 8M and a hard limit of 2G i.e., 2G address space will be reserved for the stack itself.

Those limits on system resources inhibit the growth of heap beyond 2G and may force the process to die, if it couldn't get the requested resources

So, the solution to the above mentioned problem is to increase the limits for data segment and reducing the limits for stack . From Siebel's perspective: adding the following two lines in $SIEBEL_ROOT/bin/start_server script (at any place, but before invoking siebsvc process) will fix the issue and lets the customer to configure the maxtasks to more than 9200; thereby, eliminating the need to install another Siebel server to get around this problem

ulimit -s 512
ulimit -d 4194303

Those two lines limit the stacksize to 512K and datasize to 4G respectively


Please note that the above settings may allow the customer to set more number of maxtasks; but obviously not to a very large number, as the 32-bit process can grow upto 4G only. Also the total number of allowed MaxTasks may vary from server to server depending on the available virtual memory (physical memory + swap space)


Detailed explanation of the problem & the suggested solution


srvrmgr:sdcv480s002> list param maxtasks for comp FINSObjMgr_enu

PA_ALIAS PA_VALUE PA_DATATYPE PA_SCOPE PA_SUBSYSTEM PA_SETLEVEL PA_DISP_SETLEVEL PA PA PA PA PA_NAME
-------- -------- ----------- --------- ------------------ ----------- ------------------- -- -- -- -- -------------
MaxTasks 620 Integer Subsystem Process Management Comp level Component level set N N Y N Maximum Tasks

1 row returned.

srvrmgr:sdcv480s002> change param maxtasks=9500 for comp FINSObjMgr_enu
Command completed successfully.

After this change, siebel server process, siebsvc fails to start up with an ENOMEM error i.e., there is not enough room in the address space of the process to fulfill one of the mmap() requests by siebsvc.

Truss output of siebsvc:
...
27763: open("/export/home/sunperf/18306/siebsrvr/admin/siebel.sdcv480s002.shm", O_RDWR|O_CREAT|O_EXCL, 0700) = 9
27763: write(9, "\0\0\0\0\0\0\0\0\0\0\0\0".., 1119830016) = 1119830016
27763: mmap(0x00000000, 1119830016, PROT_READ|PROT_WRITE, MAP_SHARED, 9, 0) Err#12 ENOMEM <= failed; not enough addr space
...

sdcv480s002:/tmp/%ls -lh "/export/home/sunperf/18306/siebsrvr/admin/siebel.sdcv480s002.shm"
-rwx------ 1 sunperf other 1.0G Feb 2 12:07 /export/home/sunperf/18306/siebsrvr/admin/siebel.sdcv480s002.shm*

prstat output before siebsvc dies:
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
27763 sunperf 1087M 1082M sleep 59 0 0:00:12 3.8% siebsvc/1


From the above outputs, it is clear that siebsvc couldn't grow more than 2G, though it is allowed to go upto 4G (siebsvc is a 32-bit process; so, it will have a 4G address space in the VM system when the process gets created)

So, it is worth looking at the output of "limit" (in Csh) or "ulimit -a" (in ksh), which shows the limitations on the system resources available to the current shell and its descendents

% limit
cputime unlimited
filesize unlimited
datasize unlimited
stacksize 8192 kbytes
coredumpsize unlimited
descriptors 256
memorysize unlimited

Even though the soft limit for the datasize was reported as "unlimited", it defaults to 2G on Solaris. The following simple C program illustrates it with the help of sys/resource.h API

#include <stdio.h>
#include <stdlib.h>
#include <stddef.h>
#include <sys/time.h>
#include <sys/resource.h>

static void showlimit(int resource, char* str)
{
struct rlimit lim;

if (getrlimit(resource, &lim) != 0)
{
(void)printf("Couldn't retrieve %s limit\n", str);
return;
}

(void)printf("Current/maximum %s limit is: \t%lu (%luKB Or %luMB) / %lu (%luKB Or %luMB)\n", \
str, lim.rlim_cur, (lim.rlim_cur/1024), (lim.rlim_cur/(1024 * 1024)), \
lim.rlim_max, (lim.rlim_max/1024), (lim.rlim_max/(1024 * 1024)));
}

int main()
{
showlimit(RLIMIT_DATA, "datasize");
showlimit(RLIMIT_STACK, "stacksize");

return 0;
}

%./showlimits
Current/maximum datasize limit is: 2147483647 (2097151KB Or 2047MB) / 2147483647 (2097151KB Or 2047MB)
Current/maximum stacksize limit is: 8388608 (8192KB Or 8MB) / 2147483647 (2097151KB Or 2047MB)

Output of "showlimits" confirms the fact that the datasize has been limited to 2G under the current shell; so, obviously the process cannot grow beyond 2G, and may die if it needs more than 2G address space

So, to resolve the problem under discussion, the limit for the datasize has to be increased. As the process is allowed to take upto 4G virtual address space, it is better to set the limit to the max allowed size. Fortunately, this step doesn't need the intervention of a sys admin; and a normal user can set the limits for the shell, at his wish

"ulimit -d 4194303" in ksh or "limit -datasize 4194303" in Csh would set the limit for datasize to 4G. But $SIEBEL_ROOT/bin/start_server script is the ideal place to have this command, as it invokes siebel server process (siebsvc)

So, I made the change in "start_server" script and tried to start siebel servers again. Surprisingly, siebsvc failed to startup inspite the increase in datasize limit to 4G, with the same ENOMEM error message

...
28958: open("/export/home/sunperf/18306/siebsrvr/admin/siebel.sdcv480s002.shm", O_RDWR|O_CREAT|O_EXCL, 0700) = 9
28958: write(9, "\0\0\0\0\0\0\0\0\0\0\0\0".., 1119830016) = 1119830016
28958: mmap(0x00000000, 1119830016, PROT_READ|PROT_WRITE, MAP_SHARED, 9, 0) Err#12 ENOMEM <= failed again
...

"plimit <pid of siebsvc>" showed 4G data limit for siebsvc ("plimit" command shows the resource limits of running processe(s)

% plimit 28958
28958: siebsvc -s siebsrvr -a -g sdcv480s002 -e siebel -s sdcv480s002
resource current maximum
time(seconds) unlimited unlimited
file(blocks) unlimited unlimited
data(kbytes) 4194303 4194303
stack(kbytes) unlimited unlimited
coredump(blocks) unlimited unlimited
nofiles(descriptors) 65536 65536
vmemory(kbytes) unlimited unlimited

"pmap <pid of siebsvc>" reported nearly a little over 1G as the total VM address space in use for siebsvc

% pmap 28958
00010000 16K r-x-- /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00022000 8K rwx-- /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00024000 1094592K rwx-- [ heap ]
7E000000 4856K rw-s- dev:118,0 ino:405547
7E880000 600K r-x-- /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
7E924000 16K rwx-- /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
7E930000 8K rwx-- [ anon ]
...
...
7F9D0000 8K rwx-- [ anon ]
7F9E0000 8K r-x-- /usr/lib/libnls.so.1
7F9F2000 8K rwx-- /usr/lib/libnls.so.1
7FA00000 1600K r-x-- /usr/lib/libCstd.so.1
7FB9E000 32K rwx-- /usr/lib/libCstd.so.1
7FBA6000 8K rwx-- /usr/lib/libCstd.so.1
7FBB0000 8K r-x-- /usr/lib/libxnet.so.1
7FBC0000 160K r-x-- /usr/lib/ld.so.1
7FBF6000 16K rwx-- /usr/lib/ld.so.1
FFBF4000 48K rwx-- [ stack ]
total 1112576K

However, interestingly "pmap -r " showed that 2G of virtual address space is reserved for stack of siebsvc; so, eventually memory allocation for heap was impacted by 2G reserved space for stack.

% pmap -r 28958
00010000 16K r-x-- /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00022000 8K rwx-- /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00024000 1094592K rwx-- [ heap ]
7E000000 4856K rw-s- dev:118,0 ino:405547
7E880000 600K r-x-- /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
7E924000 16K rwx-- /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
7E930000 8K rwx-- [ anon ]
...
...
7F9D0000 8K rwx-- [ anon ]
7F9E0000 8K r-x-- /usr/lib/libnls.so.1
7F9F2000 8K rwx-- /usr/lib/libnls.so.1
7FA00000 1600K r-x-- /usr/lib/libCstd.so.1
7FB9E000 32K rwx-- /usr/lib/libCstd.so.1
7FBA6000 8K rwx-- /usr/lib/libCstd.so.1
7FBB0000 8K r-x-- /usr/lib/libxnet.so.1
7FBC0000 160K r-x-- /usr/lib/ld.so.1
7FBF6000 16K rwx-- /usr/lib/ld.so.1
7FC00000 2097152K rwx-- [ stack ]
total 3209680K

To be more precise, 1G VM space is already in use by siebsvc & nearly 2G space was reserved for stack, which leaves only less than 1G for the heap of the process. So, when siebsvc tried to memory map the huge 1G+ shared memory (shm) file, the VM system couldn't allocate the required space and ultimately the process died

Again, from the output of "showlimits", it is clear that the stack could grow upto 2G in size (ie., the hard limit for stacksize). But even with very high load on the siebel servers, it is very rare to see more than 64K stack size being used by siebsvc. So, it is better to limit the stack size as well to any value < 1M

"ulimit -s 512" in ksh or "limit -stacksize 512" in Csh would set the limit for datasize to 4G. But again, $SIEBEL_ROOT/bin/start_server script is the right place to have this command, as it invokes siebel server process (siebsvc)

After making the necessary changes, the VM was system able to fulfill the 1G+ memory allocation for mmap() request of siebel server process (siebsvc). Some interesting outputs were shown below:

prstat output for siebsvc:
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
29255 sunperf 2155M 2148M sleep 59 0 0:00:31 2.7% siebsvc/3

Truss output of siebsvc:
...
29255: open("/export/home/sunperf/18306/siebsrvr/admin/siebel.sdcv480s002.shm", O_RDWR|O_CREAT|O_EXCL, 0700) = 9
29255: write(9, "\0\0\0\0\0\0\0\0\0\0\0\0".., 1119830016) = 1119830016
29255: mmap(0x00000000, 1119830016, PROT_READ|PROT_WRITE, MAP_SHARED, 9, 0) = 0xF8000000 <= succeeds
...

% plimit 29255
29255: siebsvc -s siebsrvr -a -g sdcv480s002 -e siebel -s sdcv480s002
resource current maximum
time(seconds) unlimited unlimited
file(blocks) unlimited unlimited
data(kbytes) 4194303 4194303
stack(kbytes) 512 512

coredump(blocks) unlimited unlimited
nofiles(descriptors) 65536 65536
vmemory(kbytes) unlimited unlimited

% pmap 29255
29255: siebsvc -s siebsrvr -a -g sdcv480s002 -e siebel -s sdcv480s002
00010000 16K r-x-- /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00022000 8K rwx-- /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00024000 1094664K rwx-- [ heap ]
FE000000 4856K rw-s- dev:118,0 ino:405506
FE800000 600K r-x-- /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
FE8A4000 16K rwx-- /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
FE8B0000 8K rwx-- [ anon ]
...
...
FF970000 8K rwx-- [ anon ]
FF980000 1600K r-x-- /usr/lib/libCstd.so.1
FFB1E000 32K rwx-- /usr/lib/libCstd.so.1
FFB26000 8K rwx-- /usr/lib/libCstd.so.1
FFB30000 8K r-x-- /usr/lib/libxnet.so.1
FFB40000 160K r-x-- /usr/lib/ld.so.1
FFB76000 16K rwx-- /usr/lib/ld.so.1
FFBFA000 24K rwx-- [ stack ]
total 1112624K

% pmap -r 29255
29255: siebsvc -s siebsrvr -a -g sdcv480s002 -e siebel -s sdcv480s002
00010000 16K r-x-- /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00022000 8K rwx-- /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00024000 1094664K rwx-- [ heap ]
FE000000 4856K rw-s- dev:118,0 ino:405506
FE800000 600K r-x-- /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
FE8A4000 16K rwx-- /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
FE8B0000 8K rwx-- [ anon ]
...
...
FF970000 8K rwx-- [ anon ]
FF980000 1600K r-x-- /usr/lib/libCstd.so.1
FFB1E000 32K rwx-- /usr/lib/libCstd.so.1
FFB26000 8K rwx-- /usr/lib/libCstd.so.1
FFB30000 8K r-x-- /usr/lib/libxnet.so.1
FFB40000 160K r-x-- /usr/lib/ld.so.1
FFB76000 16K rwx-- /usr/lib/ld.so.1
FFB80000 512K rwx-- [ stack ]
total 1113112K


srvrmgr:sdcv480s002> list param maxtasks for comp FINSObjMgr_enu
PA_ALIAS PA_VALUE PA_DATATYPE PA_SCOPE PA_SUBSYSTEM PA_SETLEVEL PA_DISP_SETLEVEL PA PA PA PA PA_NAME
-------- -------- ----------- --------- ------------------ ----------- ------------------- -- -- -- -- -------------
MaxTasks 9500 Integer Subsystem Process Management Comp level Component level set N N Y N Maximum Tasks

1 row returned.



Comments: Post a Comment



<< Home


2004-2019 

This page is powered by Blogger. Isn't yours?