Symptom:
On Solaris OS, none of the Object Managers start up with total MaxTasks > 9200 thereby limiting the maximum number of users to 9000 (assuming one task per user and some overhead tasks for other server components) per single instance of Siebel server

Resolution:
On Solaris, for any user process, the maximum data segment or heap will have a soft/hard limit of 2G, by default. Similarly, stack will have a soft limit of 8M and a hard limit of 2G i.e., 2G address space will be reserved for the stack itself.

Those limits on system resources inhibit the growth of heap beyond 2G and may force the process to die, if it couldn't get the requested resources

So, the solution to the above mentioned problem is to increase the limits for data segment and reducing the limits for stack . From Siebel's perspective: adding the following two lines in $SIEBEL_ROOT/bin/start_server script (at any place, but before invoking siebsvc process) will fix the issue and lets the customer to configure the maxtasks to more than 9200; thereby, eliminating the need to install another Siebel server to get around this problem

 ulimit -s 512
 ulimit -d 4194303

Those two lines limit the stacksize to 512K and datasize to 4G respectively


Please note that the above settings may allow the customer to set more number of maxtasks; but obviously not to a very large number, as the 32-bit process can grow upto 4G only. Also the total number of allowed MaxTasks may vary from server to server depending on the available virtual memory (physical memory + swap space)

---

Detailed explanation of the problem & the suggested solution

---

srvrmgr:sdcv480s002> list param maxtasks for comp FINSObjMgr_enu

PA_ALIAS PA_VALUE PA_DATATYPE  PA_SCOPE   PA_SUBSYSTEM       PA_SETLEVEL PA_DISP_SETLEVEL     PA  PA  PA  PA PA_NAME
-------- -------- -----------  ---------  ------------------ ----------- -------------------  --  --  --  -- -------------
MaxTasks 620      Integer      Subsystem  Process Management Comp level  Component level set  N   N   Y   N  Maximum Tasks

1 row returned.

srvrmgr:sdcv480s002> change param maxtasks=9500 for comp FINSObjMgr_enu
Command completed successfully.

After this change, siebel server process, siebsvc fails to start up with an ENOMEM error i.e., there is not enough room in the address space of the process to fulfill one of the mmap() requests by siebsvc.

Truss output of siebsvc:

...
27763:  open("/export/home/sunperf/18306/siebsrvr/admin/siebel.sdcv480s002.shm", O_RDWR|O_CREAT|O_EXCL, 0700) = 9
27763:  write(9, "\0\0\0\0\0\0\0\0\0\0\0\0".., 1119830016) = 1119830016
27763:  mmap(0x00000000, 1119830016, PROT_READ|PROT_WRITE, MAP_SHARED, 9, 0) Err#12 ENOMEM <= failed; not enough addr space
...

sdcv480s002:/tmp/%ls -lh "/export/home/sunperf/18306/siebsrvr/admin/siebel.sdcv480s002.shm"
-rwx------   1 sunperf  other       1.0G Feb  2 12:07 /export/home/sunperf/18306/siebsrvr/admin/siebel.sdcv480s002.shm*

prstat output before siebsvc dies:
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 27763 sunperf  1087M 1082M sleep   59    0   0:00:12 3.8% siebsvc/1

From the above outputs, it is clear that siebsvc couldn't grow more than 2G, though it is allowed to go upto 4G (siebsvc is a 32-bit process; so, it will have a 4G address space in the VM system when the process gets created)

So, it is worth looking at the output of "limit" (in Csh) or "ulimit -a" (in ksh), which shows the limitations on the system resources available to the current shell and its descendents

% limit
cputime         unlimited
filesize        unlimited
datasize        unlimited
stacksize       8192 kbytes
coredumpsize    unlimited
descriptors     256
memorysize      unlimited

Even though the soft limit for the datasize was reported as "unlimited", it defaults to 2G on Solaris. The following simple C program illustrates it with the help of sys/resource.h API

    #include <stdio.h>
    #include <stdlib.h>
    #include <stddef.h>
    #include <sys/time.h>
    #include <sys/resource.h>

    static void showlimit(int resource, char* str)
    {
        struct rlimit lim;

        if (getrlimit(resource, &lim) != 0)
        {
            (void)printf("Couldn't retrieve %s limit\n", str);
            return;
        }

        (void)printf("Current/maximum %s limit is: \t%lu (%luKB Or %luMB) / %lu (%luKB Or %luMB)\n", \
             str, lim.rlim_cur, (lim.rlim_cur/1024), (lim.rlim_cur/(1024 * 1024)), \
             lim.rlim_max, (lim.rlim_max/1024), (lim.rlim_max/(1024 * 1024)));
    }

    int main()
    {
        showlimit(RLIMIT_DATA,  "datasize");
        showlimit(RLIMIT_STACK, "stacksize");

        return 0;
    }

%./showlimits
Current/maximum datasize limit is:      2147483647 (2097151KB Or 2047MB) / 2147483647 (2097151KB Or 2047MB)
Current/maximum stacksize limit is:     8388608 (8192KB Or 8MB) / 2147483647 (2097151KB Or 2047MB)

Output of "showlimits" confirms the fact that the datasize has been limited to 2G under the current shell; so, obviously the process cannot grow beyond 2G, and may die if it needs more than 2G address space

So, to resolve the problem under discussion, the limit for the datasize has to be increased. As the process is allowed to take upto 4G virtual address space, it is better to set the limit to the max allowed size. Fortunately, this step doesn't need the intervention of a sys admin; and a normal user can set the limits for the shell, at his wish

"ulimit -d 4194303" in ksh or "limit -datasize 4194303" in Csh would set the limit for datasize to 4G. But $SIEBEL_ROOT/bin/start_server script is the ideal place to have this command, as it invokes siebel server process (siebsvc)

So, I made the change in "start_server" script and tried to start siebel servers again. Surprisingly, siebsvc failed to startup inspite the increase in datasize limit to 4G, with the same ENOMEM error message

...
28958:  open("/export/home/sunperf/18306/siebsrvr/admin/siebel.sdcv480s002.shm", O_RDWR|O_CREAT|O_EXCL, 0700) = 9
28958:  write(9, "\0\0\0\0\0\0\0\0\0\0\0\0".., 1119830016) = 1119830016
28958:  mmap(0x00000000, 1119830016, PROT_READ|PROT_WRITE, MAP_SHARED, 9, 0) Err#12 ENOMEM <= failed again
...

"plimit <pid of siebsvc>" showed 4G data limit for siebsvc ("plimit" command shows the resource limits of running processe(s)

% plimit 28958
28958:  siebsvc -s siebsrvr -a -g sdcv480s002 -e siebel -s sdcv480s002
   resource              current         maximum
  time(seconds)         unlimited       unlimited
  file(blocks)          unlimited       unlimited
  data(kbytes)          4194303         4194303
  stack(kbytes)         unlimited       unlimited
  coredump(blocks)      unlimited       unlimited
  nofiles(descriptors)  65536           65536
  vmemory(kbytes)       unlimited       unlimited

"pmap <pid of siebsvc>" reported nearly a little over 1G as the total VM address space in use for siebsvc

% pmap 28958
00010000      16K r-x--  /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00022000       8K rwx--  /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00024000 1094592K rwx--    [ heap ]
7E000000    4856K rw-s-  dev:118,0 ino:405547
7E880000     600K r-x--  /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
7E924000      16K rwx--  /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
7E930000       8K rwx--    [ anon ]
...
...
7F9D0000       8K rwx--    [ anon ]
7F9E0000       8K r-x--  /usr/lib/libnls.so.1
7F9F2000       8K rwx--  /usr/lib/libnls.so.1
7FA00000    1600K r-x--  /usr/lib/libCstd.so.1
7FB9E000      32K rwx--  /usr/lib/libCstd.so.1
7FBA6000       8K rwx--  /usr/lib/libCstd.so.1
7FBB0000       8K r-x--  /usr/lib/libxnet.so.1
7FBC0000     160K r-x--  /usr/lib/ld.so.1
7FBF6000      16K rwx--  /usr/lib/ld.so.1
FFBF4000      48K rwx--    [ stack ]
 total   1112576K

However, interestingly "pmap -r " showed that 2G of virtual address space is reserved for stack of siebsvc; so, eventually memory allocation for heap was impacted by 2G reserved space for stack.

% pmap -r 28958
00010000      16K r-x--  /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00022000       8K rwx--  /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00024000 1094592K rwx--    [ heap ]
7E000000    4856K rw-s-  dev:118,0 ino:405547
7E880000     600K r-x--  /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
7E924000      16K rwx--  /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
7E930000       8K rwx--    [ anon ]
...
...
7F9D0000       8K rwx--    [ anon ]
7F9E0000       8K r-x--  /usr/lib/libnls.so.1
7F9F2000       8K rwx--  /usr/lib/libnls.so.1
7FA00000    1600K r-x--  /usr/lib/libCstd.so.1
7FB9E000      32K rwx--  /usr/lib/libCstd.so.1
7FBA6000       8K rwx--  /usr/lib/libCstd.so.1
7FBB0000       8K r-x--  /usr/lib/libxnet.so.1
7FBC0000     160K r-x--  /usr/lib/ld.so.1
7FBF6000      16K rwx--  /usr/lib/ld.so.1
7FC00000 2097152K rwx--    [ stack ]
 total   3209680K

To be more precise, 1G VM space is already in use by siebsvc & nearly 2G space was reserved for stack, which leaves only less than 1G for the heap of the process. So, when siebsvc tried to memory map the huge 1G+ shared memory (shm) file, the VM system couldn't allocate the required space and ultimately the process died

Again, from the output of "showlimits", it is clear that the stack could grow upto 2G in size (ie., the hard limit for stacksize). But even with very high load on the siebel servers, it is very rare to see more than 64K stack size being used by siebsvc. So, it is better to limit the stack size as well to any value < 1M

"ulimit -s 512" in ksh or "limit -stacksize 512" in Csh would set the limit for datasize to 4G. But again, $SIEBEL_ROOT/bin/start_server script is the right place to have this command, as it invokes siebel server process (siebsvc)

After making the necessary changes, the VM was system able to fulfill the 1G+ memory allocation for mmap() request of siebel server process (siebsvc). Some interesting outputs were shown below:

prstat output for siebsvc:
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 29255 sunperf  2155M 2148M sleep   59    0   0:00:31 2.7% siebsvc/3

Truss output of siebsvc:
...
29255:  open("/export/home/sunperf/18306/siebsrvr/admin/siebel.sdcv480s002.shm", O_RDWR|O_CREAT|O_EXCL, 0700) = 9
29255:  write(9, "\0\0\0\0\0\0\0\0\0\0\0\0".., 1119830016) = 1119830016
29255:  mmap(0x00000000, 1119830016, PROT_READ|PROT_WRITE, MAP_SHARED, 9, 0) = 0xF8000000 <= succeeds
...

% plimit 29255
29255:  siebsvc -s siebsrvr -a -g sdcv480s002 -e siebel -s sdcv480s002
   resource              current         maximum
  time(seconds)         unlimited       unlimited
  file(blocks)          unlimited       unlimited
  data(kbytes)          4194303         4194303
  stack(kbytes)         512             512
  coredump(blocks)      unlimited       unlimited
  nofiles(descriptors)  65536           65536
  vmemory(kbytes)       unlimited       unlimited

% pmap 29255
29255:  siebsvc -s siebsrvr -a -g sdcv480s002 -e siebel -s sdcv480s002
00010000      16K r-x--  /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00022000       8K rwx--  /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00024000 1094664K rwx--    [ heap ]
FE000000    4856K rw-s-  dev:118,0 ino:405506
FE800000     600K r-x--  /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
FE8A4000      16K rwx--  /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
FE8B0000       8K rwx--    [ anon ]
...
...
FF970000       8K rwx--    [ anon ]
FF980000    1600K r-x--  /usr/lib/libCstd.so.1
FFB1E000      32K rwx--  /usr/lib/libCstd.so.1
FFB26000       8K rwx--  /usr/lib/libCstd.so.1
FFB30000       8K r-x--  /usr/lib/libxnet.so.1
FFB40000     160K r-x--  /usr/lib/ld.so.1
FFB76000      16K rwx--  /usr/lib/ld.so.1
FFBFA000      24K rwx--    [ stack ]
 total   1112624K

% pmap -r 29255
29255:  siebsvc -s siebsrvr -a -g sdcv480s002 -e siebel -s sdcv480s002
00010000      16K r-x--  /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00022000       8K rwx--  /export/home/sunperf/18306/siebsrvr/bin/siebsvc
00024000 1094664K rwx--    [ heap ]
FE000000    4856K rw-s-  dev:118,0 ino:405506
FE800000     600K r-x--  /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
FE8A4000      16K rwx--  /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.2
FE8B0000       8K rwx--    [ anon ]
...
...
FF970000       8K rwx--    [ anon ]
FF980000    1600K r-x--  /usr/lib/libCstd.so.1
FFB1E000      32K rwx--  /usr/lib/libCstd.so.1
FFB26000       8K rwx--  /usr/lib/libCstd.so.1
FFB30000       8K r-x--  /usr/lib/libxnet.so.1
FFB40000     160K r-x--  /usr/lib/ld.so.1
FFB76000      16K rwx--  /usr/lib/ld.so.1
FFB80000     512K rwx--    [ stack ]
 total   1113112K

srvrmgr:sdcv480s002> list param maxtasks for comp FINSObjMgr_enu
PA_ALIAS PA_VALUE PA_DATATYPE  PA_SCOPE   PA_SUBSYSTEM       PA_SETLEVEL PA_DISP_SETLEVEL     PA  PA  PA  PA PA_NAME     
-------- -------- -----------  ---------  ------------------ ----------- -------------------  --  --  --  -- -------------
MaxTasks 9500      Integer      Subsystem  Process Management Comp level  Component level set  N   N   Y   N  Maximum Tasks

1 row returned.

Posted by: Giri Mandalika.    # 3:01 PM Tweet