When high number of application server processes are configured in a single PeopleSoft domain or in multiple domains cumulative, it is very likely that the PeopleSoft server domain boot process may fail with errors like:
Booting server processes ...
exec PSSAMSRV -A -- -C psappsrv.cfg -D CS90SPV -S PSSAMSRV :
Failed.
113954.ben15!PSSAMSRV.29746.1.0: LIBTUX_CAT:681: ERROR: Failure to create message queue
113954.ben15!PSSAMSRV.29746.1.0: LIBTUX_CAT:248: ERROR: System init function failed, Uunixerr = :
msgget: No space left on device
113954.ben15!tmboot.29708.1.-2: CMDTUX_CAT:825: ERROR: Process PSSAMSRV at ben15 failed with /T
tperrno (TPEOS - operating system error)
In this particular example, the PeopleSoft Enterprise is running on a Solaris 10 system. Fortunately the error message is very clear in this case; and the failure is related to the message queues. During the domain boot up process, there is a call to
msgget()
to create a message queue. If the call to
msgget()
succeeds, it returns a non-negative integer that serves as the identifier for the newly created message queue. However in case of a failure, it returns -1 and sets the error number to EACCES, EEXIST, ENOENT or ENOSPC depending on the underlying reason.
From the above error messages it clear that the
msgget()
failed with the errno set to ENOSPC (No space left on device). Man page of
msgget(2)
has the following explanation for ENOSPC error code on Solaris:
ERRORS
The msgget() function will fail if:
...
...
ENOSPC A message queue identifier is to be created but
the system-imposed limit on the maximum number of
allowed message queue identifiers system wide
would be exceeded. See NOTES.
NOTES
...
...
The system-imposed limit on the number of message queue
identifiers is maintained on a per-project basis using the
project.max-msg-ids resource control.
Good stuff. It has enough clues to suspect the configured number for the message queue identifiers.
Prior to the release of Solaris 10, the /etc/system System V IPC tunable,
msgsys:msginfo_msgmni
, could be configured to control the maximum number of message queues that can be created. The default value on pre-Solaris 10 systems is 50.
With the release of Solaris 10, majority of the System V IPC tunables were obsoleted and equivalent
resource controls were created for the remaining tunables, to reduce the administrative overhead. On Solaris 10 and later versions, System V IPC can be tuned on a per project basis using the newly introduced resource controls.
On Solaris 10, The resource control,
project.max-msg-ids
, replaced the old /etc/system tunable,
msginfo_msgmni
. And the default value has been raised to 128.
Now back to the failure in PeopleSoft environment. Let's first check the current value configured for
project.max-msg-ids
.
- Get the project ID.
% id -p
uid=222227(psft) gid=2294(dba) projid=3(default)
- Examine the
project.max-msg-ids
resource control for the project with ID 3, using the prctl
utility.
% prctl -n project.max-msg-ids -i project 3
project: 3: default
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-msg-ids
privileged 128 - deny -
system 16.8M max deny -
Alternatively run the command
ipcs -q
to check the number of active message queues. Note that the project with id '3' is configured to create a maximum of 128 (default) message queues. In any case, the number of active message queues from the
ipcs -q
output may almost match with the configured value for the
project.max-msg-ids
.
Since it appears the configured PeopleSoft domain(s) needs more than 128 message queues in order to bring up all the application server processes that constitute the PeopleSoft Enterprise, the solution is to increase the value for the resource control,
project.max-msg-ids
, to any value beyond 128. For the sake of simplicity, let's increase it to 256 (2 * default value, that is). Again
prctl
utility can be used to set the new value for the resource control.
- Login as the 'root' user
% su
Password:
- Increase the maximum value for the message queue identifiers to 256 using the
prctl
utility.
# prctl -n project.max-msg-ids -r -v 256 -i project 3
- Verify the new maximum value for the message queue identifiers
# prctl -n project.max-msg-ids -i project 3
project: 3: default
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-msg-ids
privileged 256 - deny -
system 16.8M max deny -
That's all there is. With the above change, the PeopleSoft Enterprise should boot up at least with no
Failure to create message queue .. msgget: No space left on device
errors.
Before we conclude, note that the above mentioned solution is not persistent across multiple operating system reboots. To make it persistent, create a new project with
projadd
command. The man page for
projadd(1M) has an example showing the creation of a project.
_________________________
Technorati Tags:
Solaris |
OpenSolaris |
Oracle |
PeopleSoft |
Troubleshooting