Mandalika's scratchpad: OOB Large Page Selection Changes in Solaris 10 8/07

If you notice slight performance degradation of the application(s) on Solaris 10 8/07 compared to Solaris 10 1/06 & Solaris 10 11/06, it could be due to the conservative large page selection by the operating system. In S10 8/07, the maximum page size for heap/stack/mmap has been reduced to 64K (default is up to 256M depending on the underlying hardware -- eg., 256M on Niagara (US-T1)/US-IV+/SPARC64-VI systems, 32M on US-IV systems and 4M on US-III+ systems) on all systems lacking page coloring. 4M is the default maximum page size on US-T2 (Niagara2) systems. You can check this with the help of pmap -sx <pid>. I do not know exactly the motivation for this change, but I'm suspecting that bugs like default_physio/as_pagelock scales poorely with threads doing IO have something to do with this change.

Large Page Settings

If the application's heap grows beyond 4M, consider using MPSS (Multiple Page Size Support) facility to request large pages selectively for the process heap and stack. Alternatively you can set /etc/system tunable parameters max_uheap_lpsize and mmu_ism_pagesize for the system wide large page usage.

MPSS facility
1. Set the preferred page size for the heap and stack with MPSSHEAP and MPSSSTACK environment variables
2. Preload mpss.so.1
3. Run the application
  
  eg., On a Sun Enterprise M4000 system:
```
MPSSHEAP=32M
MPSSSTACK=64K
LD_PRELOAD=$LD_PRELOAD:mpss.so.1
<application> <arg1> <arg2> .. <argn>
```
Alternatively you can use the environment variable MPSSCFGFILE to set the preferred page sizes for the heap and the stack. See Solaris 9 or later: More performance with Large Pages (MPSS) for an example.
/etc/system tunables
- On systems other than OPL/APL (i.e., systems with SPARC64-VI processors), set max_uheap_lpsize to the desired maximum page size. Find the supported page sizes by running pagesize -a. Solaris tries to use the supported page sizes {wherever it can} until the maximum page size set. For example, if /etc/system has set max_uheap_lpsize = 0x2000000, Solaris tries to use a maximum page size of 32M although the system could use 256M pages.
- On OPL/APL systems (i.e., systems with SPARC64-VI processors), set mmu_ism_pagesize to the desired maximum page size. For example, if /etc/system has set mmu_ism_pagesize=0x200000, Solaris tries to use a maximum page size of 4M although the system could use 32M and 256M pages.
- Note that max_uheap_lpsize need not be set, if mmu_ism_pagesize has been set in /etc/system. Solaris kernel will change max_uheap_lpsize and others to the value set in /etc/system for mmu_ism_pagesize parameter, to be in sync with ism pagesize.

If the large pages have to be disabled for some reason, you can do so with the following /etc/system tunable parameters:

Solaris 10 1/06 & Solaris 10 11/06:

set exec_lpg_disable=1
set use_brk_lpg=0
set use_stk_lpg=0
set use_zmap_lpg=0

Solaris 10 8/07:

set max_uheap_lpsize=0x2000
set max_ustack_lpsize=0x2000
set max_privmap_lpsize=0x2000
set size_t max_shm_lpsize=0x2000
set use_brk_lpg=0
set use_stk_lpg=0