First, the symptom.

On Solaris x86 systems that support large pages (any page size > 4K), the processes may get 2M pages when the system is up for few hours, but may not obtain even a single large page (2M) when the system is up for few days.

Now, the explanation.

Memory fragmentation is the main reason for the above mentioned behavior on Solaris x86 platform. On Solaris x86, once memory has been fragmented, it is very rare that a large page can be put back together. This is a limitation of Solaris on x86. It takes 512 base pages (4K in size) that are aligned and physically contiguous to form a common large page (2M on x86). If a kernel allocation occurs within this region, it is typically not possible to ever form this large page again as kernel pages cannot be relocated.

Solaris/SPARC has the kernel cage to help memory from being fragmented with kernel allocations. However as of now kernel cage isn't fully implemented in Solaris/x86. That is the main reason for the non availability of large (2M) pages on Solaris x86 when the system is up for longer durations. The good news is that Sun Microsystems is actively working on a solution to address this issue.

Acknowledgments:
Kit Chow
_______________
Technorati tags:
Solaris | OpenSolaris | Performance