Solaris: (Undocumented) Thread Environment Variables

The new thread library (initially called T2) on Solaris 8 Update 7 and later versions of Solaris, provides a 1:1 threading model, for better performance and scalability. Just like other libraries of Solaris, the new thread library is binary compatible with the old library, and the applications linked with old library continue to work as they are, without any changes to the application or any need to re-compile the code.

The primary difference being the new thread library uses a 1:1 thread model, where as the old library implements a two-level model in which user-level threads are multiplexed over possibly fewer light weight processes. In the 1:1 (or 1x1) model, an user level (application) thread has a corresponding kernel thread. A multi-threaded application with n threads will have n kernel threads. Each user level thread has a light weight process (LWP) connecting it to the kernel thread. Due to the more number of LWPs, the resource consumption will be a little high with 1x1 model, compared to MxN model; but due to the less overhead of multiplexing and scheduling the threads over LWP, 1x1 model performs much better compared to the old MxN model.

Tuning an application with thread environment variables

Within the new thread library (libthread) framework, a bunch of tunables have been provided to tweak the performance of the application. All these tunables have default values, and often the default values are good enough for the application to perform better. For some unknown reasons, these tunables were not documented anywhere except in source code. You can have a look at the source code for the following files at OpenSolaris

• thr.c <- check the routine etest()
  
• synch.c <- read the comments and code between lines: 76 and 97

Here's a brief description of these environment variables along with their default values:

1. LIBTHREAD_QUEUE_SPIN
   
   
   • Controls the spinning for queue locks
     
   • Default value: 1000
   
   
2. LIBTHREAD_ADAPTIVE_SPIN
   
   
   • Specifies the number of iterations (spin count) for adaptive mutex locking before giving up and going to sleep
     
   • Default value: 1000
   
   
3. LIBTHREAD_RELEASE_SPIN
   
   
   • Spin LIBTHREAD_RELEASE_SPIN times to see if a spinner grabs the lock; if so, don’t bother to wake up a waiter
     
   • Default value: LIBTHREAD_ADAPTIVE_SPIN/2 ie., 500. However it can be set explicitly to override the default value
   
   
4. LIBTHREAD_MAX_SPINNERS
   
   
   • Limits the number of simultaneous spinners attempting to do adaptive locking
     
   • Default value: 100
   
   
5. LIBTHREAD_MUTEX_HANDOFF
   
   
   • Do direct mutex handoff (no adaptive locking)
     
   • Default value: 0
   
   
6. LIBTHREAD_QUEUE_FIFO
   
   
   • Specifies the frequency of FIFO queueing vs LIFO queueing (range is 0..8)
     
   • Default value: 4
     
   • LIFO queue ordering is unfair and can lead to starvation, but it gives better performance for heavily contended locks
     
     
     • 0 - every 256th time (almost always LIFO)
       
     • 1 - every 128th time
       
     • 2 - every 64th time
       
     • 3 - every 32nd time
       
     • 4 - every 16th time (default, mostly LIFO)
       
     • 5 - every 8th time
       
     • 6 - every 4th time
       
     • 7 - every 2nd time
       
     • 8 - every time (never LIFO, always FIFO)
   
   
7. LIBTHREAD_QUEUE_DUMP
   
   
   • Causes a dump of user−level sleep queue statistics onto stderr (file descriptor 2) on exit
     
   • Default value: 0
   
   
8. LIBTHREAD_STACK_CACHE
   
   
   • Specifies the number of cached stacks the library keeps for re−use when more threads are created
     
   • Default value: 10
   
   
9. LIBTHREAD_COND_WAIT_DEFER
   
   
   • Little bit of history:
     
     The old thread library (Solaris 8's default) forces the thread performing cond_wait() to block all signals until it reacquires the mutex.
     
     The new thread library on Solaris 9 (default) and later versions, behaves exact opposite. The state of the mutex in the signal handler is undefined and the thread does not block any more signals than it is already blocking on entry to cond_wait().
     
     However to accomodate the applications that rely on old behavior, the new thread library implemented the old behavior as well, and it can be controlled with the LIBTHREAD_COND_WAIT_DEFER env variable. To get the old behavior, this variable has to be set to 1.
   
   
10. LIBTHREAD_ERROR_DETECTION
    
    
    • Setting it to 1 issues a warning message about illegal locking operations and setting it to 2 issues the warning message and core dump
      
    • Default value: 0

Reference:
Roger Faulkner's New/Improved Threads Library presentation

________________
Technorati tags: Solaris | OpenSolaris