Performance of Single Vs Multi-threaded application

Memory allocation performance in single and multithreaded environments is an important aspect of any application. Some allocators, such as malloc on Solaris Operating Environment work best with single-threaded applications, but degrades performance with a multi-threaded application. As memory is being allocated concurrently in multiple threads, all the threads must wait in a queue while malloc() handles one request at a time. With a few extra threads, this can slow down performance, causing a problem known as heap contention i.e., all the threads compete for access to the same heap. If the application is making a considerably high number of calls to malloc() that will be an indication of heap contention

One solution to alleviate the problem is to use multiple heaps. (heap is nothing but a block(s) of memory that a process can allocate or free at run time). Every process will have its own heap i.e., heap is private to the process. If there are multiple heaps, the threads do not need to wait for a heap's lock; instead each thread can use the one allocated to the CPU it is running on

Heap Usage

When an application executes, the operating system creates a virtual address space for the new process. This address space contains will have the stack, application's executable code, data, and the heap. The heap is where dynamic memory allocation occurs. Various system calls are used to manage a process's heap; brk() and sbrk() system calls can be used to resize a process's data segment to accommodate requests for more memory allocations

malloc library

malloc on Solaris performs well only in single-threaded applications. Its performance is unsuitable for multithreaded applications, because its performance worsens as threads are added. This is due to the fact that malloc maintains a single heap, and thus only one thread can allocate/deallocate memory at any given time. Therefore, with the addition of more threads, we find more threads waiting, and the wait time grows longer, resulting in increasingly slow execution times. However, since only one memory operation is going on at a time, the overall heap usage is low. While low heap usage is good, the slow performance of malloc in multithreaded applications still should not justify using it in such applications.

mtmalloc library

mtmalloc library of Solaris was designed to address the problems faced by multi-threaded applications with single heap malloc library. Multi-threaded applications scale well by taking advantage of having concurrent access to heap by multiple threads. However memory footprint is a big tradeoff in using mtmalloc. Heap usage for mtmalloc was much higher compared to the traditional malloc. This is most likely because mtmalloc is a power of two allocator, i.e., it rounds the size of requested blocks up to the next power of two. For example, if a request is made to allocate 17 bytes, the actual size of the allocated block would be rounded up to 32 bytes. As these requests add up, it can lead to internal fragmentation, so that the "extra" memory that is allocated is wasted

<more about mtmalloc>
<example with pmap -x output showing heap allocation from malloc, mtmalloc libraries>