Mandalika's scratchpad [ Work blog @Oracle | Stock Market Notes | My Music Compositions ]

Old Posts: 09.04  10.04  11.04  12.04  01.05  02.05  03.05  04.05  05.05  06.05  07.05  08.05  09.05  10.05  11.05  12.05  01.06  02.06  03.06  04.06  05.06  06.06  07.06  08.06  09.06  10.06  11.06  12.06  01.07  02.07  03.07  04.07  05.07  06.07  08.07  09.07  10.07  11.07  12.07  01.08  02.08  03.08  04.08  05.08  06.08  07.08  08.08  09.08  10.08  11.08  12.08  01.09  02.09  03.09  04.09  05.09  06.09  07.09  08.09  09.09  10.09  11.09  12.09  01.10  02.10  03.10  04.10  05.10  06.10  07.10  08.10  09.10  10.10  11.10  12.10  01.11  02.11  03.11  04.11  05.11  07.11  08.11  09.11  10.11  11.11  12.11  01.12  02.12  03.12  04.12  05.12  06.12  07.12  08.12  09.12  10.12  11.12  12.12  01.13  02.13  03.13  04.13  05.13  06.13  07.13  08.13  09.13  10.13  11.13  12.13  01.14  02.14  03.14  04.14  05.14  06.14  07.14  09.14  10.14  11.14  12.14  01.15  02.15  03.15  04.15  06.15  09.15  12.15  01.16  03.16  04.16  05.16  06.16  07.16  08.16  09.16  12.16  01.17 

Tuesday, January 31, 2017
C, Solaris & SPARC M7: Get Defensive with Few Techniques .. Part 3/3

SPARC: Silicon Secured Memory (SSM)

Silicon Secured Memory (SSM) is a hardware feature available on the latest Oracle SPARC M7/T7/S7 platform that is designed to protect against invalid data accesses such as freed memory accesses, stale pointer references and buffer overflows real-time. It stops unauthorized access to memory whether that access is due to a programming error or a malicious attempt to exploit buffer overruns. This feature is designed to help prevent security vulnerabilities such as Heartbleed ♡. The hardware does it by comparing the version number stored in software managed pointer with the version number maintained in the memory cache lines§. A mismatch leads to a hardware trap that ultimately turns to an error.

This feature can be used during application development as well as in production to detect potential memory corruption issues. Applications written in programming languages such as C/C++ benefit the most as manual memory management plays an important role in those applications making them vulnerable to memory corruption triggered by programming errors. Existing applications can be enabled with SSM [without recompiling] by linking with a Oracle Solaris library during runtime. The overhead incurred by enabling this feature is minimal as the hardware has to generate a trap to report the error only when there is a memory access error.

As of now only 64-bit binaries can take advantage of SSM feature. One requirement is that allocated memory (target memory area for checking memory errors) needs to be 64-byte aligned and its size must be multiple of 64. Also watchout for other requirements and limitations that were outlined in some of the documents listed at the bottom of this post.

Rest of this blog post uses the following buggy code to demonstrate Silicon Secured Memory (SSM) feature.

% cat -n memerr.c

     1  #include <stdio.h>
     2  #include <stdlib.h>
     3  #include <string.h>
     5  void main(int argc, char *argv[])
     6  {
     7          char *longstr  = malloc( sizeof(char) * 64 );
     8          char *longstr2 = malloc( sizeof(char) * 64 );
    10          strcpy( longstr , argv[1] );
    11          strcpy( longstr2, argv[2] );
    13          for (int i = 0; i < 96; ++i)
    14                  printf( "%c ", longstr[i] ); /* read beyond boundary */
    15          printf( "\n" );
    17          free( longstr );
    18          free( longstr2 );
    20          strcpy( longstr , argv[2] ); /* freed memory access */
    21          strcpy( longstr2, argv[1] ); /* freed memory access */
    23          for (int i = 0; i < 96; ++i)
    24                  printf( "%c ", longstr[i] );
    25          printf( "\n" );
    27          free( longstr );  /* double free */
    28          free( longstr2 );  /* double free */
    29  }

% cc -g -m64 -o memerr memerr.c

SSM in Development Environment

Rely on Oracle Solaris Studio Code Analyzer and SSM to find and fix memory access errors.

Couple of options here. Both options require access to Oracle Solaris Studio 12.4 4/15 Platform Specific Enhancement (PSE) or later software.

  1. Preload libdiscoverADI library and run the application. This will run all 64-bit binaries in the application in SSM mode.

    % LD_PRELOAD_64=<install-dir>/lib/compilers/sparcv9/ ./memerr
  2. Enable discover utility to detect and understand runtime memory access errors identified by ADI. This can be done on a per binary basis in a large application.

     .. on build system ..
    % discover -i adi -A on -P on -w memerrs.txt memerr
     .. on deployment system (M7/T7/S7) ..
    % ./memerr jack jill
    j a c k                                                             j i l l
    j i l l                                                             j a c k
    % cat memerrs.txt
    ERROR 1 (ABR): reading memory beyond array bounds at address 0x2fffffff7d47e040 {memory: v3}:
            main() + 0x7c  
        was allocated at (0x2fffffff7d47e000, 64 bytes):
            main() + 0x10  
    ERROR 2 (FMW): writing to freed memory at address 0x2fffffff7d47e000 {memory: v9}:
            strcpy() + 0x4c
            _start() + 0x108
        was allocated at (0x2fffffff7d47e000, 64 bytes):
            main() + 0x10  
        freed at (0x2fffffff7d47e000, 64 bytes):
            main() + 0xc4  
    ERROR 12 (DFM): double freeing memory at address 0x3fffffff7d47e040 {memory: v10}:
            main() + 0x17c  
        was allocated at (0x3fffffff7d47e040, 64 bytes):
            main() + 0x20  
        freed at (0x3fffffff7d47e040, 64 bytes):
            main() + 0xd0  
            unique errors   : 12 (12 total)
     .. running the ADI enabled binary on a non-SSM system fails ..
    % ./memerr
    adi_set_enabled() failed: Operation not supported
    ADI/SSM is only supported on SPARC-xx machines. See discover -i adi documentation for more information

SSM in Production Environment

Preload libadimalloc(3LIB) library on target M7/T7/S7 system to enable real-time data protection. Data integrity is the ultimate goal.

Check the platform supported extensions to ensure that the target M7/T7/S7 system is SSM-ready.

% isainfo -v
64-bit sparcv9 applications
        xmpmul mwait sparc5 adi vis3b

Check the normal behavior of the above sample code that has quite a few memory access errors.

% ./memerr jack jill
j a c k    ▒                                                                A         j i l l    ▒
j i l l    ▒                                                                A         j a c k    ▒

Now check the behavior of the same buggy code under SSM surveillance.

% LD_PRELOAD_64=/lib/64/ ./memerr jack jill
Segmentation Fault (core dumped)

% echo ::status | mdb core
debugging core file of memerr (64-bit) from dummyhost
file: /var/tmp/memerr
initial argv: ./memerr jack jill
threading model: native threads
status: process terminated by SIGSEGV (Segmentation Fault), pc=100000bbc
, ADI version d mismatch for VA ffffffff7b13ff80

Notice that an existing binary was used as is in the above example to show the benefit of SSM in catching potential memory corruption issues.

For details, SEE:

Credit: various


Saturday, December 31, 2016
Blast from the Past : The Weekend Playlist #10

Previous playlists:

    #1    #8 (50s, 60s and 70s)    |    #2    #3    #4    #5 (80s)    |    #6    #7    #9 (90s)

New category: "instrumental" playlist. Audio & Widget courtesy: Spotify


Tuesday, December 13, 2016
C, Solaris & SPARC M7: Get Defensive with Few Techniques .. Part 2/3

Solaris: Address Space Layout Randomization (ASLR)

Address Space Layout Randomization is a security defense mechanism against attacks like buffer overflow or Return Oriented Programming (ROP) attacks that exploit software vulnerabilities. An attacker gaining control of the call stack of a process at runtime to manipulate program control flow to execute instructions of choice is an ROP attack.

All major operating systems including Solaris support Address Space Layout Randomization to minimize the risk of such attacks. In general, user land processes place the starting address of key areas at a known place. ASLR randomizes the starting address of the key areas of the proces address space such as the base of the executable, stack, brk-based heap, memory mappings including the mapping of libraries.

On Solaris, ASLR is configurable at the system level (global & local zones) and at the binary and process level with the help of sxadm command line utility. It is possible to enable or disable ASLR for all processes; or selectively enable/disable ASLR for certain applications by tagging related binaries. Tagging is just a special ELF entry inside target binary's dynamic section that explicitly tells whether or not to enable the defense mechanism. Binary tagging has precedence over the system-level configuration. Out of the box, many of userland binaries are tagged on Solaris to enable ASLR; and by default, Solaris boots the global and all non-global zones with ASLR enabled only for those binaries that are explicitly marked (tagged) to support it.

Rest of the post demonstrates ASLR configuration with few examples.

Current ASLR Settings

% sxadm info aslr
EXTENSION           STATUS                        CONFIGURATION
aslr                enabled (tagged-files)        default (default)

Above output shows that ASLR is currently enabled for tagged binaries, which is the default behavior on recent Solaris 11 updates.

Tag a Specific Userland Binary to Enable ASLR

Developers can rely on link-editor's "-z aslr=.." option to selectively tag certain dynamic executables and position-independent executables to enable or disable ASLR for those binaries.

% elfdump -d <somebin> | grep -i aslr
 ^^^^ binary not tagged for ASLR

 .. tag to enable ASLR ..

% cc -z aslr <obj-files> -o <somebin> 
% elfdump -d <somebin> | grep -i aslr
     [34]  SUNW_ASLR       0x2          ENABLE
                                        ^^^^^^ binary tagged to enable ASLR

 .. tag to disable ASLR ..

% cc -z aslr=disable <obj-files> -o <somebin> 
% elfdump -d <somebin> | grep -i aslr
     [34]  SUNW_ASLR       0x1          DISABLE
                                        ^^^^^^ binary tagged to disable ASLR

System-wide ASLR Configuration

ASLR is managed as a security extension by the sxadm command line utility. sxadm command configures and controls Solaris security extensions both at the system level (global zone, non-global zone) and at the process level.

The enable and disable subcommands enable and disable ASLR system-wide, and the delcust subcommand resets custom ASLR configuration to the out-of-the-box default configuration.

Please check the man page of sxadm(1M) for detailed information about all supported options.

% sxadm info aslr
EXTENSION           STATUS                        CONFIGURATION
aslr                enabled (tagged-files)        default (default)

 .. enable ASLR system-wide .. 

# sxadm enable -c model=all aslr
# sxadm info aslr
EXTENSION           STATUS                        CONFIGURATION
aslr                enabled (all)                 enabled (all)

 .. enable ASLR system-wide only for the tagged binaries ..

# sxadm enable -c model=tagged-files aslr
# sxadm info aslr
EXTENSION           STATUS                        CONFIGURATION
aslr                enabled (tagged-files)        enabled (tagged-files)

 .. disable ASLR system-wide ..

# sxadm disable aslr
# sxadm info aslr
EXTENSION           STATUS                        CONFIGURATION
aslr                disabled                      disabled

 .. reset custom/current config to default ..

# sxadm delcust aslr
# sxadm info aslr
EXTENSION           STATUS                        CONFIGURATION
aslr                enabled (tagged-files)        default (default)

Runtime Process Level ASLR Configuration

sxadm command supports runtime process configuration through "sxadm exec" interface. Child processes of the command executed via "sxadm exec" inherit the same security extension configuration unless the child process is executing a setuid binary or is more privileged.

# sxadm disable  aslr

 .. enable ASLR during runtime for pmap process ..

# sxadm exec -s aslr=enable /usr/bin/pmap self | grep heap
00000039452A8000         32K rw-----    [ heap ]
00000039452B0000         64K rw-----    [ heap ]

# sxadm exec -s aslr=enable /usr/bin/pmap self | grep heap
00000039B567E000          8K rw-----    [ heap ]
00000039B5680000         64K rw-----    [ heap ]

 .. disable ASLR during runtime for pmap process ..

# sxadm exec -s aslr=disable /usr/bin/pmap self | grep heap
000000010010A000         24K rwx----    [ heap ]
0000000100110000         64K rw-----    [ heap ]

# sxadm exec -s aslr=disable /usr/bin/pmap self | grep heap
000000010010A000         24K rwx----    [ heap ]
0000000100110000         64K rw-----    [ heap ]

Binary tagging takes precedence over runtime process conconfiguration if the binary was tagged to disable ASLR.


ASLR might give a hard time especially during debugging that require consistent and repeatable conditions including address space offsets. In such situations, disable ASLR tentatively for the target binary (best case) or for the entire system (worst case) until the debugging exercise completes.

Friday, September 30, 2016
C, Solaris & SPARC M7 : Get Defensive with Few Techniques .. Part 1/3

This post is related to security defense (sort of).

C: Buffer Overflows

C programming language has no built-in protection against accessing or overwriting data in any part of memory. Also C doesn't check whether the data written to an array is within the boundaries of that array. Consequently erroneous C code can easily trigger buffer overflows/overruns by writing more data to a buffer than it can hold. In other words, by putting data in a memory area past a buffer. Writing beyond the bounds of a block of allocated memory (buffer) is a security vulnerability that can corrupt data, crash the application, or may cause the execution of malicious code.

Now let's have a look at few examples using some of the unbounded and bounded string functions.

strcpy vs strncpy vs strlcpy


strcpy() does not check buffer lengths (hence unbounded function) and may overwrite memory zone contiguous to the intended destination (buffer) if not careful.

strcpy() automatically includes the terminating null byte ('\0'). The programmer has to ensure that the copied string is within the bounds of the destination buffer.

// no overflow
char word[5];
strcpy(word, "best");

// overflow
char word[5];
strcpy(word, "utopia");

strcpy(), strcat() and strcmp() functions in the same family are similarly vulnerable.

strncpy() is a bounded string function that prevents buffer overflows by accepting a buffer size argument and not writing past that boundary. This function copies at most bytes equal to the buffer size from source string to destination buffer. However unlike strcpy(), strncpy() does not guarantee null terminated string.

Initially all characters in the buffer are set to null bytes ('\0'). If the length of the source string is less than the buffer size argument, strncpy() overwrites the '\0' characters in the buffer only until the length of the source string. In this case, the copied string remains null terminated.

If the source string is equal to the size of buffer, all the '\0' characters in destination buffer will be overwritten and the copied string will be non-null terminated.

Similarly if the source string is longer than the size of buffer, all the '\0' characters in destination buffer will be overwritten and the copy will be non-null terminated and truncated resulting in data loss.

In all cases, it is the responsibility of the programmer to check for and include the terminating null byte if missing. Non-null terminated strings may exhibit undefined behavior that can potentially lead to incorrect program execution, and the hosting application is still vulnerable to attacks. For example, in case of non-null terminated strings, strlen() will keep searching memory until it finds a null character or hits an address that causes a memory protection fault of some sort.

// ok. null terminated string
char word[6];
strncpy(word, "best", 6);

// not ok. non-null terminated string
char word[6];
strncpy(word, "utopia", 6);

// null terminated but truncated string
// atleast we avoid the undefined behavior
char word[6];
strncpy(word, "utopia", 5); // leave last byte for '\0'
strncpy(word, "utopia", 6);
word[5] = '\0';

strlcpy() is another bounded string function that prevents buffer overflows and avoids non-null terminated strings by copying at most bytes equal to the buffer size [argument] from source string to destination buffer, and by always adding a terminating null byte.

The programmer must still handle the possibility of data loss if the destination buffer size is too small. Data loss or string truncation can be detected by comparing the return value of the function (which is the length of the source string) to the destination buffer size.

char word[6];
if (strlcpy(word, "utopia", 6) >= 6) {
        // handle data loss / string truncation
Note that strlcpy() is a non-standard string function. Therefore the resulting code may not be portable should the application making use of this function support a variety of operating platforms.

Solaris has this function implementation in C library.

To be continued ..


Wednesday, August 31, 2016
[Solaris] Memory Leak Checking with libumem

libumem is a userland memory allocator (a library) with some debugging features that enable easy identification and troubleshooting of process memory leaks and memory access errors. Apparently target application must either be linked with the library, or the library must be preloaded into the address space of the target process before users can take advantage of the diagnostic features offered by libumem.

Besides the diagnostic support, libumem strives to improve the memory allocation performance by creating and managing multiple independent caches each with a different buffer size. This results in good scaling due to reduced lock contention especially in multi-threaded applications with many threads allocating and deallocating memory concurrently. Evidently there will be a tradeoff somewhere that achieves better scalability; and the tradeoff in this case is a slight memory overhead. Keep in mind that some of this additional memory will be accounted for when examining the process for memory leaks.

Rest of this post details the steps involved in examining memory leaks with a simple native process as an example.

High-Level Steps:

  1. Runtime debugging features such as memory leak detection, buffer overflows can be controlled by UMEM_* environment variables. Check umem_debug(3MALLOC) man page for the complete list of environment variables along with brief description.

  2. Check if the target application was linked with libumem library (-lumem). If not, preload /usr/lib/* before running the application.


    1) Executable linked with libumem library

    % cc -g -o mleak -lumem leak.c
    % ldd mleak
   =>  /lib/
   =>     /lib/

    2) Executable not linked with libumem library

    % cc -g -o mleak leak.c
    % ldd mleak
   =>     /lib/
    % export LD_PRELOAD_32=/usr/lib/
    % export UMEM_DEBUG=default
    % ./mleak
  3. Memory leaks can be examined with the help of modular debugger, mdb. Couple of possibilities.

    • attach a live process to the mdb debugger and run relevant dcmds such as ::findleaks

      # echo ::findleaks | mdb -p `pgrep mleak`
    • generate a core image of the running process and examine the core file for memory leaks

      # gcore `pgrep mleak`
      gcore: core.7487 dumped
      # echo ::findleaks | mdb core.7487

Complete Example:

% cat -n leak.c

     1  #include <stdio.h>
     2  #include <stdlib.h>
     3  #include <unistd.h>
     5  int *intblk()
     6  {
     7          int *someint = malloc( sizeof(int) * 2 );
     8          return malloc( sizeof(int) );
     9  }
    11  void main()
    12  {
    13          int i = 0;
    14          while ( 1 ) {
    15                  int *ptr = intblk();
    16                  *ptr = (++i * 2);
    17                  if ( !(*ptr % 3) ) {
    18                          continue;
    19                  }
    20                  if ( *ptr > 500 ) {
    21                          abort();
    22                  }
    23                  free( ptr );
    24                  sleep( 1 );
    25          }
    26  }

% cc -g -o mleak  leak.c

# export UMEM_DEBUG=default
# export LD_PRELOAD_32=/usr/lib/

# ./mleak &
[1] 22427

High level summary memory leak report

mdb's findleaks dcmd displays the potential memory leaks. The summary leak report shows the bufctl address along with the topmost stack frame at the point when the memory was allocated.

eg., contd.,

# mdb -p `pgrep mleak`
Loading modules: [ ]

> ::findleaks

000b0008      52 002cc780 intblk+4
000b0008      17 002cc8e8 intblk+0x10
   Total      69 buffers, 1104 bytes

Stack trace for each leak

To get the stack trace for a memory allocation that resulted in a leak, dump the bufctl structure. The address of this structure can be obtained from the output of the findleaks dcmd (highlighted in blue in above output).

eg., contd.,

> 002cc780::bufctl_audit

            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
          2cc780           2c3fe0    50963bfdaab48                1
                            b0008                0                0

> 002cc8e8::bufctl_audit

            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
          2cc8e8           2c3f80    509643711ef91                1
                            b0008                0                0

Detailed report in one shot

To obtain a detailed leak report that shows the summary report along with stack traces for each memory allocation that ended up with a leak, run findleaks dcmd with -d option. -fv options provide some additional detail.

eg., contd.,

> ::findleaks -d

000b0008      52 002cc780 intblk+4
000b0008      17 002cc8e8 intblk+0x10
   Total      69 buffers, 1104 bytes

umem_alloc_16 leak: 52 buffers, 16 bytes each, 832 bytes total

            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
          2cc780           2c3fe0    50963bfdaab48                1
                            b0008                0                0

umem_alloc_16 leak: 17 buffers, 16 bytes each, 272 bytes total

            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
          2cc8e8           2c3f80    509643711ef91                1
                            b0008                0                0

Histogram of the size of the non-freed allocations

[cache]::umem_malloc_info reports information about malloc()'s by size for the memory allocations that weren't free()'d - hence leaked. ::umem_malloc_info output can be used to figureout the maximum allocation in a particular buffer.

eg., contd.,

> 000b0008::umem_malloc_info -gz

000b0008     16      8       69        7        484       5043 1041.9%

malloc size  ------------------ Distribution ------------------ count
          4 |@@@@@@@@@@@@                                       17
          8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@              52

The above output shows that there were 17 malloc( 4 ) (4 bytes) calls at intblk+0x10 with no matching free(<addr>) found anywhere. Disassembled code near that address is shown below.

eg., contd.,

> intblk+0x10::dis

__init_kernel_mode_misaligned_data_trap_handler+8:      nop
0x10994:                        illtrap   0x0
0x10998:                        illtrap   0x10000
0x1099c:                        illtrap   0x10000
0x109a0:                        illtrap   0x10000
0x109a4:                        illtrap   0x10000
intblk:                         save      %sp, -0x68, %sp
intblk+4:                       call      +0x1017c      <`malloc>
intblk+8:                       mov       0x8, %o0
intblk+0xc:                     st        %o0, [%fp - 0x8]
intblk+0x10:                    call      +0x10170      <`malloc>
intblk+0x14:                    mov       0x4, %o0
intblk+0x18:                    st        %o0, [%fp - 0x4]
intblk+0x1c:                    ld        [%fp - 0x4], %l0
intblk+0x20:                    or        %l0, %g0, %i0
intblk+0x24:                    ret
intblk+0x28:                    restore
0x109d4:                        illtrap   0x10000
0x109d8:                        illtrap   0x10000
0x109dc:                        illtrap   0x10000
0x109e0:                        illtrap   0x10000

> ::quit

Other related mdb dcmds of interest: umastat

Some of the libumem's debugging features work only for allocations that are smaller than 16 KB in size. Allocations larger than 16 KB could have reduced support. Such allocations are usually referred to as oversized allocations in memory leak reports generated by findleaks dcmd.


  1. Java Platform, Standard Edition Troubleshooting Guide -> Diagnose Leaks in Native Code -> Find Leaks with libumem Tool
  2. How Memory Allocation Affects Performance in Multithreaded Programs
  3. Sun Studio: Investigating memory leaks with dbx (from 2005)
  4. Sun Studio: Investigating memory leaks with Collector/Analyzer (from 2005)
  5. Sun Studio: Gathering memory allocations/leaks data, from a running process (from 2005)



This page is powered by Blogger. Isn't yours?