Mandalika's scratchpad [ Work blog @Oracle | Stock Market Notes | My Music Compositions ]

Old Posts: 09.04  10.04  11.04  12.04  01.05  02.05  03.05  04.05  05.05  06.05  07.05  08.05  09.05  10.05  11.05  12.05  01.06  02.06  03.06  04.06  05.06  06.06  07.06  08.06  09.06  10.06  11.06  12.06  01.07  02.07  03.07  04.07  05.07  06.07  08.07  09.07  10.07  11.07  12.07  01.08  02.08  03.08  04.08  05.08  06.08  07.08  08.08  09.08  10.08  11.08  12.08  01.09  02.09  03.09  04.09  05.09  06.09  07.09  08.09  09.09  10.09  11.09  12.09  01.10  02.10  03.10  04.10  05.10  06.10  07.10  08.10  09.10  10.10  11.10  12.10  01.11  02.11  03.11  04.11  05.11  07.11  08.11  09.11  10.11  11.11  12.11  01.12  02.12  03.12  04.12  05.12  06.12  07.12  08.12  09.12  10.12  11.12  12.12  01.13  02.13  03.13  04.13  05.13  06.13  07.13  08.13  09.13  10.13  11.13  12.13  01.14  02.14  03.14  04.14  05.14  06.14  07.14  09.14  10.14  11.14  12.14  01.15  02.15  03.15  04.15  06.15  09.15  12.15  01.16  03.16  04.16  05.16  06.16  07.16  08.16  09.16 

Friday, September 30, 2016
C, Solaris & SPARC M7 : Get Defensive with Few Techniques .. Part 1/3

This post is related to security defense (sort of).

C: Buffer Overflows

C programming language has no built-in protection against accessing or overwriting data in any part of memory. Also C doesn't check whether the data written to an array is within the boundaries of that array. Consequently erroneous C code can easily trigger buffer overflows/overruns by writing more data to a buffer than it can hold. In other words, by putting data in a memory area past a buffer. Writing beyond the bounds of a block of allocated memory (buffer) is a security vulnerability that can corrupt data, crash the application, or may cause the execution of malicious code.

Now let's have a look at few examples using some of the unbounded and bounded string functions.

strcpy vs strncpy vs strlcpy


strcpy() does not check buffer lengths (hence unbounded function) and may overwrite memory zone contiguous to the intended destination (buffer) if not careful.

strcpy() automatically includes the terminating null byte ('\0'). The programmer has to ensure that the copied string is within the bounds of the destination buffer.

// no overflow
char word[5];
strcpy(word, "best");

// overflow
char word[5];
strcpy(word, "utopia");

strcpy(), strcat() and strcmp() functions in the same family are similarly vulnerable.

strncpy() is a bounded string function that prevents buffer overflows by accepting a buffer size argument and not writing past that boundary. This function copies at most bytes equal to the buffer size from source string to destination buffer. However unlike strcpy(), strncpy() does not guarantee null terminated string.

Initially all characters in the buffer are set to null bytes ('\0'). If the length of the source string is less than the buffer size argument, strncpy() overwrites the '\0' characters in the buffer only until the length of the source string. In this case, the copied string remains null terminated.

If the source string is equal to the size of buffer, all the '\0' characters in destination buffer will be overwritten and the copied string will be non-null terminated.

Similarly if the source string is longer than the size of buffer, all the '\0' characters in destination buffer will be overwritten and the copy will be non-null terminated and truncated resulting in data loss.

In all cases, it is the responsibility of the programmer to check for and include the terminating null byte if missing. Non-null terminated strings may exhibit undefined behavior that can potentially lead to incorrect program execution, and the hosting application is still vulnerable to attacks. For example, in case of non-null terminated strings, strlen() will keep searching memory until it finds a null character or hits an address that causes a memory protection fault of some sort.

// ok. null terminated string
char word[6];
strncpy(word, "best", 6);

// not ok. non-null terminated string
char word[6];
strncpy(word, "utopia", 6);

// null terminated but truncated string
// atleast we avoid the undefined behavior
char word[6];
strncpy(word, "utopia", 5); // leave last byte for '\0'
strncpy(word, "utopia", 6);
word[5] = '\0';

strlcpy() is another bounded string function that prevents buffer overflows and avoids non-null terminated strings by copying at most bytes equal to the buffer size [argument] from source string to destination buffer, and by always adding a terminating null byte.

The programmer must still handle the possibility of data loss if the destination buffer size is too small. Data loss or string truncation can be detected by comparing the return value of the function (which is the length of the source string) to the destination buffer size.

char word[6];
if (strlcpy(word, "utopia", 6) >= 6) {
        // handle data loss / string truncation
Note that strlcpy() is a non-standard string function. Therefore the resulting code may not be portable should the application making use of this function support a variety of operating platforms.

Solaris has this function implementation in C library.

To be continued ..


Wednesday, August 31, 2016
[Solaris] Memory Leak Checking with libumem

libumem is a userland memory allocator (a library) with some debugging features that enable easy identification and troubleshooting of process memory leaks and memory access errors. Apparently target application must either be linked with the library, or the library must be preloaded into the address space of the target process before users can take advantage of the diagnostic features offered by libumem.

Besides the diagnostic support, libumem strives to improve the memory allocation performance by creating and managing multiple independent caches each with a different buffer size. This results in good scaling due to reduced lock contention especially in multi-threaded applications with many threads allocating and deallocating memory concurrently. Evidently there will be a tradeoff somewhere that achieves better scalability; and the tradeoff in this case is a slight memory overhead. Keep in mind that some of this additional memory will be accounted for when examining the process for memory leaks.

Rest of this post details the steps involved in examining memory leaks with a simple native process as an example.

High-Level Steps:

  1. Runtime debugging features such as memory leak detection, buffer overflows can be controlled by UMEM_* environment variables. Check umem_debug(3MALLOC) man page for the complete list of environment variables along with brief description.

  2. Check if the target application was linked with libumem library (-lumem). If not, preload /usr/lib/* before running the application.


    1) Executable linked with libumem library

    % cc -g -o mleak -lumem leak.c
    % ldd mleak
   =>  /lib/
   =>     /lib/

    2) Executable not linked with libumem library

    % cc -g -o mleak leak.c
    % ldd mleak
   =>     /lib/
    % export LD_PRELOAD_32=/usr/lib/
    % export UMEM_DEBUG=default
    % ./mleak
  3. Memory leaks can be examined with the help of modular debugger, mdb. Couple of possibilities.

    • attach a live process to the mdb debugger and run relevant dcmds such as ::findleaks

      # echo ::findleaks | mdb -p `pgrep mleak`
    • generate a core image of the running process and examine the core file for memory leaks

      # gcore `pgrep mleak`
      gcore: core.7487 dumped
      # echo ::findleaks | mdb core.7487

Complete Example:

% cat -n leak.c

     1  #include <stdio.h>
     2  #include <stdlib.h>
     3  #include <unistd.h>
     5  int *intblk()
     6  {
     7          int *someint = malloc( sizeof(int) * 2 );
     8          return malloc( sizeof(int) );
     9  }
    11  void main()
    12  {
    13          int i = 0;
    14          while ( 1 ) {
    15                  int *ptr = intblk();
    16                  *ptr = (++i * 2);
    17                  if ( !(*ptr % 3) ) {
    18                          continue;
    19                  }
    20                  if ( *ptr > 500 ) {
    21                          abort();
    22                  }
    23                  free( ptr );
    24                  sleep( 1 );
    25          }
    26  }

% cc -g -o mleak  leak.c

# export UMEM_DEBUG=default
# export LD_PRELOAD_32=/usr/lib/

# ./mleak &
[1] 22427

High level summary memory leak report

mdb's findleaks dcmd displays the potential memory leaks. The summary leak report shows the bufctl address along with the topmost stack frame at the point when the memory was allocated.

eg., contd.,

# mdb -p `pgrep mleak`
Loading modules: [ ]

> ::findleaks

000b0008      52 002cc780 intblk+4
000b0008      17 002cc8e8 intblk+0x10
   Total      69 buffers, 1104 bytes

Stack trace for each leak

To get the stack trace for a memory allocation that resulted in a leak, dump the bufctl structure. The address of this structure can be obtained from the output of the findleaks dcmd (highlighted in blue in above output).

eg., contd.,

> 002cc780::bufctl_audit

            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
          2cc780           2c3fe0    50963bfdaab48                1
                            b0008                0                0

> 002cc8e8::bufctl_audit

            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
          2cc8e8           2c3f80    509643711ef91                1
                            b0008                0                0

Detailed report in one shot

To obtain a detailed leak report that shows the summary report along with stack traces for each memory allocation that ended up with a leak, run findleaks dcmd with -d option. -fv options provide some additional detail.

eg., contd.,

> ::findleaks -d

000b0008      52 002cc780 intblk+4
000b0008      17 002cc8e8 intblk+0x10
   Total      69 buffers, 1104 bytes

umem_alloc_16 leak: 52 buffers, 16 bytes each, 832 bytes total

            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
          2cc780           2c3fe0    50963bfdaab48                1
                            b0008                0                0

umem_alloc_16 leak: 17 buffers, 16 bytes each, 272 bytes total

            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
          2cc8e8           2c3f80    509643711ef91                1
                            b0008                0                0

Histogram of the size of the non-freed allocations

[cache]::umem_malloc_info reports information about malloc()'s by size for the memory allocations that weren't free()'d - hence leaked. ::umem_malloc_info output can be used to figureout the maximum allocation in a particular buffer.

eg., contd.,

> 000b0008::umem_malloc_info -gz

000b0008     16      8       69        7        484       5043 1041.9%

malloc size  ------------------ Distribution ------------------ count
          4 |@@@@@@@@@@@@                                       17
          8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@              52

The above output shows that there were 17 malloc( 4 ) (4 bytes) calls at intblk+0x10 with no matching free(<addr>) found anywhere. Disassembled code near that address is shown below.

eg., contd.,

> intblk+0x10::dis

__init_kernel_mode_misaligned_data_trap_handler+8:      nop
0x10994:                        illtrap   0x0
0x10998:                        illtrap   0x10000
0x1099c:                        illtrap   0x10000
0x109a0:                        illtrap   0x10000
0x109a4:                        illtrap   0x10000
intblk:                         save      %sp, -0x68, %sp
intblk+4:                       call      +0x1017c      <`malloc>
intblk+8:                       mov       0x8, %o0
intblk+0xc:                     st        %o0, [%fp - 0x8]
intblk+0x10:                    call      +0x10170      <`malloc>
intblk+0x14:                    mov       0x4, %o0
intblk+0x18:                    st        %o0, [%fp - 0x4]
intblk+0x1c:                    ld        [%fp - 0x4], %l0
intblk+0x20:                    or        %l0, %g0, %i0
intblk+0x24:                    ret
intblk+0x28:                    restore
0x109d4:                        illtrap   0x10000
0x109d8:                        illtrap   0x10000
0x109dc:                        illtrap   0x10000
0x109e0:                        illtrap   0x10000

> ::quit

Other related mdb dcmds of interest: umastat

Some of the libumem's debugging features work only for allocations that are smaller than 16 KB in size. Allocations larger than 16 KB could have reduced support. Such allocations are usually referred to as oversized allocations in memory leak reports generated by findleaks dcmd.


  1. Java Platform, Standard Edition Troubleshooting Guide -> Diagnose Leaks in Native Code -> Find Leaks with libumem Tool
  2. How Memory Allocation Affects Performance in Multithreaded Programs
  3. Sun Studio: Investigating memory leaks with dbx (from 2005)
  4. Sun Studio: Investigating memory leaks with Collector/Analyzer (from 2005)
  5. Sun Studio: Gathering memory allocations/leaks data, from a running process (from 2005)


Tuesday, August 09, 2016
New Article on OTN: Oracle Solaris Tools for Locality Observability

Oracle Solaris provides a variety of tools and APIs to observe, diagnose, control, and even fix issues related to locality and latency. A brand new technical article describing some of the tools and APIs that can be used to examine the locality of CPUs, memory and I/O devices is currently available on Oracle Technology Network (OTN) at the following URL.

      Oracle Solaris Tools for Locality Observability

Knowledge of these commands and tools is especially beneficial when planning and maintaining virtualized environments.

Target audience: application developers, system administrators, and application administrators.


Thursday, July 28, 2016
Programming in C: Few Tidbits #7

  1    _Alignof operator

_Alignof operator [that was introduced in C Standard Revision, C11] returns the alignment in bytes for the specified type. An alignment is an integer value representing the number of bytes between successive addresses in memory at which a given object can be allocated.

Some processors (such as SPARC family) require the data to be aligned on their natural boundaries. Aligned data allows the processor to fetch data from the memory in an efficient manner. Likewise misaligned data may cause additional memory accesses or a hardware fault in some cases.

Each of the primitive C data types have an alignment associated with them. The alignment requirement for each of the types can be queried using the _Alignof operator as shown below. For instance, an int data type is required to be aligned on 32-bit (4 byte) boundaries. That is, the memory address for the int data must be divisible by 4.

% cat align.c

#include <stdio.h>
#include <stdarg.h>
printf("sizeof char        = %2d _Alignof char        = %2d\n", sizeof(char), _Alignof(char));
printf("sizeof short       = %2d _Alignof short       = %2d\n", sizeof(short), _Alignof(short));
printf("sizeof int         = %2d _Alignof int         = %2d\n", sizeof(int), _Alignof(int));
printf("sizeof long        = %2d _Alignof long        = %2d\n", sizeof(long), _Alignof(long));
printf("sizeof float       = %2d _Alignof float       = %2d\n", sizeof(float), _Alignof(float));
printf("sizeof double      = %2d _Alignof double      = %2d\n", sizeof(double), _Alignof(double));
printf("sizeof long long   = %2d _Alignof long long   = %2d\n", sizeof(long long), _Alignof(long long));
printf("sizeof long double = %2d _Alignof long double = %2d\n", sizeof(long double), _Alignof(long double));
printf("sizeof pointer     = %2d _Alignof pointer     = %2d\n", sizeof(void *), _Alignof(void *));

% cc -xlang=c11 -o align  align.c <-- latest Sun Studio 12.x C compiler
% ./align     <-- 32-bit executable

 sizeof char        =  1 _Alignof char        =  1
 sizeof short       =  2 _Alignof short       =  2
 sizeof int         =  4 _Alignof int         =  4
 sizeof long        =  4 _Alignof long        =  4
 sizeof float       =  4 _Alignof float       =  4
 sizeof double      =  8 _Alignof double      =  8
 sizeof long long   =  8 _Alignof long long   =  8
 sizeof long double = 16 _Alignof long double =  8
 sizeof pointer     =  4 _Alignof pointer     =  4

In general, the _Alignof value is the same as the value for sizeof for primitive data types. However structures are aligned to the alignment requirements of the widest type (largest element). In both cases, usually the compiler handles the alignment requirement for developers by padding where needed. In the following example, the compiler may try to construct the mix structure in such a way that member l (a long) falls on a 32-bit boundary.

typedef struct { char c; short s; long l; } mix;
printf("\n sizeof struct mix = %2d _Alignof struct mix = %2d", sizeof(mix), _Alignof(mix));

 sizeof struct mix =  8 _Alignof struct mix =  4

  2    Alternative to curly braces - { }

If you prefer to use some other symbol in place of open and close braces { } (known as punctuators)), try <% %>. These symbols/punctuators must appear in pairs and cannot be mixed -- that is, either use { } or <% %> as separators but do not mix them up.

Other possible alternatives:
<: :> in place of [ ], %: in place of # and %:%: in place of ##.

% cat -n punct.c
     1  %:include <stdio.h>
     3  %:define SOMESTR "some random string"
     5  void main()
     6  <%
     7          char somestr<::> = SOMESTR;
     8          puts(somestr);
     9  %>
% cc -o punct punct.c
% ./punct
some random string

  3    Compound Literals

Literals represent fixed values in source code. For instance, "chip" is a string literal, 2 is an integer literal of type int, 3.14 is a floating point literal of type float and so on. Similarly compound literals represent the contents for slightly complex objects/aggregate types such as arrays or structures (and unions too though union is not classified as an aggregate type given that an object with union type can only contain one member at a time). The compound literal provides an unnamed object of the specified type whose value is specified by the initializer list.

(int []){ 10, 20, 30, 40 };

is a compound literal which is identical to an array initialization such as the following with the exception that it has no name.

int tens[] = { 10, 20, 30, 40 };

Compound literals are useful when an instance of aggregate or union type are used only once. For instance, compound literals as function arguments eliminate the need for temporary variables. The following example illustrates the syntax and usage of compound literals with the help of an aggregate type, struct.

% cat -n compound.c

     1  #include <stdio.h>
     3  typedef struct woo { int x; float y; char z; } foo;
     5  void print( foo f ) {
     6          printf("x = %d y = %0.2f z = %c\n",
     7                  f.x, f.y, f.z);
     8  }
    10  void main() {
    11          foo s = { 75, 32.32, 'Z' };  /* initialization using a temp variable */
    12          print( s );
    13          print( (foo) { 123, 55.01, 'K' } ); /* compound literal; no temp variable */
    14  }

% cc -o compound compound.c
% ./compound
x = 75 y = 32.32 z = Z
x = 123 y = 55.01 z = K


Unnamed objects (compound literals) have automatic storage duration associated with the block in which it was created -- that is, the memory buffer will be deallocated automatically upon block exit. If the compound literal occurs outside the body of a function, the object has static storage duration -- that is, its lifetime is the entire execution of the program. The expressions in the global/file scope compound literal are required to be constant.


Saturday, June 11, 2016
Solaris API for Locality Group Observability

[ Related: 1st part @ Locality Group Observability on Solaris ]

As of this writing, Solaris has limited support for lgroup observability using programmatic interfaces. Applications can use the lgroup API to traverse the locality group hierarchy, discover the contents of each locality group and even affect thread and memory placement on desired lgroups.

The man page for liblgrp(3LIB) lists out the currently supported public interfaces, and a brief description for most of those interfaces can be obtained by running man -k lgroup in a shell. In order to use this API, applications must link with locality group library, liblgrp(3LIB).

The following sample code demonstrates lgroup API usage by making several lgrp_*() calls including lgrp_device_lgrps() to find the locality groups that are closest to the specified I/O device.

# cat -n iodevlgrp.c

     1  #include <stdio.h>
     2  #include <stdlib.h>
     3  #include <assert.h>
     4  #include <sys/lgrp_user.h>
     5  #include <sys/types.h>
     7  int main(int argc, char **argv) {
     9          if (argc != 2) {
    10                  fprintf(stderr, "Usage: %s \n", argv[0]);
    11                  exit(1);
    12          }
    14          /*  lgroup interface version check */
    15          if (lgrp_version(LGRP_VER_CURRENT) != LGRP_VER_CURRENT) {
    16              fprintf(stderr, "\nBuilt with unsupported lgroup interface %d",
    17                  LGRP_VER_CURRENT);
    18              exit(1);
    19          }
    21          lgrp_cookie_t cookie =lgrp_init(LGRP_VIEW_OS);
    22          lgrp_id_t node = lgrp_root(cookie);
    24          /* refresh the cookie if stale */
    25          if ( lgrp_cookie_stale(cookie) ) {
    26                  lgrp_fini(cookie);
    27                  cookie =lgrp_init(LGRP_VIEW_OS);
    28          }
    30          int nlgrps = lgrp_nlgrps(cookie);
    31          if ( nlgrps == -1 ) {
    32                  perror("\n lgrp_nlgrps");
    33                  lgrp_fini(cookie);
    34                  exit(1);
    35          }
    36          printf("Number of locality groups on the system: %d\n", (nlgrps-1));
    38          /* lgroups closest to the target device */
    39          int numlgrps = lgrp_device_lgrps(argv[1], NULL, 0);
    40          if (numlgrps == -1 ){
    41                  fprintf(stderr, "I/O device: %s. ", argv[1]);
    42                  perror("lgrp_device_lgrps");
    43          } else {
    44                  printf("lgroups closest to the I/O device %s: ", argv[1]);
    45                  lgrp_id_t *lgrpids = (lgrp_id_t *)calloc(numlgrps, sizeof(lgrp_id_t));
    46                  assert(lgrpids != NULL);
    47                  lgrp_device_lgrps(argv[1], lgrpids, numlgrps);
    48                  for (int i = 0; i < numlgrps; ++i) {
    49                          printf(" %d ", lgrpids[i]);
    50                  }
    51                  free(lgrpids);
    52          }
    53          lgrp_fini(cookie);
    54          printf("\n");
    55          return 0;
    57  }

% cc -o iodevlgrp -llgrp iodevlgrp.c

% ./iodevlgrp /dev/ixgbe0
Number of locality groups on the system: 2
lgroups closest to the I/O device /dev/ixgbe0:  1

% lgrpinfo -d /dev/ixgbe0
lgroup ID : 1

% ./iodevlgrp /dev/ixgbe1
Number of locality groups on the system: 2
lgroups closest to the I/O device /dev/ixgbe1:  2

% lgrpinfo -d /dev/ixgbe1
lgroup ID : 2

Source: Oracle Solaris 11.3 Programming Interfaces Guide



This page is powered by Blogger. Isn't yours?