The other day I was chasing some lock contention issue with a 64-bit application running on Solaris 10 Update 1; and stumbled with an unexpected dbx crash. My intention is to find the hot locks with the help of dbx's threads, syncs, sync -info <address> commands. (Of course, most of this information can be obtained with modular debugger, mdb, too - but that's a different story). So when I attached the running process to dbx, it crashed, while loading some of the libraries with the following error:

dbx: internal error: signal SIGBUS (invalid address alignment)
dbx's coredump will appear in /tmp

As the error message indicates, the problem is with the alignment of 64 bits (in this case) on 64-bit boundary of the object being loaded. From the bug report dbx dumped core when hitting a misaligned load in libdl.so.1, it appears that elfsign is responsible for the misaligned section headers of Solaris 10's 64-bit libraries. Since then Sun fixed the elfsign, and released majority of properly aligned 64-bit libraries as patches through sunsolve web site. So, if the system running Solaris 10 (or later) is not patched with the fixed libraries, it is still possible to see dbx crashes. This is because 64-bit dbx expects the libraries to be aligned properly on their boundaries -- it just crashes when it encounters any misaligned libraries.

Perl script to check misaligned section headers in 64-bit objects

(Credit: Chris Quenelle)

#!/usr/bin/perl

use File::Find;
sub wanted {
  return unless -x && -f;
  return unless /.*\.so\.[0-9]$/;
  return unless `/bin/file $_ | grep 64-bit`;
  $out = `elfdump -e $_ | grep e_shoff`;
  $out =~ m/e_shoff:\s+(0x[0-9a-f]+)\s/;
  if ($1 =~ m/[4c]$/) {
     print "bad alignment in file: $File::Find::name\n";
  }
}

print "Looking for bad ELF section header table alignment in 64-bit files\n";
find(\&wanted, ( "/lib", "/usr/lib" ));

Make this script executable with chmod +x <script>.pl command, and run it directly ie., ./<script>.pl (doesn't need root privileges)

Fix to Sun Studio 11's 64-bit dbx

If the above perl script returns some library names as output, it is very likely that the existing 64-bit dbx may not work on Solaris 10 (and later). Chris Quenelle of Sun Microsystems, posted a fixed 64-bit Studio 11 dbx binary, that doesn't crash even if it encounters misaligned elf section headers, in his blog. The binary can be downloaded from: http://mediacast.sun.com/share/quenelle/dbx.ss11.align.fix.gz.

Note that this is an unsupported binary. Do not copy it over the existing Studio 11 64-bit dbx. Just place it anywhere else, and make it an executable. Then simply attach the running process to the new 64-bit dbx binary.

Quick summary

Sun Studio's 64-bit dbx (any version) may crash on Solaris 10 (and later) platform. An unsupported fix can be downloaded from http://mediacast.sun.com/share/quenelle/dbx.ss11.align.fix.gz

More information in Chris Quenelle's web log:

1. Two bad Solaris bugs that affect dbx users
   
2. misligned ELF section headers

---

Example showing the dbx crash

% /home/dev/SS11/SUNWspro/prod/bin/sparcv9/dbx - 14901
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.1' in your .dbxrc
Reading -
Reading ld.so.1
Reading libmdxmembernamecache.so
Reading libnqcachestorage.so
Reading libnqcryptography.so
Reading libnqperf.so
Reading libnqportable.so
Reading libnqscache.so
...
...
Reading libsocket.so.1
Reading librt.so.1
Reading libCstd.so.1
Reading libCrun.so.1
Reading libm.so.1
Reading libthread.so.1
Reading libc.so.1
Reading libdl.so.1
Reading libnsl.so.1
Reading libaio.so.1
Reading libmd5.so.1
Reading libm.so.2
Reading libc_psr.so.1
Reading en_US.ISO8859-1.so.3

dbx: internal error: signal SIGBUS (invalid address alignment)
dbx's coredump will appear in /tmp
detaching from process 14901
Abort (core dumped)

Technorati Tags
Sun Studio | dbx | Solaris

Dr. Edsger W. Dijkstra, who coined the word Structured Programming, doesn't need a special introduction, to all those people in the field of computing. Shortest Path algorithm, semaphores solution to the classic Dining Philosophers problem, notion of invariants, pre- and post-conditions for the correctness of programs, strong case against GO TO statement are some of his contributions to Computer Science. Dr. Dijkstra was also a principal contributor to the development of the ALGOL programming language.

Dr. Dijkstra was known for the habit of writing his opinions on programming/computing (computer science, in general) in a series of manuscripts called EWDs. Dr. Dijkstra had an unique style of writing, that I always enjoyed reading. His outstanding work helps us understand the evolution of computer science, up to some extent. The University of Texas at Austin (UTA) is kind enough to archive majority of Dr. Dijkstra's writings, and make them available in PDF format, at: In Pursuit of Simplicity: The Manuscripts of Edsger W. Dijkstra web page.

Technorati Tags
Computer Science | Computing | Manuscript

Solaris 10 Update 1 (aka Solaris 10 1/06) is now available for download, at Sun downloads web site. One of the notable features of this major release is the large page support for instructions. It is a known fact that dTLB/dTSB, iTLB/iTSB miss rates is a bottleneck for very large applications. The system has to spend some time serving the virtual-to-physical memory translations. These translations are expensive, and the time spent in these translations are counted in usr% (CPU).

Until the release of Solaris 9, 8K is the only supported page size on Solaris. Solaris 9 introduced Multiple Page Size Support (MPSS), for the applications to use any of the possible hardware address translation sizes supported by the system (use pagesize -a to get the supported page sizes). However on Solaris 9, MPSS for user applications is provided only for anonymous memory eg., heap, stack, ISM, MAP_ANON mappings.

Solaris 10 introduced dynamic TSB support, which helps reducing the dTSB miss rate considerably. However, on Solaris 10, user iTLB misses are still a major bottleneck. In Solaris, user text segments are implemented with memory mapped files, and can potentially be mapped with large pages.

VMPSS - Large Pages for Executables and Files

Now with Solaris 10 Update 1, Sun introduced MPSS for executables (text/instructions) and files, which is referred as MPSS for Vnodes or simply VMPSS. VMPSS extended MPSS support to regular file mappings in order to reduce iTLB misses for text, dTLB misses for initialized data, and also to reduce dTLB misses in general for other memory mapped files. Due to VMPSS, large applications may get large pages automatically, for text and initialized data segments, based on the segment size and the page size support capabilities of the underlying hardware. So if you have a large application running on Solaris 10 Update 1, the application might already be enjoying the benefits of VMPSS, and hence you may observe noticeable improvement in the run-time performance of the application. To check if the application is taking advantage of VMPSS, run pmap -sx <pid>, and observe the page sizes under Pgsz column of pmap output.

Default page sizes, and tunables on different platforms

UltraSPARC II

No large pages are used by default on US-II systems. To enable the default use of 64K or 4M pages for text, add the following lines in /etc/system {and reboot the machine once}:

        set use_text_pgsz64k 1
        set use_text_pgsz4m 1

To enable the default use of 64K pages for data, add set use_initdata_pgsz64k 1 as well, to /etc/system

UltraSPARC III/III+/IV/IV+

4M is the only large page size supported by default for text mappings. If the application exhibits performance regression with this default behavior, the default use of 4M text pages can be disabled by adding the following line in /etc/system:

        set use_text_pgsz4m 0

No large pages are used by default for initialized data segments, on these platforms.

UltraSPARC T1 (aka Niagara)

64K and 4M page sizes for text, and 64K page sizes for initialized data are used by default.

In case of regressions, add the following lines in /etc/system.

To disable the defalt use of 64K or 4M text pages:

        set use_text_pgsz64k 0
        set use_text_pgsz4m 0

To disable the default use of 64K data pages:
        set use_initdata_pgsz64k 0

You can find more information in Denis Sheahan's blog post: UltraSPARC T1 large page projects in Solaris

x86

No large pages are used by default on x86. The default use of 2M text pages on PAE machines or 4M pages on non-PAE machines can be enabled with the following setting in /etc/system:

        set use_text_largepages 1

----
All the above mentioned {VMPSS} design policies should result in the use of large pages when they are most expected to help performance, and limit potential regressions from large pages when they are unlikely to improve performance.

Acknowledgements:
Aleksandr Guzovskiy, Sun Microsystems

Technorati tags
Solaris | OpenSolaris | Sun

(Note: the words "feedbin" and "feedback" are used interchangeably throughout this blog post)

It is a known fact that by default compiler arranges to create one profile feedback file for each profiled executable. A profiled binary contains compiler inserted instrumentation code, to collect run-time data (like block execution counts) from it. Compiler inserts instrumentation code, when the original code is compiled with -xprofile=collect compiler option.

The default behavior is good enough for small programs or applications with very few executables. However for very large real world applications with tens of executables, having too many feedback files poses slight inconvenience in use phase, where these feedback files are specified on compile line with -xprofile=use:<path_to_feedback_file> option, to generate highly optimized binaries. For example, if the application consists of twenty executables, we need to have twenty -xprofile=use options on the compile line, like:

% cc -xO4 -xprofile=use:feedback1 -xprofile=use:feedback2 ... -xprofile=use:feedback19 -xprofile=use:feedback20 -o optimizedbin application.c

There are two major inconveniences with this approach:

1. If make file grabs all compiler flags from environment variables like CFLAGS, it may not be possible to specify all -xprofile=use options in a single CFLAGS, due to the underlying shell restrictions on the number of characters per variable.
   
   
2. The compile line may become too long, and look ugly with too many -xprofile=use flags.

To get around these inconveniences, it is suggested to use compiler supported environment variables SUN_PROFDATA_DIR and SUN_PROFDATA in profile data collection phase, to request the profiler to write all the profile feedback data from different executables in to a single feedbin file, instead of creating one for each executable. If these environment variables are set, the profiler writes the feedback data in to the file pointed by SUN_PROFDATA, under directory SUN_PROFDATA_DIR ie., the feedback data from all executables will be written in to SUN_PROFDATA_DIR/SUN_PROFDATA.

The following example illustrates the default behavior, as well as the behavior with SUN_PROFDATA and SUN_PROFDATA_DIR environment variables.

eg.,

%cat a.c
#include <stdio.h>

int main()
{
    printf("In a.c\n");
    return (0);
}

%cat b.c
#include <stdio.h>

int main()
{
    printf("In b.c\n");
    return 0;
}

% cc -xO2 -xprofile=collect -o a a.c
% cc -xO2 -xprofile=collect -o b b.c

Default behavior

% setenv SUN_PROFDATA_DIR /tmp/default

% ./a
In a.c

% ./b
In b.c

% ls /tmp/default
a.profile/  b.profile/

Observe that there are two profiles, one for each executable. a.profile holds the feedback data for the executable "a"; and b.profile holds the feedback data for the executable "b".

To use the feedback data, the programs have to be compiled as follows:

% cc -xO2 -xprofile=use:/tmp/default/a -o a a.c
% cc -xO2 -xprofile=use:/tmp/default/b -o b b.c

Requesting for a single feedback file to hold all profile data

% setenv SUN_PROFDATA_DIR /tmp/consolidate
% setenv SUN_PROFDATA singlefeedbin.profile

These variables inform the profiler to write all feedback data from different profiled binaries into a single feedbin file: /tmp/consolidate/singlefeedbin.profile.

%mkdir /tmp/consolidate

%./a
In a.c

%./b
In b.c

% ls /tmp/consolidate
singlefeedbin.profile/

Observe that the singlefeedbin.profile's feedbin holds the feedback data for both executables "a" and "b". If there are more profiled executables, shared libraries, all the feedback data will be appended to this feedbin file.

To use this profile, simply:

% cc -xO2 -xprofile=use:/tmp/consolidate/singlefeedbin -o a a.c
% cc -xO2 -xprofile=use:/tmp/consolidate/singlefeedbin -o b b.c

Related blog posts:

1. Sun Studio C/C++: Profile Feedback Optimization (PFO): part 1 | part 2
   
2. Sun Studio 11: Asynchronous Profile Feedback Data Collection

Acknowledgements:
Chris Aoki, Sun Microsystems
___________________
Technorati tags: Sun Studio

Here's another instance to show that Solaris is gaining quite a momentum lately, on x86 and SPARC platforms. All these days we were fiddling with RealPlayer binaries targeted for another platform, just to make them work on Solaris x86 platform. Real Player native binaries are now available for both x86 and SPARC versions of Solaris, at Helix community web site. Hooray!

These binaries can be downloaded from the following location:
http://forms.helixcommunity.org/helix/builds/?category=realplay-stable
________
Technorati tag: Solaris | OpenSolaris

At optimization level 2 (ie., -O2 or -xO2) or higher, Sun Studio C/C++ compilers recognize loop-invariant code and re-arranges it in such a way that it is executed less often (mostly it may get executed only once) than it would be within the loop.

Loop-invariant code consists of statements which could be moved to before or after the loop, without affecting the semantics of the program. In simple words, loop-invariant code is the set of instructions inside a loop whose results do not change; and hence can safely be moved out of the loop.

Consider the following simple example.

% cat loopinvariant.c
#include <stdio.h>

int main()
{
        long i = 0, max = 5000000;
        int count = 20;
        const float pi = 3.14;

        while (i < (max + 100))
        {
                i = i + (count + 1) * (int) pi + 2;
                printf("\ni = %ld", i);
        }
        return (0);
}

% cc -o loopinvariant loopinvariant.c

% timex loopinvariant
...
...
i = 5000060
i = 5000125
real       10.56
user        0.11
sys         0.39

In the above example, the following instructions are loop-invariant:

1. (max + 100) in while (i < (max + 100)), and
   
2. (count + 1) * (int) pi + 2 in i = i + (count + 1) * (int) pi + 2;

At O2 optimization level (or higher), compiler recognizes this kind of code, and may rearrange it as follows (check the highlighted code -- if you can understand assembly code, check the assembly by compiling it with cc -S loopinvariant.c):

#include <stdio.h>

int main()
{
        long i = 0, max = 5000000;
        int count = 20;
        const float pi = 3.14;

        long maxvalue = (max + 100);
        int tempval = (count + 1) * (int) pi + 2;

        while (i < maxvalue)
        {
                i = i + tempval;
                printf("\ni = %ld", i);
        }
        return (0);
}

So, when the original code is compiled with -xO2 optimization (or higher), it runs a little faster.

% cc -xO2 -o loopinvariant2 loopinvariant.c

% timex loopinvariant2
...
...
i = 5000060
i = 5000125
real        9.87
user        0.10
sys         0.38

Note:
When -xprofile=use option is used, Sun Studio C/C++ compilers make sure that the loop-invariant code will not be moved from low to high execution places.
___________________
Technorati tags: C | C++ | Sun Studio | Programming