Last weekend I've upgraded the OS on my Dell Inspiron 2650 laptop, from Solaris 10 GA to Solaris Nevada Build 17 (aka snv_17 or Solaris 11). In fact, I have no plans to upgrade it in imminent future, but thanks to an unexpected power outage, that made my laptop sleep for few hours, after running 57 days continuously. So, I took that opportunity to do, some of the things that I have been keep postponing for quite a few days:

1. Prepare the system to install OpenSolaris bits
   This is the major reason for the upgrade; but I'm not sure if I'll be able to play with OpenSolaris right away.The reason being, I couldn't afford significant down time of my note book, at the moment.
   
   
2. Experiment with large pages for text and initialized data segments (aka VMPSS or MPSS for Vnodes).[To DO:// detailed blog post on VMPSS]. VMPSS maps user applications, and library text and initialized data segment with large pages by default on SPARC. ie., the user doesn't have to enable it, like the way they do it for MPSS for Data. This was integrated into Solaris Express Build 15.
   
   Since no large pages are used by default on x86 systems, I've enabled it with set use_text_largepages=1 setting in /etc/system, once the system is ready. I can see that some of the processes (eg., Xorg, X-Chat, Mozilla) are using 4M pages.

   # pgrep Xorg
   2357
   
   # pmap -sx 2357 | grep 4M
   08800000    8192    8192    8192       -   4M rw---    [ heap ]
   09400000    4096    4096    4096       -   4M rw---    [ heap ]

3. To install vold (volume daemon)
   Somehow I missed this daemon, from my Solaris 10 GA installation, and has been mounting the CD-ROMs manually ever since
   
   
4. To post instructions for the system recovery from a run-time linker failure, on Solaris 10 and later

Upgrading Solaris 10 to Solaris Express build 17 (aka Solaris 11)

I have some bad news here with the upgrade. First of all, there is no way I can force it to overwrite the previously installed packages. It would be nice to have a blind upgrade option for home users, just like blindingly fast upgrade or BFU for OpenSolaris. Installer tried to backup most of the packages, and complains that there is not enough space to install the new/updated packages, even though I've reserved 9G disk space for root file system (/). It actually needs 4.5G disk space, for the complete OEM. Since it tried to backup most of the stuff from Solaris 10 GA installation, the disk space constraints gone up to ~9G. To accomodate the 300M+ shortage, I've sacrificed the existing swap partition.

Then I kicked off the installation and slept, with the hope that the installation from CD1 will be complete by the morning. But to my surprise, it flunked; there is absolutely 0% progress on the screen. Then I thought of restarting the whole process, by killing the installer windows. As soon as I killed the status window, the installation resumed and took three hours to complete the upgrade (just) from CD1.

After the reboot, many errors showed up on the screen and was stuck in the console mode (neither asking for CD #2, nor leaving the text mode). So I've decided to make a clear install. Other influencing factors include (but not limited to):

1. /opt/csw directory is a bit messy with three different window managers (KDE, Enlightenment and Xfce) and hundreds of 3^rd party softwares; and due to this most of the allocated 9G disk space, was filled up. It is not so easy to clean it up
   
   
2. Multiple copies (different versions) of GLib, GTK+, Pango, etc., got installed under /usr/lib, /opt/sfw, /opt/csw directories
   
   
3. Accidental deletion of package config files (.pc) from /usr/lib/pkgconfig directory, made it hard to build applications
   
   
4. Parts of JDS were broken, which I never bothered to fix, due to the reluctance emerged from high memory requirements. JDS is pretty cool; but GNOME components hog most of the physical memory, leaving no choice to the user except keeping the number of open windows to a pretty low number. Moreover it makes it real hard to run it on any machine with less than 512M memory

      NPROC USERNAME  SIZE   RSS MEMORY      TIME  CPU
       37 techno    864M  559M    91%   4:54:34 8.2%

   Let's hope that GNOME Memory Reduction project yields good results.

Solaris 11 clean installation

The real nice thing is the installer realized that there are some UFS partitions with some data; and gave an option to preserve the partitions of my choice. I chose my home directory; and from there the installation was a piece of cake. At the end, I felt that the (clean) installation was fast, compared to Solaris 10 GA installation.

Here are some hints for home users:
Choose:
* Solaris Interactive installation in the initial screen.
* Networked in Network Connectivity screen
* Use DHCP
* No to IPv6 & Kerberos, if running Solaris 10 is the only priority
* None to Name Service (assuming the machine is not in corporate network)
* CD/DVD to "Specify Media"
* Default Install to "Select type of install"
* No to "Do you need to override the system's default NFS version 4 domain name?"

Notes:

1. Once the installation from CD1 is complete, system boots up from the hard disk; and it may appear that it is not going to ask for CD2. Just be patient, and wait for it to ask you about CD2
   
   
2. You have to download the companion software CD from a different location. It can't be find in the same location, where you download the Solaris OS CD/DVD images. Here's the URL: http://www.sun.com/software/solaris/freeware/s10download.xml. If this image is on hard disk, you need not burn it into a CD; but can be mounted on a loopback file device with the help of lofiadm tool. This can be done while the installer is waiting for the CD or path to the installer on local/remote disk.

Relevant resources:

1. http://docs.sun.com/app/docs/doc/817-0544/6mgbagb1b?a=view (installation instructions)
   
2. http://shots.osdir.com/slideshows/slideshow.php?release=279&slide=1 (screen shots)

Post-installation troubleshooting (Solaris & JDS)

1. Missing JDS Linux partition
   • Solaris 11 overwrote the Master Boot Record (MBR) with its GRUB; hence JDS Linux was disappeared from the GRUB menu list. Luckily I had a backup of the grub config file from Linux partition. So a simple copy/paste to /boot/grub/menu.lst, did the job. Now I can boot up both JDS Linux and Solaris 11
   
   
2. Audio driver problem is back
   • Fixed it with the audio driver from http://www.tools.de/solaris/audio/
   
   
3. Unable to login to JDS as normal user
   • Able to login as root, but not as normal user. This behavior gave a hint that it has something to do with permissions. With little effort, found that ~/.ICEauthority file holds the X authorization records, and shouldn't be owned by root for normal users to login. Fixed it by changing the ownership of ~/.ICEauthority file
   
   
4. [JDS] Unable to launch any application from command line. The following error message appears on the console:
   

    Xlib: connection to ":0.0" refused by server
    Xlib: Invalid MIT-MAGIC-COOKIE-1 key
    Thu Aug 15 21:29:59 2005 Gtk-WARNING **: cannot open display: :0.0 at (eval 1) line 1.

   • When we try to run an X11 application, it reads the $DISPLAY variable, connects to the X11 server (local host, in this case) and provides the magic cookie by reading ~/.Xauthority
     
     
   • Fixed it by deleting ~/.Xauthority file (that's not a proper way, of course)
   
   
5. Couldn't burn CDs. It failed with the following error:
   

   # /opt/sfw/bin/cdrecord -scanbus
   Warning: Using USCSI interface.
   Warning: Volume management is running, medialess managed drives are invisible.
   /opt/sfw/bin/cdrecord: No such file or directory. Cannot open '/dev/rdsk/c1t0d0s2'. Cannot open SCSI driver.
   ...
   
   # truss /opt/sfw/bin/cdrecord -scanbus
   ...
   
   open("/dev/rdsk/c1t0d0s2", O_RDONLY|O_NDELAY)   Err#16 EBUSY
   ...

   • From the warning message, and the truss output, it is clear that some thing is blocking the CD recording device; and it is the "removable media manager" process vold
     
     
   • For burning CDRs on Solaris:
     1. stop the volume manager daemon vold, because it blocks the CD recorder device
           /etc/init.d/volmgt stop
        
     2. Burn the CD using cdrecord
        
     3. Restart the daemon
           /etc/init.d/volmgt start
   
   
6. gtkam didn't recognize the digital camera
   • Fixed it by binding the device (digital camera) to ugen (USB Generic Driver)
     
     
   • There's a piece of good news lately. The fix to the RFE 6213551: libusb support should just work was incorporated in Solaris Express 8/05 (Nevada Build 19). With this, the system takes care of automatic handling of unknown USB devices; and we should be able to plug in devices with USB interface like digital camera, scanner, .. with no effort (manual configuration) at all.

Goodies

1. Replaced the default GRUB splash screen with Chandan's Solaris Express theme
   
2. Replaced the default login screen with Chandan's Open World theme for OpenSolaris
   
3. Installed Evince, a document viewer by grabbing evince-0.3.2-SunOS5.8-i386-CSW.pkg.gz from blastwave.org

Now I have a nice little shiny Solaris desktop. As Solaris is getting better (from home users perspective) with each Solaris Express build, it is definitely worth the effort. For all new potential users, Solaris Express is the way to go.
________________
Technorati tag: Solaris

With the arrival of Solaris 10, the instructions of Recovering from a Runtime Linker Failure web page became obsolete. Since this problem appears to be very common, thought it would be useful to document it.

Symptom: For all commands, system responds (only) with the following error:
        Cannot execute /usr/lib/ld.so.1

Due to the Solaris Process Model Unification, all static binaries were removed from Solaris 10 distribution. One simple workaround is to pro-actively copying the essential static executables (esp., mv and cp), from an earlier version of Solaris to Solaris 10; and using 'em when/where needed.

However even if the session is gone by accident, no need to panic. The recovery steps are very simple (assuming there is a backup ld.so.1 on the machine, running Solaris 11):

1. Turn the machine on, with the installation media in the CD/DVD drive
   
2. Select Solaris in the initial GRUB screen
   
3. Select 6. Single user shell from the interactive menu, when you are prompted to select an installation type.
   
4. The disk on which the OS was installed will be shown, before you get to the
   single user command prompt.

   SunOS Release 5.11 Version snv_17 32-bit
   Copyright 1983-2005 Sun Microsystems, Inc.  All rights reserved.
   Use is subject to license terms.
   Booting to milestone "milestone/single-user:default".
   Configuring devices.
   Searching for installed OS...
           /dev/dsk/c1d0s0 --       Solaris 11 nv_17 X86

5. Respond with a No to the next question: Do you wish to automatically update boot archives? [y,n,?]
   
6. At this point, you will be at the command prompt. The next step is to mount the
   root file system (/) on some mount point say /mnt.
           # mount /dev/dsk/c1d0s0 /mnt
   
7. Restore the original run-time linker, ld.so.1.

   eg.,
           # cd /mnt/usr/lib
           # cp ld.so.1.orig ld.so.1

8. Unmount the root file system, eject the CD/DVD, and reboot

           # cd /
           # umount /mnt
           # reboot

Recovery steps on earlier versions of Solaris:

1. Recovering from a Run-time Linker Failure I
   
2. Recovering from a Run-time Linker Failure II

________________
Technorati tags: Solaris | OpenSolaris

At last I was able to watch the movie Book of Shadows - Blair witch 2, from the beginning to the end. It is a long, boring, sick flick which offers nothing, but lots of weirdness and insane brutality (I was holding back on The Texas Chainsaw Massacre, for the same reason; that's another story). Nothing really happens in the middle {of the movie}; and it is hard (and painful) to watch the illusions and delusions of the characters throughout the movie.

Anyway the main idea of this post is not to write a review, but to provide a web resource that has plenty of spoilers for clue less movies like these. After watching some movies (eg., Mulholland Dr., Lost Highway of David Lynch), we may have to use some part of the brain, to put together all the pieces that we just watched, to see if they make any sense at all. So, in general I read the reviews by other people, to see how/what they are thinking about the same movie. The web site Amuse Yourself, has the spoilers in one or two simple sentences, for hundreds of movies (some of 'em are silly, of course).

And it really amazes me to see people putting so much effort in analyzing and publishing the material on the web. eg., http://www.castleofspirits.com/blairwitch3.html. I believe, It certainly helps us learn new things, and think in different dimensions, if we couldn't match our thoughts with others'.
___________________
Technorati tags: movies | spoilers

The other day I was trying to build a large enterprise application with Sun Studio 9 C/C++ compilers and with profile feedback optimization. For some reason, iropt (code optimizer) failed with the error: compiler(iropt) error: heap_free: misaligned argument(0x666f2a3e). As a first step to see if this is a bug, I thought of installing SS9's backend patch 115983-04; but surprisingly the patch installer refused to install the patch. Even though it doesn't make any sense, I tried to install the patch 115983-01, before installing 115983-04, with no luck. Then I tried to install the packages, that are bundled with the patch, manually; even this attempt failed, as if there is no Sun Studio 9 compiler installed on the system.

Later I realized that I've installed RR (Revenue Release) version of Sun Studio 9, almost an year back. And since GA (General Availability) version was released some time after RR was released, I just want to give it a shot with the hope that this problem was fixed in GA bits. So, I 've uninstalled the RR version, and installed the GA copy that I downloaded from Sun Microsystems web site. The same serial# works for both RR & GA. This time I was able to install the patch 115983-04, without any issues; and iropt's error was gone too.

Moral of the story:
[Generic] Install the latest compilers and keep it up-to-date with all compiler patches from sunsolve.

[Specific to the current problem] Install GA copy of Sun Studio 9 compiler collection. If you have RR copy, you may not be able to patch it.

(Just for the records)
____________________
Technorati tags: Sun Studio | C | C++

Wouldn't it be nice if we know what exactly the compiler did when we specify a set of optimization flags on compile line? We specify a wide variety of compiler options with the hope that the resulting binary performs better. But unless we know for sure that using a certain compilation flag helps, most of the times it appears that the compiler is doing nothing, and even we may think that certain options were there just to give placebo effect to the user.

Sun ships a tool called er_src with Sun Studio compilers; so the users can examine or have a look at the optimizations done by the compiler(s). With the availability of er_src, compiler optimizations are no longer treated as a "black box" optimization.

Compiler logs most of its actions in stabs/dwarf of ELF object file, when the source was compiled with -g (debug) flag. Compiler generated messages are called "compiler commentary"; and the commentary will be interspersed in the source code, where the compiler did some optimization or transformation. When compiled with debug flag (-g), compiler commentary and the location of the source code will be stored in the object file (.o). er_src tool reads the source file and interleaves the compiler commentary in the output. Obviously the original source file has to be there in the path that was stored in the object file, during compilation. [Thanks to Chris Quenelle for the correction]

With er_src <object-file>, er_src dumps all the source along with compiler commentary. It is also possible to get the commentary and the disassembly for all or selected functions. Note that er_src even accepts Java class (.class) files.

Read the man page of er_src, for an explanation of how to read compiler commentary in object files to determine for which functions the compiler actually makes a substitution.

eg.,

 % cat string.c
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int __strcmp(const char *str1, const char *str2 ) {
   int rc = 0;

   for(;;) {
      rc = *str1 - *str2;
      if(rc != 0 || *str1 == 0) {
         return (rc);
      }
      ++str1;
      ++str2;
   }
}

int __strlen(const char *str) {
        int length = 0;

        for(;;) {
                if (*str == 0) {
                        return (length);
                } else {
                        ++length;
                        ++str;
                }
        }
}

char *__strreverse(const char *str) {
        int i, length = 0;
        char *revstr = NULL;

        length = __strlen(str);
        revstr = (char *) malloc (sizeof (char) * length);

        for (i = length; i > 0; --i) {
                *(revstr + i - 1) = *(str + length - i);
        }

        return (revstr);
}

int main() {
        printf("\nstrcmp(pod, podcast) = %d", __strcmp("pod", "podcast"));
        printf("\nstrlen(Solaris10) = %d", __strlen("Solaris10"));
        printf("\nreverse(Solaris10) = %s", __strreverse("Solaris10"));

        return (0);
}

To see the impact of compiling the same code with -O4 optimization, compile this source with -g -xO4 options

% cc -c -g -xO4 string.c

% er_src string.o
Source file: ./string.c
Object file: ./string.o
Load Object: ./string.o

     1. #include <stdio.h>
     2. #include <string.h>
     3. #include <stdlib.h>
     4.
     5. int __strcmp(const char *str1, const char *str2 ) {
        

   Bounds test for loop below moved to top of loop
     6.    int rc = 0;
     7.
     8.    for(;;) {
     9.       rc = *str1 - *str2;
    10.       if(rc != 0 || *str1 == 0) {
    11.          return (rc);
    12.       }
    13.       ++str1;
    14.       ++str2;
    15.    }
    16. }
    17.
    18. int __strlen(const char *str) {
        

   Bounds test for loop below moved to top of loop
    19.         int length = 0;
    20.
    21.         for(;;) {
    22.                 if (*str == 0) {
    23.                         return (length);
    24.                 } else {
    25.                         ++length;
    26.                         ++str;
    27.                 }
    28.         }
    29. }
    30.
    31. char *__strreverse(const char *str) {
    32.         int i, length = 0;
    33.         char *revstr = NULL;
    34.

   Function __strlen inlined from source file string.c into the code for the following line
   Bounds test for loop below moved to top of loop
    35.         length = __strlen(str);
    36.         revstr = (char *) malloc (sizeof (char) * length);
    37.

   Loop below scheduled with steady-state cycle count = 3 <= indicates that 
              software pipelining (modulo scheduling) has been applied
   Loop below unrolled 1 times
   Loop below has 1 loads, 1 stores, 0 prefetches, 0 FPadds, 0 FPmuls, and 0 FPdivs per iteration
    38.         for (i = length; i > 0; --i) {
    39.                 *(revstr + i - 1) = *(str + length - i);
    40.         }
    41.
    42.         return (revstr);
    43. }
    44.
    45.
    46. int main() {

   Function __strcmp inlined from source file string.c into the code for the following line
   Bounds test for loop below moved to top of loop
    47.         printf("\nstrcmp(pod, podcast) = %d", __strcmp("pod", "podcast"));

   Function __strlen inlined from source file string.c into the code for the following line
   Bounds test for loop below moved to top of loop
    48.         printf("\nstrlen(Solaris10) = %d", __strlen("Solaris10"));

   Function __strreverse inlined from source file string.c into the code for the following line
   Function __strlen inlined from source file string.c into inline copy of function __strreverse
   Bounds test for loop below moved to top of loop
   Loop below scheduled with steady-state cycle count = 3
   Loop below unrolled 1 times
   Loop below has 1 loads, 1 stores, 0 prefetches, 0 FPadds, 0 FPmuls, and 0 FPdivs per iteration
    49.         printf("\nreverse(Solaris10) = %s", __strreverse("Solaris10"));
    50.
    51.         return (0);
    52. }

From this listing, it is clear that the compiler tried its best to optimize the code by inlining the routines, and by doing loop unrolling and transformations. Of course, these are the things it is supposed to do with the documented -xO4 option. But since the compiler predictions may not be correct all the time, it is the responsibility of the user (say developer) to find out how the code being laid out; and if not satisfied with the outcome, to give more hints to the compiler with the compiler supported pragmas, profile feedback, rearranging the code etc.,

Now let's see what the compiler thinks about the same code, if we provide some feedback about run-time behavior of the program.

 % cc -g -xO4 -xprofile=collect -o string string.c

% ./string

strcmp(pod, podcast) = -99 <- returns 0 if matches
strlen(Solaris10) = 9
reverse(Solaris10) = 01siraloS

% ls -ld string.profile
drwxrwxrwx   2 build    engr         512 Aug  4 17:23 string.profile/

% cc -g -xO4 -xprofile=use:string -c string.c

% er_src string.o
Source file: ./string.c
Object file: ./string.o
Load Object: ./string.o

     1. #include <stdio.h>
     2. #include <string.h>
     3. #include <stdlib.h>
     4.
     5. int __strcmp(const char *str1, const char *str2 ) {
        
     6.    int rc = 0;
     7.
     8.    for(;;) {
     9.       rc = *str1 - *str2;
    10.       if(rc != 0 || *str1 == 0) {
    11.          return (rc);
    12.       }
    13.       ++str1;
    14.       ++str2;
    15.    }
    16. }
    17.
    18. int __strlen(const char *str) {
        
    19.         int length = 0;
    20.
    21.         for(;;) {
    22.                 if (*str == 0) {
    23.                         return (length);
    24.                 } else {
    25.                         ++length;
    26.                         ++str;
    27.                 }
    28.         }
    29. }
    30.
    31. char *__strreverse(const char *str) {
        
    32.         int i, length = 0;
    33.         char *revstr = NULL;
    34.

   Function __strlen not inlined because the profile-feedback execution count is too low
    35.         length = __strlen(str);
    36.         revstr = (char *) malloc (sizeof (char) * length);
    37.

   Loop below scheduled with steady-state cycle count = 3
   Loop below unrolled 1 times
   Loop below has 1 loads, 1 stores, 0 prefetches, 0 FPadds, 0 FPmuls, and 0 FPdivs per iteration
    38.         for (i = length; i > 0; --i) {
    39.                 *(revstr + i - 1) = *(str + length - i);
    40.         }
    41.
    42.         return (revstr);
    43. }
    44.
    45.
    46. int main() {
        

   Function __strcmp not inlined because the profile-feedback execution count is too low
    47.         printf("\nstrcmp(pod, podcast) = %d", __strcmp("pod", "podcast"));

   Function __strlen not inlined because the profile-feedback execution count is too low
    48.         printf("\nstrlen(Solaris10) = %d", __strlen("Solaris10"));

   Function __strreverse not inlined because the profile-feedback execution count is too low
    49.         printf("\nreverse(Solaris10) = %s", __strreverse("Solaris10"));
    50.
    51.         return (0);
    52. }

This time, the compiler thought it is not very beneficial to inline the routines because the execution frequency of those routines is too low (1 in this case); and of course that's what profile feedback optimization is supposed to do ie., optimizing the code, based on the run-time feedback. In this example, both -xO4 and -xprofile (Profile Feedback Optimization) are working together to make the best decision.

Few more examples:
To list all functions from the given object:

% er_src -func string.o

Functions sorted in lexicographic order

Load Object: 

    Address     Size        Name

  0x00000000        64      __strcmp
  0x00000040        72      __strlen
  0x00000088       184      __strreverse
  0x00000140       372      main

To print the compiler commentary only for changes involved inlining:

% er_src -cc inline string.o
       ...
       ...

    29. }
    30.
    31. char *__strreverse(const char *str) {
    32.         int i, length = 0;
    33.         char *revstr = NULL;
    34.

   Function __strlen inlined from source file string.c into the code for the following line
    35.         length = __strlen(str);
    36.         revstr = (char *) malloc (sizeof (char) * length);
    37.
       ...
       ...

    43. }
    44.
    45.
    46. int main() {

   Function __strcmp inlined from source file string.c into the code for the following line
    47.         printf("\nstrcmp(pod, podcast) = %d", __strcmp("pod", "podcast"));

   Function __strlen inlined from source file string.c into the code for the following line
    48.         printf("\nstrlen(Solaris10) = %d", __strlen("Solaris10"));

   Function __strreverse inlined from source file string.c into the code for the following line
   Function __strlen inlined from source file string.c into inline copy of function __strreverse
    49.         printf("\nreverse(Solaris10) = %s", __strreverse("Solaris10"));
    50.
    51.         return (0);
    52. }

To print disassembly:

% er_src -disasm all -1 string.o
---------------------------------------
Annotated disassembly
---------------------------------------
Source file: ./string.c
Object file: ./string.o
Load Object: ./string.o

     1. #include <stdio.h>
     2. #include <string.h>
     3. #include <stdlib.h>
     4.
     5. int __strcmp(const char *str1, const char *str2 ) {
        
        [ 5]        0:  ldsb        [%o0], %o4
        [ 5]        4:  mov         %o0, %o3

   Bounds test for loop below moved to top of loop
     6.    int rc = 0;
     7.
     8.    for(;;) {
     9.       rc = *str1 - *str2;
        [ 9]        8:  ldsb        [%o1], %o5
        [ 9]        c:  subcc       %o4, %o5, %o0
    10.       if(rc != 0 || *str1 == 0) {
        [10]       10:  bne,pn      %icc,0x38
        [10]       14:  cmp         %o4, 0
        [10]       18:  be,pn       %icc,0x38
    11.          return (rc);
    12.       }
    13.       ++str1;
        [13]       1c:  inc         %o3
        [ 9]       20:  ldsb        [%o1 + 1], %o5
        [ 9]       24:  ldsb        [%o3], %o4
    14.       ++str2;
        [14]       28:  inc         %o1
        [ 9]       2c:  subcc       %o4, %o5, %o0
        [10]       30:  be,pt       %icc,0x18
        [10]       34:  cmp         %o4, 0
        [11]       38:  retl
        [11]       3c:  nop
    15.    }
    16. }
    17.
       ...
       ... 

__________________
Technorati tags: Sun Studio | C | C++