Mandalika's scratchpad [ Work blog @Oracle | Stock Market Notes | My Music Compositions ]

Old Posts: 09.04  10.04  11.04  12.04  01.05  02.05  03.05  04.05  05.05  06.05  07.05  08.05  09.05  10.05  11.05  12.05  01.06  02.06  03.06  04.06  05.06  06.06  07.06  08.06  09.06  10.06  11.06  12.06  01.07  02.07  03.07  04.07  05.07  06.07  08.07  09.07  10.07  11.07  12.07  01.08  02.08  03.08  04.08  05.08  06.08  07.08  08.08  09.08  10.08  11.08  12.08  01.09  02.09  03.09  04.09  05.09  06.09  07.09  08.09  09.09  10.09  11.09  12.09  01.10  02.10  03.10  04.10  05.10  06.10  07.10  08.10  09.10  10.10  11.10  12.10  01.11  02.11  03.11  04.11  05.11  07.11  08.11  09.11  10.11  11.11  12.11  01.12  02.12  03.12  04.12  05.12  06.12  07.12  08.12  09.12  10.12  11.12  12.12  01.13  02.13  03.13  04.13  05.13  06.13  07.13  08.13  09.13  10.13  11.13  12.13  01.14  02.14  03.14  04.14  05.14  06.14  07.14  09.14  10.14  11.14  12.14  01.15  02.15  03.15  04.15  06.15  09.15  12.15  01.16  03.16  04.16  05.16  06.16  07.16  08.16  09.16  12.16  01.17  02.17  03.17  04.17 


Wednesday, November 28, 2007
 
Solaris: Which process is associated with a socket connection?

Consider the following real world scenario (copied from an e-mail message that I got from one of our partners):

[...] one process is waiting for a very long time in a send system call. It is sending on a valid fd but the question we have is that, is there any way to find who is on the other end of that fd? We want to know to which process is that message being sent to. [...]

Here is how I proceed in finding the other end of the socket, and the state of the socket connection with Mozilla's Thunderbird mail client in one end of the socket connection:
  1. Get the process id of the application
    % prstat
    PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
    22385 mandalik 180M 64M sleep 49 0 0:05:15 0.1% thunderbird-bin/5

  2. Run pfiles on the pid - it prints a list of open files including open sockets (pfiles is the Solaris supported equivalent of lsof utility).
    % pfiles 22385
    22385: /usr/lib/thunderbird/thunderbird-bin -UILocale C -contentLocale C
    Current rlimit: 512 file descriptors
    ...
    ...

    33: S_IFSOCK mode:0666 dev:280,0 ino:31544 uid:0 gid:0 size:0
    O_RDWR|O_NONBLOCK
    SOCK_STREAM
    SO_SNDBUF(49152),SO_RCVBUF(49640)
    sockname: AF_INET 192.168.1.2 port: 60364
    peername: AF_INET 192.18.39.10 port: 993

    ...
    ...
  3. Locate the socket id and the corresponding sockname/port#, peername/port# in the output of pfiles pid (see step #2).

    Here my assumption is that I know the socket id I'm interested in. In the above output, 33 is the socket id. One end of the socket is bound to port 60364 on the local host 192.168.1.2; and the other end of the socket is bound to port 993 on the remote host 192.18.39.10.

  4. Run netstat -a | egrep "<sockport#>|<peerport#> (get the port numbers from step 3); and check the state of the socket connection. If you see anything other than ESTABLISHED, it indicates trouble.
    % netstat -a | egrep "60364|993"
    solaris-devx-iprb0.60364 mail-sfbay.sun.com.993 48460 0 49640 0 ESTABLISHED

    If you want to see the host names in numbers (IP addresses), run netstat with option -n.
    %  netstat -an | egrep "60364|993"
    192.168.1.2.60364 192.18.39.10.993 49559 0 49640 0 ESTABLISHED

    Now since we know both ends of the socket, we can easily get the state of the socket connection at the other end by running netstat -an | egrep '<localport#>|<remoteport#>.

    If the state of the socket connection is CLOSE_WAIT, have a look at the following diagnosis: CPU hog with connections in CLOSE_WAIT.

Finally to answer ... which process is that message being sent to ... part of the original question:

Follow the above steps and find the remote host (or IP) and remote port number. To find the corresponding process id on the remote machine to which the other half of the socket belongs to, do the following:
  1. Login as root user on the remote host.
  2. cd /proc
  3. Run pfiles * | egrep "^[0-9]|sockname" > /tmp/pfiles.txt.
  4. vi /tmp/pfiles.txt and search for the port number. If you scroll few lines up, you can see the process ID, name of the process along with its argument(s).

_____________________
Technorati tags:
| |


Comments:
this was a brilliant piece of information......thanks....the blog is great!!!
 
Thanks for sharing this precious information, you're the man !
 
Wow! This was EXACTLY what I was looking for. Thanks!!
 
illumos utility: https://github.com/alhazred/ptcpstat
 
Post a Comment

Links to this post:

Create a Link



<< Home


2004-2017 

This page is powered by Blogger. Isn't yours?