[Solaris] Cleanup all IPC resources

Run the following wrapper script with root user privileges.

for i in `ipcs -a | awk '{ print $2 }'`
do
 ipcrm -m $i 2> /dev/null
 ipcrm -q $i 2> /dev/null
 ipcrm -s $i 2> /dev/null
done

---

[Java, WebLogic] Find the process id (pid) of a WebLogic managed server instance

Run the following as the user who owns the process, or with root user privileges.

/usr/java/bin/jps -v | grep <WLS_server_name> | awk '{ print $1 }'

I think this tip is applicable on all supported platforms.

eg.,
Finding the pid of a managed server, bi_server1.

# /usr/java/bin/jps -v | grep bi_server1 | awk '{ print $1 }'
18659

# pargs 18659 | grep weblogic.Name
argv[7]: -Dweblogic.Name=bi_server1

---

[Oracle Database] Make Oracle ignore hints

Set the following hidden parameter.

_optimizer_ignore_hints=TRUE

(in general, Oracle does not recommend playing with hidden parameters. Check with Oracle support when in doubt).

---

[Oracle Database] Data Pump Export in a RAC environment fails with ORA-31693, ORA-31617, ORA-19505, ORA-27037 errors

eg.,

ORA-31693: Table data object "<SCHEMA>"."<TABLE>":"P_1147" failed to load/unload and is being skipped due to error:
ORA-31617: unable to open dump file "<FILE>" for write
ORA-19505: failed to identify file "<FILE>"
ORA-27037: unable to obtain file status
SVR4 Error: 2: No such file or directory
Additional information: 3

Workaround:

Add CLUSTER=N to the list of existing expdp options.

---

[Solaris, ZFS] Check the current ARC size and its breakdown

kstat -m zfs | grep size   (any user)  - OR -
echo ::arc | mdb -k | grep size  (root user)

echo ::memstat | mdb -k  (root user)


eg.,

# echo ::arc | mdb -k | grep size
size                      =       259391 MB
buf_size                  =         3218 MB
data_size                 =       249309 MB
other_size                =         6863 MB
l2_hdr_size               =            0 MB

# kstat -m zfs | grep size
        buf_size                        3375105344
        data_size                       261419672320
        l2_hdr_size                     0
        other_size                      7197048560
        size                            271991826224

# echo ::memstat | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   14646326            114424    4%
ZFS File Data            31948806            249600   10%
Anon                     24660113            192657    7%
Exec and libs                8912                69    0%
Page cache                 126052               984    0%
Free (cachelist)            24517               191    0%
Free (freelist)         263965754           2062232   79%
Total                   335380480           2620160

---

[Fusion Middleware] Disable Fusion Middleware Diagnostic Framework (DFW) Dump Sampling

The Diagnostic Framework in FMW 11g environments detect, diagnose and resolve critical errors such as uncaught exceptions, deadlocked threads and out of memory errors. It is enabled by default.

Though DFW is supposed to diagnose and fix some of the issues transparently, due to the inevitable bugs in [all kinds of] software and misconfigurations, sometimes DFW itself may become a major issue. For instance, there is a bug that reported very high system CPU time on a SPARC server where FMW 11g was running. Per the bug description, the system CPU utilization spikes every minute exactly at 00s of a minute, CPU utilization goes down within few seconds - but the pattern persists and the spiky behavior returns within a minute. Another symptom was the sudden drop in available swap space from tens of giga bytes to a few mega bytes when the CPU spike occurs. Upon close examination, it was found out that DFW in FMW is forking tens of jstack processes to collect the thread dumps from an equal number of java processes running in that FMW environment, causing the sudden spike in CPU (each process is busy gathering thread dumps at the same time) and a steep drop in swap space (each jstack process forked a jmap process. both jstack and jmap processes consume some virtual memory just like any other process). All this happened because DFW thought it found a critical issue, and it wasn't noticed or addressed by anyone including the administrators (DFW couldn't fix this particular issue on its own) - so, it kept gathering the diagnostic data continuously. In this example, DFW did the right thing but the diagnostic data collection frequency was too short - only one minute, that diminished the value of DFW and made it a liability. In such dire situations, probably it is best to disable the dump sampling feature of Diagnostic Framework tentatively while the underlying original issue is being fixed in that application environment. It can be enabled again when the critical issue was fixed, and no longer an issue.

Steps to disable Fusion Middleware Diagnostic Framework (DFW) Dump Sampling: (courtesy: Shashidhara Varamballi)

Method (1) Using WLST:

1. run wlst.sh
2. connect to the AdminServer
3. execute command: enableDumpSampling (enable=0, server='<server_name>')

Method (2) Manual editing of config file:

1. Edit $DOMAIN_HOME/config/fmwconfig/servers/<server_name>/dfw_config.xml
2. Change the "enabled" attribute from "true" to "false".
   eg.,
   <dumpSampling enabled="false">
3. Change the "useExternalCommands" attribute from "true" to "false".
   eg.,
   <threadDump useExternalCommands="false"/>
4. Save the changes

--

SEE ALSO:

Fusion Middleware Diagnostics weblog

---

[Solaris 11] Virtual-to-physical link (NIC) mapping

Check the output of /sbin/dladm show-phys (any user). By default, only those physical links that are available on the running system are displayed. Option -P shows the physical device and attributes of all physical links.

eg.,

$ /sbin/dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         1000   full      ixgbe0
net5              Infiniband           down       0      unknown   ibp2
net1              Ethernet             up         1000   full      ixgbe1
net6              Infiniband           down       0      unknown   ibp3
net4              Ethernet             up         10     full      usbecm2

$ /sbin/dladm show-phys -P
LINK              DEVICE       MEDIA                FLAGS
net8              ibp1         Infiniband           r----
net0              ixgbe0       Ethernet             -----
net7              ibp0         Infiniband           r----
net3              ixgbe3       Ethernet             r----
net5              ibp2         Infiniband           -----
net1              ixgbe1       Ethernet             -----
net6              ibp3         Infiniband           -----
net4              usbecm2      Ethernet             -----
net2              vsw0         Ethernet             r----

Labels: solaris oracle database fmw weblogic java dfw