(1) List all files in an uninstalled RPM

Try any of the following.

rpm -qlp 
rpm -qilp 
rpm2cpio  | cpio -t

(2) Extracting a specific file from an RPM

Steps:

1. List all files with path

   rpm2cpio  | cpio -t
   

   Note down the path of the file in focus as shown in above listing

2. Extract the target file

   rpm2cpio  | cpio -ivdm 
   

3. Look for the extracted file path under current directory

(3) Suspending and resuming a process

Suspend by running:

kill -TSTP 

and resume by running:

kill -CONT 

TSTP and CONT are job control signals for stopping the process and resuming it respectively.

Processes can trap and handle TSTP signal. If you don't want to give much control to the process, consider STP signal that cannot be handled, ignored, or blocked.

Related note: kill -l shows the complete list of signals.

(4) Accessing files that are shadowed by a mounted filesystem

When a directory containing one or more files becomes a mount point for a filesystem, files in the original directory are hidden and not accessible until the filesystem was unmounted. Following steps help accessing such hidden files.

High level steps:

unshare --mount
cd /
umount 
exit

(5) Socket statistics

On Linux, ss utility can be used to examine sockets statistics.

To see all open and non-listening sockets that have established connection, run ss command without any options.

-s option prints a summary of statistics.

-l option shows all listening sockets whereas -a option shows both listening and non-listening sockets.

-t, -u, -w and -x options show TCP, UDP, RAW and Unix domain sockets respectively.

ss       # all non-listening sockets in ESTAB state
ss -l    # listening sockets
ss -a    # listening and non-listening sockets
ss -t    # TCP sockets
ss -u    # UDP sockets
ss -w    # RAW sockets
ss -x    # UNIX domain sockets
ss -s    # summary of stastistics

Check man ss for all supported interesting options.

(6) Preserve symlinks when creating a Zip archive

zip --symlinks -r .zip 

(7) Exclude subdirectories when creating a Zip archive

Use -x of zip utility to exclude certain files or directories. Make sure to exclude not just directories but also the files underneath with "/*"

zip -r .zip  -x                        # exclude one file 
zip -r .zip  -x /* [-x /* .. -x /*]    # exclude one or more directories

(8) Extract specific file from a Zip archive

• Locate the path of the file of interest by listing all files in the zip archive
• Extract the file in focus with the help of -j option and optionally -d option to specify destination path

$ unzip -l oci-java-sdk-2.30.0.zip | grep full
241351083  12-21-2022 18:33   lib/oci-java-sdk-full-2.30.0-javadoc.jar
 36970074  12-21-2022 17:35   lib/oci-java-sdk-full-2.30.0-sources.jar
 93038832  12-21-2022 17:35   lib/oci-java-sdk-full-2.30.0.jar
106426811  12-21-2022 18:37   shaded/lib/oci-java-sdk-full-shaded-2.30.0.jar

$ unzip -j oci-java-sdk-2.30.0.zip lib/oci-java-sdk-full-2.30.0.jar -d /var/tmp/dep
Archive:  oci-java-sdk-2.30.0.zip
  inflating: /var/tmp/dep/oci-java-sdk-full-2.30.0.jar

(9) Renaming or copying files that start with a dash (-)

Problem and workaround with an example.

$ mv -07-01-05-40.pem api_key_pvt.pem
mv: illegal option -- 0
usage: mv [-f | -i | -n] [-v] source target
       mv [-f | -i | -n] [-v] source ... directory

$ mv -- "-07-01-05-40.pem" api_key_pvt.pem

$ ls -lrtha
total 16
-rw-r--r--@  1 gmandali  staff   1.7K Jun 30 22:41 api_key_pvt.pem

Double dash (--) is a bash builtin to indicate end of command options.

This workaround is applicable to cp (copy) and rm (remove) commands in addition to mv (move/rename) command.

Another option is to prefix the source file name with its pathname.

mv ./-somefile.txt somefile.txt

Posted by: Giri Mandalika.    # 11:53 PM 0 comments.     Tweet

[1] Check if a String Starts with Substring or a Wildcard Pattern

Reproducing from Advanced Bash-Scripting Guide -> Chapter 7. Tests.

The == comparison operator behaves differently within a double-brackets test than within single brackets.

[[ $a == z* ]]   # True if $a starts with an "z" (pattern matching).
[[ $a == "z*" ]] # True if $a is equal to z* (literal matching).

eg.,

% cat -n compare.sh
     1	#!/bin/bash
     2	
     3	echo "Wildcard matching"
     4	
     5	for name in rats star
     6	do
     7		[[ ${name} == st* ]] && echo ${name} starts with "st" \
     8	        	             || echo ${name} does not start with "st"
     9	done
    10	
    11	echo
    12	echo "Literal matching"
    13	
    14	for name in st* star
    15	do
    16		[[ ${name} == "st*" ]] && echo ${name} matches with "st*" \
    17	        	             || echo ${name} does not match with "st*"
    18	done

% ./compare.sh
Wildcard matching
rats does not start with st
star starts with st

Literal matching
st* matches with st*
star does not match with st*

[2] Floating-Point Arithmetic in Bash

Bash doesn't understand or support floating-point arithmetic; and assumes numbers containing a decimal point as strings. Following simple calculation shows incorrect result.

% let a=1/2
% echo $a
0

Rely on other tools such as bc, dc (calculators), sed, awk, python, perl, .. for floating-point arithmetic in bash.

eg.,

# bc
% bc <<< 'scale=2; 1/2'
.50

% a=$(bc <<< 'scale=2; 1/2')
% echo $a
.50

# perl
% echo print 1/2 | perl
0.5

# python
% echo 'print(1/2)' | python3
0.5

[3] Floating-Point Comparison in Bash

bash man page makes it clear that arithmetic evaluation is done in fixed-width integers so decimal/floating-point numbers need to rely on other tools such as bc for arithmetic evaluation (see previous tip).

Following examples highlight the problem with floating-point evaluation in bash. bash didn't have a problem with 1 < 2 comparision involving fixed-width integers but failed with an error when decimal numbers are involved in a similar comparison.

% echo $((1 > 2))
0

% echo $((1 < 2))
1

% echo $((1.0 < 2.0))
-bash: 1.0 < 2.0: syntax error: invalid arithmetic operator (error token is ".0 < 2.0")

Workaround or alternative is to rely on other tools. Following example performs the floating-point comparison with the help of bc (basic calculator).

% echo "(1.0 < 2.0)" | bc -l
1

% echo "(1.0 > 2.0)" | bc -l
0

[4] Multiline or Block Comments in Bash

I guess few examples are sufficient for this one.

# Example 1

: <<'ABC'
comment
comment
comment
ABC

# Example 2

: <<'EOF'
comment
comment
EOF

# Example 3

: '
comment
comment
'

# Example 4

<< '////'
comment
comment
////

Note that any line that starts with a colon (:) is a no-op. All the text between the delimiter identifier strings in heredoc will be redirected to a no-op command ("ABC" and "EOF" are delimiter identifiers in above examples 1 and 2). The space between colon and << is important and the string in quotes is equally important to stop the shell from expanding the variables and/or evaluating them.

(Source: stackoverflow)

Labels: Shell Scripting Tips UNIX Linux

Posted by: Giri Mandalika.    # 9:39 PM 0 comments.     Tweet

[1] Check if the content of file X is in another file Y

One option is to rely on comm utility to compare common lines in both files. comm requires both files to be sorted.

comm utility produces three text columns as output -> lines found only in file1, lines found only in file2 and lines found in both files. In order to check the existence of file X's content in file Y, simply run comm command with sorted conent of both files and suppress first two columns of output. Then simply match the line count from the output with the line count of file X.

eg.,

$ cat /tmp/fileX
Oh, yes I can make it now the pain is gone
All of the bad feelings have disappeared

$ cat /tmp/fileY
Gone are the dark clouds that had me blind
It's gonna be a bright bright
Bright bright sunshiny day
It's gonna be a bright bright
Bright bright sunshiny day
Oh, yes I can make it now the pain is gone
All of the bad feelings have disappeared
Here is that rainbow I've been praying for
It's gonna be a bright bright

$ comm -12 <(sort -u fileX) <(sort -u fileY)
All of the bad feelings have disappeared
Oh, yes I can make it now the pain is gone

$ comm -12 <(sort -u fileX) <(sort -u fileY) | wc -l
       2	<== number of lines common in both files

$ wc -l fileX
       2 fileX	<== numer of lines in source file

In above example, the numer of lines in source file match with the number of common lines in both files - so, we can assume that the content of file X is found in file Y.

To be clear, this is not really fool-proof but may work in majority of cases.

<(command) syntax invokes process substitution. Perhaps it is a topic for another post.

[2] Checking shell script syntax without running it

Specify -n option. No commands will be executed in this "noexec" mode.

bash -n <script>
sh -n <script>

eg.,

$ cat -n hello.sh 
     1	#!/bin/bash
     2	echo 'Hi Hola
     3	echo "Ni Hao Namaste"

Syntax Check

$ bash -n hello.sh 
hello.sh: line 2: unexpected EOF while looking for matching `''
hello.sh: line 4: syntax error: unexpected end of file

Execute to confirm

$ ./hello.sh 
./hello.sh: line 2: unexpected EOF while looking for matching `''
./hello.sh: line 4: syntax error: unexpected end of file

Labels: Shell Scripting Tips UNIX Linux

Posted by: Giri Mandalika.    # 8:23 PM 0 comments.     Tweet

Signals are used to notify processes of a synchronous or asynchronous event. When a signal is sent, operating system interrupts the flow of execution of target process to deliver the signal. If the process has a registered signal handler, the signal handler function will handle signals sent to the process. Otherwise, the default signal handler is executed. The default action for most signals is to terminate the process.

Almost all popular programming languages including Perl provide programming interfaces to send and receive signals. We will explore signal handling with a simple example in this blog post.

All Perl programs have access to a global variable %SIG hash that contains names or references of user-installed signal handlers. Each key in %SIG hash corresponds to a signal and associated value is the action to take when Perl runtime receives the corresponding signal. To ignore the signal, set the action to 'IGNORE'. To let the system perform the default action for the signal, set the action to 'DEFAULT'. To perform a desired action in response to a signal, write a signal handler with desired action(s) and set the action to execute that signal handler. To reiterate, a signal handler is just a function/subroutine including anonymous subroutines.

eg.,

$SIG{INT} = 'IGNORE' ;				# ignore signal INT
$SIG{INT} = 'DEFAULT';				# default signal handling for signal INT
$SIG{INT} =  sub { die "Caught sigint $!" };	# signal handler - anonymous subroutine
$SIG{INT}  = \&handle_sigint;			# signal handler - execute routine "handle_sigint"

Following sample code catches keyboard event Ctrl-C that generates SIGINT signal.

$ cat -n /var/tmp/sigint.pl
     1	#!/usr/bin/perl
     2	
     3	use strict;
     4	use warnings;
     5	
     6	sub handle_sigint {
     7	    print "Caught signal $! \n";
     8	}
     9	
    10	# install signal handler for SIGINT
    11	$SIG{INT}  = \&handle_sigint;
    12	
    13	print "Current time: " . localtime() . "\n";
    14	sleep(30);
    15	print "Time now: ". localtime() . "\n";

$ /var/tmp/sigint.pl
Current time: Thu Dec 31 16:35:00 2020
^CCaught signal Interrupted system call 
Time now: Thu Dec 31 16:35:07 2020

In this example, execution continued after the keyboard interrupt was handled. However note that not all signals allow execution to continue after the signal handler was executed.

Labels: Perl signals signal+handler

Posted by: Giri Mandalika.    # 4:46 PM 0 comments.     Tweet

Running pip commands encounter Could not find an activated virtualenv (required) error. Some of the options to get around this error are discussed in this blog post.

eg.,

# pip uninstall oci
ERROR: Could not find an activated virtualenv (required).

Try running the failed command by setting PIP_REQUIRE_VIRTUALENV to false. If the command succeeds and you don't want to encounter the same error in future, consider saving this environment variable in user's profile.

# PIP_REQUIRE_VIRTUALENV=false pip uninstall oci
Found existing installation: oci 2.18.0
Uninstalling oci-2.18.0:
  Would remove:
    /usr/lib/python2.7/site-packages/oci-2.18.0.dist-info/*
    /usr/lib/python2.7/site-packages/oci/*
Proceed (y/n)?

Another option is to run pip in isolated mode that ignores environment variables and user configuration.

# pip --isolated uninstall oci

Be aware that running in isolated mode won't ignore global configuration but only the user configuration. Therefore if there is a global configuration (say /etc/pip.conf) with a setting that forces virtual environment, --isolated option won't help.

eg.,

# pip --isolated uninstall oci
ERROR: Could not find an activated virtualenv (required).

In this case, examine global configuration by listing active configuration.

# pip config list
global.require-virtualenv='true'

Find out where the active configuration was loaded from by running pip with --verbose or -v (verbose) option.

# pip config list -v
For variant 'global', will try loading '/etc/xdg/pip/pip.conf'
For variant 'global', will try loading '/etc/pip.conf'
For variant 'user', will try loading '/root/.pip/pip.conf'
For variant 'user', will try loading '/root/.config/pip/pip.conf'
For variant 'site', will try loading '/usr/pip.conf'
global.require-virtualenv='true'

Finally edit global configuration to turn off the virtual environment requirement, and run the target pip command.

If editing the global configuration is not a viable option, consider ignoring global configuration tentatively while running target pip command(s).

eg.,

# PIP_CONFIG_FILE=/dev/null pip uninstall oci
Found existing installation: oci 2.18.0
Uninstalling oci-2.18.0:
  Would remove:
    /usr/lib/python2.7/site-packages/oci-2.18.0.dist-info/*
    /usr/lib/python2.7/site-packages/oci/*
Proceed (y/n)?

Labels: pip virtualenv python

Posted by: Giri Mandalika.    # 2:25 PM 0 comments.     Tweet

Audio & Widget courtesy: Spotify

Old Playlists:

    #1    #8   #14 (50s, 60s and 70s)    |    #2    #3    #4    #5 (80s)    |    #6    #7    #9    #16 (90s)    |    #11    #12 (00s)    |    #13 (10s) |    #10 (Instrumental) |    #15 (Cirque Du Soleil)    |    #17 (MJ)

Labels: New Wave 1980s Music

Posted by: Giri Mandalika.    # 12:12 AM 0 comments.     Tweet

Shuffling List Elements

Try shuffle method in random module.

shuffle() modifies the original sequence. In other words, shuffling happens in place and the function returns None.

To shuffle an immutable sequence -or- to save the shuffled sequence in a different object without disrupting original object holding the sequence, rely on random.sample() which returns a new shuffled list.

>>> from random import shuffle, sample
>>> x = [1, 2, 3, 4, 5]
>>> y = shuffle(x)
>>> y
>>> x
[3, 2, 5, 4, 1]
>>> y = sample(x, len(x))
>>> y
[5, 3, 2, 1, 4]
>>> x
[3, 2, 5, 4, 1]

Checking all Imported Modules

Try sys.modules.keys()

>>> import sys

>>> sys.modules.keys()
['copy_reg', 'sre_compile', '_sre', 'encodings', 'site', '__builtin__', 'sysconfig', '__main__', 'encodings.encodings', 'abc', 'posixpath', '_weakrefset', 
'errno', 'encodings.codecs', 'sre_constants', 're', '_abcoll', 'types', '_codecs', 'encodings.__builtin__', '_warnings', 'genericpath', 'stat', 'zipimport', 
'_sysconfigdata', 'warnings', 'UserDict', 'encodings.utf_8', 'sys', '_osx_support', 'codecs', 'readline', 'os.path', '_locale', 'signal', 'traceback', 
'linecache', 'posix', 'encodings.aliases', 'exceptions', 'sre_parse', 'os', '_weakref']

>>> import random

>>> sys.modules.keys()
['__future__', 'copy_reg', 'sre_compile', '_hashlib', '_sre', 'encodings', 'site', '__builtin__', 'sysconfig', '__main__', 'encodings.encodings', 'hashlib',
 'abc', 'posixpath', '_random', '_weakrefset', 'errno', 'binascii', 'encodings.codecs', 'sre_constants', 're', '_abcoll', 'types', '_codecs', 'encodings.__builtin__',
 '_warnings', 'math', 'genericpath', 'stat', 'zipimport', '_sysconfigdata', 'warnings', 'UserDict', 'encodings.utf_8', 'sys', '_osx_support', 'codecs', 'readline', 
'os.path', '_locale', 'signal', 'traceback', 'random', 'linecache', 'posix', 'encodings.aliases', 'exceptions', 'sre_parse', 'os', '_weakref']

Examining all Attributes in a Object

Try object.__dict__

__dict__ attribute contains all the attributes that describe the object. Helpful while debugging. It can be used to alter object's attributes too though I'm not sure if it is a recommended practice.

>>> class A:
...     def __init__(self):
...             self.a = 100
...             self.b = 200
... 
>>> a = A()
>>> a.__dict__
{'a': 100, 'b': 200}

When dealing with complex objects, __dict__ may falter with errors such as TypeError: Object of type xxx is not JSON serializable.

Labels: Python Tips

Posted by: Giri Mandalika.    # 11:28 PM 0 comments.     Tweet

Extracting the filename and extension from an absolute path

eg.,

Input: abspath="/var/tmp/localuserinfo.out"

Filename with extension:

$ basename ${abspath}
localuserinfo.out

Extension:

$ basename ${abspath##*.}
        -OR-
$ echo ${abspath##*.}
out

${variable##pattern} trims the longest match from the beginning of a string. Likewise ${variable#pattern} trims the shortest match from the beginning of a string.

Filename without extension:

$ basename ${abspath%.*}
localuserinfo

${variable%pattern} trims the shortest match from the end of a string. Similarly ${variable%%pattern} trims the longest match from the end of a string.

Ref:

• Bash Reference Manual -> Shell Parameter Expansion -> ${parameter##word}
• bash String Manipulations by Jim Dennis

Replacing commas with white space in a string

eg.,

Input: dummystr="Python,C,Java"

To replace all matches:

$ echo ${dummystr//,/ }
Python C Java

To replace only the first match:

$ echo ${dummystr/,/ }
Python C,Java

The difference is the extra '/' in pattern in first expression. If pattern begins with '/', all matches of pattern are replaced with string. Otherwise, only the first match is replaced.

Ref: Bash Reference Manual -> Shell Parameter Expansion -> ${parameter/pattern/string}

Sending array as argument to a shell function

Expand array before passing as argument to a function in a shell script. Sending the array name only passes the first element of the array.

Since $@ holds all arguments passed to the function, access the array argument with the help of $@ in the function definition. Using variables $1, $2, .., $n gives access to individual elements in the array.

$ cat numbers.sh 

#!/bin/bash

# print_array() .. prints all elements as expected
function print_array() {
 int_array=("$@")
 for elem in "${int_array[@]}"
 do
  echo $elem
 done
}

# print_array2() .. prints only one element .. first one in the array
function print_array2() {
 int_array=($1)
 for elem in "${int_array[@]}"
 do
  echo $elem
 done
}

numbers=(1 2 3 4)
echo "Attempting to print numbers array .."
print_array ${numbers[@]}
echo
echo "Attempt #2 with a different function .."
print_array2 ${numbers[@]}
echo
echo "Attempt #3 with only the array name as argument to print_array function .."
# following call prints only the first element of the array
print_array ${numbers}

$ ./numbers.sh 
Attempting to print numbers array ..
1
2
3
4

Attempt #2 with a different function ..
1

Attempt #3 with only the array name as argument to print_array function ..
1

Labels: Shell Scripting Tips UNIX Linux

Posted by: Giri Mandalika.    # 11:23 PM 0 comments.     Tweet

Length of a String Stored in a Variable

${#VARIABLE}

$ FRUIT="Orange"

$ echo ${#FRUIT}
6

In bash, variables that store values are known as parameters.

Remove First n Characters from a String

${VARIABLE:n}

$ PARAM="Quick Brown Fox"

$ echo ${PARAM:6}
Brown Fox

Remove Last n Characters from a String

${VARIABLE::-n}

$ PARAM="Quick Brown Fox"

$ echo ${PARAM::-6}
Quick Bro

Convert a String to Upper Case

${VARIABLE} | tr '[:lower:]' '[:upper:]' eg.,

$ NAME='Giri Mandalika'

$ echo ${NAME} | tr '[:lower:]' '[:upper:]'
GIRI MANDALIKA

Convert a String to Lower Case

${VARIABLE} | tr '[:upper:]' '[:lower:]' eg.,

$ NAME='Giri Mandalika'

$ echo ${NAME} | tr '[:upper:]' '[:lower:]'
giri mandalika

Labels: Shell Scripting Tips UNIX Linux

Posted by: Giri Mandalika.    # 9:22 PM 0 comments.     Tweet

The King of Pop needs no special introduction so here goes.

This playlist features music from five different albums of MJ -- Off the Wall (1979), Thriller (1982; "Best-selling album" in Guinness World Records), Bad (1987), Dangerous (1991) and HIStory (1995).

Audio & Widget courtesy: Spotify

Old Playlists:

    #1    #8   #14 (50s, 60s and 70s)    |    #2    #3    #4    #5 (80s)    |    #6    #7    #9    #16 (90s)    |    #11    #12 (00s)    |    #13 (10s) |    #10 (Instrumental) |    #15 (Cirque Du Soleil)

Labels: Micheal Jackson playlist

Posted by: Giri Mandalika.    # 1:31 AM 0 comments.     Tweet