Bash: Use for loop to iterate over file without using cat

Introduction:

In this very brief post I just wanted to demonstrate a way to use the for loop to iterate over a file without using the cat command.

Here is a file containing 10 numbers, 1 number on each line.

cat seq.txt
1
2
3
4
5
6
7
8
9
10

If I wanted to loop over it using for loop in bash I could type:

for i in `cat seq.txt`; do echo $i ;done

But what if I don't want to read in the file using cat command?

I could use input redirection to read in the file contents as shown in the below example:

for i in $(<seq.txt); do echo $i ; done

1

2

3

4

5

6

7

8

9

10

                            

One more way of avoiding the cat command would be to read the file into an array and then loop over the array as shown below.

readarray -t ARR < seq.txt

for i in "${ARR[@]}"; do echo $i; done

1

2

3

4

5

6

7

8

9

10

Conclusion

There is nothing wrong with using the cat command to read in file contents while iterating over it in a for loop. I just found redirecting directly from STDIN to be more elegant and I would try to use it more often.

Merge two consecutive lines using awk

Introduction:
As system admins we spend a lot of our time working with files. While doing so we may come across situations wherein we may need to manipulate the content of a file or the output of a command to suit our needs. I came across such a situation recently wherein I had to run nslookup on a couple of hosts and get the hostname the IP address printed on the same line with a colon and a space acting as a delimiter. As with many things in UNIX/Linux there is more than one tool for the job. My task could've been accomplished using sed or perl but I chose to go with awk.

The command:

[ssuri@ulabtestpinfra09:~] $ for i in `cat<<EOF
> ulabtestdinfap31
> ulabtestdinfap35
> ulabtestdinfap37
> EOF`
> do nslookup   $i | awk '/Name|Address: 10/  {print $2}' | awk '!(NR%2){print p ": " $0 }{p=$0}'
> done
ulabtestdinfap31.example.org: 10.47.84.34
ulabtestdinfap35.example.org: 10.47.64.58
ulabtestdinfap37.example.org: 10.47.216.14
[ssuri@ulabtestpinfra09:~] $


Explanation:
As you might've noticed I've used awk twice. The first use is basic so I won't get into it. Now let's talk about the second awk. NR represents the number of rows. % is the modulus operator (i.e. a%b is the remainder when a is divided by b)... (NR%2) is the modulus of NR by two, i.e. is true when NR is even and false when odd. !(NR%2) is true when NR is odd, thus. !(NR%2){print p ": " $0 } means the program will print the line concatenated with the variable p, only on odd lines. {p=$0} means that on every line, p is set to be the current line (but only after printing the current and previous line if the current line is odd).


Conclusion:
This concludes our quick article on how we could use awk to merge or join two consecutive lines. I hope that you found this post to be useful.

Perl one liner to extract LUNid and disk alias from /etc/multipath.conf file

Introduction:

We may run into situations wherein we need to fetch the LUN id and alias mapping for disks under multipath on a Linux machine. Obtaining this data manually would prove to be cumbersome. One way to fetch this data would be to use the combination of grep and paste commands. But I felt that my Perl was getting a bit rusty so I decided to go the Perl way.

First take a look at the entries from the sample file.

        multipath {
                wwid                    36000d3100008f20000000000000001f4
                alias                   dvd-rhel5-2-64
        }
        multipath {
                wwid                    36000d3100008f20000000000000001f6
                alias                   dvd-rhel5-2-32
        }
        multipath {
                wwid                    36000d3100008f20000000000000003e2
                alias                   dvd-rhel4-7-32
        }

The above output shows the multipath stanzas for a couple of disks. We are basically interested in the wwid and alias section. To extract the required information we will be using the below combination of two Perl one liners.

[root@sahil-lab1 ~]# cat mpath.cf  | perl -ne 'print if(/wwid|alias/);' | perl -pne 'if($.%2){s/\n/\t/;}'

                wwid                    36000d3100008f2000000000000000356                       alias                   aleppo

                wwid                    36000d3100008f2000000000000000a3a                       alias                   dc2tst

                wwid                    36000d3100008f2000000000000000b02                       alias                   dc1tst

                wwid                    36000d3100008f20000000000000003cf                       alias                   algiers

                wwid                    36000d3100008f2000000000000000397                       alias                   algiers_local

                wwid                    36000d3100008f200000000000000004b                       alias                   chicago

                wwid                    36000d3100008f200000000000000004c                       alias                   chicago_mysql

                wwid                    36000d3100008f200000000000000004d                       alias                   chicago_local

                wwid                    36000d3100008f200000000000000004e                       alias                   chicago_assets

                wwid                    36000d3100008f20000000000000001f4                       alias                   dvd-rhel5-2-64

                wwid                    36000d3100008f20000000000000001f6                       alias                   dvd-rhel5-2-32

                wwid                    36000d3100008f20000000000000003e2                               alias                   dvd-rhel4-7-32

[root@sahil-lab1 ~]#

You could further add an additional Perl one liner to print only the alias and LUN id as shown below.

[root@sahil-lab1~]# cat mpath.cf  | perl -ne 'print if(/wwid|alias/);' | perl -pne 'if($.%2){s/\n/\t/;}' | perl -F"\s+" -lane 'print "$F[4]  $F[2]"' 

 aleppo  36000d3100008f2000000000000000356

dc2tst  36000d3100008f2000000000000000a3a

dc1tst  36000d3100008f2000000000000000b02

algiers  36000d3100008f20000000000000003cf

algiers_local  36000d3100008f2000000000000000397

chicago  36000d3100008f200000000000000004b

chicago_mysql  36000d3100008f200000000000000004c

chicago_local  36000d3100008f200000000000000004d

chicago_assets  36000d3100008f200000000000000004e

dvd-rhel5-2-64  36000d3100008f20000000000000001f4

dvd-rhel5-2-32  36000d3100008f20000000000000001f6

dvd-rhel4-7-32  36000d3100008f20000000000000003e2

[root@sahil-lab1~]#

Explanation:

The first one liner simply prints lines containing the strings wwid or alias.

The next one liner loops over the content piped from the previous one liner and uses $. variable denoting the line number. If the remainder of the division of the line number by 2 is not 0 i,e. the line is odd, then the new line after the end of the line gets replaced by a tab thereby combining the even and odd numbered lines together. 

The last one liner invokes the awk like functionality available with Perl one liners. The -F flag in conjunction with -a flag allow us to split lines based on a delimiter and the individual strings in the line get stored in an array variable named @F and we can extract the fields by using the scalar elements that make up the @F array.

Conclusion:

I'm sure there are easier and perhaps more compact versions of Perl one liners out there to accomplish this task. I would appreciate any suggestions and feedback on this approach of extracting the required fields from the /etc/multipath.conf file.

Get result of two or more commands in the same line and redirect the result to a file

Forgive me for the long title of the post but titles should be descriptive of the content that follows and I just wanted to make sure of it for this article.

Generally when we chain commands by using semicolons, the output of each distinct command follows on a new line as shown below:

[root@pbox ~]# uname -a;date
Linux pbox 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Sun Sep 10 13:46:26 IST 2017

But what if we have a scenario in which we require the resulting output of the commands we execute to be on the same line. A scenario where I've had this requirement was to interpolate multiple variable names into a csv file. A quick and easy way would be:

[root@pbox ~]# echo "$UID, $PWD" > text.csv

[root@pbox ~]# cat text.csv

0, /root

[root@pbox ~]#

For some odd reason which I can't recall this simple redirect wasn't working for me so I had to come up with something fancier.

I decided to use a subshell, enclose the commands within that subshell and redirect them to the required file. Here's an example:

[root@pbox ~]# ( echo -n "Logged into `hostname`"; echo ", on `date`" ) > text

[root@pbox ~]# cat text

Logged into pbox, on Sun Sep 10 13:57:47 IST 2017

[root@pbox ~]#

The -n flag with the echo removes the new line it add at the end of it's output.

We could do something similar without invoking a subshell and that is by using curly braces {}.

Here's an example:

[root@pbox ~]# { echo -n "This is a `uname -s` box "; echo ", btw Today is `date`" ;} > text

[root@pbox ~]# cat text

This is a Linux box , btw Today is Sun Sep 10 14:01:37 IST 2017

[root@pbox ~]#

Run a command repeatedly to view its progress

Linux provides a neat utility called watch which we can use to run a command continuously at a specified intervals. This is useful when we have a long running task and we want to run a command repeatedly to monitor the progress of that task. A disk mirror operation is a good example.

I wrote a quick one liner to do something similar for UNIX (Solaris) as watch does for Linux.
So, here it is:

while true; do echo "lpq -P B-inst-printer1 at $(date)"; lpq -P B-inst-printer1; sleep 360; done
lpq -P B-inst-printer1 at Monday, July 31, 2017 01:44:03 AM GMT
no entries
lpq -P B-inst-printer1 at Monday, July 31, 2017 01:50:03 AM GMT
no entries
lpq -P B-inst-printer1 at Monday, July 31, 2017 01:56:05 AM GMT
no entries

This is a simple infinite while loop which will continue to run the specified command (print job status of a printer in this case) at an interval of 5 minutes implemented via a sleep.

You can further refine this one liner suited to your requirements. I hope this trick will prove useful to you in future.

Shell script to add a public key to multiple users

In this article I'd like to share a script I wrote to add the rsa_id.pub public key of a user belonging to one server to a given list of users present on another server. The script will be executed on the destination server.

The task sounds simple enough to accomplish with the help of a simple for loop but I've added some checks to make the script more user friendly  so that you may be in a position to share this script with your clolleagues and perhaps even end users if they have sufficient privileges to add keys. I've added some code that I might like to reuse in the future.

So, here is the script:

#!/bin/bash

###############################################
#Purpose: Add public ssh key to multiple users                #
#Author; Sahil Suri                                                            #
#date: 24/06/2017                                                              #
###############################################

cat /dev/null > sftp_user_not_found.txt

cat /dev/null > sftp_user_key_add.txt

#This will only check for existance of the user and .ssh directory within home directory of the user#

function check_dir() {
echo "Please supply list of users: "
read USER_LIST

for USERNAME in `cat ${USER_LIST}`

do

echo "Checking if user ${USERNAME} exists"

grep -w ${USERNAME} /etc/passwd > /dev/null

if [ $? -eq 0 ]
 then
   echo -e "The user ${USERNAME} exists on server $(hostname) \n"
   echo "The user ${USERNAME} exists on server $(hostname)"
   HOME_DIR=$(grep -w ${USERNAME} /etc/passwd | awk -F: '{print $6}')
   HOME_DIR_PERM=$(ls -ld ${HOME_DIR} | awk '{print $1}' | tr -d "d")
   GID_NUMERIC=$(grep -w ${USERNAME} /etc/passwd | awk -F: '{print $4}')
   GID_WORD=$(grep -w ${GID_NUMERIC} /etc/group | awk -F: '{print $1}')

   echo -e "Home directory for user ${USERNAME} is ${HOME_DIR} and its permissions are ${HOME_DIR_PERM} \n"
   echo "Home directory for user ${USERNAME} is ${HOME_DIR} and its permissions are ${HOME_DIR_PERM}" >> sftp_user_key_add.txt

   echo "Checking if .ssh directory exists for user ${USERNAME}"
   ls -l ${HOME_DIR}/.ssh > /dev/null 2> /dev/null
    if [ $? -ne 0 ]
         then
          echo -e ".ssh directory does not exist for user ${USERNAME} \n"
    fi

else
   echo -e "The user ${USERNAME} does not exist on server $(hostname) \n"
   echo "The user ${USERNAME} does not exist on server $(hostname) " >> sftp_user_not_found.txt
fi

done

}


#This will check for existance of the user and .ssh directory within home directory of the user#
#and add the key supplied to it in authorized_keys file for the user list provided to it#

function add_key() {

echo "Please supply list of users: "
read USER_LIST

echo "Please supply the key file you wish to append authorized_keys files of these users: "
read KEY_FILE

for USERNAME in `cat ${USER_LIST}`

do

echo "Checking if user ${USERNAME} exists"

grep -w ${USERNAME} /etc/passwd > /dev/null

if [ $? -eq 0 ]
 then
   echo -e "The user ${USERNAME} exists on server $(hostname) \n"
   echo "The user ${USERNAME} exists on server $(hostname)"
   HOME_DIR=$(grep -w ${USERNAME} /etc/passwd | awk -F: '{print $6}')
   HOME_DIR_PERM=$(ls -ld ${HOME_DIR} | awk '{print $1}' | tr -d "d")
   GID_NUMERIC=$(grep -w ${USERNAME} /etc/passwd | awk -F: '{print $4}')
   GID_WORD=$(grep -w ${GID_NUMERIC} /etc/group | awk -F: '{print $1}')

   echo -e "Home directory for user ${USERNAME} is ${HOME_DIR} and its permissions are ${HOME_DIR_PERM} \n"
   echo "Home directory for user ${USERNAME} is ${HOME_DIR} and its permissions are ${HOME_DIR_PERM}" >> sftp_user_key_add.txt

   echo "Checking if .ssh directory exists for user ${USERNAME}"
   ls -l ${HOME_DIR}/.ssh > /dev/null 2> /dev/null
    if [ $? -ne 0 ]
         then
          echo -e ".ssh directory does not exist for user ${USERNAME} \n"
          echo "Creating .ssh directory and setting permissions for ${USERNAME} user"
          sudo mkdir ${HOME_DIR}/.ssh
          sudo chmod 700 ${HOME_DIR}/.ssh
          sudo chown ${USERNAME}:${GID_WORD} ${HOME_DIR}/.ssh
          sudo touch ${HOME_DIR}/.ssh/authorized_keys
          sudo chown ${USERNAME}:${GID_WORD} ${HOME_DIR}/.ssh/authorized_keys
          sudo chmod 644 ${HOME_DIR}/.ssh/authorized_keys
          sudo cat ${KEY_FILE} >> ${HOME_DIR}/.ssh/authorized_keys
          sudo chmod 750 ${HOME_DIR}
        fi

else
   echo -e "The user ${USERNAME} does not exist on server $(hostname) \n"
   echo "The user ${USERNAME} does not exist on server $(hostname) " >> sftp_user_not_found.txt
fi

done

}

#pass an argument to the script suggesting how you'd like to execute it#

OPTION="$1"

if [ "$#" -ne 1 ]
 then
  echo "You must specify an option to run the script"
  echo "-h for help or -r to run the script. Exiting now"
  exit
fi

#Required case statement logic to implement user option#

case $OPTION in
 "-h")
    echo "Displaying help"
        echo "To run the script type: ./add_public_key.bash -r"
        echo "You need to become root via sudo su - before running the script"
        echo "Enter the file containing list of users within whose home key is to be added when prompted"
        echo "Enter the file containing the public key when prompted"
        ;;
 "-r")
    echo "Running the script"
        echo "type 1 and press enter if you wish to check for the presence of .ssh directory"
        echo "type 2 and press enter if you wish to copy the key file right now"
        read USER_SEL

        if [ $USER_SEL -eq 1 ]
         then
          check_dir
        elif [ $USER_SEL -eq 2 ]
          then
           add_key
        else
           echo "Invalid option"
          exit
         fi
        ;;
  *)
    echo "invalid execution"
        ;;
esac

The script can be run in two modes:

Help mode: If you add the -h option with the script it'll run in help mode displaying information on how to use the script.

Run/execution mode: If you add the -r option with the script then it'll run in execution mode and you'll be prompted for yet another selection this time you can choose to check for the presence of .ssh directory for the list of users or add the key for the source user from where password less authentication needs to be configured.

The script has built in checks to look for the presence of the user supplied in the list, check for the presenece of .ssh directories of the users, a check to terminate the script if the user does not supply any options and I've also added logging of some of the command output.

I hope this has been a nice read and look forward towards your feedback.

Script to check open ports in Linux

In this brief article I'd like to share a short script I recently wrote to check for port status for different ports in Linux. This script may prove useful as a pre-check or post-check after a maintenance activity or you can also put the script in cron if you need to monitor the port number corresponding to a service at regular intervals and relying on conventional monitoring tools is not an option.

Here is the script:

[root@still ~]# cat port_check.bash
#!/bin/bash

##Add a file containing a list of port numbers to chek##

PORT_LIST="/root/plist"

while read PNUM
do

netstat -tulpn | grep -w ":${PNUM}" > /dev/null

if [ $? -eq 0 ]
 then
  echo "port number ${PNUM} is listening on `hostname`"
else
 echo "port number ${PNUM} is not listening on `hostname`"
fi

done < ${PORT_LIST}

To test it I've created a file ptest with some port numbers to test the script.

[root@still ~]# cat /root/plist

80

22

30

Let's run the script:

[root@still ~]# ./port_check.bash

port number 80 is listening on still

port number 22 is listening on still

port number 30 is not listening on still

This is more of an arbritrary setup. You can add logic to send an email to you if any of the ports are not found to be in listening state

Script to check if all fstab entries are mounted

In this article I'd like to share a script which would compare file system entries in the /etc/fstab file with the file systems that are currently mounted and tell us if there is any file system which has an entry in the /etc/fstab file but is not mounted.

So, here is the script:

[root@still ~]# cat tab.bash
#!/bin/bash

FSTAB_ENTRIES=$(cat /etc/fstab | awk '$1 !~/#|^$|swap/ {print $2}')

for FS in ${FSTAB_ENTRIES}

do

df -hPT | grep -wq ${FS}

if [ $? -eq 0 ]
 then
  echo "The file system ${FS} has an entry in /etc/fstab file and is mounted"
 else
  echo "The file system ${FS} has an entry in /etc/fstab file but is not mounted"
fi

done
[root@still ~]#

Let's test it out.

This is my /etc/fstab file and the current df output.

[root@still ~]# cat /etc/fstab ; df -hTP

#

# /etc/fstab

# Created by anaconda on Sat Dec 24 12:03:40 2016

#

# Accessible filesystems, by reference, are maintained under '/dev/disk'

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

#

/dev/mapper/centos-root /                       xfs     defaults        1 1

UUID=93b957e6-f8f9-42f9-a7ad-9927f694f1ce /boot                   xfs     defaults        1 2

/dev/mapper/centos-swap swap                    swap    defaults        0 0

##testing##

/dev/vg1/lv01   /test_dir1                       ext4     defaults        1 1

/dev/vg1/lv02   /test_dir2                       ext4     defaults        1 1

/dev/vg1/lv03   /test_dir3                       ext4     defaults        1 1

/dev/vg1/lv04   /test_dir4                       ext4     defaults        1 1

Filesystem              Type      Size  Used Avail Use% Mounted on

/dev/mapper/centos-root xfs        18G  6.0G   12G  35% /

devtmpfs                devtmpfs  481M     0  481M   0% /dev

tmpfs                   tmpfs     490M   80K  490M   1% /dev/shm

tmpfs                   tmpfs     490M  7.1M  483M   2% /run

tmpfs                   tmpfs     490M     0  490M   0% /sys/fs/cgroup

/dev/sda1               xfs       497M  134M  363M  27% /boot

/dev/mapper/vg1-lv01    ext4      283M  2.1M  262M   1% /test_dir1

/dev/mapper/vg1-lv02    ext4      283M  2.1M  262M   1% /test_dir2

You can notice that /test_dir3 and /test_dir4 file systems have entries in the /etc/fstab file but are not mounted at the moment.

Let's run the script to verify.

[root@still ~]# ./tab.bash

The file system / has an entry in /etc/fstab file and is mounted

The file system /boot has an entry in /etc/fstab file and is mounted

The file system /test_dir1 has an entry in /etc/fstab file and is mounted

The file system /test_dir2 has an entry in /etc/fstab file and is mounted

The file system /test_dir3 has an entry in /etc/fstab file but is not mounted

The file system /test_dir4 has an entry in /etc/fstab file but is not mounted

There you have it.

I hope this script helps you in being used as a pre-check or post-check during maintenance activities.

Bash script to update rows in a table in a MS word file

In this article I'll explore something I thought for a long time could not be done correctly. I'll update a MS word document in Linux using a shell script without  the use of open office or any other similar software. A MS word document is essentially composed of a bunch of compressed XML file and some other data. We can uncompress the doc file in Linux via zip to obtain the XML file and also extract the text from that XML file using sed. But inserting text into a word document is a whole different story altogether.

The scenario I'm about to demonstrate is that I have a word file with a table in it and update the rows in the table.

Given below is the screenshot of the document content.

Now I wish to update each cell in the entire row and given below is the script to do it.

#!/bin/bash

echo "enter absolute path of MS WORD .docx file"

read DOC_LOCATION

DOC_FILE=$(echo $DOC_LOCATION |  awk -F/ '{print $NF}')

cp ${DOC_LOCATION} /tmp/XML

cd /tmp/XML

unzip ${DOC_FILE} >> /dev/null

cd /tmp/XML/word

cp document.xml /tmp/append

cd /tmp/append

echo "Enter user name that was created:"
read USER

echo "Enter ticket number:"
read TNO

echo "Enter Approver name:"
read APP

echo "Enter your full name"
read ADMIN

sed -e "s#USERNAME#${USER}#" -e "s#TICKET#${TNO}#" -e "s#APPROVER#${APP}#" -e "s#UNIX#${ADMIN}#" text2insert > text2add

sed -i "s#</w:tr></w:tbl>#</w:tr>$(cat text2add)</w:tbl>#" document.xml

mv -f document.xml /tmp/XML/word

cd /tmp/XML/

rm -f ${DOC_FILE}
zip -r ${DOC_FILE} * >> /dev/null

echo "updated document is available at location /tmp/XML/"

The logic used within the script uses a file named text2insert with XML tags corresponding to a row with arbitrary/default text values inserted in the tags. 

When the script is run the user is prompted to enter some input values. Based on the received inputs the text file text2insert is modified and the updated file text2add is created.

The content of text2add file is inserted into the word document XML file in between the closure the last row tag and the closure of the table tag thereby inserting an entire row of text.

Here is the text within text2insert file.

[root@cent7 append]# cat text2insert

<w:tr><w:tc><w:tcPr><w:tcW w:type="dxa"/></w:tcPr><w:p ><w:r><w:t>USERNAME</w:t></w:r></w:p></w:tc><w:tc><w:tcPr><w:tcW w:type="dxa"/></w:tcPr><w:p ><w:r><w:t>TICKET</w:t></w:r></w:p></w:tc><w:tc><w:tcPr><w:tcW w:type="dxa"/></w:tcPr><w:p ><w:r><w:t>APPROVER</w:t></w:r></w:p></w:tc><w:tc><w:tcPr><w:tcW w:type="dxa"/></w:tcPr><w:p ><w:r><w:t>UNIX</w:t></w:r></w:p></w:tc></w:tr>

This is a screenshot of a run of the script:

After the script completes execution the content of text2add is as follows:

root@cent7 append]# cat text2add

<w:tr><w:tc><w:tcPr><w:tcW w:type="dxa"/></w:tcPr><w:p ><w:r><w:t>forth</w:t></w:r></w:p></w:tc><w:tc><w:tcPr><w:tcW w:type="dxa"/></w:tcPr><w:p ><w:r><w:t>Q779911</w:t></w:r></w:p></w:tc><w:tc><w:tcPr><w:tcW w:type="dxa"/></w:tcPr><w:p ><w:r><w:t>The TSM</w:t></w:r></w:p></w:tc><w:tc><w:tcPr><w:tcW w:type="dxa"/></w:tcPr><w:p ><w:r><w:t>Sahil Suri</w:t></w:r></w:p></w:tc></w:tr>

The script will not create a new document after adding the new row. Instead I performed an in place edit with sed to update the original document.

After running the script, the updated document looks like this:

I hope that this has been an intuitive read and reinforces the philosophy that when there is a shell there is a way.

Add users via receipt of user and password inputs from different files

I recently came across a question where in an admin wanted to create multiple users on Linux servers and had the password and user information contained in different files. If it were the same password for all users then that would've been easy to loop over but this was different because the user name and passwords were both unique.

So, here are sample files for lists of user names and password.

[root@pbox ~]# cat users
james
john
[root@pbox ~]# cat passwords
james123
john123

Now instead of coming up with a logic to run a loop with two variables I decided it would be easier to just combine the two files via paste command.

[root@pbox ~]# paste -d ' ' users passwords > creds
[root@pbox ~]#

So now the creds file looks like this:

[root@pbox ~]# cat creds
james james123
john john123


With the two file problem sorted the user creation process was a one line while loop away.

[root@pbox ~]# cat creds | while read user pass; do useradd $user ; echo $pass | passwd --stdin $user;done
Changing password for user james.
passwd: all authentication tokens updated successfully.
Changing password for user john.
passwd: all authentication tokens updated successfully.
[root@pbox ~]#

I hope this article was helpful to you and gives some interesting scripting ideas!
Thank you for reading.

Script to calculate storage space allocated to a Solaris zone

In this article I'd like to share a quick script to calculate the total storage RAW plus LOFS allocated to a Solaris zone.

The script would need to be run from the global zone and it's usage will be:

./zone_storage_cal.sh <zone name>


Here is the script:

[ssuri@localhost:~] $ cat zone_storage_cal.sh
#!/usr/bin/bash

ZONE=$1

DISK_LIST=$(cat /etc/zones/${ZONE}.xml | grep "device match" | awk -F/ '{print $4}')

FORMAT_OUT="/export/home/`whoami`/format_out"

>$FORMAT_OUT

echo | sudo  format >> $FORMAT_OUT

DISK_OUT="/export/home/`whoami`/disk_out"
>$DISK_OUT

for DISK_NAME in ${DISK_LIST}

do

 DISK=$(cat $FORMAT_OUT | grep -w ${DISK_NAME%s0\"} | awk '{print  $2}')
 SIZE=$(cat $FORMAT_OUT | grep -w ${DISK_NAME%s0\"} | awk '{print  $3}' | tr -d "<>" | awk -F- '{print $4}' | tr -d "GB")

  echo "${DISK} ${SIZE}" >> $DISK_OUT

done

RAW_DISK=$(cat $DISK_OUT  | awk '{sum+=$2} END {print sum}')

echo "total RAW disk device size is ${RAW_DISK} GB"


##LOFS mount size calculation##

LOFS_LIST=$(cat /etc/zones/${ZONE}.xml | grep lofs | egrep -v "/var|/usr" | awk '{print $2}' | awk -F= '{print $2}' | tr -d \")

TOTAL_LOFS_SIZE=0

for LOFS in ${LOFS_LIST}
do

SIZE=$(df -k | cat | egrep "${LOFS}" | awk '{print $2}')

let "TOTAL_LOFS_SIZE+=$SIZE"

done

TOTAL_LOFS_SIZE_GB=$( echo "scale=2; ${TOTAL_LOFS_SIZE}/1024/1024" | bc -l)

if [ ! -z ${TOTAL_LOFS_SIZE_GB} ]
then
 echo "Total size of LOFS mounts is ${TOTAL_LOFS_SIZE_GB} GB"
fi

##end LOFS calculation##

##print total size##

if [ ! -z ${TOTAL_LOFS_SIZE_GB} ] && [ ! -z ${RAW_DISK} ]
then
 TOTAL_SIZE=$(echo "scale=2; ${TOTAL_LOFS_SIZE_GB} + ${RAW_DISK}" | bc -l)
 echo "RAW disk and LOFS size combined is ${TOTAL_SIZE} GB"
fi

The script uses the zones's XML file to get the details of any raw disk devices exported to the zone and also gather information about loop back file systems mounted on the zones.

The addition part is done using let. I didn't have to use bc because I opted not to be floating point data for initial calculations.

In the final calculations I've used bc to provide a more accurate final result.

Getting disk size info from iostat in Solaris/Using lsblk to find total size of disks allocated in Linux

I know the title of the post is long. With that out of the way lets get started.

Today I'll share a quick one liner to provide the size of attached disks in a neat format with iostat command in Solaris.

So, here's what the iostat -En command output looks like :

[ssuri@:~] $ sudo iostat -En
c21d0            Soft Errors: 0 Transport Errors: 0 Protocol Errors: 0
Vendor: SUN Product: VDSK Size: 137.67GB <137665445888 bytes>

c21d1            Soft Errors: 0 Transport Errors: 0 Protocol Errors: 0
Vendor: SUN Product: VDSK Size: 137.67GB <137665445888 bytes>

And this is the one liner to format it so that we get a two column output consisting of the disk name and disk size only.

sudo iostat -En | awk '{print $1, $6}' | tr -d "Errors:" | sed 's/Vend//g' | sed '$!N;$!N;s/\n/ /g'

Given below is the output of the above one liner:

c21d0   137.67GB

c21d1   137.67GB

c21d2   274.88GB

c21d11   34.36GB

c21d12   34.36GB

c21d13   34.36GB

c21d14   34.36GB

c21d15   34.36GB

c21d16   34.36GB

c21d17   34.36GB

c21d18   68.72GB

c21d19   274.88GB


Now coming to the Linux part.
I was recently tasked with getting information on total connected SAN storage excluding the root & swap disks for some Linux servers.
I could've used fdisk but I used lsblk instead.


Here is the default output of lsblk without any filters:

[ssuri@:~] $ lsblk
NAME                        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0                          11:0    1 1024M  0 rom
sdb                           8:16   0   32G  0 disk
`-grid_vg-grid_lv (dm-4)    253:4    0   32G  0 lvm  /grid
sdc                           8:32   0   24G  0 disk
`-kdump_vg-kdump_lv (dm-3)  253:3    0   24G  0 lvm
sdd                           8:48   0   32G  0 disk
`-elmerd5db_vg-lvol1 (dm-2) 253:2    0   32G  0 lvm  /elmerd5db/db
sde                           8:64   0   32G  0 disk
`-sde1                        8:65   0   32G  0 part
sdf                           8:80   0  256G  0 disk
`-sdf1                        8:81   0  256G  0 part
sdg                           8:96   0   32G  0 disk
`-sdg1                        8:97   0   32G  0 part
sda                           8:0    0   60G  0 disk
|-sda1                        8:1    0  512M  0 part /boot
`-sda2                        8:2    0 59.5G  0 part
  |-os_vg-root_lv (dm-0)    253:0    0   16G  0 lvm  /
  |-os_vg-swap_01_lv (dm-1) 253:1    0    4G  0 lvm  [SWAP]
  |-os_vg-tmp_lv (dm-5)     253:5    0    2G  0 lvm  /tmp
  |-os_vg-var_lv (dm-6)     253:6    0   10G  0 lvm  /var
  `-os_vg-hpds_lv (dm-7)    253:7    0   12G  0 lvm  /var/opt/perf/datafiles
sdi                           8:128  0   32G  0 disk
`-sdi1                        8:129  0   32G  0 part
sdh                           8:112  0   32G  0 disk

`-sdh1                        8:113  0   32G  0 part

I know that I'm using disk sda and sdc for root and swap space respectively.

So lets limit the output to disk devices and filter out disks /dev/sda and /dev/sdc.

[ssuri@:~] $ lsblk | egrep disk  | grep -v "sd[ac]"

sdb                           8:16   0   32G  0 disk

sdd                           8:48   0   32G  0 disk

sde                           8:64   0   32G  0 disk

sdf                           8:80   0  256G  0 disk

sdg                           8:96   0   32G  0 disk

sdi                           8:128  0   32G  0 disk

sdh                           8:112  0   32G  0 disk

Now, I'll use awk to calculate the total size of these disks:

[ssuri@:~] $ lsblk | egrep disk  | grep -v "sd[ac]" | tr -d "G" | awk '{sum+=$4} END {print sum}'

448

A quick and easy time saver.

Script to check if File Systems are in read only mode

Today I'll share a quick script with which you can check if the file systems on your server have gone into read only mode. One way would be to touch a file and check but doing that on every file system on every server would involve a lot of time and overhead.
From some experimentation I've learned that if the file systems go into read only mode while a process is writing to it then the read only mode gets reflected in /proc/mounts command.

Here is a quick script to check the status and send out an email in case of issues:

#!/bin/bash

##########################################

#date: 01/02/2017                        #

#purpse: check Linux file systems are RW #

##########################################

cat /proc/mounts | grep ext4| awk '{print $2, $4}' | awk -F "," '{print $1}' | while read FS_NAME PERM

do

if [ ${PERM} != "rw" ] ; then

        echo -e "FS $FS_NAME is mounted with $PERM permissions. Please check \n" | mail -s "Read only FS on `hostname`" unixadmins@example.com

fi

done

You can integrate this script with a monitoring tool like HP OV or nagios.

Best options for using SSH within scripts

As system administrators we frequently write scripts involving authenticating to more than one server to accomplish tasks within a single script. We want our scripts to run fast but when working with ssh within scripts, we may encounter a very slow execution time in case there are hosts on which some password less authentication setup does not exist or the server is unresponsive.
For example if a server is down & we are trying to connect to it within our script then by default ssh will continue to make connection attempts for 60 seconds before giving up & move to the next server. Take another example wherein you get prompted for password while logging into the server. The script will remain hung until you don't enter the password.

In this article I'd like to share some ssh options that I use within my scripts for quick execution.

StrictHostKeyChecking:
With this option set to no, destination host key checking is turned off so we don't get prompted to accept the key fingerprint while connecting.

BatchMode:
With this option set to yes, ssh will effectively skip any hosts for which password less ssh connectivity is not set up. You can always redirect the hostnames you were unable to connect to another file.

ConnectTimeout:
This option determines the time in seconds for which ssh will continue to initiate a connection to a host. By reducing this value to under 10 seconds, we can reduce the waiting time before moving on to another host. You should redirect the host names you were unable to connect to another file to check them later.

q(quite):
By specifying quite mode, we can effectively skip motd & banner messages seen on the terminal when we log in thereby making room for a more cleaner output.

Usage in scripts:

We can specify the above mentioned options with our ssh command. But I've realized with experience that a better way is to variableize the options so that if we have multiple ssh commands within the script then we don't have to specify the options every time.

Here is an example of how I used these options in a script:


SSH_OPTIONS=" -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=3 -q"


ssh $SSH_OPTIONS ${name} "bash -s" < /export/home/ssuri/automount_check_solaris.sh


i disabled host key checking, set batch mode to yes, reduced connection timeout to a mere 3 seconds & finally specified -q to signify that I wanted a quite ssh login.

I assigned all the ssh options I wanted to use in a variable & then used that variable with the ssh command to connect to some servers, the names of which were in a list looped over in a for loop & run a local script remotely by using "bash -s".

Using AWK to match columns from multiple files

I came across an interesting requirement at a facebook forum today. The requirement was to match columns two & three of one file with columns one & two of another file & print the entities from the first file which do not have nay matches.

Here are the files:

[root@alive ~]# cat file1
d1,40,gold
d2,30,silver
d3,20,bronze
d4,10,iron
d5,5,wood
d6,20,gold
d7,10,wood
d8,5,gold
d9,10,silver
[root@alive ~]# cat file2
gold,40
silver,30
bronze,20
iron,10
wood,5

The AWK one liner that works is:

awk -F',' 'NR==FNR{c[$1$2]++;next};!c[$3$2]' file2 file1

The following will be the output of he above AWK statement:

d6,20,gold

d7,10,wood

d8,5,gold

d9,10,silver

Now, let's do a step by step breakdown of what just happened.

-F ',' 

(The file columns are comma separated. So we changed the field separator)

NR==FNR 

(When you have two input files to awk, FNR will reset back to 1on the first line of the next file whereas NR will continuing increment from where it left off. By checking FNR==NR we are essentially checking to see if we are currently parsing the first file.)

 

c[$1$2]++;next

(Here we are parsing the first file file2 creating an associative array with columns one & two & post increment the value by one. The next command tells AWK not to process any further commands and proceed to next record.)

!c[$3$2] 

(This line only executes when FNR==NR is false, i.e. we are not parsing file2 and thus must be parsing file1. We then use the first fields $2 and $3 of file1 as the key to index into our 'seen' list created earlier.  If the value returned is 0 it means we didn't see it in file1 and therefore we should print this line. Conversely, if the value is non-zero then we did see it in file1 and thus we should not print its value. Note that !c[$2$3] is equivalent to !a[$2$3]{print} because the default action when one is not given is to print the entire line.)

In case the requirement changes and we needed to print the matching lines instead of those that didn't match, we'd modify our AWK expression as follows:

awk -F',' 'NR==FNR{c[$1$2]++;next};c[$3$2]' file2 file1

And the resulting output will be:

d1,40,gold

d2,30,silver

d3,20,bronze

d4,10,iron

d5,5,wood

I hope this has been an interesting AWK read.