Using select to create menu driven shell scripts

Menu driven scripts are an important aspect to consider when writting large scripts which would require user input for further processing. We can use an infinite while loop to create a menu for a shell script & that is quite common. If you have an X windows system available then you can make use of dialog or whiptail to create nice menus & dialog boxes for your scripts. In this article I'll domonstrate how we can use the select command for creating simple menu driven scripts.

The syntax for select command is:

select var in list
do
    commands..
done

The select command runs an inifinite loop. var is the variable you'd use in your case statement for the menu. The list would be the menu options. The commands would be part of the case statements being issued during the execution of the menu selection.
We can write the commands to be executed for a menu selection within the select loop but for a cleaner code I'll write the commands within separate functions & just call the functions in the case statements within the select loop.

So, here is the example script:

root@buntu:~# cat select.sh
#!/bin/bash

trap '' 2  # ignore ctrl+c

##set PS3 prompt##
PS3="Enter your selection: "

function diskspace {
        df -h
}

function users {
        who
}

select opts in "check disk usage" "check users" "exit script"
do
        case $opts in
                "check disk usage")
                                echo -e "\e[0;32m $(diskspace) \e[m"
                                ;;
                "check users")
                                echo -e "\e[0;32m $(users) \e[m"
                                ;;
                "exit script")
                                break
                                ;;
                *)
                                 echo "invalid option"
                                ;;
        esac
done

The script when executed will present three options to the user. Selecting option 1 will call the diskspace function, option 2 will the users function & option 3 will exit out of the loop.

I've added a trap to ignore ctrl+c presses or the INT signal & added some color to the output. 

Common file operations in perl part 2

In the previous article I demonstrated the usage of some common file manipulation operations we can perform in perl using a single perl script. Keeping in line with that in this article The functions I'll use in this article will be glob, sysopen, link, symlink & grep.

Given below is a short description for each of these functions before getting to the actual script:

Glob: This function is used to insert the * wildcard character with file names etc.

Sysopen: We can use the open function for opening files for reading & writting but we might risking clobbering some existing files if we use the open function. The sysopen function allows us to indicate conditions such that the new file will be created only if it does not exist & also allows us to set permissions during file creation.

Link & symlink: These functions are used for creating hard links & soft links respectively.

Grep: This is the perl variant of the grep command available in UNIX/Linux platforms.

So, here's the perl script illustrating the usage of the aforementioned functions.


root@buntu:~# cat file2op.pl
#!/usr/bin/perl -w

use Fcntl;

##testing system and glob functions##

my $test_var = system ("/bin/ls -l /tmp");

my $test_var2 = qx{ls -l /} ;

my $test_glob = system ('ls' ,'-l', glob('/root/test*') ) ;


print qq{$test_var} ;
print "$test_var2" ;
print "$test_glob" ;

##demonstrating sysopen##

#umask 0022;
umask 0000;

sysopen (newfile, "newfile", O_WRONLY|O_CREAT|O_EXCL, 0777) || die "could not create file" ;

print newfile "this is to demonstrate sysopen function usage";


##demonstrating creation of file links##

my $file_name = "testfile" ;
my $soft_link = "testfile_soft" ;
my $hard_link = "testfile_hard" ;

link $file_name, $hard_link ;

symlink $file_name, $soft_link ;


##demonstrating perl grep function usage##

@file_list = `ls -l /root` ;

print grep /test/, @file_list ;

##second example using perl grep##

@art = ("1", "4", "10", "14", "7") ;

@filter =  grep  ($_ > 4 , @art)  ;

print "@filter", "\n" ;


The qq function is analogous to double quotes & the qx{} function is analogous to backticks used for command substitution.

There is one more function that I need to mention before concluding this article & that is the map function. The map function takes a list or an array as its input & transforms that list element by element into a new list or array. This is in a way functioanlly similar to writting a foreach loop iterating through each element of the array & performing some actions on it. The map function operates on the _ variable.

Here is an example of using the map function:

root@buntu:~# cat map.pl
#!/usr/bin/perl

use File::Copy;


open (t1, "afile") || die "could not open file $!";

open (t2, ">> afile.bkp") || die "could not open file $!";

my @array = <t1> ;

my $var = `hostname` ;

my @new = map {$_ =~ s/server/$var/g ; $_ } @array ;

print t2 @new ;

my $f1 = "afile" ;
my $f2 = "afile.bkp" ;

move ("$f2", "$f1") ;

This script opens a file named afile & assigns the corresponding file handle to an array. We then change each occurance of the word server & replace it with the hostname of our system & apply the changes to a new array. We then copy the contents of a new array to a file & finally replace the original with the newly created file.

I hope this has been a good read.

Common file operations in perl part 1

Perl empowers the user with several useful functions for interacting with and manipulating files. In this atricle I'll talk about & demonstrate file copy, move, rename, delete operations. modify file permissions/ownerships & also about the stat function using which we can get some useful information about files.

I'll not take too much time explaining the syntax & usage of each of the functions since we can understand the usage from the function name itself but I'll talk a bit about the stat function.

So, the stat function is basically a thirteen element array & the array elements store information about different attributes of the file. Here is the list of elements:

0 Device number of file system
1 Inode number
2 File mode (type and permissions)
3 Number of (hard) links to the file
4 Numeric user ID of file.s owner
5 Numeric group ID of file.s owner
6 The device identifier (special files only)
7 File size, in bytes
8 Last access time since the epoch
9 Last modify time since the epoch
10 Inode change time (not creation time!) since the epoch
11 Preferred block size for file system I/O
12 Actual number of blocks allocated

To sum up the file manipulation operations, I've written the following perl script:

root@buntu:~# cat fileop.pl
#!/usr/bin/perl -w

use File::Copy;
use File::stat;

$file_name = "oldfile";
$copy = "oldfile_copy";
$second_copy = "oldfile_copy2";
$old_name = "oldfile";
$new_name = "newfile";
$new_path = "/tmp";

if ( -e $file_name )
 {
        ##copy file contents##

        copy ($file_name, $copy) || die "couldn't copy file $!";
        copy ($file_name, $second_copy) || die "couldn't copy file $!";
        copy ("$file_name", "${file_name}.bkp") || die "couldn't take a backup copy $!";

        ##copy file contents to the screen i.e STDOUT##

        copy ($file_name, \* STDOUT ) || die "couldn't print file to stdout  $!";

        ##move a file##

        move ("$old_name", "$new_path/") || die "couldn't move file $!";

        ##rename a file##

        rename ("$new_path/$old_name", "$new_path/$new_name") || die "couldn't rename file $!";

        ##list file & delete it##
        system ("ls /root/oldfile_copy2");
        unlink $second_copy ;

        ##change ownership & permissions##
        $uid = "1000" ;
        $gid = "1000" ;
        chown $uid, $gid, "$copy" ;
        chmod 0777, "$copy" ;

        system ("ls -l $copy") ;


        ##using stat function##

        $file_size = (stat ($copy))->size;
        $file_owner = (stat ($copy))->uid;

        print "file is owned by user with uid ", $file_owner, " size of the file is: ",  $file_size , " bytes \n"  ;

 }


The output of the execution of the script is as follows:

root@buntu:~# ./fileop.pl
this is a test file
/root/oldfile_copy2
-rwxrwxrwx 1 sa sa 20 Dec  6 21:52 oldfile_copy
file is owned by user with uid 1000 size of the file is: 20 bytes

This script performs the following tasks in order:

• copies a file
• prints the contents of the file to the screen
• moves & renames the file
• deletes a copy of the file
• changes permissions of a copy of the file
• uses stat function to get size of the file & the uid of the file owner & prints them

Getting to know IFS (internal field separator) variable in Bash shell

This variable defines the list of characters which bash considers as field separators & is particularly useful while iterating data in the form of lists with loops.
By default the space, tab & newline characters are considered as field separators.

Consider a file test.txt with the following lines:

unix
linux
solaris
sahil suri

If we iterate through this file in a simple for loop the result will be:

bash-4.1$ for i in $(<test.txt)
> do
> echo $i
> done
unix
linux
solaris
sahil
suri
bash-4.1$

As you notice the words sahil & suri were on the same line but got interpreted as separate fields by the for loop.
To remidiate this we bring the IFS variable in play here. We'll change the value of IFS to only consider new lines as separate fields.

bash-4.1$ IFS.OLD=$IFS ; IFS=$'\n'; for i in $(<test.txt); do  echo $i; done; IFS=$IFS.OLD
unix
linux
solaris
sahil suri

With IFS set to a new line, the space between sahil & suri does not get interpreted as separate fields.
We can assign multiple field separators as welll. For example:

IFS=$'\n':;"

This will set new line, colon, semi-colon & double quotes as field separaters.

Now let's take another example using a colon as a delimeter with the below file.

bash-4.1$ cat test.txt
sahil:unix admin:since 2016
bond:linux admin:since 2000


IFS.OLD=$IFS ; IFS=: ; while read name desig join_date; do echo -e "name is $name \n" "designation is $desig \n" "joined in $join_date \n" ; done < test.txt; IFS=$IFS.OLD

One thing to remember while manipulating the IFS variable is to set it back to the original value after we are done with our required task else it may lead to some unexpected results in future scripts being run in the session.

Let's take another example to using colon as a delimiter but this time we'll use two different values of IFS in the same script but in different loops.

#!/bin/bash

## changing the IFS value ##

IFS.OLD=$IFS
IFS=$'\n'
for entry in $(cat /etc/passwd)
do
echo "Values in $entry –"
IFS=:
for value in $entry
do
echo " $value"
done
done

##Resetting back to original IFS value##

IFS=$IFS.OLD

Use ssh in a perl script without using any modules

In a previous article I talked about how we could use the openssh module in perl for establishing ssh connections to remote hosts. In this article I'll talk about ssh based communication without the use of any modules.

Using modules is the right way to go but take an isolated scenario wherein you are not allowed to install perl modules from CPAN on your servers. This is a workaround for that.

root@buntu:~# cat test6.pl
#!/usr/bin/perl -w

open ( HAN1, "ssh -l sa buntu uptime;uname -a; whoami|") || die "not connecting $!";
open (HAN2, ">> outfile.txt") || die "could not write to file $!";

while (<HAN1>) {
        print HAN2;
}
close HAN1;

The above  example uses the open function to interact with the ssh process & prints the output to file outfile.txt.

Here's another example to iterate through a list of hosts.

root@buntu:~# cat test9.pl
#!/usr/bin/perl -w

@hlist = `cat hostlist` ;

foreach my $list  (@hlist) {
#       system ("ssh -l sa $list uptime") ;
        open ( HAN1, "ssh -l sa $list uptime;uname -a; whoami|") || die "not connecting $!";
        while(<HAN1>) {
        print ;
        }
        close HAN1;

}

This iterates through a list of servers contained in the file hostlist. This example uses nested loops.

Here are two more methods of running commands on multiple servers which I found much easier than the ones listed above:

[root@cent6 ~]# cat abc.pl
#!/usr/bin/perl -w

my @hosts = ("cent6", "cent6", "cent6") ;

my $comm = "uname -a;uptime" ;

foreach my $server (@hosts) {
        my $command_list = `ssh $server '$comm'` ;
        print "$command_list \n" ;
        }
[root@cent6 ~]#
[root@cent6 ~]#
[root@cent6 ~]# cat test.pl
#!/usr/bin/perl -w

open (FH, "file") || die ;
my @list = <FH> ;       #or use my @list = `cat file` ; to skip file handles


foreach my $server (@list) {
        chomp $server ;
        system ("ssh -l root $server 'uname -a; uptime'");
        }

Installing & using the openssh module in perl

I will start this article by saying that I'm failry new to perl & have recently begun my quest to become proficient in perl scripting for automating some of my system administration related tasks. Anyone whose familiar with perl would be aware that perls' flexibility stems from it's enormous commnuity base & modules with CPAN comprising of over 10,000 packages. That's a lot. In this article, I dwelve into one such module which is extremely useful for us system administrators & that's the Net::OpenSSH module.

Perl is installed in most UNIX/Linux based operating systems. I'll be using Ubuntu 16.04 for the tasks being performed.

Let's start by confirming that we in fact have perl & perldoc installed.

root@buntu:~# dpkg-query -l perl*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                        Version            Architecture       Description
+++-===========================-==================-==================-============================================================
ii  perl                        5.22.1-9           amd64              Larry Wall's Practical Extraction and Report Language
ii  perl-base                   5.22.1-9           amd64              minimal Perl system
un  perl-cross-config           <none>             <none>             (no description available)
ii  perl-doc                    5.22.1-9           all                Perl documentation
un  perl-modules                <none>             <none>             (no description available)
ii  perl-modules-5.22           5.22.1-9           all                Core Perl modules
un  perlapi-5.22.1              <none>             <none>             (no description available)

Now let's install cpanminus, which is basically a script that allows us to communicate with CPAN & install packages/modules from there.

curl -L http://cpanmin.us | perl - --sudo App::cpanminus

Let's confirm that it has been installed correctly.

root@buntu:~# dpkg-query -l cpan*

Desired=Unknown/Install/Remove/Purge/Hold

| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend

|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)

||/ Name                        Version            Architecture       Description

+++-===========================-==================-==================-============================================================

ii  cpanminus                   1.7040-1           all                script to get, unpack, build and install modules from CPAN

root@buntu:~#

I installed the openssh module via apt-get from the default internet repositories made available post installtion.

root@buntu:~# apt-get install libnet-openssh-perl -y

We can check list of installed module via the cpan -l command.

root@buntu:~# cpan -l | grep -i openssh

Net::OpenSSH    0.70

Net::OpenSSH::Constants 0.51_07

Net::OpenSSH::OSTracer  0.65_06

Net::OpenSSH::ShellQuoter       undef

Net::OpenSSH::ConnectionCache   undef

Net::OpenSSH::ModuleLoader      undef

Net::OpenSSH::ObjectRemote      undef

Net::OpenSSH::SSH       undef

Net::OpenSSH::ShellQuoter::Chain        undef

Net::OpenSSH::ShellQuoter::POSIX        undef

Net::OpenSSH::ShellQuoter::MSWin        undef

Net::OpenSSH::ShellQuoter::MSCmd        undef

Net::OpenSSH::ShellQuoter::csh  undef

Net::OpenSSH::ShellQuoter::fish undef

I used the cpan shell to download & install one of the dependent modules IO::Pty

root@buntu:~# cpan

Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v2.11)

Enter 'h' for help.

cpan[1]> install IO::Pty

Reading '/root/.cpan/Metadata'

  Database was generated on Sun, 04 Dec 2016 05:29:02 GMT

Running install for module 'IO::Pty'

Checksum for /root/.cpan/sources/authors/id/T/TO/TODDR/IO-Tty-1.12.tar.gz ok

Scanning cache /root/.cpan/build for sizes

............................................................................DONE

The cpan shell will install any dependent modules for the module being installed. Note that we do need make & gcc to be installed because under the hood module installation via CPAN downloads the tar.gz files for the modules & compiles & builds them to make them available for use.

Given below is the first program I wrote using the Net::Openssh modules:

root@buntu:~# cat test7.pl

#!/usr/bin/perl -w

use Net::OpenSSH;

my $cmd1 = " uptime ";

my $cmd2 = "uptime";

my $hostname = "buntu";

my $username = "sa";

my $password = "123";

my $timeout  = 20;

my $ssh = Net::OpenSSH->new(

    $hostname,

    user        => $username,

    password    => $password,

    timeout     => $timeout,

    master_opts => [ -o => "StrictHostKeyChecking=no" ]

);

$ssh->error and die "Unable to connect to remote host: " . $ssh->error;

my ( $fh, $pid ) = $ssh->open2pty( { stderr_to_stdout => 1 } );

 $ssh->system("ls /tmp") or

    die "remote command failed: " . $ssh->error;

It's failry simple. It logs in to host name buntu with the supplied credentials & uses the system function to run the "ls /tmp" command & display its output.

As I progress in my learning, I'll definitely try to post more articles on perl scripting for system administrration.

Script to group together similar file names

In this post, I'd like to share a little script I wrote to club together similarly named contents in a file. The query was posted on social media & I'm sharing the same answer here which I wrote there.

So, here' s the problem statement:

Have a look at the below file testfile:

[root@centops ~]# cat testfile
test1.txt
test2.html
test4.sql
test3.txt
test4.txt
test5.sql
test7.html
test5.txt
test9.sql

The file contains names ending with txt, sql & html. What if I needed to group together the names ending with txt & similarly for sql & html. Here is a simple script which does exactly that:

[root@centops ~]# cat group.sh

#!/bin/bash

for i in $(< testfile)

do

##separate the file names into tmp files##

 r1=$(echo $i | grep txt$);  echo $r1 | sed '/^$/d' >> tmp1

 r2=$(echo $i | grep sql$);  echo $r2 | sed '/^$/d' >> tmp2

 r3=$(echo $i | grep html$); echo $r3  | sed '/^$/d' >> tmp3

done

##group the contents##

cat tmp1 tmp2 tmp3 >> result.txt

cat result.txt

rm -f tmp1 tmp2 tmp3

Here's the script output:

[root@centops ~]# ./group.sh

test1.txt

test3.txt

test4.txt

test5.txt

test4.sql

test5.sql

test9.sql

test2.html

test7.html

This particular script may cater to a very specific requirement but this is some food or thought. Through in a couple of variable to substitute the type of entities that need to be grouped & a parameter substitution for the file name & this little script may do a lot.