Barking at daemons with Monit

Monit is a light weight open source utility which can be used for monitoring & maintenance of a wide variety of system attributes. We can use monit to check for resource utilization, service status, inode count, existence of scripts & even file/directory permissions. We can configure monit to restart services if they are not found to be in a running state.

We can also setup centralized monitoring using Nmonit. In this setup the server on which we install nmonit would be the centralized management host from where we can manage other hosts on which monit is installed under a 'single pane of glass' view.

Installation & setup:

Monit is available in the epel repository as well as a source. I downloaded the source from the official website & it essentially consisted of the monit binary along with the monit configuration file monitrc.

The setup was fairly straightforward. I copied the monit binary to /etc/init.d & monitrc file to /etc. I commented out the 'use address' & 'allow' directives so that the monit GUI interface is accessible from other hosts within the same network.
To start the monit service just type 'monit'.

Once started the monit GUI interface will be accessible on

http://<IP address>:2812

Once started we can view the status of the attributes monitored by monit by running the command monit status. Monit polls the attributes every 5 seconds & the status of the monitored resources is refreshed.

Configuration:

The monit configuration file monitrc has a lot of sample configurations that we can use. A lot of configuration examples & other information can be found on the Monit wiki here.

Here are a few attributes I configured in the monitrc file for monit to keep an eye on:

[root@centops ~]# tail -15 /etc/monitrc
#

 check process sshd with pidfile /var/run/sshd.pid
   start program  "/etc/init.d/sshd start"
   stop program  "/etc/init.d/sshd stop"
   if failed port 22 protocol ssh then restart

 check process jobber with pidfile /var/run/jobber.pid
   start program  "/etc/init.d/jobber start"
   stop program  "/etc/init.d/jobber stop"

  check directory check_dir with path /check_dir
    if failed permission 755 then alert
    if failed uid 0 then alert
    if failed gid 0 then alert
[root@centops ~]#

In the above example, monit will monitor the status of sshd & jobber services & the ownership/permissions of directory /check_dir.

Here's a view of the GUI:

Demo:

Off course the article would be incomplete if there isn't a demo in the end. So, from the attributes we configured to be monitored lets pick the directory.

Here is the status of /check_dir directory:

Now If I play around with the permissions & change them to something other tahn 755 monit we notice this the next time it does polls the attribute & will throw an alert like we see in the screen below:

Exploring oterm (onion terminal)

Onion terminal is a UNIX terminal in a web browser. It's designed to be used securely with SSL but the user can skip the SSL part although lesser security is never recommended.

Its main feature are:

1. Standalone: Just the executable contains everything you need to serve the terminal.
2. Secure: It uses SSL to ensure you use a secure channel.
3. Authenticated: It uses PAM to check user authentication.
4. Session management: It exports as many sessions as wanted, per user. So its also a screen replacement.

Here are the implementation details:

This example uses opack to embed jquery and custom javascript and HTMLs. The handler itself creates the pty (forkpty, set permissions...) and stores all that on the user session. The input of data is made at oterm/PID/in, and it is a POST with the data at the parameter data. The output is in another channel, oterm/PID/out, and it blocks until there is some data available, then it reads as much as possible and sends it. The AJAX client when receives that data processes it, and asks for new data. With this simple in/out trick we can have the same functionality as WebSockets, but asynchronously.

Installation:

Oterm can be downloaded as a statically linked binary from the projects' github page here. On the github page it mentions that the binary was compiled on Ubuntu 11.04 machine but as it worked on my Centos test machine flawlessly. 

Usage:

The statically linked binary when downloaded runs 'right out of the box'. Executing it without any options lays out the usage details similar to a man page.

[root@centops ~]# ./oterm

[EB9E] [2016-09-24 11:22:59] [ERROR onion.c:710] Error setting the certificate (Error while reading file.)

Cant set certificate and key files (/etc/pki/tls/certs/pound.pem, /etc/pki/tls/certs/pound.key)

 oterm - Linux terminal over HTTP

It uses HTTP(S)+HTML+CSS+JavaScript to show a remote terminal inside a browser.

Options:

   -p|--port <port_number>      -- Set the port number to use. Default 8080

   -i|--ip   <server_ip>        -- Set the ip to attach to. Default ::

   -c|--cert <certificate file> -- Set the SSL certificate file. Default: /etc/pki/tls/certs/pound.pem

   -k|--key  <key file>         -- Set the SSL key file. Default: /etc/pki/tls/certs/pound.key

   --no-ssl                     -- Do not uses SSL. WARNING! Very unsecure

[root@centops ~]#

Since I did not have SSL keys setup with me, I tested the usage without them. To start it up I typed:

[root@centops ~]# ./oterm  --no-ssl --ip 192.168.44.138 --port 6789

Disabling SSL!

Using ip 192.168.44.138

Using port 6789

Listening at 6789

After this go to a web browser & type http://<IP address>:<port> . It will give you a password prompt.

I tried to login directly as root when presented with the prompt for credentials & was not allowed to do so. I logged in as a normal user & was able to switch to root.

Here's the page that immediately appeared after I logged in:

I clicked on create a new remote session & it created a session & returned a session id.

I clicked on the session id & this launched a terminal session for my logged in user in a new tab. After that I was able to run normal commands in the terminal window as the user & after I switched to the root user.

Conclusion:

I found oterm to be a fast, light & definitely cool web based terminal emulator.

Record & Replay superuser sessions with sudoreplay & sudossh

In this article I'll talk about two methods to record & replay terminal activity for commands executed as root user & replay the session at a later time.

• Method 1: Update required entries in sudoers file & replay the session using sudoreplay.
• Method 2: Use sudosh as a shell to record sessions and replay them with sudosh-replay.

Method 1: Using sudo/sudoreplay:

Sudo command logging is not enabled by default. To enable it add the following entries in /etc/sudoers file:

• log_output
• log_input
• iolog_dir

Here is a sample entry for users test & test1 in /etc/sudoers file:

Defaults:test,test1 log_output
Defaults:test,test1 log_input

Defaults iolog_dir=/var/log/sudo-io/%{user}

• log_output : log command output activity
• log_input: log command input activity
• ilog_dir: path to save session output

That's it. The next time the user logs in & executes command with sudo privileges, they will be logged & saved in /var/log/sudo-io/<user name> file.

Method 2: Using sudosh

The source code for sudosh can be obtained from github & is compatible with a wide range of *nix platforms. Sudosh was designed specifically to be used in conjunction with sudo or by itself as a login shell. 

Installation:

Download the source code from githhub. Compile & install the package:

1) ./configure

2) make

3) make install

4) sudosh -i

Usage:

To start logging via sudosh type the following command:

#sudo /usr/local/bin/sudosh

You can also add an entry in /etc/sudoers file as follows (example taken from README file):

-- /etc/sudoers begin --

User_Alias      ADMINS=admin1,admin2,admin3

User_Alias      DBAS=dba1,dba2,dba3

Cmnd_Alias      SUDOSH=/usr/local/bin/sudosh

ADMINS          ALL=SUDOSH

DBAS            ALL=(oracle)/usr/local/bin/sudosh

To start a sudosh logging session type /usr/local/bin/sudosh or sudo /usr/local/bin/sudosh on the command line & it'll start logging your session. To stop logging type exit.

The session logs get saved in /var/log/sudosh:

[root@centdb ttyrec-1.0.8]# ls -l /var/log/sudosh

total 24

-rw------- 1 root  root  714 Sep 21 21:48 root-root-script-1474519688-QGv8iZ9sXKhs830a

-rw------- 1 root  root  371 Sep 21 21:48 root-root-time-1474519688-QGv8iZ9sXKhs830a

-rw------- 1 test1 test1 380 Sep 22 01:24 root-test1-script-1474532682-LmbZCWgpi5xXBrrc

-rw------- 1 test1 test1 338 Sep 22 01:24 root-test1-time-1474532682-LmbZCWgpi5xXBrrc

-rw------- 1 test  test  617 Sep 21 21:53 root-test-script-1474519973-Y9mXUS3t2UY4wzl9

-rw------- 1 test  test  172 Sep 21 21:53 root-test-time-1474519973-Y9mXUS3t2UY4wzl9

To replay a session, first type sudosh-replay on the command line to get a list of available sessions.

[root@centdb ttyrec-1.0.8]# sudosh-replay

Date                Duration From         To           ID

====                ======== ====         ==           ==

09/21/2016 21:48:08 22s      root         root         root-root-1474519688-QGv8iZ9sXKhs830a

09/21/2016 21:52:53 8s       root         test         root-test-1474519973-Y9mXUS3t2UY4wzl9

09/22/2016 01:24:42 10s      root         test1        root-test1-1474532682-LmbZCWgpi5xXBrrc

Usage: sudosh-replay ID [MULTIPLIER] [MAXWAIT]

See 'sudosh-replay -h' for more help.

Example: sudosh-replay root-test1-1474532682-LmbZCWgpi5xXBrrc 1 2

Then from the ID column select the session you want to play & replay the session by typing sudosh followed by ID as in the example below:

[root@centdb ttyrec-1.0.8]# sudosh-replay root-test1-1474532682-LmbZCWgpi5xXBrrc

This will replay the recording of the session.

Demo:

Logging terminal activity with TILT & snoopy

In this article I'll explore usage to TILT & snoopy tools for logging terminal activity.

TILT (terminal interaction logging tool):

tilt logs user activity on the terminal via telnet/ssh to log files provided users login the systems using the telnet/ssh variants provided by tilt during the installation process. tilt provides a dumplog command using which we can view commands executed along with their output in the log as if they were executed on the terminal.

Installation:

tilt source code tilt-0.1.1.tgz can be downloaded from here. Once downloaded extract the source code.

# tar xvf tilt-0.1.1.tgz

Add a new user named tilt & create the directory /var/log/tilt.

#useradd tilt
#mkdir /var/log/tilt
#chmod 777 /var/log/tilt
# ls -ld /var/log/tilt
drwxrwxrwx 3 root root 4096 Sep 21 19:59 /var/log/tilt

To install the package run configure followed by make & make install.

#./configure

#make 

#make install

Usage:

Installing tilt will create the following binaries:

[root@centdb tilt-0.1.1]# ls -l /usr/local/tilt/bin/telnet

---s--x--x 1 tilt root 35472 Sep 21 19:56 /usr/local/tilt/bin/telnet

[root@centdb tilt-0.1.1]# ls -l /usr/local/tilt/bin/ssh

---s--x--x 1 tilt root 35472 Sep 21 19:56 /usr/local/tilt/bin/ssh

[root@centdb tilt-0.1.1]# ls -l /usr/local/tilt/bin/bash

---s--x--x 1 tilt root 35472 Sep 21 19:56 /usr/local/tilt/bin/bash

So, now to log a terminal session started via ssh, the user would need to use /usr/local/tilt/bin/ssh to initiate the ssh session. For example:

[root@centdb ~]# /usr/local/tilt/bin/ssh test@centdb

test@centdb's password:

Last login: Tue Sep 20 00:43:52 from centdb

[test@centdb ~]$ whoami

test

[test@centdb ~]$ date

Wed Sep 21 19:59:54 PDT 2016

[test@centdb ~]$ uptime

 19:59:56 up 6 min,  2 users,  load average: 0.08, 0.32, 0.18

[test@centdb ~]$ exit

logout

Connection to centdb closed.

Once the session is terminated a log of the session will be generated in /var/log/tilt.

[root@centdb ~]# ls -l /var/log/tilt/2016-09-21/

total 4

-r--r----- 1 tilt root 1949 Sep 21 19:59 B-root-2016-09-21-19:59:46--usr-local-tilt-bin-ssh test@centdb

[root@centdb ~]#

To view the contents of the log file use dumplog command provided as part of tilt installation.

[root@centdb ~]# /usr/local/tilt/bin/dumplog /var/log/tilt/2016-09-21/B-root-2016-09-21-19\:59\:46--usr-local-tilt-bin-ssh\ test\@centdb

test@centdb's password:

Last login: Tue Sep 20 00:43:52 from centdb

[test@centdb ~]$ whoami

test

[test@centdb ~]$ date

Wed Sep 21 19:59:54 PDT 2016

[test@centdb ~]$ uptime

 19:59:56 up 6 min,  2 users,  load average: 0.08, 0.32, 0.18

[test@centdb ~]$ exit

logout

Connection to centdb closed.

[root@centdb ~]#

Snoopy to log command line interaction:

Snoopy is designed to increase visibility of user activity being performed on the system by providing a log of commands  executed. Snoopy is completely transparent  to the user and applications it hooks in as a library providing a wrapper around calls to execve() calls. Logging is done via syslogd and written to authpriv allowing secure off site logging of activity. This article serves as a brief introduction towards snoopy usage. A more in depth article can be found here.

Installation:

You can get the snoopy source code from github & the rpm variant is available in the EPEL repository. I went with the rpm way.

[root@centdb /]# rpm -ivh snoopy-1.7.10-1.el6.x86_64.rpm

warning: snoopy-1.7.10-1.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY

Preparing...                ########################################### [100%]

   1:snoopy                 ########################################### [100%]

[root@centdb /]#

Snoopy can be enabled in one of two ways:

1) To enable for the whole system with a global preload.

echo "/lib64/snoopy.so" >> /etc/ld.so.preload

2) To enable in user space for a particular user then

   set LD_PRELOAD=/lib64/snoopy.so

Afterwards logs will be made via syslog to  /var/log/secure

Usage:

I updated my systems' /etc/ld.so.preload file to add an entry for snoopy.

[root@centdb sudosh]# cat /etc/ld.so.preload

/lib64/snoopy.so

I started a new session & ran a couple of commands & I could see the commands being logged in /var/log/secure. Here's a log snippet:

Sep 21 20:17:55 centdb snoopy[10094]: [uid:0 sid:10006 tty:/dev/pts/0 cwd:/root filename:/bin/ls]: ls --color=auto

Sep 21 20:18:00 centdb snoopy[10107]: [uid:0 sid:10006 tty:/dev/pts/0 cwd:/root filename:/usr/bin/whoami]: whoami

Sep 21 20:18:02 centdb snoopy[10113]: [uid:0 sid:10006 tty:/dev/pts/0 cwd:/root filename:/usr/bin/uptime]: uptime

Sep 21 20:18:46 centdb snoopy[10203]: [uid:0 sid:10006 tty:/dev/pts/0 cwd:/root filename:/usr/bin/sudo]: sudo su - test

Logging terminal activity with tlog

While researching for ways to record & replay a ssh terminal session, I came across a tool called tlog.
`Tlog` is a terminal I/O recording and playback package & I have come across other different ways to accomplish this. But the way tlog differs from these utilities lies in the fact that log is essentially designed to be used a centralized user activity logging mechanism.
The makers have mentioned in the README file that presently tlog is not a production ready software & should be considered as a development preview only.
Under default settings tlog sends its recorded sessions to a centralized recording facility (syslog). Its recordings can also be re-directed to ElasticSearch. The recordings are in JSON format.

`Tlog` is naturally split into two tools: `tlog-rec` and `tlog-play` - for recording and playback respectively. `Tlog-rec` is intended to be the user's login shell. It puts itself between the actual user's shell and the terminal upon user login, logging everything that passes through. At the moment,
`tlog-play` can playback recordings from ElasticSearch or from a file written by `tlog-rec` with `file` writer selected.

Testing

You can test that session recording and playback work in general by recording
into and playing back from a file.

To record into a file execute tlog-rec on the command line as such:

    tlog-rec --writer=file --file-path=tlog.log

After exiting the recorded session you can play it back with tlog-play:

    tlog-play --reader=file --file-path=tlog.log

Recommended usage:

Change the shell of the user to be recorded to `tlog-rec`:

    sudo chsh -s /usr/bin/tlog-rec <user>

Login as the user on a text terminal. The recorded terminal data will be

delivered to syslog with facility "authpriv" and priority "info", and may

appear in `/var/log/auth.log` on Debian-based systems, or in `/var/log/secure`

on Fedora and derived systems.

Demo:

I first tried with the recommended usage & created a user called test & changed the shell.

[root@centops log]# getent passwd test

test:x:501:501::/home/test:/usr/bin/tlog-rec

I then logged in to the server as test user & ran a couple of commands & exited.

The session output did get saved to /var/log/secure file on my centos 6 server but not in a very readable format:

Sep 21 10:30:40 centops tlog: {"host":"centops","user":"test","session":7,"id":1,"pos":0,"timing":"=120x45+14>45+1642<1+2>1+93<1+1>1+129<1+2>1+64<1+2>1+247<1+3>1+588<1+1>2+5>318+956<1+2>1+186<1+3>1+227<1+2>1+189<1+3>1+153<1+2>1+157<1+3>1+402<1+2>73+1302<1+2>1+187<1+2>1+219<1+4>1+431<1+1>1+157<1+2>1+248<1+2>1+280<1+1>45+1366<1+1>8","in_txt":"df -h\rwhomai\rwhoami\r\u0004","in_bin":[],"out_txt":"\u001b]0;test@centops:~\u0007\u001b[?1034h[test@centops ~]$ df -h\r\nFilesystem            Size  Used Avail Use% Mounted on\r\n/dev/sda2              18G  9.4G  7.2G  57% /\r\ntmpfs                 491M   68K  491M   1% /dev/shm\r\n/dev/sda1             283M   34M  234M  13% /boot\r\n/dev/mapper/vg01-lv00\r\n                      2.0G  352M  1.5G  19% /R_D\r\n\u001b]0;test@centops:~\u0007[test@centops ~]$ whomai\r\n-bash: whomai: command not found\r\n\u001b]0;test@centops:~\u0007[test@centops ~]$ whoami\r\ntest\r\n\u001b]0;test@centops:~\u0007[test@centops ~]$ logout\r\n","out_bin":[]}

Sep 21 10:30:40 centops su: pam_unix(su-l:session): session closed for user test

It seems that the timing & character information along with session output is being re-directed together to a single file.

To replay the session via tlog-play I needed to extract the session log from the /var/log/secure file, remove the time stamp & put it in a new file & then play the recording:

Here's the log extracted & put into a file:

[root@centops ~]# cat t.log

{"host":"centops","user":"test","session":7,"id":1,"pos":0,"timing":"=120x45+14>45+1642<1+2>1+93<1+1>1+129<1+2>1+64<1+2>1+247<1+3>1+588<1+1>2+5>318+956<1+2>1+186<1+3>1+227<1+2>1+189<1+3>1+153<1+2>1+157<1+3>1+402<1+2>73+1302<1+2>1+187<1+2>1+219<1+4>1+431<1+1>1+157<1+2>1+248<1+2>1+280<1+1>45+1366<1+1>8","in_txt":"df -h\rwhomai\rwhoami\r\u0004","in_bin":[],"out_txt":"\u001b]0;test@centops:~\u0007\u001b[?1034h[test@centops ~]$ df -h\r\nFilesystem            Size  Used Avail Use% Mounted on\r\n/dev/sda2              18G  9.4G  7.2G  57% /\r\ntmpfs                 491M   68K  491M   1% /dev/shm\r\n/dev/sda1             283M   34M  234M  13% /boot\r\n/dev/mapper/vg01-lv00\r\n                      2.0G  352M  1.5G  19% /R_D\r\n\u001b]0;test@centops:~\u0007[test@centops ~]$ whomai\r\n-bash: whomai: command not found\r\n\u001b]0;test@centops:~\u0007[test@centops ~]$ whoami\r\ntest\r\n\u001b]0;test@centops:~\u0007[test@centops ~]$ logout\r\n","out_bin":[]}

Here's a quick demo of the playback:

Record & replay Linux terminal sessions using script/scriptreplay & convert them to GIFs using congif.

The script command is available by default on most *nix distributions & is a neat tool for capturing the output of our terminal session to a file. To run the script command type

#script <file name>

If no file name is specified then the output is saved to a file called typescript in the directory from where script command was run.

We can also append session output to the script command by typing

#script -a <file name>

Script combined with scriptreplay command provides an easy to use yet powerful functionality to replay a terminal session recorded with script command.

The catch is that we need to capture the timing information in a time file. The time file contains two pieces of information. The time elapsed since characters were last displayed on the screen & the number of characters typed on the display now.

The timing information is redirected to standard error. So to start the script session while capturing timing information type:

script -t 2> time.txt cmd.txt

To use scriptreplay to replay the session type:

scriptreplay time.txt cmd.txt

Here's a live demo:

Using congif to convert a script session recording into a GIF.

Congif is a tool tool to convert script session recordings to GIF files. You can download congif from github. Once downloaded extract the source code & run make to install the congif binary. 

To create a GIF using congif type:

./congif -l 0 /root/time.txt /root/cmd.txt

• -l 0 means to replay the GIF in an infinite loop.
• time.txt is the timing information file
• cmd.txt is the script command output file.

The output of this command will be a GIF file called con.gif created in the current working directory.

Here's a quick demo:

Setting up Icinga monitoring tool on CentOS

Icinga is an open source computer system and network monitoring application. It has been forked from Nagios & is compatible with most of Nagios plugins & as we'll see in this article NRPE package which is installed at client end for Nagios can be used with Icinga as well.

In this article we'll install icinga on Centos 6.

Pre-requisites:

Since this is a lab setup, I've taken the liberty to disable SELinux & IPtables.
We need to install some packages for the web interface & database backend.

# yum -y install wget httpd mod_ssl gd gd-devel mysql-server php-mysql php-xmlrpc gcc mysql libdbi libdbi-devel libdbi-drivers libdbi-dbd-mysql

Create icinga user and icinga-cmd group, add  icinga and apache user to the part of icinga-cmd group.

#useradd icinga
#groupadd icinga-cmd
#usermod -a -G icinga-cmd icinga
#usermod -a -G icinga-cmd apache

Download the icinga source code icinga-1.10.1.tar.gz & extract it.

Compile & install icinga:


# ./configure --with-command-group=icinga-cmd --enable-idoutils
# make all
# make install
# make install-init
# make install-config
# make install-commandmode
# make install-webconf
# make install-idoutils

Configure icinga:

Move sample idoutils configuration files to Icinga base directory.

# cd /usr/local/icinga/etc/
# mv idomod.cfg-sample idomod.cfg
# mv ido2db.cfg-sample ido2db.cfg
# cd modules/

# mv idoutils.cfg-sample idoutils.cfg

Create database for idoutils:

[root@centops etc]# service mysqld start
Initializing MySQL database:  Installing MySQL system tables...
OK

[root@centops etc]# chkconfig mysqld on

# mysql -u root
mysql> CREATE DATABASE icinga;
Query OK, 1 row affected (0.00 sec)
mysql> GRANT USAGE ON icinga.* TO 'icinga'@'localhost' IDENTIFIED BY 'icinga';
Query OK, 0 rows affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
mysql> quit

Bye

Import the database:

[root@centops etc]# mysql -u root icinga < /R_D/icinga-1.10.1/module/idoutils/db/mysql/mysql.sql

[root@centops etc]#

Configure the web interface:

Create a icingaadmin user for logging into the web interface. 

[root@centops etc]#  htpasswd -c /usr/local/icinga/etc/htpasswd.users icingaadmin
New password:
Re-type new password:

Adding password for user icingaadmin

Download & install Nagios plugins:

#tar zxvf nagios-plugins-2.1.3.tar.gz
#cd nagios-plugins-2.1.3
# ./configure --prefix=/usr/local/icinga --with-cgiurl=/icinga/cgi-bin --with-nagios-user=icinga --with-nagios-group=icinga
# make
# make install

Start icinga:

Before we start the service it's good to be a sanity check to make sure that our configuration is ok.

[root@centops etc]# /usr/local/icinga/bin/icinga -v /usr/local/icinga/etc/icinga.cfg

If there are no errors we can start up the services & configure them to start on boot.

[root@centops ~]# /etc/init.d/icinga start

Running configuration check...OK

Starting icinga: Starting icinga done.

[root@centops ~]# /etc/init.d/ido2db start

Starting ido2db: done.

[root@centops ~]#

#chkconfig ido2db on

#chkconfig icinga on

[root@centops ~]# service httpd start

Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.44.138 for ServerName

                                                           [  OK  ]

[root@centops ~]# chkconfig httpd on

[root@centops ~]#

[root@centops ~]# service mysqld status

mysqld (pid  2961) is running...

[root@centops ~]# chkconfig mysqld on

[root@centops ~]#

The web interface will be accessible via the URL http://<IP address>/icinga/

Client side configuration:

A cool thing about icinga is that perhaps since it's a Nagios fork the NRPE package serves as a client for icinga too. The details for installing NRPE can be found in the Nagios core set up article.

Once NRPE is installed we move to the icinga server side configuration to add the client.

Create the client template file:

Go to the path /usr/local/icinga/etc/objects/ & make a copy of the localhost.cfg file. This is the client template file for linux.

#cd /usr/local/icinga/etc/objects/

#cp localhost.cfg icingaclient.cfg

In the file linuxclient.cfg replace all occurrences of localhost with the hostname of the client & update the address filed with the IP address of the client.

# sed -i 's/localhost/rheldb/g' icingaclient.cfg

# grep address icingaclient.cfg

        address                 192.168.44.135

#chown icinga.icinga icingaclient.cfg

Update the path to the client template file in the icinga.cfg file:

#  grep icingaclient.cfg icinga.cfg

cfg_file=/usr/local/icinga/etc/objects/icingaclient.cfg

This needs to be done every time a new client is added.

Restart the icinga service:

#service icinga restart

#service ido2db restart

Now open up the URL again. Now we should now be able to see the client we configured to be monitored by icinga.

Logging user activity with pam_tty_audit

A specific components of the audit system uses the pam_tty_audit.so PAM module to enable or disable auditing of TTY input for specified users.
When the user logs in, pam_tty_audit.so records the exact keystrokes the user makes into the /var/log/audit/audit.log file.
This module is flexible in the sense that it allows us to enable or diable logging for specific users.
pam_tty_audit.so captures keystrokes including backspaces etc so we can expect some junk characters in the output.

Since this is a auditd component, we can use the aureport with the -tty parameters to report all record logged by the module.

The set up:

Add the line "session     required      pam_tty_audit.so enable=*" in the following files to enable keystroke logging for all users:

• /etc/pam.d/password-auth
• /etc/pam.d/system-auth
• /etc/pam.d/sshd

The file password-auth/system-auth will be as follows:

[root@centdb pam.d]# cat password-auth
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 500 quiet
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 500 quiet
account     required      pam_permit.so

password    requisite     pam_cracklib.so try_first_pass retry=3 type=
password    sufficient    pam_unix.so md5 shadow nullok try_first_pass use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
session     required      pam_tty_audit.so enable=*
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

The /etc/pam.d/sshd file will be:

[root@centdb pam.d]# cat sshd

#%PAM-1.0

auth       required     pam_sepermit.so

auth       include      password-auth

account    required     pam_nologin.so

account    include      password-auth

password   include      password-auth

# pam_selinux.so close should be the first session rule

session    required     pam_selinux.so close

session    required     pam_loginuid.so

session    required      pam_tty_audit.so enable=*

# pam_selinux.so open should only be followed by sessions to be executed in the user context

session    required     pam_selinux.so open env_params

session    optional     pam_keyinit.so force revoke

session    include      password-auth

[root@centdb pam.d]#

Restart the auditd service.

We can now view the user keystroke logging reports in aureport/ausearch as well as view logs in /var/log/audit/audit.log.

We can view the terminal logs of users for today as follows:

[root@centdb audit]#  aureport --tty -ts today

TTY Report

===============================================

# date time event auid term sess comm data

===============================================

1. 09/19/2016 01:14:42 1800 503 ? 211 bash "df -h",<ret>

2. 09/19/2016 01:14:45 1806 503 ? 211 bash "date",<ret>

3. 09/19/2016 01:14:44 1803 503 ? 211 bash "uptime",<ret>

4. 09/19/2016 01:15:00 1810 503 ? 211 bash <^D>

5. 09/19/2016 01:28:16 1869 503 ? 211 bash <ret>,"uptime",<ret>,"date",<ret>,"date",<ret>,"sudo su -",<ret>,"  whoami",<ret>,"w",<ret>,<ret>,<ret>,"echo ??",<backspace>,<backspace>,"$?",<ret>,<ret>,<ret>,"df -h",<ret>,"uname -a",<ret>,"whoami",<ret>,<ret>,<ret>,<ret>,<ret>,<up>,<ret>,<ret>,<up>,<up>,<ret>,<ret>,<ret>,<up>,<up>,<up>,<up>,<ret>,<^D>

6. 09/19/2016 01:33:25 1943 503 ? 214 bash <ret>

7. 09/19/2016 01:33:28 1945 503 ? 214 bash "whoami",<ret>

8. 09/19/2016 01:49:27 1989 503 ? 214 bash <^D>

9. 09/19/2016 01:49:28 1994 503 ? 214 bash <nl>,"df -h",<ret>,"uname -a",<ret>,"date",<ret>,"uptime",<ret>,"sudo su -",<ret>,<^D>

10. 09/19/2016 01:50:29 2054 503 ? 219 bash "uptime",<ret>

11. 09/19/2016 01:50:30 2057 503 ? 219 bash "date",<ret>

12. 09/19/2016 01:50:32 2060 503 ? 219 bash "eit",<ret>

13. 09/19/2016 01:50:35 2068 503 ? 219 bash <ret>,"uptime",<ret>,"sudo su -",<ret>,"exit",<ret>

14. 09/19/2016 01:50:34 2062 503 ? 219 bash "exit",<ret>

15. 09/19/2016 01:51:16 2127 503 ? 221 bash "fdisk -l",<ret>

16. 09/19/2016 01:51:18 2130 503 ? 221 bash <ret>

17. 09/19/2016 01:51:19 2132 503 ? 221 bash "date",<ret>

18. 09/19/2016 01:51:21 2135 503 ? 221 bash <ret>

19. 09/19/2016 01:51:27 2137 503 ? 221 bash "dmesg",<ret>

20. 09/19/2016 01:51:31 2140 503 ? 221 bash "exit",<ret>

21. 09/19/2016 01:51:33 2146 503 ? 221 bash "uptime",<ret>,<ret>,"date",<ret>,"sudo su -",<ret>,"exit",<ret>

[root@centdb audit]#

We can view the logs for a specific time window as well as shown below:

[root@centdb audit]#  aureport --tty -ts 09/19/2016 -te 09/19/2016

TTY Report

===============================================

# date time event auid term sess comm data

===============================================

1. 09/19/2016 01:14:42 1800 503 ? 211 bash "df -h",<ret>

2. 09/19/2016 01:14:45 1806 503 ? 211 bash "date",<ret>

3. 09/19/2016 01:14:44 1803 503 ? 211 bash "uptime",<ret>

The cool thing I found in the aureport output was that even after I switched to root, the original UID of the logged in user was still retained while running commands.

We can also filter events by a specific user as shown in the following example:

[root@centdb audit]#  ausearch -ui 503 --interpret  | more

----

type=USER_CMD msg=audit(09/19/2016 01:14:41.237:1782) : user pid=58871 uid=test auid=test ses=211 msg='cwd=/home/test cmd=su - termina

l=pts/2 res=success'

----

type=PATH msg=audit(09/19/2016 01:14:41.228:1781) : item=1 name=(null) inode=659194 dev=08:02 mode=file,755 ouid=root ogid=root rdev=0

0:00 nametype=NORMAL

type=PATH msg=audit(09/19/2016 01:14:41.228:1781) : item=0 name=/usr/bin/sudo inode=808774 dev=08:02 mode=file,suid,111 ouid=root ogid

=root rdev=00:00 nametype=NORMAL

type=CWD msg=audit(09/19/2016 01:14:41.228:1781) :  cwd=/home/test

Logging root user activity with rootsh

Rootsh is a tool that can help to log all activities performed by a user assuming sudo privileges while working on the system. This is useful when there is a "WHO DID IT??" situation.
To use roothsh start a shell with logging of input/output. Rootsh must be started via sudo if you want to become root. It does not raise your privileges on itâs own.You can run rootsh as a standalone application if you only want  to log your own user session. If you call rootsh with additional commands, these will be passed to the shell.

The rootsh source code is available on source forge. Download & extract the source code.

[root@centdb /]# tar xvf rootsh-1.5.3.tar.gz
rootsh-1.5.3/
rootsh-1.5.3/contrib/
rootsh-1.5.3/contrib/linux/
rootsh-1.5.3/contrib/linux/rootsh.spec
rootsh-1.5.3/contrib/aix/
rootsh-1.5.3/contrib/aix/rootsh/
rootsh-1.5.3/contrib/aix/rootsh/rootsh/
rootsh-1.5.3/contrib/aix/rootsh/rootsh/config
rootsh-1.5.3/contrib/aix/rootsh/rootsh/root/
rootsh-1.5.3/contrib/aix/rootsh/rootsh/root/usr/
rootsh-1.5.3/contrib/aix/rootsh/rootsh/root/usr/share/
rootsh-1.5.3/contrib/aix/rootsh/rootsh/root/usr/share/man/
rootsh-1.5.3/contrib/aix/rootsh/rootsh/root/usr/share/man/man1/
rootsh-1.5.3/contrib/aix/rootsh/rootsh/root/usr/share/man/man1/copy_your_rootsh.1_here
rootsh-1.5.3/contrib/aix/rootsh/rootsh/root/usr/bin/
rootsh-1.5.3/contrib/aix/rootsh/rootsh/root/usr/bin/COPY_YUR_ROOTSH_BINARY_HERE

------------------------------------------------------------------------------------------------------------------------

Now we compile the source source & create the binaries from the compiled source code:

[root@centdb rootsh-1.5.3]# ./configure --disable-syslog --disable-linenumbering

checking for a BSD-compatible install... /usr/bin/install -c

checking whether build environment is sane... yes

checking for gawk... gawk

checking whether make sets $(MAKE)... yes

checking build system type... x86_64-unknown-linux-gnu

checking host system type... x86_64-unknown-linux-gnu

checking for syslog... no

checking for logfile... enabled

checking for logdir... /var/log/rootsh

checking for maximum logfile size... 0

checking for defaultshell... /bin/bash

checking for gcc... gcc

checking for C compiler default output file name... a.out

------------------------------------------------------------------------------------------------------------------------

[root@centdb rootsh-1.5.3]# make

Making all in src

make[1]: Entering directory `/rootsh-1.5.3/src'

make  all-am

make[2]: Entering directory `/rootsh-1.5.3/src'

if gcc -DHAVE_CONFIG_H -I. -I. -I.     -g -O2 -Wall -Wstrict-prototypes -Wmissing-declarations   -Wnested-externs -Wpointer-arith -pedantic -pedantic -Wstrict-prototypes -MT rootsh.o -MD -MP -MF ".deps/rootsh.Tpo" -c -o rootsh.o rootsh.c; \

        then mv -f ".deps/rootsh.Tpo" ".deps/rootsh.Po"; else rm -f ".deps/rootsh.Tpo"; exit 1; fi

rootsh.c:183: warning: function declaration isnât a prototype

rootsh.c:299:2: warning: C++ style comments are not allowed in ISO C90

rootsh.c:299:2: warning: (this will be reported only once per input file)

rootsh.c: In function âmainâ:

rootsh.c:345: warning: ISO C90 forbids mixed declarations and code

------------------------------------------------------------------------------------------------------------------------

[root@centdb rootsh-1.5.3]# make install

Making install in src

make[1]: Entering directory `/rootsh-1.5.3/src'

make[2]: Entering directory `/rootsh-1.5.3/src'

test -z "/usr/local/bin" || mkdir -p -- "/usr/local/bin"

  /usr/bin/install -c 'rootsh' '/usr/local/bin/rootsh'

make[2]: Nothing to be done for `install-data-am'.

make[2]: Leaving directory `/rootsh-1.5.3/src'

make[1]: Leaving directory `/rootsh-1.5.3/src'

make[1]: Entering directory `/rootsh-1.5.3'

make[2]: Entering directory `/rootsh-1.5.3'

make[2]: Nothing to be done for `install-exec-am'.

test -z "/usr/local/man/man1" || mkdir -p -- "/usr/local/man/man1"

 /usr/bin/install -c -m 644 './rootsh.1' '/usr/local/man/man1/rootsh.1'

make[2]: Leaving directory `/rootsh-1.5.3'

make[1]: Leaving directory `/rootsh-1.5.3'

------------------------------------------------------------------------------------------------------------------------

Create a directory /var/log/rootsh where rootsh will store all its logs.

#mkdir /var/log/rootsh

Now for users having sudo access which you'd like to log via rootsh, add the following line in their .bash_profile file like I've done below for user test:

[root@centdb rootsh]# cat /home/test/.bash_profile

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

        . ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

sudo /usr/local/bin/rootsh

Now log in to the system as the user & run some commands. Rootsh will log the commands as shown below:

[root@centdb rootsh]# pwd

/var/log/rootsh

[root@centdb rootsh]# ll

total 4

--w-r-s--x 1 root root 865 Sep 18 23:51 test.20160918235054.0b36c.closed

[root@centdb rootsh]# cat test.20160918235054.0b36c.closed

rootsh session opened for test as root on /dev/pts/2 at Sun Sep 18 23:50:54 2016

[root@centdb test]# sudo su -

[root@centdb ~]# df -h

Filesystem      Size  Used Avail Use% Mounted on

/dev/sda2        18G   11G  5.9G  65% /

tmpfs           495M  255M  240M  52% /dev/shm

/dev/sda1       283M   37M  232M  14% /boot

.host:/         369G  144G  225G  39% /mnt/hgfs

[root@centdb ~]# date

Sun Sep 18 23:51:10 PDT 2016

[root@centdb ~]# uptime

 23:51:11 up  4:13,  3 users,  load average: 2.35, 1.68, 0.77

[root@centdb ~]# hostname

centdb

[root@centdb ~]# logout

[root@centdb test]# exit

*** rootsh session ended by user

rootsh session closed for test on /dev/pts/2 at Sun Sep 18 23:51:20 2016

[root@centdb rootsh]#

The file name breakup for the file test.20160918235054.0b36c.closed is:

<user>.<timestampt>.<PID>.<session status>