Recovering from a 'chmod -r 777 /'

Running a 'chmod -r 777 /' will have catastrophic results on the functioning of any solaris system.

This will prevent most of the OS services & functions from working correctly including ssh & the server will enter maintenance mode if rebooted.

It happened with me a few days ago & this post is about the tasks performed to get the system back up to a stable state.

The first corrective action was to restore /etc to the default mask i.e 755 for directories & 644 for files.

find /etc -type d -exec chmod 755 {} \;

find /etc -type f -exec chmod 644 {} \;

The next critical task was to restore services & associated binaries to their expected permissions & ownerships. This task was accomplished with pkgchk command.

pkgchk checks the accuracy of installed files or, by using the  -l  option, displays information about package files. pkgchk checks the  integrity of  directory  structures  and files. Discrepancies are written to standard error along with a detailed explanation of the problem.

To fix the permissions I ran

# /usr/sbin/pkgchk -f 

This command correct file attributes if possible. (permission, datestamp, ownership) 

then to check the binaries: 

# /usr/sbin/pkgchk -n 

After that reboot the server

# /usr/sbin/init 6 

Once the server is up, check the status of smf services: 

# /usr/bin/svcs -xv

After the reboot the services were operating optimally & the server was accessible remotely via ssh. There were some other changes needed which I came across via trial & error but pkgchk did most of the heavy lifting. 

ZFS rename

With 'zfs rename'subcommand we can do than renaming the file system.
We can accomplish the following operations:

• Change the file systems' name
• Relocate the file system within ZFS hierarchy.
• Rename the file system & relocate it within ZFS hierarchy.

For relocating the file system the following conditions must be satisfied:

• The file system can be relocated in the hierarchy of the same pool.
• If there are quota/reservation properties set then there must be enough space available in the destination location in the hierarchy to complete the move operation.

Examples:

I have the following ZFS file system hierarchy under the pool ztest:

root@solarislab:~# zfs list -r ztest
NAME               USED  AVAIL  REFER  MOUNTPOINT
ztest              302K   976M    32K  none
ztest/z1           124K   976M    31K  none
ztest/z1/z2         93K   976M    31K  none
ztest/z1/z2/z3      62K   976M    31K  none
ztest/z1/z2/z3/z4   31K   976M    31K  none

Now to rename z4 to z5 type the following:

root@solarislab:~# zfs rename ztest/z1/z2/z3/z4 ztest/z1/z2/z3/z9

root@solarislab:~#

root@solarislab:~#  zfs list -r  ztest

NAME               USED  AVAIL  REFER  MOUNTPOINT

ztest              304K   976M    32K  none

ztest/z1           124K   976M    31K  none

ztest/z1/z2         93K   976M    31K  none

ztest/z1/z2/z3      62K   976M    31K  none

ztest/z1/z2/z3/z9   31K   976M    31K  none

ztest/zfstest       31K   976M    31K  /test

To move z9 file system from its current location to immediately after z2 in the hierarchy, type:

root@solarislab:~# zfs rename ztest/z1/z2/z3/z9 ztest/z1/z2/z9

root@solarislab:~#  zfs list -r  ztest

NAME            USED  AVAIL  REFER  MOUNTPOINT

ztest           304K   976M    32K  none

ztest/z1        124K   976M    31K  none

ztest/z1/z2      93K   976M    31K  none

ztest/z1/z2/z3   31K   976M    31K  none

ztest/z1/z2/z9   31K   976M    31K  none

ztest/zfstest    31K   976M    31K  /test

Configuring client side authentication with LDAP in CentOS 7

In order to allow clients to be able to authenticate LDAP users we need to perform the following series of steps:

# yum install -y openldap-clients nss-pam-ldapd
# authconfig-tui

Do the following to enable LDAP authentication:

1. Put '*' Mark on "Use LDAP"
2. Put '*' Mark on "Use LDAP Authentication"
3. Select "Next" and Enter.
4. Enter the server field as "ldap://linux1.learnitguide.net/"
5. Enter the Base DN Field as "dc=learnitguide,dc=net"
6. Select "OK" and Enter

2. Test the Client Configuration.

Search the ldap user using the below command and check the output. If you get output, then our LDAP Configurations are working properly.

[root@cent ~]# getent passwd ldapuser1
ldapuser1:x:1005:1006:ldapuser1:/home/ldapuser1:/bin/bash
[root@cent ~]#

For the home directory to be mounted automatically you'd need to configure autofs or add an NFS entry in the /etc/fstab file.

Configuring openLDAP in CentOS 7

LDAP, or Lightweight Directory Access Protocol is a protocol for centrally managing related information. The most common & well known use of LDAP is as for centralized authentication. LDAP is a directory service. A directory service is a specialized database optimized for read access i.e. searching & browsing.
In LDAP information is managed in the form of entries that are arranged in a hierarchical structure. An entry is a collection of attributes that have a globally-unique Distinguished Name (DN).

With that clarified we'll start with LDAP configuration. I'll be explaining common terms as we come across them during the configuration.

I've used host instructor.example.com for my LDAP server configuration. There is no DNS so I'm using /etc/hosts file for name resolution. SElinux is in permissive mode & the firewall is turned off.

LDAP has many configuration files & the entries are case sensitive. One must be very cautious while editing the files.

Installation process:

Install the required packages:

yum install openldap openldap-clients openldap-servers migrationtools

Generate a LDAP password (in this case redhat)

slappasswd -s redhat -n > /etc/openldap/passwd

The encrypted password is in the file.

# cat /etc/openldap/passwd
{SSHA}YrzPc3BFXAcWhEgMVmhrQoZ03h5INxix

Generate a self signed certificate valid for 365 days:

openssl req -new -x509 -nodes -out /etc/openldap/certs/cert.pem -keyout /etc/openldap/certs/priv.pem -days 365
Generating a 2048 bit RSA private key
.....+++
..........................................................+++
writing new private key to '/etc/openldap/certs/priv.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:
State or Province Name (full name) []:
Locality Name (eg, city) [Default City]:
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:instructor.example.com

Email Address []:

Modify the permissions/ownership of the certificates in /etc/openldap/certs directory:

# cd /etc/openldap/certs
# chown ldap:ldap *
# chmod 600 priv.pem

Now prepare the LDAP database:

# cp /usr/share/openldap-servers/DB_CONFIG.example /var/lib/ldap/DB_CONFIG

Verify the configuration:

#slaptest

Change LDAP database ownership:

# chown ldap:ldap /var/lib/ldap/*


To start the configuration of the LDAP server, add the cosine & nis LDAP schemas:

 ldapadd -Y EXTERNAL -H ldapi:/// -D "cn=config" -f cosine.ldif
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
adding new entry "cn=cosine,cn=schema,cn=config"
ldap_add: Other (e.g., implementation specific) error (80)

        additional info: olcAttributeTypes: Duplicate attributeType: "0.9.2342.19200300.100.1.2"

ldapadd -Y EXTERNAL -H ldapi:/// -D "cn=config" -f nis.ldif

SASL/EXTERNAL authentication started

SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth

SASL SSF: 0

adding new entry "cn=nis,cn=schema,cn=config"

ldap_add: Other (e.g., implementation specific) error (80)

        additional info: olcAttributeTypes: Duplicate attributeType: "1.3.6.1.1.1.1.2"

Schemas are packaging units & are used for referencing multiple objects together rather than having to reference the objects as individual entities.

Schemas consist of Objectclasses & attributes. Objectclasses contain sets of attributes & the attributes typically contain some data.

Now, create the /etc/openldap/changes.ldif file and paste the following lines. 

Replace olcRootPW with the encrypted password you created with slappasswd command

[root@instructor ~]# cat /etc/openldap/changes.ldif

dn: olcDatabase={2}hdb,cn=config

changetype: modify

replace: olcSuffix

olcSuffix: dc=example,dc=com

dn: olcDatabase={2}hdb,cn=config

changetype: modify

replace: olcRootDN

olcRootDN: cn=Manager,dc=example,dc=com

dn: olcDatabase={2}hdb,cn=config

changetype: modify

replace: olcRootPW

olcRootPW: {SSHA}YrzPc3BFXAcWhEgMVmhrQoZ03h5INxix

dn: cn=config

changetype: modify

replace: olcTLSCertificateFile

olcTLSCertificateFile: /etc/openldap/certs/cert.pem

dn: cn=config

changetype: modify

replace: olcTLSCertificateKeyFile

olcTLSCertificateKeyFile: /etc/openldap/certs/priv.pem

dn: cn=config

changetype: modify

replace: olcLogLevel

olcLogLevel: -1

dn: olcDatabase={1}monitor,cn=config

changetype: modify

replace: olcAccess

olcAccess: {0}to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" read by dn.base="cn=Manager,dc=example,dc=com" read by * none

Using the file created above, update the LDAP server configuration:

# ldapmodify -Y EXTERNAL -H ldapi:/// -f /etc/openldap/changes.ldif

SASL/EXTERNAL authentication started

SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth

SASL SSF: 0

modifying entry "olcDatabase={2}hdb,cn=config"

modifying entry "olcDatabase={2}hdb,cn=config"

modifying entry "olcDatabase={2}hdb,cn=config"

modifying entry "cn=config"

modifying entry "cn=config"

modifying entry "cn=config"

modifying entry "olcDatabase={1}monitor,cn=config"

Create the /etc/openldap/base.ldif with the following content:

[root@instructor ~]# cat /etc/openldap/base.ldif

dn: dc=example,dc=com

dc: example

objectClass: top

objectClass: domain

dn: ou=People,dc=example,dc=com

ou: People

objectClass: top

objectClass: organizationalUnit

dn: ou=Group,dc=example,dc=com

ou: Group

objectClass: top

objectClass: organizationalUnit

Build the directory service structure:

# ldapadd -x -w redhat -D cn=Manager,dc=example,dc=com -f /etc/openldap/base.ldif

adding new entry "dc=example,dc=com"

adding new entry "ou=People,dc=example,dc=com"

adding new entry "ou=Group,dc=example,dc=com"

Create a user for testing:

# mkdir /home/guests

# useradd -d /home/guests/ldapuser01 ldapuser01

# passwd ldapuser01

Now we migrate this user from local file to LDAP:

cd /usr/share/migrationtools

Edit the migrate_common.ph file and replace in the following lines:

$DEFAULT_MAIL_DOMAIN = "example.com";

$DEFAULT_BASE = "dc=example,dc=com";

Import the user to the LDAP database:

# grep ":10[0-9][0-9]" /etc/passwd > passwd

# ./migrate_passwd.pl passwd users.ldif

# ldapadd -x -w redhat -D cn=Manager,dc=example,dc=com -f users.ldif

adding new entry "uid=sa,ou=People,dc=example,dc=com"

adding new entry "uid=stack,ou=People,dc=example,dc=com"

adding new entry "uid=james,ou=People,dc=example,dc=com"

adding new entry "uid=test,ou=People,dc=example,dc=com"

adding new entry "uid=sftptest,ou=People,dc=example,dc=com"

adding new entry "uid=ldapuser1,ou=People,dc=example,dc=com"

adding new entry "uid=ldapuser2,ou=People,dc=example,dc=com"

adding new entry "uid=ldapuser01,ou=People,dc=example,dc=com"

# grep ":10[0-9][0-9]" /etc/group > group

# ./migrate_group.pl group groups.ldif

# ldapadd -x -w redhat -D cn=Manager,dc=example,dc=com -f groups.ldif

adding new entry "cn=sa,ou=Group,dc=example,dc=com"

adding new entry "cn=stack,ou=Group,dc=example,dc=com"

adding new entry "cn=libvirtd,ou=Group,dc=example,dc=com"

adding new entry "cn=james,ou=Group,dc=example,dc=com"

adding new entry "cn=test,ou=Group,dc=example,dc=com"

adding new entry "cn=sftpusers,ou=Group,dc=example,dc=com"

adding new entry "cn=ldapuser1,ou=Group,dc=example,dc=com"

adding new entry "cn=ldapuser2,ou=Group,dc=example,dc=com"

adding new entry "cn=ldapuser01,ou=Group,dc=example,dc=com"

To test the configuration for ldapuser01:

# ldapsearch -x cn=ldapuser01 -b dc=example,dc=com

# extended LDIF

#

# LDAPv3

# base <dc=example,dc=com> with scope subtree

# filter: cn=ldapuser01

# requesting: ALL

#

# ldapuser01, People, example.com

dn: uid=ldapuser01,ou=People,dc=example,dc=com

uid: ldapuser01

cn: ldapuser01

sn: ldapuser01

mail: ldapuser01@example.com

objectClass: person

objectClass: organizationalPerson

objectClass: inetOrgPerson

objectClass: posixAccount

objectClass: top

objectClass: shadowAccount

userPassword:: e2NyeXB0fSQ2JGhWREtZSUtoJDRaajBqYlU3MzI3ZTJ3YkdmYW8uNkhQb2NZdmp

 hNGpzaHRpUEdkQ0pJU0dIenVFb2FEd2FMUUR3Mi5Ic3A1a1dJdVRSUzJTUTUxOFZsV0hJTFJmYWUu

shadowLastChange: 17019

shadowMin: 0

shadowMax: 99999

shadowWarning: 7

loginShell: /bin/bash

uidNumber: 1007

gidNumber: 1008

homeDirectory: /home/guests/ldapuser01

# ldapuser01, Group, example.com

dn: cn=ldapuser01,ou=Group,dc=example,dc=com

objectClass: posixGroup

objectClass: top

cn: ldapuser01

userPassword:: e2NyeXB0fXg=

gidNumber: 1008

# search result

search: 2

result: 0 Success

# numResponses: 3

# numEntries: 2

Creating a chrooted sftp account

SFTP is a utility similar to scp as it is used to download & upload files but sftp offers a lot of inbuilt commands & additional customization.

In this example, I'm going to create a chrooted sftp account. So the user will have sftp access only i.e no login shell & will be restricted to its home directory.

First we create the user & set it's shell to /sbin/nolign & change ownership of its home directory. Then we create a directory under the user's home directory for uploading/downloading of files.

[root@cfeclient ~]# groupadd sftpusers

[root@cfeclient ~]# useradd -g sftpusers sftptest -s /sbin/nologin

[root@cfeclient ~]# passwd sftptest

Changing password for user sftptest.

New password:

Retype new password:

passwd: all authentication tokens updated successfully.

[root@cfeclient ~]#

[root@cfeclient ~]# ls -ld /home/sftptest/

drwx------. 2 sftptest sftpusers 59 Aug  5 10:04 /home/sftptest/

[root@cfeclient ~]# chown root:root /home/sftptest/

[root@cfeclient ~]# chmod 755 /home/sftptest/

[root@cfeclient ~]# grep sftptest /etc/passwd

sftptest:x:1004:1005::/home/sftptest:/sbin/nologin

[root@cfeclient ~]# mkdir /home/sftptest/folder

[root@cfeclient ~]# chown sftptest /home/sftptest/folder/

[root@cfeclient ~]# ls -ld /home/sftptest/folder/

drwxr-xr-x. 2 sftptest root 6 Aug  5 10:06 /home/sftptest/folder/

[root@cfeclient ~]#

That does it with the user setup. Now we need to edit our sshd_config file:

Change the subsystem directive from 

 

Subsystem      sftp    /usr/libexec/openssh/sftp-server

To 

Subsystem       sftp    internal-sftp

Add the following lines:

Match Group sftpusers

                ChrootDirectory %h

                ForceCommand internal-sftp

The name “internal-sftp” implements an in-process “sftp” server.  This may simplify configurations using ChrootDirectory to force a different filesystem root on clients. I'll demonstrate the practical working in a demo.

In the Match Group stanza we imply that for any member of the group sftpusers the chrootdirectory will be their home directory denoted by %h & will be able to run internal-sftp only.

So, here's the demo:-

Let's login as sftptest user.

[root@cent ~]# sftp sftptest@cfeclient

sftptest@cfeclient's password:

Connected to cfeclient.

sftp> pwd

Remote working directory: /

Notice when we ran pwd command it returned / as the output & not /home/sftptest/

If we run ls to check the contents

sftp> ls

folder

sftp> cd folder/

sftp> put file

Uploading file to /folder/file

file                                                                                                                                                        100%    0     0.0KB/s   00:00

sftp> ls

file

sftp>

sftp> ls -l

-rwxr-xr-x    1 1004     1005            0 Aug  5 14:50 file

So we are indeed in the /home/sftptest/ directory & we were able to upload a file to the folder directory.

If we check on the server as root user we'll find that the file is in /home/sftptest/folder 

[root@cfeclient folder]# pwd

/home/sftptest/folder

[root@cfeclient folder]# ll

total 0

-rwxr-xr-x. 1 sftptest sftpusers 0 Aug  5 10:50 file

If we try to change to any directory & go beyond the realm of our home directory the session won't allow it & we'll get the following error:

sftp> cd /etc

Couldn't canonicalise: No such file or directory

sftp> cd /var

Couldn't canonicalise: No such file or directory

sftp>

Password less login to a remote machine from Windows via putty & putty gen

So to remotely login to a server via putty without being prompted for a password we need a simple tool called putty gen.
It can be downloaded from the same website as putty.
Once you download it just double click the icon & click on run.

A window like the once in the screen shot shown below will open.

Click on generate & click on Save private key once the keys get generated. This will prompt you to save the keys in a file in .ppk format.

Once you save the file open it with notepad & you'll see that the file contains a public key & a private key. Copy & save the public key in the authorized_keys file under .ssh directory of the user you want to login as.

Make sure that there are no spaces or junk characters in the file & add the suffix ssh-rsa in front of the key.

Heres is an example in which I added the key in ~/.ssh/authorized_keys file fo user sa.

'cat -vet' helps verify that there are no junk characters in the file.

Now open putty.

Click on data & enter user name in the 'auto login username' field.

Click on ssh & then click on auth. Under the private key file for authentication browse to the .ppk key file that you had saved earlier.

Now just enter the server name in the session section & you'll be directly logged in:

Passwordless ssh between different users in a single command

There can many be situations wherein we login to a destination machine with a different username than the source machine.
To exchange ssh keys between different users is easy with the ssh-copy-id command.

In this example, I have users user1 & user2 on source & destination servers.
I've generated ssh keys for user1 on the source server. Now, I want to login to destination server as user2 without being prompted for a password.

To do this I need to run the command ssh-copy-id <destination user>@<destination server>

When we do this what happens is that on the destination server, the authorized_keys file for the destination user gets updated with the keys of the source user on source server:

Setting specific client options in ssh

Suppose we want to use a specific set of options while connecting to a particular host without having to enter them on the command line again & again.

We can accomplish this task by setting a client specific directive in the ssh_config file.

The order in which ssh_config file read entries is mentioned in the file but I'll write it here anyway:

# Configuration data is parsed as follows:
1. command line options
2. user-specific file
3. system-wide file
Any configuration value is only changed the first time it is set. Thus, host-specific definitions should be at the beginning of the configuration file, and defaults at the end.

So, if we want to connect to a system cfeclient using port 876 & ssh protocol 2 version 2, we'll enter the following lines in the ssh_config file:

To verify that the options have been set do a ssh login in verbose mode to cfeclient:

From the output you can gather that the connection used protocol version 2 & port number 876.

Hardening ssh

SSH is designed to be a secure encrypted communication method by default.
But here are a few ways to increase the security provided by ssh.

Given below are a few directives present in the /etc/ssh/sshd_config file which can be tweaked to enhance security:

• Port (the default port is 22. We can modify it so that ssh listens for incoming connections on a different port)
• PermitRootLogin (disallow direct root login if set to no)
• ListenAddress (For servers with multiple IP addresses, this limits the ssh connectivity to one address only)
• Protocol (value can be 1 or 2. Its 1 by default but 2 is more secure)
• AllowUsers (With this extreme directive set only the users part of this directive will be allowed to login to the server via ssh denying access to any other users)
• DenyUsers (Users added to this directive will be denied ssh login access even if they have a fully functional account existing on the server)
• PermitTTY (If this is set to no then a user will not get a shell on login but will be able to run command remotely via ssh. I'll be demonstrating its usage later in this tutorial)
• ForceCommand ( allow a user to only run a specified command remotely)

Here is a sample of the tweaks that I did in the sshd_config file:

Once the directives are set you need to restart the ssh service so that the config file is read.

With these directives in place only the sa user can login to the server via port 876.

To take things further if we want to disable the use of password all together & rely on ssh keys only then we can set the PasswordAuthentication directive to no followed by a restart of the ssh service.

With this directive set to no, only those users will be allowed access whose public keys are available on the destination server:

In the above example when I try to login as test user I'm denied access because there are not keys on the destination server. On the other hand I had exchanged the keys for sa user earlier so its allowed access.

To add another layer of security we can also modify the authorized keys file on the destination server to specify that a key pair will be accepted & authenticated only if the connection request comes from a specific host. To do this add from "hostname", before the public key in the authorized_keys file.

Given below is an example:

Now, to go a step even further we can actually change the default file that ssh uses to run.

This is slightly tricky & should be preferably done from the server's console.

Match Address & Match User stanzas:

To allow custom directives to be implemented for connections originating from a specific network or user we apply the Match Address 7 Match User stanzas.

Here is an example of a Match Address stanza:

The result of this stanza is the following for any ssh connections originating from IP address 192.168.44.134:

Only sa user can access the server via ssh

sa user cannot login directly but can only run commands remotely.

Here what happened when I tested the settings:

So from the above screen shot we can understand that the user sa was denied direct login but could run commands remotely.

Now for a quick demo of the Match User stanza.

I created the following Match User stanza in my sshd_config file:

This will allow user sa to only run the script /tmp/test.sh remotely if the user tries to gain access to the server via ssh. I tried it & here's the result:

Please note that adding or updating the stanzas requires a restart of the ssh service.

So, stop the ssh service & run the command /usr/sbin/sshd -f <config file name> to start the service & direct it to use the config file that you've mentioned. For example, I used the file my_sshd_config to configure ssh & the same can be verified by looking for the sshd process in ps -ef output:

For the purposes of this tutorial I had disabled the firewall & selinux. I know it's sounds weird disabling these security measures in a tutorial aimed at hardening. But I wanted the focus here to be solely on hardening of the ssh service. I hope this has been an informative read for you.

You can also allow or deny ssh access to a server for a single node or an entire network as a whole using TCP wrappers i.e, editing the /etc/hosts.allow or /etc/hosts.deny file.