Monday 12 July 2021

Using capture groups in grep in Linux

Introduction

Let me start by saying that this article isn't about capture groups in grep per se. What we are going to do here with grep is something similar to capture groups in other programming languages. We'll be printing only the text that is matched with grep and not the entire line containing the matched text.
I go into this because I was trying to accomplish a task using a Perl one-liner but when I couldn't get that to work I thought of turning back to grep and I was pleasantly surprised to achieve that desired result with relative ease.

The scenario

I needed to filter the output of the df command and then print just the matched text. Here's a quick grep of the matched text.

df -hTP | grep 'db'

/dev/mapper/ethoss0db_vg-lvol1                   ext4   126G   85G   35G  71% /ethoss0db/db
/dev/mapper/yuris0db_vg-lvol1                   ext4    32G   22G  8.3G  73% /yuris0db/db

The piece of text I need is ethoss0 and yuris0 which I intended to use further down the line in another script.

GREP to the rescue

To use some enhanced regular expression magic with grep we had to awaken the Perl within it i.e. use the -oP flags with the grep command.

-o, --only-matching
Print only the matched (non-empty) parts of a matching line, with each such part on a separate output line.

 -P, --perl-regexp
Interpret PATTERN as a Perl regular expression. 

Here is the final grep command I used to get the desired output.

df -hTP | grep -oP '% /\K(.*)(?=db/db$)'

ethoss0
yuris0

So little typing! Elegant isn' it?

Now let's break it down

  • That’s the magic of \K which excludes the token immediately prior to it from the matched text (without rewinding the stream and attempting a different match for that text). \K tells the engine to pretend that the match attempt started at this position.
  • In the expression (?=db/db$), the (?=<expression>), matches the characters immediately before the expression and does not include the expression itself.
  • Using the combination of \K also known as zero-width lookbehind and the  zero-width lookahead  i.e. (?=<expr>), we are able to print only the matched text from a line.


Conclusion

I hope you found this article to be useful and I hope you would consider using all your grep options before resorting to something else while working with regular expressions for your future scripts.

Introducing Puppet Bolt Part 2: Understanding tasks

Introduction

In the last post we discussed the motivation behind using Puppet Bolt in your environment and demonstrated how to install Puppet Bolt and run one off ad hoc commands using some of the options provided by the tool. In this post we'll take things a little further from running commands and start creating and running tasks.

What is a task?
Tasks are basically scripts that perform single ad-hoc functions. Tasks can however be orchestrated using plans. All tasks are stored in Puppet modules. In case you do not wish to create your own tasks you could also download them from Puppet forge or GitHub and use them directly. You can also pass arguments to Bolt tasks.

Display available tasks:
Bolt comes with a couple of pre-installed tasks that you could use out of the box. To view a list of available tasks type:

[sahil@bolt-lab ~]$ bolt tasks show
Expected subcommand 'tasks' to be one of command, script, task, plan, file, puppetfile, secret, apply
[sahil@bolt-lab ~]$ bolt task show
facts                    Gather system facts
package                  Manage and inspect the state of packages
puppet_agent::install    Install the Puppet agent package
puppet_agent::version    Get the version of the Puppet agent package installed. Returns nothing if none present.
puppet_conf              Inspect puppet agent configuration settings
reboot                   Reboots a machine
reboot::last_boot_time   Gets the last boot time of a Linux or Windows system
service                  Manage and inspect the state of services

MODULEPATH:
/home/sahil/.puppetlabs/bolt/modules:/home/sahil/.puppetlabs/bolt/site-modules:/home/sahil/.puppetlabs/bolt/site

Use `bolt task show <task-name>` to view details and parameters for a specific task.
[sahil@bolt-lab ~]$

To view information about a particular task just append the task name at the end of the above executed command. Here's an example.

[sahil@bolt-lab ~]$ bolt task show facts

facts - Gather system facts

USAGE:
bolt task run --nodes <node-name> facts

MODULE:
built-in module
[sahil@bolt-lab ~]$


Executing a task:
To demonstrate how to execute tasks, lets take use the service task.

[sahil@bolt-lab ~]$ bolt task show service

service - Manage and inspect the state of services

USAGE:
bolt task run --nodes <node-name> service action=<value> name=<value> provider=<value>

PARAMETERS:
- action: Enum[start, stop, restart, enable, disable, status]
    The operation (start, stop, restart, enable, disable, status) to perform on the service.
- name: String[1]
    The name of the service to operate on.
- provider: Optional[String[1]]
    The provider to use to manage or inspect the service, defaults to the system service manager. Only used when the 'puppet-agent' feature is available on the target so we can leverage Puppet.

MODULE:
built-in module

The above description states that we need to pass the action and name parameters to tell bolt the name of the service we want to work with and the action we want to take on the service. Let's run use the service task to check the status of the nfs service.

[sahil@bolt-lab ~]$ bolt task run --nodes bolt-lab service name=nfs action=status --user sahil
Started on bolt-lab...
Finished on bolt-lab:
  {
    "status": "MainPID=0,LoadState=loaded,ActiveState=active",
    "enabled": "enabled"
  }
Successful on 1 node: bolt-lab
Ran on 1 node in 1.45 seconds
[sahil@bolt-lab ~]$


Before we can start writing tasks we need to understand what is a module?
A puppet module consists of self contained code and data residing in a simple file structure. Modules can easily be shared and reused. You could even distribute your own modules as tar.gz files.

Given below are some rules we need to adhere to while creating puppet modules:

  • The directory name of the module is the name of the module itself.
  • It should contain a manifests directory with an init.pp file.
  • The init.pp file must contain a single class definition the same as the module's name.
  • A metadata.json file should be present in the parent directory.

If you do not intend to distribute your modules and use them primarily for running tasks then you could afford to ignore most of the above mentioned rules.


Creating our first task:
Since a module is basically some files arranged in a directory hierarchy, to create a module we just need to create that directory structure.

For our first module and task we'll be keeping things simple and create a very basic script to work with packages using yum.

We'll be naming our task my_pkg. When you run the 'bolt task show' command it'll show you the default directories where Bolt looks for modules. We could use a different directory and then specify that with the --modulepath parameter while using Bolt. but for this demonstration we'll use one of the default ones.

Let's create a directory named my_pkg under one of the default module paths.

[sahil@bolt-lab ~]$ mkdir -p /home/sahil/.puppetlabs/bolt/modules/my_pkg

Now we'll create a tasks sub-directory under this directory to store the tasks that we'll be creating under this module.

[sahil@bolt-lab ~]$ mkdir -p /home/sahil/.puppetlabs/bolt/modules/my_pkg/tasks

I would like to pass some parameters to the task that I'll be creating. To add parameters to your task as environment variables, pass the argument prefixed with the Puppet task prefix PT_ . There are more ways to pass parameters to Puppet tasks but for now we'll stick to this approach using environment variables.

I've created a small bash script named yum in the tasks directory and populated it with the following content:

[sahil@bolt-lab ~]$ cat /home/sahil/.puppetlabs/bolt/modules/my_pkg/tasks/yum.bash
#!/bin/bash

action="$PT_action"
name="$PT_name"

yum $action $name

[sahil@bolt-lab tasks]$
$

To make Bolt aware that we've added a new module and to able to use it, we'll need to install the module. To do that we first need to create a file named Puppetfile that has the following content.

[sahil@bolt-lab ~]$ cat Puppetfile
#Local modules
mod 'my_pkg', local:true
[sahil@bolt-lab ~]$

Before we can use our Puppet file we need to move it to the default Bolt directory.

[sahil@bolt-lab ~]$ mv Puppetfile /home/sahil/.puppetlabs/bolt/
[sahil@bolt-lab ~]$ ls /home/sahil/.puppetlabs/bolt/Puppetfile
/home/sahil/.puppetlabs/bolt/Puppetfile
[sahil@bolt-lab ~]$

To install the module we need to run the below command:

[sahil@bolt-lab ~]$ bolt puppetfile install
Successfully synced modules from /home/sahil/.puppetlabs/bolt/Puppetfile to /home/sahil/.puppetlabs/bolt/modules
[sahil@bolt-lab ~]$

If we run Bolt task show again we will be able to see the my_pkg module defined with the yum task:

[sahil@bolt-lab ~]$ bolt task show
facts                    Gather system facts
my_pkg::yum
package                  Manage and inspect the state of packages
puppet_agent::install    Install the Puppet agent package
puppet_agent::version    Get the version of the Puppet agent package installed. Returns nothing if none present.
puppet_conf              Inspect puppet agent configuration settings
reboot                   Reboots a machine
reboot::last_boot_time   Gets the last boot time of a Linux or Windows system
service                  Manage and inspect the state of services

MODULEPATH:
/home/sahil/.puppetlabs/bolt/modules:/home/sahil/.puppetlabs/bolt/site-modules:/home/sahil/.puppetlabs/bolt/site

Use `bolt task show <task-name>` to view details and parameters for a specific task.
[sahil@bolt-lab ~]$

If we weren't going to pass any parameters to the task then this would be enough. But since we do intend to pass parameters and would like to inform the users about the functionality of the task, we'll add a JSON metadata file in our tasks directory and populate it with the following content:

[sahil@bolt-lab ~]$ cat /home/sahil/.puppetlabs/bolt/modules/my_pkg/tasks/yum.json
{
  "description": "Allow users to manipulate software packages using yum",
  "parameters": {
    "action": {
      "description": "Task to perofmr on the package",
      "type": "String"
    },
    "name": {
      "description": "The package name",
      "type": "String"
    }
  }
}


Now let's execute the bolt task show command for our my_pkg::yum task.

[sahil@bolt-lab tasks]$ bolt task show my_pkg::yum

my_pkg::yum - Allow users to manipulate software packages using yum

USAGE:
bolt task run --nodes <node-name> my_pkg::yum action=<value> name=<value>

PARAMETERS:
- action: String
    Task to perform on the package
- name: String
    The package name

MODULE:
/home/sahil/.puppetlabs/bolt/modules/my_pkg
[sahil@bolt-lab tasks]$


Now we are all done with the prep work. Let's run our task.

[sahil@bolt-lab tasks]$ bolt task run --nodes 10.31.19.151 my_pkg::yum action=info name=httpd --user sahil --run-as root
Started on 10.31.19.151...
Finished on 10.31.19.151:
  Loaded plugins: fastestmirror
  Loading mirror speeds from cached hostfile
   * base: mirror.aktkn.sg
   * epel: d2lzkl7pfhq30w.cloudfront.net
   * extras: mirror.aktkn.sg
   * nux-dextop: mirror.li.nux.ro
   * updates: mirror.aktkn.sg
  Available Packages
  Name        : httpd
  Arch        : x86_64
  Version     : 2.4.6
  Release     : 89.el7.centos
  Size        : 2.7 M
  Repo        : updates/7/x86_64
  Summary     : Apache HTTP Server
  URL         : http://httpd.apache.org/
  License     : ASL 2.0
  Description : The Apache HTTP Server is a powerful, efficient, and extensible
              : web server.

  {
  }
Successful on 1 node: 10.31.19.151
Ran on 1 node in 3.03 seconds
[sahil@bolt-lab tasks]$ bolt task run --nodes 10.31.19.151 my_pkg::yum action=list name=httpd --user sahil --run-as root
Started on 10.31.19.151...
Finished on 10.31.19.151:
  Loaded plugins: fastestmirror
  Loading mirror speeds from cached hostfile
   * base: mirror.aktkn.sg
   * epel: d2lzkl7pfhq30w.cloudfront.net
   * extras: mirror.aktkn.sg
   * nux-dextop: mirror.li.nux.ro
   * updates: mirror.aktkn.sg
  Available Packages
  httpd.x86_64                     2.4.6-89.el7.centos                     updates
  {
  }
Successful on 1 node: 10.31.19.151
Ran on 1 node in 3.60 seconds


As this point the action can only be info or list because I haven't added the -y option with yum to allow for install or erase operations. But we can tweak the script to suit our requirements.


Conclusion

This concludes our article on how to start writing tasks to run using Puppet Bolt. We hope you found it to be useful and look forward towards your feedback and suggestions.




Docker: Getting started

Introduction
In this article, we be covering a basic introduction to the Docker Tool along with its installation on a Centos 7 system. Docker has become synonymous with containerization of applications in Linux and thousands of applications have successfully been dockerized and deployed in Production since Docker's inception.

History of Docker:
Docker, Inc. is a technology company based in San Francisco. It was founded by French born american Solomon Hykes and start as a PaaS provider called dotCloud. dotCloud leveraged Linux containers and their internal tool used to manage containers was nick-named Docker. In 2013 dotCloud was re branded as Docker.

What is Docker?
To understand what is Docker we must first know what are containers. A container is a special type of process that is isolated from other processes. Containers are assigned resources that no other process can access, and they cannot access any resources not explicitly assigned to them. Docker provides a Runtime and Orchestration Engine to allow users to deploy and manage containers. There are tools available that also allow for the orchestration of containers.

Docker is available in the following main editions:

Enterprise Edition (EE): Paid and supported for 12 months. Provides additional features like certified images, same day support and vulnerability scanning of images.
Community Edition (CE) : Free and supported for 4 months.
Moby: Open-Source upstream project of Docker and breaks Docker down into more modular components

Why should you use Docker?

Dev and Production environment are the same
Bugs in Production can be replicated in Development
Lets you put your environment and configuration into code and deploy it
Allows the same Docker configuration to be used in a variety of environments
Build standards and repeatable processes
Developer Productivity
App Isolation
Server Consolidation

Installing Docker:

Before installing Docker we must install some of the pre-requisite packages as shown below:

yum install -y yum-utils device-mapper-persistent-data lvm2

Now we will add the stable repository for the Docker community edition of the system:

[root@dokcer-test ~]# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
Loaded plugins: fastestmirror
adding repo from: https://download.docker.com/linux/centos/docker-ce.repo
grabbing file https://download.docker.com/linux/centos/docker-ce.repo to /etc/yum.repos.d/docker-ce.repo
repo saved to /etc/yum.repos.d/docker-ce.repo
[root@dokcer-test ~]#

Now that the Docker repository has been configured, we can install Docker CE using yum.

 yum -y install docker-ce

The above command installs the following packages:

===========================================================================================================================================================
 Package                                 Arch                         Version                                 Repository                              Size
===========================================================================================================================================================
Installing:
 docker-ce                               x86_64                       3:19.03.1-3.el7                         docker-ce-stable                        24 M
Installing for dependencies:
 container-selinux                       noarch                       2:2.99-1.el7_6                          extras                                  39 k
 containerd.io                           x86_64                       1.2.6-3.3.el7                           docker-ce-stable                        26 M
 docker-ce-cli                           x86_64                       1:19.03.1-3.el7                         docker-ce-stable                        39 M


After the packages are successfully installed, we'll start the docker service and also enable it so that it starts automatically on boot.

[root@dokcer-test ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-07-30 04:38:00 UTC; 6s ago
     Docs: https://docs.docker.com
 Main PID: 9008 (dockerd)
   CGroup: /system.slice/docker.service
           └─9008 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Jul 30 04:37:58 dokcer-test.example.com dockerd[9008]: time="2019-07-30T04:37:58.649760132Z" level=info msg="Creating filesystem xfs on de...emapper
Jul 30 04:37:59 dokcer-test.example.com dockerd[9008]: time="2019-07-30T04:37:59.064964291Z" level=info msg="Successfully created filesyst...emapper
Jul 30 04:37:59 dokcer-test.example.com dockerd[9008]: time="2019-07-30T04:37:59.211113995Z" level=warning msg="[graphdriver] WARNING: the...elease"
Jul 30 04:37:59 dokcer-test.example.com dockerd[9008]: time="2019-07-30T04:37:59.269462315Z" level=info msg="Loading containers: start."
Jul 30 04:37:59 dokcer-test.example.com dockerd[9008]: time="2019-07-30T04:37:59.567854253Z" level=info msg="Default bridge (docker0) is a...ddress"
Jul 30 04:37:59 dokcer-test.example.com dockerd[9008]: time="2019-07-30T04:37:59.654696034Z" level=info msg="Loading containers: done."
Jul 30 04:38:00 dokcer-test.example.com dockerd[9008]: time="2019-07-30T04:38:00.354472432Z" level=info msg="Docker daemon" commit=74b1e89...19.03.1
Jul 30 04:38:00 dokcer-test.example.com dockerd[9008]: time="2019-07-30T04:38:00.355436639Z" level=info msg="Daemon has completed initialization"
Jul 30 04:38:00 dokcer-test.example.com systemd[1]: Started Docker Application Container Engine.
Jul 30 04:38:00 dokcer-test.example.com dockerd[9008]: time="2019-07-30T04:38:00.527177004Z" level=info msg="API listen on /var/run/docker.sock"
Hint: Some lines were ellipsized, use -l to show in full.
[root@dokcer-test ~]#


It is considered best practice to not manage containers directly as the root user. Therefore, I'll add a user named sahil to the docker group and use this user account for running and managing containers on this server.

usermod -aG docker sahil

Let's verify that we can run Docker related commands as the user sahil:

[sahil@docker-test ~]$ docker version
Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        74b1e89
 Built:             Thu Jul 25 21:21:07 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89
  Built:            Thu Jul 25 21:19:36 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
[sahil@docker-test ~]$


Conclusion
This concludes our introduction to Docker where we talked about the history of Docker along with it's most popular use cases and also installed Docker on a Centos 7 system.

Friday 14 May 2021

Create VM in Google Cloud using cloud shell

 Introduction

In this article, 'll explain how we can create a virtual machine in Google Cloud using the command line tool gcloud which is pre-installed in google cloud shell. Cloud Shell is a Debian-based virtual machine loaded with all the development tools you'll need (gcloud, git, and others) and offers a persistent 5-GB home directory. Let's get started.


Step 1: Open cloud Shell

To open cloud shell, click on the shell icon on the top let corner of the google cloud home page.


Step 2: Authenticate 

After clicking on the shell icon, google cloud shell is launched in a terminal window towards the bottom of the home page. Here type 'gcloud auth list' to authenticate your user account to use cloud shell to access and manipulate google cloud resources. You may also type 'gcloud config list project' to check the project you are working under. The term project is analogous to the term organization in AWS.




Step 3: Create virtual machine

Now, we will create our virtual machine using the following command:

gcloud compute instances create gcelab2 --machine-type n1-standard-2 --zone us-central1-f

  • "compute instances" is the google cloud component we'd like to work with.
  • "create" is the action that we'd like to perform.
  • gcelab2 is the name of the VM
  • "machine-type" defines the CPU/Memory configuration that we would like to use and this is divided into entities called series wherein each series comprises CPU/Memory settings for VMs. In this example we've selected n1-standard-2.
  • "zone" attribute specifies the location where the VM will reside.



Step 4: Verify login to the VM

We should now be able see our VM in the VM instances section under the Compute Engine.


You can either click on ssh under the connect section to login to the VM or use the following command in the cloud shell:

gcloud compute ssh gcelab2 --zone us-central1-f


Conclusion

We hope that you found this post to be useful and we look forward towards your feedback and suggestions

Using capture groups in grep in Linux

Introduction Let me start by saying that this article isn't about capture groups in grep per se. What we are going to do here with gr...