Thursday, April 18, 2019

How to monitor disk performance on RHEL or CentOS 7 ?

How to monitor disk performance on RHEL or CentOS 7 ?


iotop : Watches I/O usage information output by the Linux kernel (requires 2.6.20 or later) and displays a table of current I/O usage by processes or threads on the system. At least the CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING, CONFIG_TASKSTATS and CONFIG_VM_EVENT_COUNTERS options need to be enabled in your Linux kernel build configuration. iotop displays columns for the I/O bandwidth read and written by each process/thread during the sampling period. It also displays the percentage of time the thread/process spent while swapping in and while waiting on I/O. For each process, its I/O priority (class/level) is shown.

[root@localhost ~]# iotop

Total DISK READ :       0.00 B/s | Total DISK WRITE :       0.00 B/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:       0.00 B/s
   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 14856 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.08 % [kworker/3:0]
     1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % systemd --switched-root --system --deserialize 22
     2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
     3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
     5 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H]
     6 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/u256:0]
     7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
     8 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_bh]
     9 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_sched]
    10 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [lru-add-drain]
    11 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
    12 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/1]
    13 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
    14 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]
 10925 be/4 hpe         0.00 B/s    0.00 B/s  0.00 %  0.00 % gsd-xsettings [dconf worker]
    16 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/1:0H]
    17 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/2]
    18 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/2]
    19 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/2]


iostat : It report CPU statistics and IO statistics for devices and partitions. This command is used for monitoring system input/output device loading by observing the time devices are active in relation to their average transfer rates. The iostat command generates reports that can be used to change system configuration to better balance the input/output load between physical disks.

[root@localhost ~]# iostat
Linux 3.10.0-957.el7.x86_64 (localhost.localdomain)     Friday 29 March 2019    _x86_64_        (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.84    0.91    1.70    0.94    0.00   95.61

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              16.88       426.35       198.58    1477954     688381
scd0              0.01         0.30         0.00       1050          0
dm-0             16.66       415.36       197.98    1439850     686291
dm-1              0.03         0.71         0.00       2460          0


To check iostats on /dev/sda:

[root@localhost ~]# iostat -p sda
Linux 3.10.0-957.el7.x86_64 (localhost.localdomain)     Friday 29 March 2019    _x86_64_        (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.79    0.85    2.01    1.01    0.00   95.34

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              19.93       404.65       185.43    1584346     726011
sda1              0.52         8.15         0.53      31892       2090
sda2             19.41       396.08       184.89    1550782     723921


To check detailed stats:

[root@localhost ~]# iostat -x
Linux 3.10.0-957.el7.x86_64 (localhost.localdomain)     Friday 29 March 2019    _x86_64_        (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.72    0.57    1.58    0.74    0.00   96.39

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.03     0.37   12.37    1.45   274.77   137.89    59.72     0.18   13.40    6.47   72.30   3.92   5.41
scd0              0.00     0.00    0.00    0.00     0.18     0.00    72.41     0.00    1.72    1.72    0.00   1.31   0.00
dm-0              0.00     0.00   12.02    1.82   268.25   137.54    58.63     0.24   17.31    6.65   87.75   3.90   5.39
dm-1              0.00     0.00    0.02    0.00     0.42     0.00    54.67     0.00    0.20    0.20    0.00   0.11   0.00


atop : atop is a Advanced System & Process Monitoring tool, which displays live status of the critical hardware resources (from a performance point of view) on system level, i.e. cpu, memory, disk and network in a arranged manner.

NOTE: To install this tool you will have to install EPEL repository and download it separately as it doesn't comes with RHEL or CentOS OS ISO's.

To install EPEL on CentOS and atop:

[root@localhost ~]# yum --enablerepo=extras install epel-release

[root@localhost ~]# yum install atop

To run atop :

[root@localhost ~]# atop




ioping : This tool generates various I/O patterns and lets you monitor I/O speed and latency in real time.

NOTE: To install this tool you will have to install EPEL repository and download it separately as it doesn't comes with RHEL or CentOS OS ISO's.

To install EPEL on CentOS and ioping:

[root@localhost ~]# yum --enablerepo=extras install epel-release

[root@localhost ~]# yum install ioping

To run ioping :

[root@localhost ~]# ioping -c 10 .
4 KiB <<< . (xfs /dev/dm-0): request=1 time=236.0 us (warmup)
4 KiB <<< . (xfs /dev/dm-0): request=2 time=496.4 us
4 KiB <<< . (xfs /dev/dm-0): request=3 time=592.1 us
4 KiB <<< . (xfs /dev/dm-0): request=4 time=630.0 us
4 KiB <<< . (xfs /dev/dm-0): request=5 time=384.8 us
4 KiB <<< . (xfs /dev/dm-0): request=6 time=696.7 us
4 KiB <<< . (xfs /dev/dm-0): request=7 time=1.22 ms (slow)
4 KiB <<< . (xfs /dev/dm-0): request=8 time=580.1 us
4 KiB <<< . (xfs /dev/dm-0): request=9 time=778.8 us
4 KiB <<< . (xfs /dev/dm-0): request=10 time=726.6 us

--- . (xfs /dev/dm-0) ioping statistics ---
9 requests completed in 6.11 ms, 36 KiB read, 1.47 k iops, 5.76 MiB/s
generated 10 requests in 9.01 s, 40 KiB, 1 iops, 4.44 KiB/s
min/avg/max/mdev = 384.8 us / 678.6 us / 1.22 ms / 222.8 us


[root@localhost ~]# ioping -c 10 -i 5 -s 16k .
16 KiB <<< . (xfs /dev/dm-0): request=1 time=374.2 us (warmup)
16 KiB <<< . (xfs /dev/dm-0): request=2 time=732.5 us
16 KiB <<< . (xfs /dev/dm-0): request=3 time=932.0 us
16 KiB <<< . (xfs /dev/dm-0): request=4 time=574.3 us
16 KiB <<< . (xfs /dev/dm-0): request=5 time=667.4 us
16 KiB <<< . (xfs /dev/dm-0): request=6 time=588.0 us
16 KiB <<< . (xfs /dev/dm-0): request=7 time=903.8 us
16 KiB <<< . (xfs /dev/dm-0): request=8 time=666.1 us
16 KiB <<< . (xfs /dev/dm-0): request=9 time=886.9 us
16 KiB <<< . (xfs /dev/dm-0): request=10 time=594.5 us

--- . (xfs /dev/dm-0) ioping statistics ---
9 requests completed in 6.55 ms, 144 KiB read, 1.37 k iops, 21.5 MiB/s
generated 10 requests in 45.0 s, 160 KiB, 0 iops, 3.55 KiB/s
min/avg/max/mdev = 574.3 us / 727.3 us / 932.0 us / 135.9 us

NOTE: 

-c Stop after count requests
-i Set time between requests to interval (Default 1s)
-s Request size (Default 4k)

How to add more swap space by adding new swap partition in RHEL or CentOS 7 ?

How to add more swap space by adding new swap partition in RHEL or CentOS 7


What is swap?


Swap space on a disk is used when the amount of physical RAM is full in the Linux system. When the system runs out of RAM, inactive pages are moved from the RAM to the swap space. Swap space can be either a dedicated swap partition or a swap file. So here we will see how you can add more swap space as a swap file.


In this article we will see how we can add more swap space using a new swap partition.


Steps :

Let's check how much swap is available on the system currently:

[root@localhost ~]# swapon --show

[root@localhost ~]# free -h

I have a new 10GB disk named "/dev/sdb" and i want to use it as a swap space.


[root@localhost ~]# fdisk -l
Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Step 1 : Create a disk partition on /dev/sdb and set partition type as swap:


[root@localhost ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xceaf232e.

Command (m for help): n  ==> CREATE NEW PARTITION
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p  ==> PRIMARY PARTITION
Partition number (1-4, default 1): ==> PRESS ENTER
First sector (2048-20971519, default 2048): ==> PRESS ENTER
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519): ==> PRESS ENTER
Using default value 20971519
Partition 1 of type Linux and of size 10 GiB is set

Command (m for help): p  ==> PRINT PARTITION TABLE

Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xceaf232e

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048    20971519    10484736   83  Linux

Command (m for help): l  ==> LIST TYPE OF SUPPORTED FILESYSTEM

 0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris
 1  FAT12           27  Hidden NTFS Win 82  Linux swap / So c1  DRDOS/sec (FAT-
 2  XENIX root      39  Plan 9          83  Linux           c4  DRDOS/sec (FAT-
 3  XENIX usr       3c  PartitionMagic  84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
 4  FAT16 <32M      40  Venix 80286     85  Linux extended  c7  Syrinx
 5  Extended        41  PPC PReP Boot   86  NTFS volume set da  Non-FS data
 6  FAT16           42  SFS             87  NTFS volume set db  CP/M / CTOS / .
 7  HPFS/NTFS/exFAT 4d  QNX4.x          88  Linux plaintext de  Dell Utility
 8  AIX             4e  QNX4.x 2nd part 8e  Linux LVM       df  BootIt
 9  AIX bootable    4f  QNX4.x 3rd part 93  Amoeba          e1  DOS access
 a  OS/2 Boot Manag 50  OnTrack DM      94  Amoeba BBT      e3  DOS R/O
 b  W95 FAT32       51  OnTrack DM6 Aux 9f  BSD/OS          e4  SpeedStor
 c  W95 FAT32 (LBA) 52  CP/M            a0  IBM Thinkpad hi eb  BeOS fs
 e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a5  FreeBSD         ee  GPT
 f  W95 Ext'd (LBA) 54  OnTrackDM6      a6  OpenBSD         ef  EFI (FAT-12/16/
10  OPUS            55  EZ-Drive        a7  NeXTSTEP        f0  Linux/PA-RISC b
11  Hidden FAT12    56  Golden Bow      a8  Darwin UFS      f1  SpeedStor
12  Compaq diagnost 5c  Priam Edisk     a9  NetBSD          f4  SpeedStor
14  Hidden FAT16 <3 61  SpeedStor       ab  Darwin boot     f2  DOS secondary
16  Hidden FAT16    63  GNU HURD or Sys af  HFS / HFS+      fb  VMware VMFS
17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE
18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fd  Linux raid auto
1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid fe  LANstep
1c  Hidden W95 FAT3 75  PC/IX           be  Solaris boot    ff  BBT
1e  Hidden W95 FAT1 80  Old Minix

Command (m for help): t  ==> CHANGE PARTITION TYPE
Selected partition 1
Hex code (type L to list all codes): 82 ==> ENTER "82" FOR SWAP
Changed type of partition 'Linux' to 'Linux swap / Solaris'

Command (m for help): p  ==> PRINT PARTITION TABLE

Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xceaf232e

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048    20971519    10484736   82  Linux swap / Solaris

Command (m for help): w  ==> WRITE PARTITION TABLE CHANGES
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.


Step 2 : Run "partprobe" command to re-read the partition table after changes:

[root@localhost ~]# partprobe /dev/sdb


Step 3 : Format the new /dev/sdb1 swap disk:

[root@localhost ~]# mkswap /dev/sdb1
Setting up swapspace version 1, size = 10484732 KiB
no label, UUID=8b375896-99fe-4b54-8c78-9ef41accef9b


Step 4 : Activate swap space on "/dev/sdb1" partition:

[root@localhost ~]# swapon /dev/sdb1


Step 5 : To make these new changes persistent across reboot add an fstab entry:

[root@localhost ~]# vi /etc/fstab

/dev/sdb1 swap swap defaults 0 0

save and exit.


Step 6 : Verify if new disk is added as a swap space:

[root@localhost ~]# swapon --show
NAME      TYPE      SIZE USED PRIO
/dev/dm-1 partition 3.9G   0B   -2
/dev/sdb1 partition  10G   0B   -3


[root@localhost ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.7G        421M        3.0G         14M        281M        3.0G
Swap:           13G          0B         13G


Removing swap partition gracefully:


Step 1 : Deactivate swap on /swapfile:

[root@localhost ~]# swapoff -v /dev/sdb1
swapoff /dev/sdb1


Step 2 : Remove swap entry from /etc/fstab file:


Step 3 : Delete "/dev/sdb1" partition or remove disk [OPTIONAL] :


Verify if swap space from "/dev/sdb1" partition is deactivated and removed:


[root@localhost ~]# swapon --show
NAME      TYPE      SIZE USED PRIO
/dev/dm-1 partition 3.9G   0B   -2


[root@localhost ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.7G        413M        3.0G         14M        281M        3.0G
Swap:          3.9G          0B        3.9G

How to add more swap space as a swapfile on RHEL or CentOS 7 ?

How to add more swap space as a swapfile on RHEL or CentOS 7

What is swap?

Swap space on a disk is used when the amount of physical RAM is full in the Linux system. When the system runs out of RAM, inactive pages are moved from the RAM to the swap space. Swap space can be either a dedicated swap partition or a swap file. So here we will see how you can add more swap space as a swap file.

In this article we will see how we can add more swap space using a swapfile.

Steps :

Let's check how much swap is available on the system currently:

[root@localhost ~]# swapon --show
NAME      TYPE      SIZE USED PRIO
/dev/dm-1 partition 3.9G   0B   -2


[root@localhost ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.7G        414M        1.1G         14M        2.2G        3.0G
Swap:          3.9G          0B        3.9G


Step 1 : Create a swapfile, which can be used as swap memory:

[root@localhost ~]# dd if=/dev/zero of=/swapfile bs=1G count=2
2+0 records in
2+0 records out
2147483648 bytes (2.1 GB) copied, 56.732 s, 37.9 MB/s


OR you can use "fallocate" command to create a new swapfile:


[root@localhost ~]# fallocate -l 2G /swapfile


[root@localhost ~]# ls -lkh /swapfile
-rw-r--r--. 1 root root 2.0G Apr 18 17:22 /swapfile



Step 2 : Ensure that only the root user have read and write permissions on the /swapfile:

[root@localhost ~]# chmod 600 /swapfile


Step 3 : Create swap on /swapfile :

[root@localhost ~]# mkswap /swapfile
Setting up swapspace version 1, size = 2097148 KiB
no label, UUID=97900239-4711-4cfd-90a2-e46be60a34e4


Step 4 : Activate swap on the file :

[root@localhost ~]# swapon /swapfile


Step 5 : To make this swap file persistent across reboot, add an fstab entry:

# vi /etc/fstab

/swapfile swap swap defaults 0 0

save and exit.


Verify if this new swap file is active:

[root@localhost ~]# swapon --show
NAME      TYPE      SIZE USED PRIO
/dev/dm-1 partition 3.9G   0B   -2
/swapfile file        2G   0B   -3


[root@localhost ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.7G        418M        1.1G         14M        2.2G        3.0G
Swap:          5.9G          0B        5.9G

The new swapfile is now added as a swap memory.

Removing swapfile gracefully:


Step 1 : Deactivate swap on /swapfile:

[root@localhost ~]# swapoff -v /swapfile
swapoff /swapfile

Step 2 : Remove swap entry from /etc/fstab file:


Step 3 : Delete /swapfile :

[root@localhost ~]# rm -rf /swapfile


Verify if the /swapfile is deactivated and removed:

[root@localhost ~]# swapon --show
NAME      TYPE      SIZE USED PRIO

/dev/dm-1 partition 3.9G   0B   -2


[root@localhost ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.7G        415M        1.1G         14M        2.2G        3.0G
Swap:          3.9G          0B        3.9G


Monitoring Memory, CPU and swap on RHEL or CentOS 7

top : The top provides a dynamic real-time view of running processes, CPU, Memory and SWAP utilization on a system. It can display system summary information as well as a list of processes or threads currently being managed by the Linux kernel.

[root@localhost ~]# top


In case as a root you want to check it for some particular user then :

[root@localhost ~]# top -u hpe



pmap : This command report memory map of a process or processes.

[root@localhost ~]# pmap -d 22745



vmstat : This command reports information about processes, memory, paging, block IO, traps, disks and cpu activity.

[root@localhost ~]# vmstat -w

[root@localhost ~]# vmstat -as

[root@localhost ~]# vmstat -dw


[root@localhost ~]# vmstat -D

            5 disks
            2 partitions
        48470 total reads
          104 merged reads
      9879363 read sectors
      6320494 milli reading
        57453 writes
        39762 merged writes
     46514707 written sectors
     61995147 milli writing
            0 inprogress IO
         1162 milli spent IO


mpstat : The  mpstat  command  writes  to  standard output activities for each available processor, processor 0 being the first one.  Global average activities among all processors are also reported. The mpstat command can be used both on SMP and UP machines, but in the latter, only global average activities will be printed. If no activity has been selected, then the default report is the CPU utilization report.

[root@localhost ~]# mpstat -A



free : free displays the total amount of free and used physical and swap memory in the system, as well as the buffers and caches used by the kernel. The information is gathered by parsing /proc/meminfo.

[root@localhost ~]#  free -mlt




swapon : It displays name of the swap device along with total/used size and it's priority:


[root@localhost ~]# swapon --show
NAME      TYPE      SIZE  USED PRIO
/dev/dm-1 partition 3.9G 70.5M   -2


ps : This command gives you the list of running process on the system.

[root@localhost ~]# ps -eaf
UID         PID   PPID  C STIME TTY          TIME CMD
root          1      0  0 13:14 ?        00:00:08 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
root          2      0  0 13:14 ?        00:00:00 [kthreadd]
root          3      2  0 13:14 ?        00:00:11 [ksoftirqd/0]
root          5      2  0 13:14 ?        00:00:00 [kworker/0:0H]
root          7      2  0 13:14 ?        00:00:07 [migration/0]
root          8      2  0 13:14 ?        00:00:00 [rcu_bh]
root          9      2  0 13:14 ?        00:00:08 [rcu_sched]
root         10      2  0 13:14 ?        00:00:00 [lru-add-drain]
root         11      2  0 13:14 ?        00:00:03 [watchdog/0]
root         12      2  0 13:14 ?        00:00:03 [watchdog/1]
root         13      2  0 13:14 ?        00:00:14 [migration/1]
root         14      2  0 13:14 ?        00:00:09 [ksoftirqd/1]
root         16      2  0 13:14 ?        00:00:00 [kworker/1:0H]
root         17      2  0 13:14 ?        00:00:04 [watchdog/2]
root         18      2  0 13:14 ?        00:00:19 [migration/2]
root         19      2  0 13:14 ?        00:00:03 [ksoftirqd/2]
root         20      2  0 13:14 ?        00:00:08 [kworker/2:0]
root         21      2  0 13:14 ?        00:00:00 [kworker/2:0H]
root         22      2  0 13:14 ?        00:00:06 [watchdog/3]
root         23      2  0 13:14 ?        00:00:24 [migration/3]
root         24      2  0 13:14 ?        00:00:27 [ksoftirqd/3]
root         26      2  0 13:14 ?        00:00:00 [kworker/3:0H]


Use of "--forest" switch in the ps command displays parent and chile process tree.

[root@localhost ~]#  ps -e -o pid,args --forest





pstree : It shows running processes as a tree. The tree is rooted at either pid or init if pid is omitted. If a user name is specified, all process trees rooted at processes owned by that user are shown.


[root@localhost ~]# pstree



[root@localhost ~]# pstree -u hpe
bash───vim

bash───vim



lsof : This displays the lists of files currently opened on your system.

[root@localhost ~]# lsof | more



If you want to check list of opened files for some user:

[root@localhost ~]# lsof -u hpe



strace : Strace is used for debugging and troubleshooting the execution of an executable. It displays the system calls used by the process, and the signals received by the process. It also provides execution of a binary from start to end.

[root@localhost ~]# strace ls
execve("/usr/bin/ls", ["ls"], [/* 31 vars */]) = 0
brk(NULL)                               = 0x1b95000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe9a053f000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=122100, ...}) = 0
mmap(NULL, 122100, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fe9a0521000
close(3)                                = 0
open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320i\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=155784, ...}) = 0
mmap(NULL, 2255184, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe9a00f8000
mprotect(0x7fe9a011c000, 2093056, PROT_NONE) = 0
mmap(0x7fe9a031b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7fe9a031b000
mmap(0x7fe9a031d000, 6480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe9a031d000
close(3)                                = 0
open("/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \26\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=20032, ...}) = 0
.
.
.



Space reclamation / UNMAP on RHEL or CentOS 7

What is Space Reclamation?

This feature supports Thin Provisioned environment, and it is the process of freeing space from the storage system that has already been freed from the host file system.

A host file system contains metadata to keep track to know which blocks are available to store new data and which blocks contain actual data and must not be overwritten. This metadata is stored within the LUN. When a file is deleted in the host file system, the file system metadata is updated to mark that file's blocks as free space. Total file system free space is then recalculated to include the newly-freed blocks. To the storage system, these metadata updates appear no different than any other writes being performed by the host. Therefore, the storage system is unaware that any deletions have occurred.


What is Unmapping?

Unmapping is a process which de-allocates relationship between LBA and a physical block in a logical unit.


Reclamation within RHEL and CentOS:

RHEL / CentOS 6.x and later versions of operating systems have introduced SCSI UNMAP command in their propiratory EXT4 filesystem, to support releasing of free space from SAN platforms. This is being done by UNMAP command. The Linux kernel discard requests to inform the storage that a given range of blocks is no longer in use or not really having any valid data.

This can be done by two ways:

Method 1: Create EXT4 file-system and mount it with "discard" option, which automatically sends SCSI UNMAP command from host to array as soon as blocks on the storage frees up.

[root@localhost ~]# mkfs.ext4 /dev/sdb
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdb is entire device, not just one partition!
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1310720 inodes, 5242880 blocks
262144 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2153775104
160 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done


[root@localhost ~]# mkdir /space_reclaim_dir

[root@localhost ~]# mount -o discard /dev/sdb /space_reclaim_dir/

[root@localhost ~]# mount | grep "/dev/sdb"
/dev/sdb on /space_reclaim_dir type ext4 (rw,relatime,seclabel,discard,data=ordered)


Method 2: Another method is running "fstrim" command as per user requirement on a mounted partition or disk to free up unused blocks:

I have a mounted disk which is not mounted using "discard" option.

[root@localhost ~]# mount | grep "/dev/sdb"
/dev/sdb on /space_reclaim_dir type ext4 (rw,relatime,seclabel,data=ordered)

[root@localhost ~]# fstrim -v /space_reclaim_dir

In case you want to freeup unused blocks from all of the mounted partitions then run:

[root@localhost ~]# fstrim -a

NOTE:

If your mounted device doesn't support UNMAP operation then you might see this error:

[root@localhost ~]# fstrim -v /no_space_reclaim_dir
fstrim: /no_space_reclaim_dir: the discard operation is not supported

Also to verify TRIM support on all of the devices run :

[root@localhost ~]# lsblk --discard
NAME            DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sda                    0        0B       0B         0
├─sda1                 0        0B       0B         0
└─sda2                 0        0B       0B         0
  ├─centos-root        0        0B       0B         0
  └─centos-swap        0        0B       0B         0
sdb                    0        0B       0B         0
sr0                    0        0B       0B         0

check the values of DISC-GRAN (discard granularity) and DISC-MAX (discard max bytes) columns. Non-zero values indicate TRIM support. In current output none of the disks supports "TRIM" or UNMAP feature.

Wednesday, April 17, 2019

What are /dev/random and /dev/urandom files?

/dev/random will block after the entropy pool is exhausted. It will remain blocked until additional data has been collected from the sources of entropy that are available. This can slow down random data generation. Randomness is critical to the security of cryptography in your application – one-time pads, key generation.

[root@localhost ~]# dd if=/dev/random of=random.txt bs=2048 count=2048
dd: warning: partial read (82 bytes); suggest iflag=fullblock
0+2048 records in
0+2048 records out
150665 bytes (151 kB) copied, 21.6704 s, 7.0 kB/s


/dev/urandom will not block. Instead it will reuse the internal pool to produce more pseudo-random bits. It's best when we have to a large file with random data. Also if you are using "dd" command to wipe data of a disk by replacing it with random data.

[root@localhost ~]# dd if=/dev/urandom of=urandom.txt bs=2048 count=2048
2048+0 records in
2048+0 records out
4194304 bytes (4.2 MB) copied, 0.0361976 s, 116 MB/s

Difference between /var/log/messages and /var/log/dmesg log files

/var/log/messages – This file contains global system messages, including the messages that are logged during system startup. There are several things that are logged in /var/log/messages including mail, cron, daemon, kern, auth, etc.

[root@localhost ~]# cat /var/log/messages
Apr 17 20:30:03 localhost rsyslogd: [origin software="rsyslogd" swVersion="8.24.0-34.el7" x-pid="9423" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Apr 17 20:40:02 localhost systemd: Started Session 9 of user root.
Apr 17 20:40:32 localhost dhclient[9217]: DHCPREQUEST on ens33 to 192.168.189.254 port 67 (xid=0xde4e9a7)
Apr 17 20:40:32 localhost dhclient[9217]: DHCPACK from 192.168.189.254 (xid=0xde4e9a7)
Apr 17 20:40:32 localhost NetworkManager[8908]: <info>  [1555513832.8097] dhcp4 (ens33):   address 192.168.189.128
Apr 17 20:40:32 localhost NetworkManager[8908]: <info>  [1555513832.8107] dhcp4 (ens33):   plen 24 (255.255.255.0)
Apr 17 20:40:32 localhost NetworkManager[8908]: <info>  [1555513832.8107] dhcp4 (ens33):   gateway 192.168.189.2
Apr 17 20:40:32 localhost NetworkManager[8908]: <info>  [1555513832.8107] dhcp4 (ens33):   lease time 1800
Apr 17 20:40:32 localhost NetworkManager[8908]: <info>  [1555513832.8107] dhcp4 (ens33):   nameserver '192.168.189.2'
Apr 17 20:40:32 localhost NetworkManager[8908]: <info>  [1555513832.8107] dhcp4 (ens33):   domain name 'localdomain'
Apr 17 20:40:32 localhost NetworkManager[8908]: <info>  [1555513832.8108] dhcp4 (ens33): state changed bound -> bound
Apr 17 20:40:32 localhost dbus[8773]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
Apr 17 20:40:32 localhost dhclient[9217]: bound to 192.168.189.128 -- renewal in 735 seconds.
Apr 17 20:40:32 localhost systemd: Starting Network Manager Script Dispatcher Service...
.
.
.


/var/log/dmesg – This file contains kernel ring buffer information. When the system boots up, it prints number of messages on the screen that displays information about the hardware devices that the kernel detects during boot process. These messages are available in kernel ring buffer and whenever the new message comes the old message gets overwritten. You can also view the content of this file using the dmesg command.

[root@localhost ~]# cat /var/log/dmesg
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-957.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Thu Nov 8 23:39:32 UTC 2018
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-957.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_IN.UTF-8
[    0.000000] Disabled fast string operations
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ebff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009ec00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000dc000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bfecffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bfed0000-0x00000000bfefefff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000bfeff000-0x00000000bfefffff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000bff00000-0x00000000bfffffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000f0000000-0x00000000f7ffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffe0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.7 present.
[    0.000000] DMI: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
[    0.000000] Hypervisor detected: VMware
[    0.000000] vmware: TSC freq read from hypervisor : 2903.999 MHz
[    0.000000] vmware: Host bus clock speed read from hypervisor : 66000000 Hz
[    0.000000] vmware: using sched offset of 9162497474 ns
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0x140000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
.
.
.

What are /dev/zero and /dev/null files ?

"/dev/zero" and "/dev/null" are two dummy devices files which are useful for creating empty files.


"/dev/zero": 

It is used to create a file with no data but with required size(A file with all zero’s written on it).

Let's create file with /dev/zero

[root@localhost ~]# dd if=/dev/zero of=/opt/zero.txt bs=2048 count=2048
2048+0 records in
2048+0 records out
4194304 bytes (4.2 MB) copied, 0.150465 s, 27.9 MB/s

it creates a file that has continuous zeros in it. So we can say that /dev/zero is a file which is used to create a new file with some required size without any meaning to the data.


"/dev/null":

This file is useful in cases like redirecting unwanted output/error etc to this file. Actually it acts as a black hole which absorbs this unwanted input. So whenever you feed some data to this file, you can not retrieve the data which is fed to it. This file even useful for creating files with zero size.

e.g.

If you see i have few files and folders in current directory:

[root@localhost ~]# ls
anaconda-ks.cfg  Documents  initial-setup-ks.cfg  Pictures  Templates
Desktop          Downloads  Music                 Public    Videos

If i will redirect "ls" output to "/dev/null" file, the output data can't be retrive again even after reading null file:

[root@localhost ~]# ls > /dev/null

[root@localhost ~]# cat /dev/null
[root@localhost ~]#

Tuesday, April 16, 2019

How to install and configure Docker on RHEL or CentOS 7 ?

How to install and configure Docker on RHEL or CentOS 7 ?


Step 1. Install the required packages. yum-utils provides the yum-config-manager utility, and device-mapper-persistent-data and lvm2 are required by the device-mapper storage driver.

Verify if centos "extras" repository is enabled in "/etc/yum.repos.d/CentOS-Base.repo", bydefault it would be enabled:

[extras]
name=CentOS-$releasever - Extras
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7


NOTE: If installing on RHEL 7.5 or ater version, please mount RHEL 7.6  or later ISO to /mnt and add local repository for yum, note that RHEL 7.6 iso is needed to update few of the packages and dependencies as 7.5 ISO has old version of packages which are not compatible to the latest docker releases.


[root@localhost ~]# yum install -y yum-utils device-mapper-persistent-data lvm2


Add docker repository, so that docker can be installed with its dependencies:

[root@localhost ~]# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

Note: This will download and add "/etc/yum.repos.d/docker-ce.repo" file.

[root@localhost ~]# yum update

[root@localhost ~]# yum install docker-ce docker-ce-cli containerd.io

OR

In case if you manually want to download and install "docker-ce, docker-ce-cli, containerd.io" rpm's those RPM packages are available at:




(ERROR IN RHEL 7) : In case during "yum install docker-ce" you are observing below error:

[root@localhost ~]# yum install docker-ce docker-ce-cli containerd.io
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Resolving Dependencies
--> Running transaction check
---> Package containerd.io.x86_64 0:1.2.5-3.1.el7 will be installed
---> Package docker-ce.x86_64 3:18.09.6-3.el7 will be installed
--> Processing Dependency: container-selinux >= 2.9 for package: 3:docker-ce-18.09.6-3.el7.x86_64
---> Package docker-ce-cli.x86_64 1:18.09.6-3.el7 will be installed
--> Finished Dependency Resolution
Error: Package: 3:docker-ce-18.09.6-3.el7.x86_64 (docker-ce-stable)
           Requires: container-selinux >= 2.9
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest


then install "container-selinux" rpm first:

[root@localhost ~]# yum install http://vault.centos.org/centos/7.3.1611/extras/x86_64/Packages/container-selinux-2.9-4.el7.noarch.rpm

Then again try to install docker:

[root@localhost ~]# yum install docker-ce


Step 2. Start Docker daemon:

[root@localhost ~]# systemctl start docker

[root@localhost ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2019-03-29 20:56:16 IST; 10s ago
     Docs: https://docs.docker.com
 Main PID: 77237 (dockerd)
    Tasks: 13
   Memory: 36.0M
   CGroup: /system.slice/docker.service
           └─77237 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Mar 29 20:56:11 localhost.localdomain dockerd[77237]: time="2019-03-29T20:56:11.297944396+05:30" level=i...rpc
Mar 29 20:56:11 localhost.localdomain dockerd[77237]: time="2019-03-29T20:56:11.298059662+05:30" level=i...rpc
Mar 29 20:56:11 localhost.localdomain dockerd[77237]: time="2019-03-29T20:56:11.370193837+05:30" level=i...ds"
Mar 29 20:56:11 localhost.localdomain dockerd[77237]: time="2019-03-29T20:56:11.443415096+05:30" level=i...t."
Mar 29 20:56:13 localhost.localdomain dockerd[77237]: time="2019-03-29T20:56:13.734544797+05:30" level=i...ss"
Mar 29 20:56:16 localhost.localdomain dockerd[77237]: time="2019-03-29T20:56:16.427547518+05:30" level=i...e."
Mar 29 20:56:16 localhost.localdomain dockerd[77237]: time="2019-03-29T20:56:16.522272850+05:30" level=i...9.5
Mar 29 20:56:16 localhost.localdomain dockerd[77237]: time="2019-03-29T20:56:16.522575499+05:30" level=i...on"
Mar 29 20:56:16 localhost.localdomain systemd[1]: Started Docker Application Container Engine.
Mar 29 20:56:16 localhost.localdomain dockerd[77237]: time="2019-03-29T20:56:16.538844946+05:30" level=i...ck"
Hint: Some lines were ellipsized, use -l to show in full.


The docker daemon is up and running, now lets make it persistent across reboot:

[root@localhost ~]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.


To check if docker is installed and configured properly run:

[root@localhost ~]# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.09.5
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-957.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.683GiB
Name: localhost.localdomain
ID: KDKJ:IZFD:G32Q:DHWP:ORN6:EBWW:Q466:OAF7:MN6K:6WVY:T3M2:4VID
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine


[root@localhost ~]# docker -v
Docker version 18.09.5, build e8ff056