Bash: Read File Line By Line – While Read Line Loop

Posted on by admin

The while loop is the best way to read a file line by line in Linux.

If you need to read a file line by line and perform some action with each line – then you should use a while read line construction in Bash, as this is the most proper way to do the necessary.

In this article i will show the general syntax of the while read line construction in Bash and an example of how to read a file line by line from the Linux command line.

I will also show an example of the Bash script that reads an input file line by line and prints each line with some appended text. (more…)

6 Comments

HowTo: View a Config File Without Comments

Posted on by admin

With the help of grep command it is quite easy just to look at the active setting of configuration files, omitting comments.

The following command will display the content of the file somefile.conf, omitting blank lines and the lines started with #: 

$ grep -v '^\s*$\|^\s*\#' somefile.conf

Also, you can pipe the output to a new file:

$ grep -v '^\s*$\|^\s*\#' somefile.conf > newfile.conf

To make filtering easier, you can add an alias for ‘clean config’:

$ alias cf='grep -v '^\s*$\|^\s*\#'

And use it as follows:

$ cf /etc/ntp.conf
driftfile /var/lib/ntp/drift
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1 
restrict -6::1
server 0.centos.pool.ntp.org
server 1.centos.pool.ntp.org
server 2.centos.pool.ntp.org
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
No comments yet

HowTo: Remove Blank Lines From a File

Posted on by admin

Use one of the following commands to remove blank lines from a file.

1. Using the grep command:

$ grep -v "^$" file.txt

2. Using the sed command:

$ sed '/^$/d' file.txt

3. Using the awk command:

$ awk '/./' file.txt

4. Using the tr command:

$ tr -s '\n' < file.txt

You also can pipe the output of each command to a new file, like follows:

$ grep -v "^$" input.txt > output.txt
No comments yet

HowTo: Set a Warning Message (Banner) in SSH

Posted on by admin

SSH warning banners and welcome messages are necessary when organization wishes to prosecute an unauthorized user or just give out some information or announcement.

Display SSH Warning Message BEFORE the Login

Pre login SSH warning banner shows before the password prompt, during an interactive session using SSH. It usually uses for legal warnings to establish the terms and conditions by which someone is allowed to use the system.

The SSH warning messages are commonly located in the files ‘/etc/issue’ and ‘/etc/issue.net’, but you can also use your custom file like ‘/etc/ssh/sshd-banner’. The content of the specified file is sent to the remote user before authentication.

Create an SSH login banner file:

$ vi /etc/ssh/sshd-banner

Append some Warning text:

WARNING:  Unauthorized access to this system is forbidden and will be
prosecuted by law. By accessing this system, you agree that your actions
may be monitored if unauthorized usage is suspected.

Open the sshd_config file:

$ vi /etc/ssh/sshd_config

Edit the path to the banner file:

Banner /etc/ssh/sshd-banner

Save the file and reload the sshd:

$ service sshd reload

Display SSH Welcome Message AFTER the Login

The content of the file ‘/etc/motd’ is displayed after successful authentication, but just before the shell. It is used for system announcements and other important information, that you want authenticated users to know about before they start using the system.

Edit the file ‘/etc/motd’:

$ vi /etc/motd

Place the announcement message and save the file.

Now this message will be shown after the successful authentication through SSH.

No comments yet

3 Steps to Perform SSH Login Without Password

Posted on by admin

It is very easy to perform SSH login to the remote server without prompting a password.

With a help of utilities from OpenSSH package, you can generate authentication keys on your local machine, copy public key to the remote server and add identities to your authentication agent.

Just three simple steps separate you from the possibility of connecting to a remote server without prompting a password.

Step 1: Generate a key pair on the local server

Use ssh-keygen to generate authentication keys for SSH.

$ ssh-keygen

Output:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa): 
Created directory '/home/user/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/user/.ssh/id_rsa.
Your public key has been saved in /home/user/.ssh/id_rsa.pub.
The key fingerprint is:
8c:2a:ed:82:98:6d:12:0a:3a:ba:b2:1c:c0:25:be:5b

Step 2: Install your public key on the remote server

Use ssh-copy-id to connect to the remote machine and install your public key by adding it the authorized_keys file.

$ ssh-copy-id -i ~/.ssh/id_rsa.pub UserName@RemoteServer

Output:

UserName@RemoteServer's password: ********
Now try logging into the machine, with "ssh 'username@remoteserver'", and check in:
~/.ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.

Step 3: Add a private key to the authentication agent on the local server

Use ssh-add to add identities to the ssh-agent – the authentication agent.

$ ssh-add

Output:

Identity added: /home/user/.ssh/id_rsa (/home/user/.ssh/id_rsa)

Now you can log into the remote server via the SSH protocol without prompting a password.

7 Comments

HowTo: Configure CISCO port description

Posted on by admin

Type the following commands to create/edit the description of the port ‘FastEthernet 0/1’ on CISCO switch or router:

# enable
# configure terminal
(config)# interface FastEthernet 0/1
(config-subif)# description This is 1st Port of My Switch
(config-subif)# end
# write

To display a description and a status of an interface, use the ‘show interfaces description‘ command:

# show interfaces FastEthernet 0/1 description
Interface                      Status         Protocol Description
FastEthernet 0/1                     up           up     This is 1st Port of My Switch
No comments yet

HowTo: Find Large Files in Linux

Posted on by admin

Finding the largest files is extremely useful especially when you you’re low on disk space and want to free it up.

The best way to find the largest files on your Linux system is to use the command line.

Actually there is no simple command to list the largest files in Linux.

However, you can easily find the largest files, using a combination of the several simple commands.

Find The Largest Files in Linux

Run the combination of the following commands to find the largest files on your Linux system, beginning from the <DIR> directory (change <DIR> to whatever directory you need), and list top 10 of them.

$ find <DIR> -mount -type f -ls 2> /dev/null | sort -rnk7 | head -10 | awk '{printf "%10d MB\t%s\n",($7/1024)/1024,$NF}'

Find 10 largest files, starting from ‘/’ (root)

$ find / -mount -type f -ls 2> /dev/null | sort -rnk7 | head -10 | awk '{printf "%10d MB\t%s\n",($7/1024)/1024,$NF}'

       106 MB	/var/lib/mysql/ibdata1
        94 MB	/usr/lib/locale/locale-archive
        41 MB	/scripts/20130206-015833.tar.gz
        41 MB	/scripts/20130206-004839.tar.gz
        41 MB	/scripts/20130206-130400.tar.gz
        41 MB	/scripts/20130206-000442.tar.gz
        41 MB	/scripts/20130206-132019.tar.gz
        41 MB	/root/20130208-133954.tar.gz
        33 MB	/var/log/messages-20130303
        32 MB	/var/lib/rpm/Packages

Find 10 largest files, starting from ‘/home’

$ find /home -mount -type f -ls 2> /dev/null | sort -rnk7 | head -10 | awk '{printf "%10d MB\t%s\n",($7/1024)/1024,$NF}'

      3007 MB	/home/user/Desktop/share/linux-65835.iso
       448 MB	/home/user/Pictures/Turkey/SAM_4590.AVI
       266 MB	/home/user/Pictures/Turkey/SAM_4588.AVI
       173 MB	/home/user/Camera/VID_20130909_120713.mp4
       152 MB	/home/user/Camera/VID_20130909_115427.mp4
       133 MB	/home/user/Camera/VID_20130909_210904.mp4
       133 MB	/home/user/Pictures/Paris/VID_20130928_182431.mp4
       131 MB	/home/user/Pictures/Turkey/SAM_4597.AVI
       129 MB	/home/user/Pictures/Turkey/SAM_4641.AVI
       127 MB	/home/user/Desktop/tmp/Camera/VID_20130911_164440.mp4
No comments yet

HowTo: Restore MySQL Database From the Command Line

Posted on by admin

Do not use phpMyAdmin to restore your MySQL database if the MySQL database is large.

phpMyAdmin has limit on total upload size and, there is also maximum execution time which may cause browser to time out.

The solution of how to restore large MySQL database from the sql dump file is to use Linux/Unix command line.

Restoring an existent Database

If you need to restore a database that has already been existed on the server, than you have to use ‘mysqlimport‘ command.

The syntax for ‘mysqlimport’:

$ mysqlimport -u [username] -p[password] [dbname] [backupfile.sql]

Note: There is no space between -p and the password.

Example:

$ mysqlimport -u root -pSeCrEt customers_db customers_db_backup.sql

Restoring a new Database

1. Create an appropriately named Database on the target server.

Example:

mysql> CREATE DATABASE customers_db;

2. Load the sql dump file using the ‘mysql‘ command:

$ mysql -u [username] -p[password] [db_to_restore] < [backupfile.sql]

Example:

$ mysql -u root -pSeCrEt customers_db < customers_db_backup.sql

If the sql dump file is compressed, you can do the following:

$ gunzip < [backupfile.sql.gz] | mysql -u [username] -p[password] [dbname]

Example:

$ gunzip < customers_db_backup.sql.gz | mysql -u root -pSeCrEt customers_db
No comments yet

HowTo: BackUp MySQL Database From the Command Line

Posted on by admin

The basic installation of MySQL Server provides a great command line utility to take backup of MySQL databases

The ‘mysqldump‘ command is used for creation “dumps” of databases managed by MySQL.

These dumps are just files with all the SQL commands needed to recreate the database.

Backup a Database

Use the following command to backup a database:

$ mysqldump --opt -u [username] -p[password] [dbname] > [backupfile.sql]

Note: There is no space between -p and the password.

Example:

$ mysqldump --opt -u root -pSeCrEt customers_db > customers_db_backup.sql

Backup multiple Databases

Use the following command to backup several databases:

$ mysqldump --opt -u [username] -p[password] --databases [dbname_1] [dbname_2] ... > [backupfile.sql]

Example:

$ mysqldump --opt -u root -pSeCrEt --databases customers_db employees_db > backup.sql

Backup all Databases

Use the following command to backup all databases:

$ mysqldump --opt -u [username] -p[password] --all-databases > [backupfile.sql]

Example:

$ mysqldump --opt -u root -pSeCrEt --all-databases > all_db_backup.sql

Backup a Database with a Compression

Use the following command to backup a database with compression:

$ mysqldump --opt -u [username] -p[password] [dbname] |  gzip -9 > [backupfile.sql]

Example:

$ mysqldump --opt -u root -pSeCrEt customers_db |  gzip -9 > customers_db_backup.sql.gz

If you want to extract the .gz file, use the command below:

$ gunzip customers_db_backup.sql.gz
No comments yet

Installing VSftpd FTP Server on CentOS/RHEL

Posted on by admin

VSftpd is an FTP server for Linux.

According to its authors, it is very secure, stable and fast.

In this article, I’ll show how to install and configure VSftpd FTP Server in Centos / RHEL and how to add new FTP user.

The FTP protocol is insecure. If it possible, use SFTP – Secure FTP which is inbuilt into OpenSSH SSHD server.

Install the VSftpd Server

Type the following command to install the VSftpd FTP Server:

yum install vsftpd

Backup the default configuration file ‘vsftpd.conf’:

cp /etc/vsftpd/vsftpd.conf /etc/vsftpd/vsftpd.conf.back

We also need to install the FTP client, so that we can connect to an FTP server:

yum install ftp

Configure the VSftpd Server

Once the VSftpd Server is installed, you can adjust the basic configuration.

Anonymous FTP is allowed by default. Set the anonymous_enable=NO to secure your server.

Open the configuration file:

vi /etc/vsftpd/vsftpd.conf

Make the following changes:

Option Description
anonymous_enable=NO Disable anonymous login
local_enable=YES Enable local users
write_enable=YES Give FTP users permissions to write data
connect_from_port_20=NO Port 20 need to be turned off. It makes VSftpd run less privileged
chroot_local_user=YES Chroot everyone
local_umask=022 Set umask to 022, to make sure that all the files (644) and folders (755) you upload, get the proper permissions

Check vsftpd.conf man pages, for all configuration options.

man vsftpd.conf

Add New FTP User

Let’s add the new user called ‘ftpuser’ and set ‘/var/www/path/to/your/dir’ as his home directory:

useradd -d '/var/www/path/to/your/dir' -s /sbin/nologin ftpuser

Setup a password for the new user:

passwd ftpuser

Create the home directory for the new user, if you haven’t done it before:

mkdir -p /var/www/path/to/your/dir

To enable the ‘ftpuser’ to read and write the data in his home directory, change the permissions and the ownership:

chown -R ftpuser '/var/www/path/to/your/dir'
chmod 775 '/var/www/path/to/your/dir'

Add the group ‘ftpusers’ for an FTP users and add the ‘ftpuser’ to it:

groupadd ftpusers
usermod -G ftpusers ftpuser

Configure the Firewall for VSftpd

Add the rule to the IPTABLES, if you use it:

vi /etc/sysconfig/iptables

Append the following line, before the REJECT line, to open the port 21:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 21 -j ACCEPT

Save and close the file. Restart the firewall.

service iptables restart

Set the VSftpd service to Start At Boot

chkconfig --levels 235 vsftpd on

Start the VSftpd FTP service.

service vsftpd start

Test the VSftpd Server

Test the FTP Server locally.

ftp localhost

Output:

Trying 127.0.0.1...
Connected to localhost (127.0.0.1).
220 (vsFTPd 2.2.2)
Name (localhost:root): ftpuser
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
***

Test it remotely.

ftp your.ftp.server.com

Output:

Connected to your.ftp.server.com.
220 (vsFTPd 2.2.2)
Name (your.ftp.server.com:yourname):
Name (localhost:root): ftpuser
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
***
No comments yet