AWK command in UNIX and LINUX with examples

awk command in unix and linux with examples
Awk command in UNIX and LInux

1.Awk Introduction
2.Awk Installation
3.Basic examples of awk
4.Count and Print number of lines in file
5.Print all lines matches the pattern
6.Counting number of matched pattern
7.Print lines based on given characters
8.Operators in Awk
9.Arrays in awk
10.Regular expressions
11.Awk condition
12.Awk loop
13.Awk Statements
14.Awk functions
15.Awk redirection


1. Awk command in UNIX and Linux Introduction

The tutorial Awk command in UNIX and Linux is meant to teach about the basic and the advance use of awk as well. At the end of the tutorial with enough practice you will be able to use awk for your general purposes.
According to GNU abbreviation, awk is derived from its authors Aho, Weinberger and Kernighan.

AWK is a programming language and is extensively used for data extraction and text processing.
It is more popularly used in OS like UNIX and Linux.


2. Awk installation

Before we start working on awk we need to check weather awk is pre-installed or not in your distribution of Linux/UNIX. Use which command, it will show you the installed path of awk.
You should see the following line if it is already installed.

$ which awk
/usr/bin/awk

Install awk by package manager by the following way in Debian and Redhat based Linux.

To install awk in redhat based system

# yum install gwk

To install in debian based system.

# apt-get install gwk

You can even download the awk source code if it's not available in your system.
You can get all the previous abd latest version of awk from here
Download the latest version.

# wget https://ftp.gnu.org/gnu/gawk/gawk-4.1.4.tar.xz

Extract the file with tar command

# tar -xfv gawk-4.1.4.tar.xz

A directory of gwk will be created. Go inside the directory with 'cd' command and run 'configure'

# ./configure

After that run make command for the successful compilation of source code.

# make

Now install awk with this command.

# make install


3. Basic examples of awk

The most common example of awk would be printing the content of file on the terminal.
The option '{print}' is used here with awk command.

# awk '{print}' emp.txt 
SERIAL         NAME         DESIGNATION    
---------    --------    ------------------

1.             JOHN          Engineer
2.             Katty         Steno
3.             MARC          Software Developer
4.             JORGE         Manager

There is another way of printing all lines, but in this case it won't print headings of the field.

# awk '/a/' emp.txt

In the above example you saw how to print the content of any text file. Now imagine there are many columns and you want to print only few columns or
just a single column. The below example will illustrate how to use awk command to print particular columns.

To print only second column.

# awk '{print $2}' file4.txt 
NAME
--------
JOHN
Katty
MARC
JORGE

To print column first and second.

# awk '{print $1"\t"$2}' file4.txt 
SERIAL         NAME
---------    --------
1.              JOHN
2.              Katty
3.              MARC
4.              JORGE

Print columns without the field name. The below command would print column 2 without its field.

# awk '/a/ {print $2}' file4.txt 

Note: $ (Dollar sign) in the above examples. It's is used to specify the column we want to print. "\t" was used to give the tabs space.


4. Count and Print number of lines in file

The below commands can be used to count number of lines present in file. It will also count spaces btweeen lines.

Example:1

# awk '{++a} END {print "Lines=", a}' file4.txt 
Lines= 7

Example:2

# awk 'END { print "Lines=", NR }' file4.txt
Lines= 7

Example:3

# awk 'BEGIN{a=0}{a++;}END{print "Lines=", a}' file4.txt 
Lines= 7

The above examples also includes spaces while counting lines. Use the below awk command to ignore spaces while counting lines.

# awk 'NF{++a} END{print "Lines=", a}' file4.txt
Lines= 6


5. Print all lines matches the pattern

We will see an example of awk where we want to print from the file which matches the pattern 'Software Developer'

# awk '/Software Developer/' file4.txt 
3.    Marc    Software Developer
5.    Lucy    Software Developer

To print multiple pattern matches. For example printing lines which matches pattern 'Software Developer' and 'Engineer'.

# awk '/Software Developer|Engineer/' file4.txt 
1.    John    Engineer
3.    Marc    Software Developer
5.    Lucy    Software Developer
6.    Raun    Engineer    

Similarly, you can extend your search with three or four pattern matches.


6. Counting number of matched pattern

To count number of pattern matched in a file. Execute the following command. The below command will look for the pattern 'Katty' in the file and print the number of pattern matched.

# awk '/Katty/{++b} END {print "Matched=", b}' file4.txt
Matched= 1


7. Print lines based on given characters

Print lines which has less than 15 character

# awk 'length($0) < 15' file4.txt 
2.    Katty    Steno

Print details of the person having ID = 521.

# awk '$1==521' file4.txt 
521.    Jorge    Manager

Print details of all the persons in an organization having ID less than 300.

# awk '$1 < 300' file4.txt 
210.    John    Engineer
163.    Marc    Software Developer
056.    Lucy    Software Developer

 

8. Operators in Awk

There are various operators you can use with awk. We are going to learn about some of the most common operators which can be used with awk.

a) Arithmetic operators

As the name arithmetic implies that we can do arithmetic operations, such as additions (+), substractions (-), multiplications (*), division (/), etc.

Addition

To add two numbers we can use the awk command in the following way.

$ awk 'BEGIN{c=2; d=4; print "Total=", (c+d)}'
Total= 6

Subtraction

To do the subtraction.

$ awk 'BEGIN{c=5; d=2; print "Total=", (c-d)}'
Total= 3

Multiplication

To multiply two numbers.

$ awk 'BEGIN{g=5; h=2; print "Total=", (g*h)}'
Total= 10

Division

For dividing two numbers we can use awk in the given way.

$ awk 'BEGIN{e=14; f=7; print "Total=", (e/f)}'
Total= 2

b) Assignment operator

Assignment operators are represented by equal sign (=). The common assignment operators that we will be discussion about are
=, +=, *=, /=, ^=

= sign

With this '=' operator you can assign any value. For example.

$ awk 'BEGIN { name="Raby"; print "Hello", name }'
Hello Raby

+= sign

Assigned value will be added with n.

$ awk 'BEGIN { n=3; n += 4; print "OUTPUT:", n }'
OUTPUT: 7

*= sign

In this case, the assigned value will be multiplied with n.

$ awk 'BEGIN { n=5; n *= 6; print "OUTPUT:", n }'
OUTPUT: 30

/= sign

Here, n will be divided by the assigned value that is 5.

$ awk 'BEGIN { n=25; n /= 5; print "OUTPUT:", n }'
OUTPUT: 5

^= sign

In this case n has the power 5.

$ awk 'BEGIN { n=2; n ^= 5; print "OUTPUT:", n }'
OUTPUT: 32

c) Logical operators

There are three logical operators you can use with awk tool and they are AND(&&), OR(||), NOT(!). Below are some examples to demonstrate the
use of each of these operators.

AND (&&)

Consider having a file 'file1.txt'

$ cat file1.txt
4 3 7 2
8 6 4 5
3 1 9 7
2 5 6 9

The below example will check column 1 for number less than 4 and column 4 for number greater than 5 and then print the line.

$ awk '$1 < 4 && $4 > 5' file1.txt 
3 1 9 7
2 5 6 9

OR (||)

OR operator is used to execute when either of the condition is true. The below example will check for either of the conditions and prints.

$ awk '$2 == 3 || $3 == 1' file1.txt 
4 3 7 2

NOT (!)

NOT operator is used to return 0 if the condition is true. For example, the below example will check column 2 for number equal to 3 and exclude that line.

The below example will check column 2 and print lines having character greater than 4.

$ awk 'length($2) > 4' file4.txt
1.    Steve         Engineer
3.    Samantha      Software Developer

Note the use of NOT (!) operator now in this example. It prints nothing and returns zero.

$ awk '!length($2) > 4' file4.txt


9. Arrays in awk

In a simple language arrays is used to collect data of similar types and arranged them in order so that it can be accessed later.

Syntax:

array[]=""

Let's see an example to print the value of an array in run time.

$ awk 'BEGIN {
name["n"]="Aman"
print name["n"]     
}'
OUTPUT: Aman

In the same way you can create two arrays at a time and print their values.

$ awk 'BEGIN {
name["n"]="Aman"
name["s"]="Sam"
print name["n"]
print name["s"]     
}'
OUTPUT: 
Aman
Sam

Multi-dimensional array

Similarly you can use multi-dimensional arrays in awk to store multiple values.

awk 'BEGIN {
name["0,0"]="Aman"
name["0,1"]="Sam"
name["1,0"]="Raymond"
name["1,1"]="Sonia"
print name["0,0"]
print name["0,1"]
print name["1,0"]
print name["1,1"]
}'
OUTPUT:
Aman
Sam
Raymond
Sonia

Deleting an array

You can also delete an array in run time. In the below example we are going to delete an array and then try to print it.

$ awk 'BEGIN {
name["n"]="Aman"
name["s"]="Sam"
delete name["n"]
print name["s"] 
print name["n"]    
}'
OUTPUT:
Sam

We have used 'delete' to delete the array, and it's clear by the example that it prints a blank space in place of the array.


10. Regular expressions

^ (Start)-- This expression matches the start of any line. For example.

Create a file 'file.txt'.

# vi file.txt
Namon works as a software developer
Nainital is a beautiful place
Martha is trying to learn awk command in UNIX
Jeneth is coming

Now run awk as follows to search every line and print lines which starting with character 'Na'.

# awk '/^Na/' file1.txt 
OUTPUT:
Namon works as a software developer
Nainital is a beautiful place

$ (Ending)-- This expression will check ending of lines for the matching character.

Look at the below example where awk searches for 'wk' at the end of each line and print the same.

# awk '/wk$/' file1.txt 
OUTPUT: Martha is trying to learn awk command in UNIX

. (DOT)-- Dot is used to match any character between words.

For instance, if we search for 'd.v'. It prints the below line as the word developer contains the character 'd.v' and dot is anything which comes between.

# awk '/d.v/' file1.txt 
OUTPUT: Namon works as a software developer

? -- Let's learn this expression by an example.

It makes the preceding character optional and ingnores it when searching for the string.
In the below example awk will look for any word contains 'ender' in the file and completely ingnores the preceding character 'g' in it.

# cat file.txt | awk '/x?ender/'
xender
gender

This time we have executed the above command with a pipe.

* (Asterisk) -- It matches any occurance of the preceding character.

The below command executes and matches any preceding characters.

# cat file1.txt | awk '/lea*/'
OUTPUT:
Martha is trying to learn awk command in UNIX
Sumit is learning Linux everyday

11. Awk condition

Like any other programming language condition statements are used in awk.

If

Its syntax is.

if (condition){

codes to be executed

}

For example, the below awk code checks the value of a and b and prints if the value is equal.

# awk 'BEGIN {a = 5; b = 5; if (a == b) printf "Both the numbers are equal.\n" }'
OUTPUT: Both the numbers are equal.

if else

Syntax:

if (){

code to execute

else

code to execute

}

Example:

# awk 'BEGIN {a = 5; b = 4; if (a == b) printf "Both numbers are equal.\n"; else printf "numbers are not equal.\n" }'
OUTPUT: numbers are not equal.

if else if

Syntax:

if (condition 1){

code to execute

else if (condition 2)

code to execute

else

code to execute

}

Example:

# awk 'BEGIN {str = "learning awk"; if (str == "learning linux") printf "learning linux.\n"; else if (str == "learning
awk") printf "learning awk.\n"; else printf "Not learning anything.\n"}'
OUTPUT: learning awk.

 

12. Awk loop

For loop

Example:

$ awk 'BEGIN {
for (i=0;
i<=3;
i++)
print i
}'
OUTPUT:
0
1
2
3

While loop

Example:

$ awk 'BEGIN {
i=0;
while (i<3){
print i;
i++}
}'
OUTPUT:
0
1
2

Do while loop

Example:

$ awk 'BEGIN {
i=0;
do
{
print i;
i++
}
while (i<3)
}'
OUTPUT:
0
1
2

 

13. Awk Statements

Break statement

Example:

$ awk 'BEGIN {
addition=0;
for (i=0;
i<6;
i++)
{
addition +=i;
if (addition>5)
break;
else
print "addition=",addition
}
}'
OUTPUT:
addition= 0
addition= 1
addition= 3

Continue statement

Example:

$ awk 'BEGIN {
for (i=0;
i<5;
i++)
{
if (i==3){
continue;
}
print "value=",i;
}
}'
OUTPUT:
value= 0
value= 1
value= 2
value= 4

Exit statement

Example:

$ awk 'BEGIN {
for (i=1;
i<7;
i++)
{
if (i==5){
exit;
}
print "value=",i;
}
}'
OUTPUT:
value= 1
value= 2
value= 3
value= 4


14. Awk functions

With awk you can define functions. We have created a file called 'test_funcion.awk'.

Example:

$ vi test_function.awk
#!/bin/awk -f
function test_func(a, b){
print "Value a=",a
print "Value b=",b
}
BEGIN {
test_func(1, 2)
}
OUTPUT: 
Value a= 1
Value b= 2

The first line in the file is to let the shell know we are running an awk file. In the above example, we have created a function called test_func(), then we are calling that function.


15. Awk redirection

Create a new file called 'file1.txt'. However, if the file is not created already, the below command will create a file when it's executed.

# awk 'BEGIN { print "This ia a test" > "/root/file1.txt" }'
# cat file1.txt
OUTPUT:
This ia a test

With the append operator '>>', we can add another sentence to the next line.

# awk 'BEGIN { print "This ia a second test" >> "/root/file1.txt" }'
# cat file1.txt
OUTPUT:
This ia a test
This ia a second test

 


This is it with the tutorial Awk command in UNIX and Linux with examples.

 

Download Our Free eBook now

Linux and UNIX Shell scripting ebook

Be the first to comment

Leave a Reply

Your email address will not be published.


*


shares