Process data with awk

awk is a record-oriented stream processing language. Where grep filters whole lines and cut extracts fixed columns, awk works on the structure of the data: it splits each line into fields, applies conditions per field, accumulates values in variables and reformats the output. No need for a Python script to compute the total of a CSV or count the error codes in a log: awk does it in one or two lines.

No curriculum names awk: the tool serves the RHCSA objective "Use grep and regular expressions to analyze text", as soon as the analysis is about columns rather than whole lines. Two implementations coexist, mawk by default on Debian and Ubuntu, gawk on RHEL: every example on this page gives the same output with both.

What you will learn

• Access the automatic fields $1, $NF, NR and NF
• Change the separator with -F to parse :, , or any delimiter
• Filter lines with numeric conditions ($3 >= 1000) and regex ($1 ~ /pattern/)
• Use BEGIN and END to print headers and compute totals
• Build an associative array to count occurrences per value
• Reformat output cleanly with printf

In which context?

awk comes in as soon as you need to reason about the structure of a line rather than its raw text:

• extract the login and shell of every account from /etc/passwd
• compute the total revenue from a CSV export
• count the 4xx and 5xx HTTP codes per IP in an access.log
• print only one user's processes from the output of ps
• reformat a command output as a readable table with aligned columns

awk is not a full scripting language like Python, but for stream processing on the command line, it is faster to write, more readable than a complex pipeline, and installed on every distribution.

The two datasets of this guide

Every output of this guide comes from the two files below. Create them in a working directory to reproduce each command and compare your result with the one shown. The first is a log excerpt in the Combined Log format produced by Apache and nginx; the second is a sales CSV export with a header line, exactly the kind of file you get from a spreadsheet.

cat > access.log <<'EOF'
192.168.1.10 - alice [09/Apr/2026:08:12:03 +0200] "GET /index.html HTTP/1.1" 200 1043
10.0.0.5 - bob [09/Apr/2026:08:12:44 +0200] "POST /login HTTP/1.1" 401 219
192.168.1.10 - alice [09/Apr/2026:08:13:01 +0200] "GET /dashboard HTTP/1.1" 200 5120
10.0.0.99 - - [09/Apr/2026:08:13:37 +0200] "GET /admin HTTP/1.1" 403 162
10.0.0.5 - bob [09/Apr/2026:08:14:02 +0200] "POST /login HTTP/1.1" 200 874
172.16.0.3 - charlie [09/Apr/2026:08:14:50 +0200] "GET /api/users/42 HTTP/1.1" 404 145
192.168.1.10 - alice [09/Apr/2026:08:15:12 +0200] "DELETE /api/orders/7 HTTP/1.1" 500 302
EOF

cat > sales.csv <<'EOF'
product,region,quantity,unit_price
laptop,north,5,1200
server,south,2,3500
switch,north,10,450
laptop,east,3,1200
server,north,1,3500
switch,south,8,450
disk,east,20,80
EOF

The <<'EOF' delimiter is between single quotes as a reflex: the text is then written as is, without the shell expanding a $ or a backtick command that a real log may contain.

The automatic fields

awk splits each line into fields according to a separator (spaces by default) and numbers them:

Variable	Content
$0	The whole line
$1	First field
$2	Second field
$NF	Last field (NF = number of fields)
NR	Number of the current line
NF	Number of fields on the current line

Swipe to see more

Example on an access.log

The Combined Log format always puts information at the same rank: the client address first, the HTTP status code in ninth position. That is what makes awk usable here without a regex: you designate a piece of data by its position, not by its shape. The program between braces runs once per line read, and print separates its arguments with a space.

awk '{print NR, $1, $9}' access.log

1 192.168.1.10 200
2 10.0.0.5 401
3 192.168.1.10 200
4 10.0.0.99 403
5 10.0.0.5 200
6 172.16.0.3 404
7 192.168.1.10 500

• $1 = IP address
• $9 = HTTP status code (the 9th field in the Combined Log format)
• NR = line number

Change the separator with -F

By default, awk splits fields on spaces. The -F option changes this separator:

Parse /etc/passwd with -F:

The /etc/passwd file lists the local accounts of the system, one per line, with seven fields separated by colons. cut -d: -f1,7 would do the same extraction, but awk also lets you filter on the value of a field, for example keep only the accounts whose shell is /bin/bash. Careful, -F: must come before the program on the command line. Placed after, the option is taken for a file name: gawk stops on fatal: cannot open file `-F:' for reading, mawk on cannot open "-F:". Your values will differ from these: they are the accounts of the test machine.

awk -F: '{print $1, $7}' /etc/passwd

root /bin/bash
daemon /usr/sbin/nologin
bob /bin/bash
alice /bin/zsh
sshd /usr/sbin/nologin

• $1 = user name
• $7 = login shell (7th field separated by :)

Compute on a CSV with -F,

# Print product and total (quantity × unit price), skipping the header
awk -F, 'NR > 1 {print $1, $3*$4}' sales.csv

laptop 6000
server 7000
switch 4500
laptop 3600
server 3500
switch 3600
disk 1600

NR > 1 skips the first line (header). Fields $3 and $4 are multiplied directly: awk handles arithmetic natively.

NR == 1 { next }, an equivalent alternative

NR == 1 { next } drops line 1 and moves to the next one. The benefit shows as soon as the program has several blocks: the header is discarded once, at the top of the program, instead of repeating NR > 1 in front of each block. It is the form used for the total computation below.

Filter with conditions

An awk program is written as condition { action }. When the condition is omitted, the action applies to every line; when the action is omitted, awk prints the whole line. The three forms of condition that follow, numeric, string and regex, cover nearly all administration filtering.

Numeric conditions

awk treats a field that looks like a number as a number in a comparison. It is the decisive difference with grep, which only knows text: grep 4 would find 404 as well as 1043, whereas $9 >= 400 really compares values.

The trap comes when the field does not look like a number. awk then switches to string comparison, and the result is surprising: echo abc | awk '$1 >= 400' prints the line on mawk as on gawk, because the string "abc" is lexically greater than "400". No error message, just spurious lines in your result. On a file whose cleanliness you do not control, force the conversion with $1+0 >= 400: the +0 imposes a numeric comparison and discards non-numeric values.

# Requests with an HTTP code >= 400 (errors)
awk '$9 >= 400 {print $1, $9, $7}' access.log

10.0.0.5 401 /login
10.0.0.99 403 /admin
172.16.0.3 404 /api/users/42
192.168.1.10 500 /api/orders/7

String comparison

The == operator requires strict equality on the whole field. Here the difficulty comes from the format: in the Combined Log, the request is surrounded by double quotes, so the sixth field is literally "POST with its opening quote attached. The "\"POST" sequence builds this string in the awk program, the backslash protecting the inner quote. If you are looking for a substring rather than an exact value, use the regex operator presented right after.

# POST requests only
awk '$6 == "\"POST" {print $1, $7, $9}' access.log

10.0.0.5 /login 401
10.0.0.5 /login 200

Filter by regex with ~ and !~

The ~ and !~ operators apply a regular expression to a specific field, which grep cannot do: grep 192.168 would also return a line where this address appears in the URL or the user agent. The pattern goes between slashes and is not anchored by default, hence the ^ in the first example to require the start of the field. Note the escaped dots as \.: without them, the regex dot accepts any character.

# IPs of the 192.168 network only
awk '$1 ~ /^192\.168\./ {print $1, $9}' access.log

192.168.1.10 200
192.168.1.10 200
192.168.1.10 500

# Exclude alice's requests
awk '$3 !~ /alice/ {print $1, $3, $9}' access.log

10.0.0.5 bob 401
10.0.0.99 - 403
10.0.0.5 bob 200
172.16.0.3 charlie 404

• ~: the field matches the pattern
• !~: the field does not match the pattern

BEGIN and END: global processing

BEGIN runs before the first line. END runs after the last one. They are used for headers, totals and summaries.

Compute a total with END

It is the pattern to remember for any aggregate computation: accumulate line by line in a variable, then print the result in END. Two details matter. awk implicitly initialises every numeric variable to 0, so there is nothing to declare before the loop. And the program spreads here over several lines inside a single pair of single quotes: the shell only takes over again at the closing quote, which allows this readable layout. The '\'' sequence inserts an apostrophe into the printed text without breaking that quote.

awk -F, 'NR == 1 { next }
{
  total += $3 * $4
  lines++
}
END {
  print "Lines processed:", lines
  print "Today'\''s total revenue:", total, "€"
}' sales.csv

Lines processed: 7
Today's total revenue: 29800 €

• total += $3 * $4: accumulation in a global variable
• lines++: counter of processed lines
• The END block is called only once, after every line

Associative arrays: count per value

awk natively supports arrays whose keys are strings, the perfect tool to count occurrences per value.

Count the HTTP codes

This command does in one line what would require a sort | uniq -c on a previously extracted field. The array key is the value read in the field, not a numeric index: awk creates the entry on the first encounter and increments it afterwards. As a result, the number of counters adapts by itself to the values present in the file, without you having to know the list of HTTP codes in advance.

awk '{codes[$9]++} END {for (c in codes) print c, codes[c]}' access.log | sort

200 3
401 1
403 1
404 1
500 1

• codes[$9]++: increments the counter indexed by the HTTP code value
• for (c in codes): iterates over every key of the array
• | sort: awk does not guarantee the order of the keys, so sort at the end

Summary table per IP

Here is the real case that justifies learning associative arrays: spot at a glance which client generates errors. The program keeps two counters indexed on the same key, the IP address, and prints them side by side. The column to watch is the third one: an IP with many requests and few errors is a normal user, an IP with a ratio close to 1 looks like a scan or a broken client. The final sort cannot be replaced by an awk option, since the traversal order of an array is not defined.

awk '{ip[$1]++; if ($9 >= 400) err[$1]++}
END {
  for (i in ip) printf "%-16s %6d %6d\n", i, ip[i], err[i]+0
}' access.log | sort -k2 -rn

192.168.1.10          3      1
10.0.0.5              2      1
172.16.0.3            1      1
10.0.0.99             1      1

• Two parallel arrays: ip[] for the total, err[] for the errors
• err[i]+0: a missing key holds the empty string, and +0 converts it to 0. With %6d, printf already does this conversion (measured on mawk and gawk); the +0 makes the intent explicit and becomes essential if you switch to print
• sort -k2 -rn: sort by the second column (requests), descending

Reformat with printf

print separates fields with the OFS (a space by default). printf gives precise control over alignment and format.

awk -F, 'NR > 1 {printf "%-10s %-6s %6d €\n", $1, $2, $3*$4}' sales.csv

laptop     north    6000 €
server     south    7000 €
switch     north    4500 €
laptop     east     3600 €
server     north    3500 €
switch     south    3600 €
disk       east     1600 €

Common printf formats:

Format	Effect
%-10s	String, left-aligned, width 10
%10s	String, right-aligned, width 10
%6d	Integer, width 6
%.2f	Float, 2 decimals
%s	String without fixed width

Swipe to see more

awk vs the other tools

Tool	When to use it
grep	Filter lines by text pattern, no reasoning on fields
cut	Extract fixed-position columns or a simple delimiter, no computation
sed	Transform text, substitute patterns, edit in place
awk	Filter by field, compute, count, reformat, build summary tables
Python	Complex logic, rich data structures, JSON/YAML files

Swipe to see more

The practical rule: if you need to reason about the value of a field (not just extract it), start with awk.

Frequent traps

The first line of the table is about the shell, not awk: it is by far the most frequent mistake, because the error message talks about awk syntax when the program was already mangled by bash before reaching the interpreter. The next four come from awk's own behaviour and trigger no error: the result is simply wrong or incomplete, which is far more dangerous in a production script. The last one, NR versus FNR, only shows up if you pass several files to the same command.

Trap	Symptom	Solution
Quotes in the shell	" in the awk pattern interpreted by bash	Always wrap the awk program in '...'
Mixing up print and printf	Missing newline with printf	Add \n explicitly in printf
Missing key in an array	Empty value printed by print, and the key is created along the way (it then shows up in for (k in arr))	Use arr[k]+0 for integers, arr[k]"" for strings; test with if (k in arr), which creates nothing
Order of array keys	Unsorted result	Pipe into sort after the END block
NR vs FNR with several files	NR keeps incrementing across files	Use FNR for the line number within the current file

Swipe to see more

Key points

• $1, $NF, NR: the basic variables to know by heart
• -F: or -F,: changes the field separator
• $3 >= 400: direct numeric conditions on a field
• $1 ~ /regex/ and $1 !~ /regex/: pattern filtering on a field
• array[$key]++: count occurrences per value
• BEGIN / END: before the first line / after the last
• printf "%-10s %6d\n": format output as an aligned table
• | sort: always sort after for (k in array) if the order matters

Knowledge check

Check that the essentials of this guide stuck. The questions cover only what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →