awk: print columns and filter lines

When you work with log files, CSV exports or tabular data, you often need to extract columns, filter lines or compute statistics. awk is a small text-processing language that reads a file line by line, splits each line into fields ($1, $2...), and lets you apply conditions and actions to those fields.

What you will learn

• Extract specific columns from a file
• Filter lines on conditions (value, regex)
• Perform calculations (sum, average, count)
• Use the BEGIN and END blocks to structure your processing
• Handle CSV files with a custom separator

The awk command in the Linux ecosystem

awk belongs to the text-processing commands of Linux. Each tool has its speciality, and knowing which one fits saves you from forcing awk where grep or cut would do:

Command	Speciality	When to use it
awk	Process column data	Extract, filter, compute on fields
sed	Edit files as a stream	Replace text, delete lines
grep	Search by content	Filter lines matching a pattern
cut	Extract simple columns	Basic split on a delimiter
sort / uniq	Sort and deduplicate	Ranking, counting occurrences
xargs	Run in bulk	Pass lines to a command

Swipe to see more

Frequent command combinations:

The awk command is often combined with other tools through a pipe for more complex processing:

# awk + sort: sort awk's results
awk -F';' '{ print $2, $3 }' ventes.csv | sort -k2 -n

# grep + awk: filter then extract
grep "ERROR" app.log | awk '{ print $1, $NF }'

# awk + xargs: bulk processing
awk '/pattern/ { print $1 }' fichier | xargs commande

Understand the awk command in 2 minutes

The mental model below sums up how awk reasons, before the details of syntax, variables and blocks:

🧠 Mental model: How awk works

awk = For each line → If the condition is true → Run the action

Source · SVG

Esc to close, + and − to zoom, scroll to move around. 100 % − Fit + Close

Key points

• awk reads the file line by line automatically
• Each line is split into fields: $1, $2, $3... ($0 = whole line)
• By default, the separator is a space or a tab
• Without a condition, the action applies to EVERY line
• BEGIN runs BEFORE reading the file (initialisation)
• END runs AFTER reading the whole file (summary, totals)

Golden rules

1

Always set -F for CSV files By default awk splits on spaces. For a CSV: awk -F';'...

2

Test on a few lines first Use head -5 file | awk '...' to check your command.

Essential vocabulary

$0

The whole line

$1, $2, $N

Field 1, 2, N of the line

$NF

Last field of the line

NR

Number of the current line

NF

Number of fields in the line

FS

Field Separator

📚 Going further: 12 advanced options

OFS

Output Field Separator

RS

Record Separator

ORS

Output Record Separator

FILENAME

Name of the file being processed

FNR

Line number within the current file

gsub()

Replace every occurrence

sub()

Replace the first occurrence

split()

Split a string into an array

length()

Length of a string

substr()

Extract a substring

printf

Formatted output (as in C)

arrays[]

Associative arrays (key/value)

Unlike grep (which filters lines) or sed (which edits a stream), awk is a real language that treats each line as a record with numbered fields.

Full syntax

Two calling forms coexist, and the choice depends on the length of the program. On the command line, everything fits between single quotes, mandatory to stop the shell from replacing $1 with a positional argument before awk sees it. Beyond a few lines, use an .awk file with the -f option: you get back your editor's indentation, comments and syntax highlighting.

awk [options] 'programme' fichier(s)     # program written on the command line, for simple cases
awk [options] -f script.awk fichier(s)   # awk program read from an external file

Program structure:

awk 'BEGIN { init } condition { action } END { final }' fichier

Main options:

Option	Description	Example
-F sep	Sets the field separator	awk -F';' '...' (CSV)
-v var=val	Sets a variable before execution	awk -v seuil=100 '$2 > seuil'
-f script	Reads the program from a file	awk -f analyse.awk data.txt

Swipe to see more

Common forms:

# Minimal form: action on every line
awk '{ print $1 }' fichier

# With a condition: filter lines
awk '/pattern/ { print }' fichier
awk '$3 > 100 { print $1, $3 }' fichier

# With BEGIN/END blocks: initialisation and wrap-up
awk 'BEGIN { FS=";" } { sum += $2 } END { print sum }' fichier

# From a pipe
cat fichier | awk '{ print $NF }'

# Several files
awk '{ print FILENAME, $0 }' fichier1 fichier2

The 3 parts of an awk program

The awk command always follows the same logic: an input, an optional condition, an optional action. The table shows what each part answers and its default value:

Part	Question	Examples	Required?
Input	Which file or stream to process?	fichier.txt, stdin	File or pipe
Condition (pattern)	Which lines to process?	/ERROR/, $3 > 100, NR > 1	No (default: all)
Action	What to do with those lines?	{ print $1 }, { sum += $2 }	No (default: { print })

Swipe to see more

Input: awk reads the file or stream line by line. Each line is automatically split into fields according to the separator (space by default).

Condition (optional): if you set no condition, the action applies to every line. Otherwise, only the matching lines are processed by the action.

Action (optional): between braces { }. Without an explicit action, awk runs { print } (prints the line), the implicit action.

Built-in variables: understand the splitting

awk automatically splits each line into numbered fields, without you writing a single line of code for it. Two variables of the table stand out and deserve to be memorised first: NF counts the fields of the current line, and $NF therefore points to the last field whatever the number of columns. This is what lets you process lines of variable length, a common case in system logs. Do not confuse NR (the line number) with NF (the number of fields): swapping the two is the most frequent typo among beginners.

Variable	Meaning	Example
$0	The whole line	print $0
$1, $2, $N	Field 1, 2, N	print $1, $3
$NF	Last field	print $NF
NR	Line number (global)	NR > 1 (skip header)
NF	Number of fields	NF >= 3
FS	Input separator	BEGIN { FS=";" }

Swipe to see more

Concrete example:

# File: "Alice 25 Paris"
# $1=Alice, $2=25, $3=Paris, NF=3
awk '{ print $1, "a", $2, "ans" }' fichier.txt
# Result: Alice a 25 ans

Set the field separator

By default, awk splits on spaces and tabs. A CSV file uses another separator, which you declare with -F or in a BEGIN block:

# CSV file with a ; separator
awk -F';' '{ print $1, $3 }' ventes.csv

# Equivalent with BEGIN
awk 'BEGIN { FS=";" } { print $1, $3 }' ventes.csv

Watch out, common pitfall: if $2 is empty or wrong, check that you actually set -F, the separator option!

Filters: target the lines to process

Conditions let you select the lines the action applies to. Spot the difference between the last two families of the table: /pattern/ tests the whole line, whereas $N ~ /regex/ restricts the test to one specific field. This precision avoids false positives, for instance when you look for an error code that could also appear in a URL. The condition NR > 1 tests no content at all: it only drops the header line of a CSV file, a step to take every time before any calculation.

Condition	What it tests	Example
$N == "val"	Strict equality	$1 == "admin"
$N != "val"	Different from	$3 != "OK"
$N > val	Greater than (numeric)	$2 > 100
$N ~ /regex/	Matches the regex	$1 ~ /^192/
$N !~ /regex/	Does not match	$4 !~ /error/
/pattern/	Pattern on the whole line	/ERROR/
NR > 1	From line 2 on	Skip the CSV header

Swipe to see more

Combine conditions, AND (&&) / OR (||) / NOT (!):

# AND: both conditions must be true
awk '$2 > 100 && $3 == "OK" { print }' fichier.txt

# OR: at least one condition is true
awk '$1 == "admin" || $1 == "root" { print }' fichier.txt

# NOT: invert a condition
awk '!($3 ~ /error/) { print }' fichier.txt

Actions: what to do with the selected lines?

Once the lines are filtered, you can act on them. The first two actions of the table print something immediately, the next two accumulate in memory without producing anything before the END block. This distinction explains a classic surprise: a program that only contains { sum += $2 } prints nothing at all, since nobody ever asks for the result. awk variables do not even need to be declared: they are worth 0 or the empty string on first use, which lets you write sum += $2 without prior initialisation.

Action	What it does	Example
{ print }	Prints the line (default)	{ print }
{ print $1, $3 }	Prints some fields	{ print $1, $3 }
{ sum += $2 }	Accumulates in a variable	Total calculation
{ count[$1]++ }	Associative array	Count per key
{ next }	Moves to the next line	Skip this line

Swipe to see more

The special blocks: BEGIN and END

awk offers two blocks that run before and after the file is processed. They run only once, unlike the body of the program which runs on every line. BEGIN prepares the ground, usually setting FS or printing a report title, and it runs even if the file is empty or missing. END returns what was accumulated: this is where totals, averages and the walk through associative arrays happen.

awk 'BEGIN { actions_init } condition { actions } END { actions_fin }' fichier

Block	When it runs	Use case
BEGIN { }	Before reading the file	Set FS, print a title
END { }	After reading the whole file	Print a total, an average

Swipe to see more

Full example, report with a total:

awk -F';' '
  BEGIN { print "=== Rapport ===" }
  NR > 1 { total += $3; print $1, $3 }
  END { print "Total:", total }
' ventes.csv

Perform calculations

awk excels at simple statistics on files. The four examples below share the same structure: accumulate in the body, return in END. Two precautions apply to all of them. Add NR > 1 as soon as the file has a header, otherwise a line of text enters the calculation with the value 0 and skews the average. Also protect the division: if no line is processed, n is 0, and the result depends on the implementation. gawk (the awk of RHEL) stops on a fatal error division by zero attempted, whereas mawk (the awk installed by default on Debian) prints -nan without reporting any error.

# Sum of a column
awk '{ sum += $2 } END { print "Total:", sum }' fichier.txt

# Average
awk '{ sum += $2; n++ } END { print "Moyenne:", sum/n }' fichier.txt

# Min and Max
awk 'NR == 1 || $2 > max { max = $2 } END { print "Max:", max }' fichier.txt

# Count lines matching a condition
awk '$3 == "OK" { count++ } END { print "OK:", count }' fichier.txt

Associative arrays: group and count

Associative arrays are the most powerful feature of awk for analysis: on their own they replace a sort | uniq -c and handle aggregations those tools cannot do. The syntax count[$1]++ creates the key if it does not exist and increments it otherwise, in a single expression. One point of attention on the walk: for (k in arr) returns the keys in a non-guaranteed order, which is neither the order of appearance nor alphabetical order. For a ranked result, pipe the output to sort, as the third example shows.

# Count occurrences per value
awk '{ count[$1]++ } END { for (k in count) print k, count[k] }' access.log

# Sum per category
awk -F';' '{ ventes[$4] += $2 } END { for (v in ventes) print v, ventes[v] }' data.csv

# Combine with an external sort
awk '{ count[$1]++ } END { for (k in count) print k, count[k] }' log | sort -k2 -rn

Format the output with printf

print automatically adds spaces and a newline. printf offers precise control over column width, which becomes essential as soon as you produce a readable report rather than a simple extraction. The principle is that of the C language printf: a format string containing markers, then the values in order. Remember that the width is expressed in characters and that the minus sign aligns left: %-15s reserves fifteen positions and pads with spaces on the right.

# Alignment and format
awk '{ printf "%-15s %8.2f€\n", $1, $2 }' prix.txt

# Formatted table
awk 'BEGIN { printf "%-10s %s\n", "Nom", "Valeur" }
     { printf "%-10s %d\n", $1, $2 }' fichier.txt

Format	Meaning	Example
%s	String	printf "%s", $1
%d	Integer	printf "%5d", $2 (5 characters)
%.2f	Decimal, 2 digits	printf "%.2f", $3
%-10s	10 characters, left-aligned	printf "%-10s", $1
\n	Newline	Mandatory with printf!

Swipe to see more

Watch out, pitfall: printf does not add a newline automatically. Add \n at the end of the format!

Typical errors (and fixes)

These five situations cover most awk programs that "do not work". None of them produces an error message: the command runs and returns a wrong result, which is much harder to spot than a crash. The diagnostic reflex is always the same: print NF and $0 to check the actual splitting before suspecting the logic of the program.

Error	Cause	Fix
$2 is empty or wrong	Wrong separator	Set -F';' for CSV
The shell interprets $1	Double quotes	Use single quotes: awk '...'
The header is included in calculations	No filter	Add NR > 1
printf glues everything on one line	No \n	Add \n at the end of the format
String comparison instead of number	"9" > "100" is true (string)	Force with ($2 + 0) > 100

Swipe to see more

Common processing patterns

Now that you understand awk, here are ten ready-to-use patterns, from the simplest to the most advanced. Each one gives its general formula and a concrete example.

Extract a column

Print a given field of each line. A field number that does not exist causes no error: awk returns an empty string, which produces blank lines rather than an explicit message.

awk '{ print $N }' fichier        # formula
awk '{ print $2 }' fichier.txt    # example

• $N: field number (1, 2, 3…)
• $0: the whole line
• $NF: the last field

Process a CSV

Read a file with a custom separator. This form does not handle quoted fields containing the separator ("Dupont, Jean"): for a complex CSV, use a real parser rather than awk.

awk -F'<sep>' '{ print $1, $3 }' fichier.csv   # formula
awk -F';' '{ print $1, $3 }' ventes.csv        # example

• -F: sets the field separator
• ;: CSV separator (or ,, or \t)

Filter on a condition

Only print the lines that meet a condition. The condition goes before the braces; inside them, it would become a mere expression with no effect on line selection.

awk '<condition> { action }' fichier            # formula
awk '$3 > 100 { print $1, $3 }' fichier.txt     # example

• $N == "val": strict equality
• $N > val: numeric comparison
• $N ~ /regex/: regular expression match

Skip the header

Ignore the first line, the headers of a CSV file. With several files as arguments, NR keeps counting without resetting: then use FNR > 1, which starts again at each file.

awk 'NR > 1 { action }' fichier              # formula
awk -F';' 'NR > 1 { print $1 }' data.csv     # example

• NR: current line number (1, 2, 3…)
• NR > 1: from line 2 on

Sum a column

Compute the total of a numeric column. The sum variable does not need to be declared: it is 0 on first use. Add NR > 1 if the file has a header.

awk '{ sum += $N } END { print sum }' fichier       # formula
awk '{ sum += $2 } END { print sum }' ventes.txt    # example

• sum += $N: accumulates the value in a variable
• END: block run after reading the file

Compute an average

Average of a column. The counter n is essential, because NR would also count the lines skipped by the condition. Protect the division if the file can be empty.

awk '{ sum += $N; n++ } END { print sum/n }' fichier            # formula
awk 'NR > 1 { sum += $3; n++ } END { print sum/n }' data.csv    # example

• n++: counts the number of processed lines
• sum/n: average = total ÷ count

Count per group

Group and count occurrences with an associative array. It is the equivalent of a GROUP BY in one line. The output order is not guaranteed: add | sort for a stable result.

awk '{ count[$1]++ } END { for (k in count) print k, count[k] }' fichier        # formula
awk '{ count[$1]++ } END { for (ip in count) print ip, count[ip] }' access.log  # example

• count[$1]++: associative array indexed by $1
• for (k in arr): walks the keys of the array

Sum per group

Add up a value per category. The same program can carry several aggregates in parallel (one array for the sum, one for the count), which lets you compute an average per group in the END block.

awk '{ total[$1] += $2 } END { for (k in total) print k, total[k] }' fichier             # formula
awk -F';' '{ vente[$4] += $2 } END { for (v in vente) print v, vente[v] }' data.csv      # example

• total[$1] += $2: accumulates the value of $2 per key $1

Filter with a regular expression

Select the lines that match a pattern. The pattern is written between slashes, without quotes. Writing "/ERROR/" would turn the regex into a plain string, always true, and every line would come out.

awk '/<pattern>/ { action }' fichier      # formula
awk '/ERROR|WARN/ { print }' app.log      # example

• /regex/: pattern tested on the whole line
• $N ~ /regex/: pattern tested on a specific field
• !~: the opposite, does not match

Format the output

Print with a precise alignment and format. The number of markers must match the number of values supplied: a marker without a value stops the program, gawk on not enough arguments to satisfy format string and mawk on not enough arguments passed to printf.

awk '{ printf "%-10s %5d\n", $1, $2 }' fichier        # formula
awk '{ printf "%-15s %8.2f€\n", $1, $2 }' prix.txt    # example

• %-10s: string on 10 characters, left-aligned
• %5d: integer on 5 characters
• %.2f: decimal with 2 digits after the point

Pitfalls to avoid

Six errors come back constantly with awk. For each one: the faulty command, the symptom, the cause and the correction.

Wrong field separator

The most frequent awk mistake is to forget the separator of a CSV: the command runs, but $2 returns nothing useful.

awk '{ print $2 }' fichier.csv   # WRONG: prints empty lines

Symptom: $2 is empty or does not contain what you expect. Cause: the file uses ; or ,, but awk splits on spaces by default.

awk -F';' '{ print $2 }' fichier.csv   # RIGHT: explicit separator

Confusing single and double quotes

The quotes around the program decide who reads the $: the shell or awk.

awk "{ print $1 }" fichier   # WRONG: the shell replaces $1 before awk

Symptom: empty or unexpected result. Cause: double quotes let the shell interpret the $.

awk '{ print $1 }' fichier   # RIGHT: single quotes, awk receives the program intact

Always wrap the awk program in single quotes.

Header included in calculations

A CSV header is text, and text counts as 0 in an addition: the result is silently wrong.

awk -F';' '{ sum += $2 } END { print sum }' data.csv   # WRONG: wrong result

Symptom: the total or the average is wrong. Cause: the header line (text) is added to the calculation, where it is worth 0.

awk -F';' 'NR > 1 { sum += $2 } END { print sum }' data.csv   # RIGHT: header skipped

String comparison instead of numbers

A comparison can be numeric or lexicographic, and awk chooses according to the operands, not according to your intent.

awk '$2 > "100"' fichier   # WRONG: "9" > "100" is true in string comparison

Symptom: a line whose field is 9 comes out although you were looking for values above 100. Cause: as soon as one of the two operands is a quoted constant, awk switches to lexicographic comparison, and "9" comes after "100" in character order. The same pitfall triggers if the field contains something other than a number, for example a glued unit (100Mo) or a thousands separator.

awk '$2 > 100' fichier            # RIGHT: numeric constant, numeric comparison
awk '($2 + 0) > 100' fichier      # RIGHT: the + 0 forces the numeric context

Without quotes around 100, awk applies its strnum rules: a field that looks like a number is treated as one, so the comparison is indeed numeric. The + 0 stays useful for doubtful fields, since it converts explicitly and makes the intent visible on review.

Condition without an explicit action

A condition alone is valid awk, but it relies on an implicit action that a reader does not see.

awk '$3 > 100' fichier   # Watch out: works, but hard to read

Symptom: the command prints lines with no visible action. Cause: without { }, awk applies { print } by default, an implicit behaviour that is unclear.

awk '$3 > 100 { print }' fichier   # RIGHT: explicit action

printf without a newline

printf gives full control over the output, including the newline that print would have added for you.

awk '{ printf "%s %s", $1, $2 }' fichier   # WRONG: the whole output on a single line

Symptom: the output is glued, with no line breaks. Cause: unlike print, printf does not add a line break automatically.

awk '{ printf "%s %s\n", $1, $2 }' fichier   # RIGHT: explicit \n at the end of the format

Hands-on lab

Nothing replaces practice. Create the test environment below, then go through the eight steps. At the end, clean up with rm -rf ~/awk-lab.

Prepare the ground

This script creates four test files in ~/awk-lab: a sales CSV, an Apache log, a grade sheet and an employee file.

# Create the lab
mkdir -p ~/awk-lab
cd ~/awk-lab

# CSV file with a ; separator
cat > ventes.csv << 'EOF'
produit;quantite;prix_unitaire;vendeur
laptop;5;899.99;Alice
souris;50;19.99;Bob
clavier;30;49.99;Alice
ecran;10;299.99;Charlie
laptop;3;899.99;Bob
souris;100;19.99;Alice
EOF

# Apache log file
cat > access.log << 'EOF'
192.168.1.10 - - [01/Jan/2024:10:00:00] "GET /index.html HTTP/1.1" 200 1234
192.168.1.20 - - [01/Jan/2024:10:01:15] "GET /api/users HTTP/1.1" 200 567
192.168.1.10 - - [01/Jan/2024:10:02:30] "POST /api/login HTTP/1.1" 401 89
192.168.1.30 - - [01/Jan/2024:10:03:45] "GET /index.html HTTP/1.1" 200 1234
192.168.1.20 - - [01/Jan/2024:10:04:00] "GET /api/users HTTP/1.1" 500 45
192.168.1.10 - - [01/Jan/2024:10:05:10] "GET /images/logo.png HTTP/1.1" 200 5678
EOF

# Student grades file
cat > notes.txt << 'EOF'
Alice 15 18 12
Bob 10 14 16
Charlie 18 17 19
Diana 12 11 13
Eve 20 19 18
EOF

# File with spaces (more complex)
cat > employes.txt << 'EOF'
Jean Dupont;Développeur;45000
Marie Martin;Chef de projet;55000
Pierre Durand;Administrateur;50000
Sophie Bernard;Développeur;48000
EOF

echo "Lab créé dans ~/awk-lab"
ls -la ~/awk-lab

The eight steps

These eight commands modify no file: they print their result on screen, so you can replay them as often as needed. The progression follows the order in which a real processing job is built: first split correctly, then filter, finally compute and format. Always compare the output you get with the announced result; a gap almost always points to a separator or field number problem.

1. Print a specific column. Prints only the first column (product) of the CSV file.

   awk -F';' '{ print $1 }' ~/awk-lab/ventes.csv

   Expected result: the list of products (laptop, souris, clavier, ecran…). -F';' sets the separator, $1 targets the first field.

2. Skip the header. Prints the products without the header line.

   awk -F';' 'NR > 1 { print $1 }' ~/awk-lab/ventes.csv

   NR is the line number; NR > 1 skips the first line (produit).

3. Filter on a value. Prints only the sales made by Alice.

   awk -F';' '$4 == "Alice" { print }' ~/awk-lab/ventes.csv

   $4 == "Alice" tests whether the fourth field is exactly Alice, three lines match.

4. Compute a total. Computes the total of the quantities sold.

   awk -F';' 'NR > 1 { total += $2 } END { print total }' ~/awk-lab/ventes.csv

   Expected result: 198 (5+50+30+10+3+100). total += $2 accumulates, END prints after reading.

5. Compute an average. Computes the average of the three grades of each student.

   awk '{ moy = ($2 + $3 + $4) / 3; print $1, moy }' ~/awk-lab/notes.txt

   The calculation happens directly in awk. No need for -F here: the default separator (space) fits.

6. Count per category. Counts the number of sales per seller.

   awk -F';' 'NR > 1 { count[$4]++ } END { for (v in count) print v, count[v] }' ~/awk-lab/ventes.csv

   Expected result: Alice 3, Bob 2, Charlie 1, in an order that varies from one implementation to another. count[$4]++ uses an associative array indexed by seller.

7. Filter with a regular expression. Prints the log lines containing an error (HTTP code 4xx or 5xx).

   awk '$8 ~ /^[45]/ { print }' ~/awk-lab/access.log

   Expected result: two lines, codes 401 and 500. Count the fields before writing the command: in this log format, $8 holds the HTTP code and $9 the response size. In a full Apache log, the timestamp contains a time zone ([01/Jan/2024:10:00:00 +0100]) that adds a field and shifts everything by one. The command awk '{ print NF, $0 }' settles the question in a second.

8. Format the output. Prints the employees with their salary followed by the € symbol, in aligned columns.

   awk -F';' '{ printf "%-20s %s€\n", $1, $3 }' ~/awk-lab/employes.txt

   printf gives precise formatting: %-20s reserves 20 left-aligned characters.

Progressive exercises

Eight exercises to validate your mastery, from the simplest to the most complete. All of them work on the files of the ~/awk-lab lab (or on /etc/passwd). Search on your own first, then unfold the solution to check yourself.

Foundation level

These three exercises cover the basic move: split and print. Two of them use /etc/passwd, a file present on every distribution and separated by colons, which forces you to think of the -F option from the first command. Check your result by comparing it with cut -d: -f7 /etc/passwd: both tools must produce exactly the same output.

Exercise 1, print the last column. Print the last column of each line of /etc/passwd. Hint: $NF stands for the last field.

Show the solution

awk -F':' '{ print $NF }' /etc/passwd

/etc/passwd uses : as its separator; $NF targets the last field, the shell of each user.

Exercise 2, filter users with a shell. Print the users whose shell is /bin/bash. Hint: filter on $7.

Show the solution

awk -F':' '$7 == "/bin/bash" { print $1 }' /etc/passwd

$7 is the shell field in /etc/passwd; you test strict equality, then print the name ($1).

Exercise 3, number the lines. Print each line of the notes.txt file preceded by its number. Hint: NR holds the line number.

Show the solution

awk '{ print NR, $0 }' ~/awk-lab/notes.txt

NR = current line number, $0 = whole line.

Composition level

Here you move on to accumulation: the result no longer shows line by line but only once, in the END block. Both exercises require skipping the header or mastering the formatting. If your total looks a few units too high, the header was probably counted; if it is a decimal when you expected an integer, check the target column.

Exercise 4, total of a CSV column. Compute the total of the prix_unitaire column of ventes.csv (separator ;). Hint: accumulate with += and print in END.

Show the solution

awk -F';' 'NR > 1 { total += $3 } END { print total }' ~/awk-lab/ventes.csv

NR > 1 skips the header; the END block runs after the whole file has been read.

Exercise 5, average with formatting. Compute the average of column 2 of notes.txt and print it with 2 decimals. Hint: printf "%.2f".

Show the solution

awk '{ sum += $2; n++ } END { printf "Moyenne : %.2f\n", sum/n }' ~/awk-lab/notes.txt

printf formats the output; %.2f forces two decimals.

Industrialisation level

These two exercises reproduce a real log analysis: aggregate by key and isolate errors. The first introduces the associative array, the second asks you to locate the right field before writing the regex. Neither can be solved in your head: start by printing the numbered fields to locate the information.

Exercise 6, count requests per IP. Count how many requests each IP made in access.log. Hint: associative array indexed by $1.

Show the solution

awk '{ count[$1]++ } END { for (ip in count) print ip, count[ip] }' ~/awk-lab/access.log

count[$1]++ creates one counter per IP; for (k in arr) walks the keys.

Exercise 7, filter HTTP errors. Print only the lines of access.log whose HTTP code is an error (4xx or 5xx). Hint: start by counting the fields with awk '{ print NF, $0 }' to locate the code.

Show the solution

awk '$8 ~ /^[45][0-9][0-9]$/ { print }' ~/awk-lab/access.log

~ tests a regular expression; ^[45][0-9][0-9]$ requires exactly three digits starting with 4 or 5. The code sits in $8 in this file: $9 holds the response size, and filtering it would return lines with 200.

Synthesis level

The last exercise combines everything: BEGIN for the header, the body for the data and the accumulation, END for the total. It is the exact structure of any report produced with awk, and being able to write it from memory is a good mastery criterion. Take care of the alignment: identical widths between the header and the data lines make all the readability.

Exercise 8, formatted report with totals. From ventes.csv, generate a report with a header, aligned data and a final total. Hint: BEGIN for the header, printf for alignment, END for the total.

Show the solution

awk -F';' '
  BEGIN { printf "%-15s %10s\n", "Produit", "Prix" }
  NR > 1 { printf "%-15s %10.2f\n", $1, $3; total += $3 }
  END    { printf "%-15s %10.2f\n", "TOTAL", total }
' ~/awk-lab/ventes.csv

BEGIN sets the titles, the body processes each line, END prints the summary, the typical structure of an awk report.

Troubleshooting

When an awk program does not give the expected result, the problem almost always comes from the field splitting. Here is how to diagnose it, from the actual input to the error messages.

Diagnostic methods

These four commands answer a single question, asked in order: what does awk actually see? The second one pays off the most, because an unexpected NF immediately reveals a wrong separator or irregular lines. The third one numbers each field and helps when you no longer know which column holds the information you want. The fourth limits the work to a few lines, a useful reflex on a multi-gigabyte log.

# See what awk actually receives (line number + line)
awk '{ print NR, $0 }' fichier

# Check the number of fields detected on each line
awk '{ print NF, $0 }' fichier

# Print each field separately, numbered
awk -F';' '{ for (i=1; i<=NF; i++) print i, $i }' fichier

# Test on a few lines only, without processing the whole file
head -5 fichier | awk '...'

Frequent errors

Unlike the logic errors seen above, these cases produce an explicit message and a non-zero return code. The wording of the message depends on the implementation: gawk on RHEL and AlmaLinux, mawk by default on Debian and Ubuntu. Both give the line of the faulty program, which helps above all when the program lives in an .awk file. Division by zero is particularly treacherous: it happens in the END block after a seemingly successful run, when no line fed the counter, and mawk does not even report it.

Error message (gawk / mawk)	Probable cause	Fix
awk: cmd. line:1: ... syntax error / awk: line 1: syntax error at or near ...	Badly closed braces { } or parentheses	Check that each { has its } and that single quotes wrap the whole program
unexpected newline or end of string / missing } near end of file	Program cut before its end, final brace forgotten	Close the brace before the final single quote
fatal: not enough arguments to satisfy format string / not enough arguments passed to printf	More % markers than values in printf	Supply one value per marker
fatal: division by zero attempted / no message, result -nan	The divisor is 0 (empty file, zero counter)	Protect the calculation: END { if (n > 0) print sum/n }

Swipe to see more

A program written between double quotes produces no message at all: the shell replaces $1 before awk sees it, { print $1 } becomes { print }, and the command prints the whole line instead of the requested column.

awk cheat sheet

The five tables that follow gather what is used daily, sorted by nature. They are designed for quick lookup while writing a command, not for learning: if an item surprises you, go back to the matching section above.

General syntax

The three forms below cover every use. The third is the full structure, the one any slightly ambitious processing job tends towards: initialise, process each line, conclude. Note that the blocks are all optional and can appear in any order, even if the natural order remains the most readable.

Element	Role	Example
awk [options] '{ programme }' fichier(s)	General form of the command	awk '{ print $1 }' data.txt
awk [options] -f script.awk fichier(s)	Program read from an external file	awk -f analyse.awk data.csv
awk 'BEGIN {…} condition {…} END {…}' fichier	Full structure with special blocks	awk 'BEGIN{s=0} {s+=$1} END{print s}'

Swipe to see more

Options

Three options are enough in practice. -F sets the input separator and replaces a more verbose BEGIN { FS=";" }. The -v option is the only clean way to bring a shell value into the program: it avoids opening double quotes, and therefore letting the shell interpret the $. It also serves to redefine a built-in variable, for example -v OFS=",".

Option	Role	Example
-F sep	Sets the field separator	awk -F';' '...' data.csv
-v var=val	Sets a variable before execution	awk -v seuil=100 '$2 > seuil'
-f script.awk	Reads the program from a file	awk -f analyse.awk data.txt

Swipe to see more

Built-in variables

These variables are updated automatically on each line read, you never have to feed them. The distinction to remember sets FS, the input separator, against OFS, the output separator. Redefining OFS only has an effect if you use the comma in a print: print $1, $2 honours OFS, whereas print $1 $2 glues the two fields together without inserting anything, the concatenation behaviour.

Variable	Role	Example
$0	The whole line	print $0
$1, $2, $N	Field 1, 2, N	print $1, $3
$NF	The last field	print $NF
NR	Line number (global)	NR > 1
NF	Number of fields in the line	NF >= 3
FS	Input separator	BEGIN { FS=";" }
OFS	Output separator	BEGIN { OFS="," }

Swipe to see more

Conditions

Any awk condition comes down to an expression that is true or false, which allows combining them freely with &&, || and the negation !. Watch the equality comparison: == compares, = assigns. Writing $1 = "admin" replaces the first field instead of testing it, and the program runs without the slightest error while rewriting your data.

Condition	Role	Example
$N == "val"	Strict equality	$1 == "admin"
$N ~ /regex/	Matches a regex	$1 ~ /^[A-Z]/
$N !~ /regex/	Does not match	$2 !~ /error/
/pattern/	Matches on the whole line	/ERROR/
&& ||	Logical AND / OR	$2 > 0 && $3 < 100

Swipe to see more

Blocks, functions and output

This last table gathers the most useful built-in functions. Three of them make awk self-sufficient on jobs often handed to sed: gsub() replaces every occurrence directly in a field, substr() extracts a portion of a string, and split() splits a value into an array, which allows a second level of splitting inside a field already separated.

Element	Role	Example
BEGIN { }	Block run before reading	BEGIN { print "Titre" }
END { }	Block run after reading	END { print total }
length()	Length of a string	length($1)
substr(s,i,n)	Extracts a substring	substr($1,1,3)
gsub(r,s)	Replaces every occurrence	gsub(/old/,"new")
split(s,a,fs)	Splits a string into an array	split($1,parts,"-")
tolower() / toupper()	Changes the case	tolower($1)
print	Prints (with a newline)	print $1, $2
printf	Formatted output, no automatic newline	printf "%-10s", $1
%s / %d / %.2f	String / integer / decimal	printf "%.2f", $3
arr[$1]++	Counter per key	count[$1]++
for (k in arr)	Walks the keys of an array	for (ip in count)
delete arr[k]	Deletes a key	delete data["tmp"]

Swipe to see more

Mastery checklist

You master awk when you can tick each of these points, from field extraction to a complete report with BEGIN and END:

• I can extract a column with $N
• I can set a separator with -F
• I can filter lines with a condition
• I can use NR, NF and $0
• I can compute a sum with the END block
• I can format output with printf
• I can use associative arrays
• I can filter with a regular expression (~)
• I can structure a processing job with BEGIN and END
• I can generate a complete report with a header and a total

Conclusion

The awk command is a fundamental tool for anyone working with text data. Thanks to its automatic splitting into fields, its flexible conditions and its calculation abilities, it lets you process, analyse and transform structured files in a few lines. Whether to parse logs, process CSV files or generate reports, mastering awk will save you considerable time.

Articles you may find useful

• Edit files with sed
• Search with grep
• Find files with find
• Bulk processing with xargs

Key points

• awk reads its input line by line and splits it into fields: $1, $2, $NF, $0 for the whole line.
• -F sets the separator: without it, a ; CSV gives an empty $2.
• The program goes between single quotes, otherwise the shell replaces $1 before awk.
• NR > 1 drops the CSV header before any calculation.
• You accumulate in the body and return the result in END; BEGIN prepares before reading.
• Associative arrays (count[$1]++) group and count, in a non-guaranteed output order.
• printf does not add a newline: end the format with \n.
• Error messages differ between gawk (RHEL) and mawk (Debian), which does not even report a division by zero.

Knowledge check

Check that the essentials of this guide are acquired. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

FAQ: frequently asked questions

The answers below cover the questions that come up most often about awk, from column extraction to shell variables and associative arrays.

What is the awk command and what is it for?

How do I extract a specific column with awk?

How do I filter lines containing a specific pattern?

How do I compute on numeric columns?

How do I use shell variables in awk?

How do I format the output with printf in awk?

Can awk process several files at once?

How do I use associative arrays with awk?

How do I write a complete awk script in a file?

How do I handle CSV files with awk?

How do I use regular expressions in awk?

What are the main special variables in awk?

×

See in the glossary →