Sort, count and deduplicate on Linux

You have the output of ps aux, an Apache log file, or a list of packages to analyse, and you need to know who appears most often, which lines are duplicated, how many 404 errors there are. On Linux, a pipeline of three commands is enough: sort to order, uniq to deduplicate, wc to measure.

These tools have been part of the Unix foundation since the seventies. They are present on every Linux system without installation. Their real power lies in composition: chained together, they replace whole scripts in a single line.

You have seen the basics of sort, uniq and wc in the fundamentals. This guide covers the advanced options and the real administration patterns: column sorting, stability, versions, list comparison and log analysis.

What you will learn

• Sort on a precise column, in numeric or version order
• Identify duplicates, unique lines, the most frequent ones
• Measure: lines, words, bytes, UTF-8 characters
• Compare two sorted lists with comm
• Build the sort | uniq -c | sort -rn pipeline to analyse any log

In which context?

An incident has just occurred. You have a 500,000-line access.log and you want to know which IP made the most requests in the last 10 minutes. Or you have just migrated a package and you want to compare the list of active services before and after. Or you are auditing a system and looking for the user accounts that appear in several files at once. In these situations, sort, uniq and comm do in 30 seconds what a script would do in 30 lines, with less risk of error. They are also the foundation of every log-analysis pipeline: as soon as you must answer "how many times?" or "who/what is duplicated?", this is where you start.

sort, sort precisely

sort reads its whole input before writing anything: it loads the lines, orders them, then gives them back. By default it compares lines character by character according to the locale's order, which gives surprising results on numbers. The options that follow all bear on the same choice: what to compare (-k, -t) and how to compare (-n, -V, -r).

Reminder: alphabetical vs numeric sort

The most frequent trap with sort:

printf "10\n2\n20\n1\n100\n" | sort

Output: alphabetical sort (default)

1
10
100
2
20

Without options, sort sorts as text. 10 comes before 2 because 1 < 2 in ASCII, and the comparison stops at the first character that differs.

printf "10\n2\n20\n1\n100\n" | sort -n

Output: numeric sort

1
2
10
20
100

The -n option is essential as soon as you sort numbers, whatever their number of digits.

Sort on a column

The -k option chooses the sort column. By default, the separator is a space or a tab.

# scores.txt file: "name score"
sort -k2 -n scores.txt

Output

frank 60
bob 72
diana 72
charlie 88
alice 95
eve 95

• -k2: sort on the second field
• -n: numeric sort (without this flag, 60, 72, 88 would be sorted as text)

To sort in descending order:

sort -k2 -rn scores.txt

Output

eve 95
alice 95
charlie 88
diana 72
bob 72
frank 60

Sort on a column with a custom delimiter

Use -t to define the field separator:

# Sort /etc/passwd by UID (field 3, delimiter :)
sort -t: -k3 -n /etc/passwd | head -5

Output

root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync

It is the same mechanism as cut -d:: the sort -t: -k3 -n | cut -d: -f1,3 pair is very common.

Sort and deduplicate in one pass: -u

sort -u is the equivalent of sort | uniq but more efficient: it sorts and removes duplicates in a single operation.

sort -u fruits.txt

Output

apple
banana
cherry
date

Sort version numbers: -V

Alphabetical sort and numeric sort both give wrong results for version numbers such as v1.9, v1.10. The -V (version) option handles them correctly:

printf "v1.10\nv1.9\nv1.2\nv2.0\n" | sort -V

Output

v1.2
v1.9
v1.10
v2.0

Useful to sort Git tags, package names or versioned logs.

Keep the order of ties: --stable

When two lines have the same value on the sort field, sort may reorder them arbitrarily. The --stable option guarantees that their original order is kept:

sort -k2 -n --stable scores.txt

The bob 72 and diana 72 lines keep the order of the source file, which matters when you chain successive sorts.

---

uniq, count and filter duplicates

uniq handles consecutive repetitions. It must therefore almost always be preceded by sort, which groups identical lines together.

Count occurrences: -c

uniq -c replaces each group of identical lines with a single line, preceded by its number of occurrences. The count bears on the whole line, spaces included.

sort fruits.txt | uniq -c

Output

      2 apple
      3 banana
      2 cherry
      1 date

The number on the left is the occurrence count. The variable indentation is normal: it is the width of the largest number.

Find the most frequent items

Combine uniq -c with sort -rn to get a ranking:

sort fruits.txt | uniq -c | sort -rn

Output

      3 banana
      2 cherry
      2 apple
      1 date

It is the top-N pattern, probably the most used pipeline in Linux log analysis.

Show only duplicates: -d

-d shows only one copy of each repeated line and hides everything else. Handy to spot a duplicate entry in an inventory file or a user list.

sort fruits.txt | uniq -d

Output

apple
banana
cherry

Only the lines that appear more than once.

Show only unique lines: -u

-u does the opposite of -d: it keeps only the lines with no repetition at all. Beware of the naming trap: this -u does not have the same effect as sort's -u, which keeps one occurrence of each value.

sort fruits.txt | uniq -u

Output

date

Only the lines that appear exactly once.

-d and -u are complementary

uniq -d returns the duplicates, uniq -u returns the non-duplicates. Together, they split a list into two disjoint groups.

---

wc, measure precisely

wc (word count) counts lines, words or bytes, depending on the option requested. Without options, it shows all three at once. Its strength is to sit at the end of a pipeline: whatever survived the previous filters becomes a number usable in a script or a report.

Reminder of the basic options

These three options cover almost every administration need. They work on a file passed as an argument as well as on standard input.

wc -l file.txt    # number of lines
wc -w file.txt    # number of words
wc -c file.txt    # number of bytes

Count several files at once

wc accepts several files and prints a total:

wc -l fruits.txt scores.txt access.log

Output

  8 fruits.txt
  6 scores.txt
  8 access.log
 22 total

Useful to quickly audit the size of configuration files or logs.

Bytes vs characters: -c vs -m

-c counts bytes, -m counts characters (Unicode code points). The difference shows with non-ASCII characters:

echo "héllo" | wc -c    # → 7 (é takes 2 bytes in UTF-8 + newline)
echo "héllo" | wc -m    # → 6 (5 characters + newline)

In Linux administration, -c is enough to check a file's size. Use -m only if you work with files containing accented characters or non-Latin scripts.

Count filtered items

wc -l at the end of a pipeline is a reliable way to count results:

# How many users with a UID >= 1000?
awk -F: '$3 >= 1000' /etc/passwd | wc -l

# How many processes is the user bob running?
pgrep -c -u bob

# How many .conf files in /etc?
find /etc -name "*.conf" | wc -l

For processes, pgrep -c -u bob replaces the ps aux | grep "^bob" | wc -l reflex, which counts itself: run by bob, it also counts ps, grep and wc. Measured on Debian 13 for a shell and two sleep processes, the ps | grep form answers 7, pgrep answers 3.

grep -c rather than grep | wc -l

To count the lines matching a pattern in a file, grep -c pattern file is more direct and faster than grep pattern file | wc -l. Both give the same result: prefer -c when possible.

---

comm, compare two sorted lists

comm compares two files line by line and shows three columns:

• column 1: lines present only in file 1
• column 2: lines present only in file 2
• column 3: lines common to both files

# list_a.txt: apple, banana, cherry
# list_b.txt: banana, cherry, date

comm <(sort list_a.txt) <(sort list_b.txt)

Output

apple
    banana
    cherry
  date

comm requires sorted files

comm does not sort its inputs. If the files are not sorted, the result is incorrect and comm prints warnings. Always use <(sort file) to guarantee sorting on the fly.

Select a column with -1, -2, -3

The options remove a column:

# Intersection: lines common to both (remove columns 1 and 2)
comm -12 <(sort list_a.txt) <(sort list_b.txt)

Output

banana
cherry

# In A only (remove columns 2 and 3)
comm -23 <(sort list_a.txt) <(sort list_b.txt)

Output

apple

# In B only (remove columns 1 and 3)
comm -13 <(sort list_a.txt) <(sort list_b.txt)

Output

date

Use case: packages installed before/after

Comparing two states of a system is the most rewarding use case of comm. The principle: capture a sorted list before the operation, the same one after, then ask comm what appeared and what disappeared. The output of dpkg --get-selections is already sorted by name, but the explicit sort guarantees the result whatever the source.

# Capture the state before
dpkg --get-selections | awk '{print $1}' | sort > before.txt

# ... after an operation ...

# Capture the state after
dpkg --get-selections | awk '{print $1}' | sort > after.txt

# Packages added
comm -13 before.txt after.txt

# Packages removed
comm -23 before.txt after.txt

---

Administration pipelines

The examples that follow all rest on the same mechanism: awk extracts a column, sort groups identical values, uniq -c counts them, sort -rn ranks them from the most to the least frequent. Only the field number passed to awk changes from one case to the next, because it depends on the format of the log analysed. Check this number on a real line before running the full analysis.

Analyse logs: top IPs

The sort | uniq -c | sort -rn pattern applies to any column:

# Top IPs in an access log
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10

Output (lab)

      3 192.168.1.10
      3 192.168.1.5
      2 192.168.1.20

Analyse HTTP codes

The same pipeline applied to the return code field gives the distribution of the server's responses. An unusual proportion of 404 or 403 stands out at once.

awk '{print $4}' access.log | sort | uniq -c | sort -rn

Output (lab)

      4 200
      2 404
      1 403
      1 201

Find the most called endpoints

Ranking the URLs called shows where to put the optimisation or caching effort: the first lines usually concentrate most of the traffic.

awk '{print $3}' access.log | sort | uniq -c | sort -rn

Output (lab)

      4 /api/users
      2 /api/products
      2 /api/orders

The 5 users with the highest UID

Here, no counting: sort -t: -k3 -n orders /etc/passwd by increasing UID, tail -5 keeps the end of the ranking and cut keeps only the name and the identifier. Useful to spot the last accounts created on a machine.

sort -t: -k3 -n /etc/passwd | tail -5 | cut -d: -f1,3

Count the active SSH connections per IP

ss -tn state established '( sport = :22 )' lists the established TCP connections on the server's port 22; without this filter, every established connection would be counted, SSH or not. Beware of the field number: as soon as you filter with state, ss removes the State column, and the remote address ends up in fourth position, not fifth. NR>1 skips the header line, cut -d: -f1 isolates the IPv4 address from the port, and the ranking brings out an IP that would open an abnormal number of sessions.

ss -tn state established '( sport = :22 )' | awk 'NR>1 {print $4}' | cut -d: -f1 | sort | uniq -c | sort -rn

---

Troubleshooting

The errors met with these three commands almost always come from the same misunderstanding: they compare text, not values. A badly sorted number, an undetected duplicate or an inconsistent comm are fixed by stating the nature of the data (-n, -V) or by guaranteeing the sort upstream.

Problem	Likely cause	Solution
sort: numbers badly ordered	Alphabetical sort by default	Add -n (numeric) or -V (version)
uniq does not remove every duplicate	Non-consecutive lines	Go through sort before uniq
comm: "not in sorted order" warnings	Unsorted files	Use comm <(sort f1) <(sort f2)
wc -l: result one lower	Last line without \n	Normal, wc -l counts newlines
sort -k2: wrong sort on a column	Alphabetical sort on the column	Add -n if the column contains numbers
sort -u does not remove everything	Different case (Apple vs apple)	Add -f (ignore case): sort -uf

Swipe to see more

---

Key points

• sort -n is essential for numbers. Without it, 10 comes before 2.
• sort -k2 -n sorts numerically on the second column. -t: sets the delimiter.
• sort -V sorts version numbers correctly (v1.9 before v1.10).
• sort -u = sort | uniq in a single pass.
• uniq -c counts occurrences. uniq -d returns the duplicates. uniq -u returns the unique lines.
• sort | uniq -c | sort -rn | head -N is the universal pipeline to find the N most frequent items.
• comm requires sorted files: always use <(sort ...).
• wc -c counts bytes, wc -m counts Unicode characters.
• To count a user's processes, pgrep -c -u user: a ps | grep | wc -l counts itself.

Knowledge check

Check that the essentials of this guide stuck. The questions cover only what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →