Transform text with cut, tr and paste

A log file gives you too much information at once. A CSV has its columns in the wrong order. Two lists must be put side by side before being processed. In these situations, graphical tools are no help: cut, tr and paste do the job in a single command line.

These three tools belong to the family of Unix filters. They read text, transform it and write it, with no intermediate file, no interface, fully composable through pipes. cut extracts, tr replaces, paste assembles. Together, they prepare data for other tools (sort, awk, sed, Bash scripts) with an efficiency that is hard to beat.

You have seen cut and tr in the fundamentals. This guide goes further: advanced options, real administration use cases, frequent mistakes and practical combinations.

What you will learn

• Extract multiple fields, ranges and complements with cut
• Convert character sets, delete and squeeze with tr
• Merge several files or streams into columns with paste
• Combine these three tools in common administration pipelines

In which context?

You rarely handle text for fun. You do it because a tool gives you more than you need, and you must extract the essentials before passing them to another tool. In practice: you get a CSV export of an inventory with 20 columns and only need 3; an application log mixes upper and lower case and your comparison script is case-sensitive; two reference files must be merged side by side before being loaded into a report. These three situations come up every week in administration or CI/CD work, and cut, tr, paste settle them in one line, without a temporary file or a Python script.

cut, extract precisely

cut splits each line and keeps only the requested portion. It works in two exclusive modes: by field (-f), with a single-character delimiter set by -d, or by character position (-c). The default delimiter is the tab, which explains most of the empty outputs seen on a file separated by colons or commas.

Reminder of the basic syntax

These two forms cover most uses, one by field, the other by position.

cut -d: -f1 /etc/passwd          # field 1, delimiter :
cut -c1-10 file.txt              # characters 1 to 10

If these two lines are not familiar, start with the Filter and transform text guide in the fundamentals.

Extract several fields and ranges

cut accepts a list of comma-separated fields, or a range with a dash:

# Fields 1, 3 and 5 of the passwd file
cut -d: -f1,3,5 /etc/passwd

# Fields 1 to 3 (range)
cut -d: -f1-3 /etc/passwd

# From field 4 to the end of the line
cut -d: -f4- /etc/passwd

# From the first character to the 5th
cut -c-5 file.txt

From field N to the end

The -f4- notation (no number after the dash) means "from field 4 to the end of the line". It is handy when you do not know the exact number of columns.

Invert the selection with --complement

The --complement option selects everything except the fields given. It is only available in GNU cut (Linux):

# Everything except field 2 (the password) in /etc/passwd
cut -d: --complement -f2 /etc/passwd

It is useful to quickly anonymise a sensitive column in a file without rewriting the others.

Change the output separator

By default, cut reuses the same delimiter on output. GNU cut picks another one with --output-delimiter, which directly gives CSV output from a colon-delimited file:

cut -d: -f1,3,6 --output-delimiter=',' /etc/passwd

On a cut that is not GNU (macOS, BSD), the option does not exist: you then convert the separator afterwards with tr, cut -d: -f1,3,6 /etc/passwd | tr ':' ',', which produces the same output.

Output (excerpt)

root,0,/root
bob,1000,/home/bob

The limits of cut

cut works on fixed positions: it is not suited when columns have variable widths (multiple spaces). In that case, awk is more powerful.

# Wrong result if several spaces separate the columns
ps aux | cut -d' ' -f3    # unpredictable result with multiple spaces

# Better with awk
ps aux | awk '{print $3}'  # always correct

---

tr, convert and clean

tr (translate) reads from standard input and transforms or deletes characters. It never works with files directly, always through a pipe or a redirection.

Convert character sets

The basic syntax: tr 'set1' 'set2'

# Upper case → lower case
echo "CRITICAL ERROR" | tr 'A-Z' 'a-z'

# Lower case → upper case
cat /etc/hostname | tr 'a-z' 'A-Z'

# Replace spaces with underscores (file names)
echo "my log file.txt" | tr ' ' '_'

Delete characters with -d

With -d, tr only expects a single set: any character that belongs to it disappears from the stream, the others go through unchanged.

# Delete all the digits of a line
echo "error-404-page-not-found" | tr -d '0-9'

# Delete the Windows carriage returns (\r) of a file copied from Windows
tr -d '\r' < windows-file.txt > unix-file.txt

# Delete spaces
echo "  root  :  0  " | tr -d ' '

Windows files on Linux

Text files created on Windows use \r\n as the line ending. On Linux, this causes strange behaviour in scripts. The command tr -d '\r' < file.txt > clean.txt is the quick fix before dos2unix.

Squeeze repetitions with -s

The -s (squeeze) option reduces consecutive repetitions of a character to a single occurrence:

# Squeeze multiple spaces into one
echo "root    :    0    :    /root" | tr -s ' '

Output

root : 0 : /root

# Squeeze the empty lines of a file (several consecutive \n)
tr -s '\n' < file-with-empty-lines.txt

Handy to normalise the output of commands such as df, ps or netstat, which use variable spacing.

Use the POSIX character classes

tr accepts POSIX classes between [: and :]:

tr '[:upper:]' '[:lower:]'    # same as A-Z → a-z, without accented letters
tr -d '[:space:]'             # delete all whitespace (space, tab, newline...)
tr -d '[:punct:]'             # delete all punctuation
tr -d '[:digit:]'             # delete the digits

Take care to use single quotes around strings containing !, # or $ to avoid interpretation by the shell:

echo 'abc123!@#' | tr -d '[:digit:]'   # → abc!@#
echo 'abc123!@#' | tr -d '[:punct:]'   # → abc123

tr and accented letters

GNU tr works byte by byte: an accented letter, which takes two bytes in UTF-8, escapes it, POSIX classes included. echo 'ÉCOLE' | tr '[:upper:]' '[:lower:]' returns École. To change the case of accented text, go through GNU sed, which understands UTF-8: sed 's/.*/\L&/' turns it to lower case, \U to upper case. Classes remain preferable to ranges for readability, [:digit:] stating its intent better than 0-9.

Administration examples

Three recurring cases illustrate the real uses of tr: normalising a separator, extracting a URL segment in an Apache log, and putting a vertical list back on a single line to pass it to a command. Note the < export.csv redirection in the first example: tr never takes a file name as an argument.

# Normalise a CSV file exported from Excel (separator ; → ,)
tr ';' ',' < export.csv > export-normalised.csv

# Extract the first path segment of the URLs in an Apache log
# (combined format: ... "GET /blog/page HTTP/1.1" ...)
grep 'GET' /var/log/apache2/access.log | cut -d'"' -f2 | cut -d' ' -f2 | cut -d'/' -f2 | sort -u

# Prepare a list of packages (one per line → a single line for apt install)
cat packages.txt | tr '\n' ' '

---

paste, merge files into columns

paste is less known than cut and tr, but very useful to assemble several files or streams side by side.

Merge two files

Imagine two files:

names.txt

alice
bob
charlie

roles.txt

admin
user
user

paste names.txt roles.txt

Output

alice  admin
bob  user
charlie  user

By default, paste separates the columns with a tab.

Change the delimiter with -d

The option replaces the tab with the character of your choice, which is enough to produce CSV directly.

paste -d',' names.txt roles.txt

Output

alice,admin
bob,user
charlie,user

You can chain several different delimiters to assemble three files or more:

paste -d':,' names.txt roles.txt uid.txt

paste alternates between : and , at each field.

Turn lines into columns with -s

The -s (serial) option merges all the lines of a file into a single line:

paste -s names.txt

Output

alice  bob  charlie

It is the equivalent of tr '\n' '\t', with control over the delimiter and one difference: paste ends the line with a newline, where tr leaves a trailing separator.

# Generate an inline list for a command
paste -sd',' names.txt

Output

alice,bob,charlie

Number each line

Combine paste with the nl command (or cat -n) to number lines:

nl -ba names.txt | paste -d' ' - roles.txt

Or more simply:

paste <(nl -ba names.txt) roles.txt

Process substitution

<(command) is a Bash process substitution. Bash runs the command, plugs its output into a file descriptor and replaces the expression with the matching path, such as /dev/fd/63. paste therefore receives a perfectly valid file name, which it requires, without any temporary file being written to disk.

Assemble the output of two commands

By combining paste with two process substitutions, you put the result of two commands side by side without writing any intermediate file.

# Compare the local users with their UIDs
paste <(cut -d: -f1 /etc/passwd) <(cut -d: -f3 /etc/passwd)

Output (excerpt)

root  0
daemon  1
bob  1000

---

Common administration pipelines

The five chains below match real operational needs. They all follow the same progression: a command produces too wide an output, tr normalises it, cut extracts the useful column, and a last filter sorts or deduplicates. Always test a pipeline step by step, removing the last link, before freezing it in a script.

Extract the IPs of the active SSH connections

This pipeline assumes an ss output whose columns are separated by a variable number of spaces, hence the mandatory tr -s.

ss -Htn state established '( sport = :22 )' | tr -s ' ' | cut -d' ' -f4 | cut -d: -f1 | sort -u

1. ss -Htn state established '( sport = :22 )': established TCP connections on local port 22, the SSH server's, without a header line (-H), otherwise the word Address ends up in the results. Without the port filter, the pipeline would list every established connection. Beware, ss shortens the state to ESTAB: a grep ESTABLISHED would never return anything. The native state established filter is safer than a grep.
2. tr -s ' ': normalises multiple spaces
3. cut -d' ' -f4: extracts the remote address column. The state filter removes the state column from the output, so the remote address becomes the fourth field, not the fifth.
4. cut -d: -f1: removes the port. Valid for IPv4 addresses: an IPv6 address itself contains :
5. sort -u: deduplicates

Create a CSV file from /etc/passwd

The fields kept are the login name, the UID and the home directory; tr then converts the historical : separator into a comma.

cut -d: -f1,3,6 /etc/passwd | tr ':' ',' > users.csv

Compare two lists and find the differences

comm requires two sorted inputs, hence the sort in the substitutions. The -13 options hide columns 1 and 3, leaving only the lines present only in the second file, in other words the packages added.

# Two sorted lists of installed packages (before/after)
comm -13 <(sort before.txt) <(sort after.txt)

Clean a script generated on Windows

A \r at the end of the shebang causes the bad interpreter error, a fault nothing makes visible on screen. The redirection writes a new file: the original script stays intact, and the execution permission has to be set again on the copy.

tr -d '\r' < script.sh > script-clean.sh
chmod +x script-clean.sh

Generate an inline list of services to start

paste -sd' ' puts the service names on a single line, which xargs then passes as arguments to systemctl. Check the content with an echo before really running the command: xargs waits for no confirmation.

cat services.txt | paste -sd' ' | xargs systemctl start

---

Troubleshooting

Most surprises with these three tools come from characters that are invisible on screen: a tab taken for a space, a \r at the end of a line, multiple spaces between columns. The first reflex is therefore cat -A file, which makes these characters visible. The lines below cover the seven most frequent symptoms.

Problem	Likely cause	Solution
cut returns empty fields	Wrong delimiter	Check with cat -A file, tabs show as ^I
cut --complement not recognised	Non-GNU system (macOS, BSD)	Use awk instead
tr: tr: extra operand	Too many arguments	tr takes only two sets, check the syntax
tr: accented letters not transformed	GNU tr works byte by byte, UTF-8 escapes it	Use GNU sed: sed 's/.*/\L&/' (lower case) or \U (upper case)
paste: shifted columns	Files with a different number of lines	Check with wc -l, paste fills with empty fields if they differ
paste: unwanted tabs	Default delimiter	Add -d',' or -d' ' to choose
Nothing shows with tr	Forgotten pipe or redirection	tr reads stdin, use echo "..." | tr ... or tr ... < file

Swipe to see more

---

Put it into practice

Turning a raw file into exact facts is the daily routine of administration scripting. This lab makes you cut columns, deduplicate lines, count occurrences, sum a field and rewrite a delimiter with cut, sort, uniq, sed and awk. Your output is compared with the expected result, line for line.

Guided lab Practice

Learn how to do it

Transform and aggregate text with cut, sort, uniq, sed and awk

$ dsoxlab run l1-text-processing

Environment: Shell·Duration: ~30 min

How it is checked: the output you produce is compared with the expected result.

First time? What DSOXLab is and how to install it

DSOXLab is the command line that prepares a disposable environment (an isolated shell or a VM), runs the lab scenario, then checks the real state of the system and computes a score. Nothing is installed or broken on your workstation.

It needs Python 3.11 or later, uv or pipx, and git. The installation guide details these prerequisites; on Windows, or on a Mac without Python, use the appliance instead.

Install the CLI and add the catalogue, once:

uv tool install dsoxlab   # or: pipx install dsoxlab
dsoxlab catalog add https://github.com/stephrobert/linux-dsoxlab-training

Before you run the lab, check:

• DSOXLab installed
• catalogue available
• compatible environment (isolated shell or VM)

To check, nothing is probed from this page: run dsoxlab doctor. Read the diagnosis

Run this lab, then validate your work:

dsoxlab run l1-text-processing
dsoxlab check l1-text-processing

See every lab: dsoxlab list-labs. Track your progress: dsoxlab progress. The full dsoxlab guide covers scoring and the pitfalls.

Key points

• cut extracts columns by position or by delimiter. -f4- selects from field 4 to the end. --complement inverts the selection and --output-delimiter changes the output separator (GNU only).
• tr transforms byte by byte from stdin. -d deletes, -s squeezes repetitions. POSIX classes ([:upper:], [:digit:]) are more readable than ranges, but none converts accented letters: for those, GNU sed (\L, \U).
• paste assembles files into columns. -d controls the delimiter, -s merges all the lines of a file onto a single line.
• These three tools are designed for the pipe: they never modify the source files.
• When columns have variable widths (multiple spaces), prefer awk to cut.

Knowledge check

Check that the essentials of this guide stuck. The questions cover only what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →