GNU Parallel: run jobs in parallel

When you have to process hundreds of files, convert images in bulk or run scripts on several servers, launching the tasks one by one is too slow. GNU Parallel runs several commands simultaneously, using every core of your CPU. Where xargs works sequentially by default, parallel spreads the load and drastically speeds up your batch jobs.

What you will learn

• Run commands in parallel with :::
• Control the number of jobs with -j
• Use the placeholders ({}, {.}, {/})
• Handle files with spaces and special characters
• Monitor progress and handle errors
• Combine parallel with find and other commands

The parallel command in the Linux ecosystem

GNU Parallel is the logical evolution of xargs for bulk processing. Here is how it fits among the parallelism tools of a Linux system:

Command	Speciality	When to use it
parallel	Parallel execution	Bulk processing, maximum performance
xargs	Sequential execution	Simple commands, one by one
make -j	Parallel build	Compiling projects
& / wait	Background jobs	A few independent tasks

Swipe to see more

Advantages of parallel over xargs:

• Parallelism by default, one job per argument, where xargs -P needs -n or -I to form several batches
• Progress bar (--bar, --eta)
• Advanced error handling (--halt)
• Rich placeholders ({.}, {/}, {#}...)
• Execution on remote servers (-S)

Understand GNU Parallel in 2 minutes

parallel reads a list of arguments, then launches one command instance per argument, several at a time. It constantly keeps the requested number of jobs running: as soon as one job ends, the next one starts, without waiting for the whole wave to finish. This queue mechanism explains both the time saving and the surprises in the order of the outputs.

🧠 Mental model: How parallel works

parallel = For each argument → Run N commands simultaneously → Collect the results

Source · SVG

Esc to close, + and − to zoom, scroll to move around. 100 % − Fit + Close

Key points

• parallel reads the arguments and runs several instances of the command simultaneously
• By default, parallel runs as many jobs as there are logical processors (nproc)
• {} stands for the current argument, {.} without extension, {/} base name
• ::: passes inline arguments, :::: reads them from a file
• The order of outputs is NOT guaranteed (parallel execution)

Golden rules

1

Always test with --dry-run first Checks the commands that will run without launching them.

2

Limit the jobs with -j to avoid overload Too many parallel jobs can saturate CPU, memory or disk.

Essential vocabulary

:::

Passes arguments directly on the command line

::::

Reads the arguments from a file

{}

Placeholder for the current argument

-j N

Number of jobs in parallel (default: nproc)

--dry-run

Prints the commands without running them

--bar

Shows a progress bar

📚 Going further: 12 advanced options

{.}

Argument without extension

{/}

Base name (without path)

{//}

Directory path

{/.}

Base name without extension

{#}

Job number (sequence)

{%}

Slot number (worker)

--colsep

Column separator for several fields

--halt

Behaviour on error (now,fail=1...)

--results

Saves stdout/stderr in a directory

--eta

Shows the estimated remaining time

--null

NUL-separated arguments (for find -print0)

-S

Run on remote servers (SSH)

Unlike xargs, which passes the arguments to a single instance of the command, parallel launches N simultaneous instances (N = number of logical processors by default, the value of nproc).

Full syntax

Options go before the command to run, otherwise parallel passes them to that command. Three forms exist depending on the origin of the arguments, and they only differ by their separator: ::: inline, :::: from a file, or nothing at all when the arguments arrive through a pipe.

parallel [options] commande ::: arguments           # Inline arguments
parallel [options] commande :::: fichier.txt        # Arguments from a file
cat liste.txt | parallel [options] commande {}      # Arguments from stdin

Main options:

Option	Description	Example
-j N	N jobs in parallel (default: nproc)	parallel -j 4 ...
--dry-run	Prints without running	parallel --dry-run rm ...
--bar	Progress bar	parallel --bar cmd ...
--halt	Behaviour on error	--halt now,fail=1
-k	Keeps the order of outputs	parallel -k echo ...

Swipe to see more

The 3 parts of a parallel command

Every parallel line breaks down the same way, which helps locate where an error comes from: a badly split argument comes from the source, a command not found comes from the second block, a saturated machine comes from the third.

Part	Question	Examples	Required?
Source	Where do the arguments come from?	:::, ::::, stdin	Yes
Command	Which command to run?	gzip {}, convert	Yes
Control	How much parallelism?	-j 4, --bar	No (defaults)

Swipe to see more

Placeholders: the heart of parallel

A placeholder says where the argument must be inserted in the command. If you set none, parallel automatically appends the argument at the end of the command, which explains why parallel gzip ::: *.log works without {}. The variants below split the argument into path, name and extension, without calling basename or dirname.

Placeholder	Meaning	Example (/path/photo.jpg)
{}	Full argument	/path/photo.jpg
{.}	Without extension	/path/photo
{/}	Base name (without path)	photo.jpg
{//}	Directory path	/path
{/.}	Base name without extension	photo
{#}	Job number (1, 2, 3...)	1
{%}	Slot number (worker)	2

Swipe to see more

Conversion example:

# Convert photo.jpg → photo.png
parallel convert {} {.}.png ::: *.jpg

# Result for photo.jpg:
# convert photo.jpg photo.png

Argument sources

The source determines the argument separator: the newline for a file or a pipe, the shell's splitting for the inline form. A name containing spaces therefore goes through a pipe intact; it is a name containing a newline that breaks the splitting.

Source	Syntax	Example
Inline	::: arg1 arg2	parallel echo ::: a b c
File	:::: fichier.txt	parallel wget :::: urls.txt
Stdin	cmd | parallel	find . | parallel gzip
Multi-column	--colsep	parallel --colsep ',' cp {1} {2}

Swipe to see more

Control the parallelism

Without -j, parallel runs as many simultaneous jobs as the machine has logical processors (threads), that is the value of nproc: measured on a processor with 10 cores and 16 threads, 32 one-second jobs finish in 2.1 s, so 16 at a time. This default suits a compute-heavy job, but it is too high for tasks bound by the disk or the network, and far too low for network waits where several dozen jobs fit without effort.

# Default: as many jobs as logical processors (nproc)
parallel gzip ::: *.log

# Limit to 4 simultaneous jobs
parallel -j 4 gzip ::: *.log

# A single job (sequential, for debugging)
parallel -j 1 gzip ::: *.log

# As many jobs as arguments (careful!)
parallel -j 0 gzip ::: *.log

Watch out: too many parallel jobs can saturate the system (CPU, memory, disk I/O). Start with -j 4 and adjust.

Handling files with spaces

Unlike xargs, parallel splits its input on newlines and protects each argument before passing it to the shell: a name containing spaces arrives intact. The real trap is a name that itself contains a newline, which ls splits into two arguments.

# WRONG: a name containing a newline becomes two arguments
ls *.jpg | parallel echo {}

# RIGHT: null separator with find -print0 + --null
find . -name "*.jpg" -print0 | parallel --null echo {}

The find -print0 | parallel --null combination is the safe pattern for every file processing job.

Typical errors (and fixes)

These five situations cover almost every support call about parallel. Two of them are silent: badly split files and out-of-order outputs produce no error message, only a wrong result.

Error	Cause	Fix
Badly split files	Newline in a name	find -print0 | parallel --null
Mixed outputs	Parallel execution	Add -k or --line-buffer
Saturated system	Too many jobs	Limit with -j 4
Files deleted by mistake	No test	ALWAYS --dry-run first
SSH asks for a password	Key not configured	ssh-copy-id user@serveur

Swipe to see more

Installation

GNU Parallel is not always installed by default, and on the RHEL family it comes from EPEL, not from the base repositories:

# Debian/Ubuntu
sudo apt install parallel

# Fedora
sudo dnf install parallel

# RHEL/AlmaLinux/Rocky: the package comes from EPEL, missing from the base repositories
sudo dnf install epel-release
sudo dnf install parallel

# macOS (Homebrew)
brew install parallel

# Check
parallel --version

Essential patterns

Now that you understand the logic of parallel, here are ten ready-to-use patterns. Each one gives its general formula and a concrete example.

Basic parallel execution

Run the same command on several arguments, one job per argument.

parallel commande ::: arg1 arg2 arg3   # formula
parallel gzip ::: *.log                # example

• :::: separates the command from the list of arguments

From standard input

Read the arguments from a pipe, one per line.

cat liste.txt | parallel commande {}                 # formula
find . -name "*.jpg" | parallel convert {} {.}.png   # example

• {}: placeholder of the current argument

Limit the parallelism

Control the number of simultaneous jobs with -j, the main throttle.

parallel -j N commande ::: arguments   # formula
parallel -j 4 gzip ::: *.log           # example

• -j N: N jobs at most (default: number of logical processors, nproc)
• -j 0: as many jobs as possible

Simulation mode

Check the commands before running them, with --dry-run, the simulation mode.

parallel --dry-run commande ::: arguments   # formula
parallel --dry-run rm ::: *.bak             # example

• --dry-run: prints the commands without launching anything, essential before rm

File transformation

Convert files by changing their extension, with the {.} placeholder.

parallel commande {} {.}.new_ext ::: fichiers   # formula
parallel ffmpeg -i {} {.}.mp3 ::: *.wav         # example

• {.}: argument without extension, {/}: base name, {//}: directory

Files with spaces (safe)

Safely process every name, including those containing a newline, with the null separator.

find . -print0 | parallel --null commande {}                       # formula
find . -name "*.jpg" -print0 | parallel --null convert {} {.}.png  # example

• -print0 + --null: null separator, the safe pair for files

Progress monitoring

Show the progress of the tasks with --bar.

parallel --bar commande ::: arguments      # formula
parallel --bar --eta gzip ::: *.log        # example

• --bar: progress bar, --eta: estimated remaining time

Error handling

Control the behaviour when a task fails, with --halt.

parallel --halt now,fail=1 commande ::: arguments      # formula
parallel --halt now,fail=1 ./script.sh ::: *.txt       # example

• now,fail=1: stops at the first error
• soon,fail=20%: stops beyond 20% of failures

Multi-column arguments

Use several fields per line of a file, with --colsep.

parallel --colsep ',' commande {1} {2} :::: fichier.csv    # formula
parallel --colsep ',' cp {1} {2} :::: batch.csv            # example

• --colsep: column separator, {1}, {2}: the columns
• ::::: reads the arguments from a file

Execution on remote servers

Spread the tasks over several machines through SSH.

parallel -S serveur1,serveur2 commande ::: arguments    # formula
parallel -S srv1,srv2,srv3 'echo {}; hostname' ::: {1..10}   # example

• -S: list of SSH servers (passwordless SSH key required)

Pitfalls to avoid

Eight errors come back constantly with parallel. Some are destructive, others simply saturate the machine.

No test before rm/mv

It is the only pitfall of this list that is irreversible: rm goes through no trash, and the glob is expanded by the shell before parallel sees anything.

parallel rm ::: *.bak   # WRONG: deletes immediately, no way back

Symptom: files deleted by mistake. Cause: no check of the commands before running them.

parallel --dry-run rm ::: *.bak   # RIGHT: preview, then rerun without --dry-run

System overload

Beyond the number of cores, extra jobs do not compute faster: they fight over the CPU and memory, and the total time can become longer than that of a sequential run.

parallel -j 100 gzip ::: *.log   # WRONG: too many jobs: saturated system

Symptom: slow system, saturated memory, heavy swap. Cause: too many parallel processes for the available resources.

parallel -j 4 gzip ::: *.log   # RIGHT: limit to a reasonable number

Files with spaces

The trap is silent, and rarer than you might think: parallel splits on newlines, so mon fichier.jpg goes through intact. Only a name that itself contains a newline is split into two non-existent arguments, without any error.

ls *.jpg | parallel echo {}   # WRONG: breaks on a name containing a newline

Symptom: one file is processed as two arguments. Cause: the newline is the default separator, and nothing forbids putting one in a file name.

find . -name "*.jpg" -print0 | parallel --null echo {}   # RIGHT: null separator

Output order not guaranteed

The display order follows the order in which jobs finish, not their launch order: a short job started third shows up before a long job started first.

parallel echo ::: 1 2 3 4 5   # WRONG: output in a random order

Symptom: out-of-order results. Cause: parallel execution guarantees no order.

parallel -k echo ::: 1 2 3 4 5   # RIGHT: -k keeps the order of outputs

Interleaved outputs

By default, parallel applies --group: the output of each job is kept whole and shown only when the job ends. Mixing only appears with -u (--ungroup), which writes as it goes and can cut a line in two.

parallel -u commande ::: args   # WRONG: a line can be cut in two

Symptom: interleaved, unreadable output lines. Cause: -u bypasses the internal buffering of parallel.

parallel --line-buffer commande ::: args   # RIGHT: each line stays whole

SSH without a configured key

parallel has no terminal to offer remote jobs: the password prompt never shows up, the job simply stays blocked until the SSH timeout.

parallel -S serveur commande ::: args   # WRONG: blocks on the password prompt

Symptom: frozen run, timeout. Cause: SSH key authentication not configured.

ssh-copy-id user@serveur   # RIGHT: set up the key, then rerun parallel

Badly handled quotes

Between double quotes, it is the shell that interprets the string before passing it on: variables are substituted and $(...) run only once, before parallel has replaced {}. Single quotes pass the text as is to parallel, which evaluates it for each job.

parallel echo "Taille de {} : $(wc -c < {})" ::: *.txt   # WRONG: $(...) run by the shell, on a file named {}

Symptom: {}: No such file or directory, then an empty size for each file. Cause: the shell runs $(wc -c < {}) before parallel, on the literal text {}.

parallel 'echo "Taille de {} : $(wc -c < {})"' ::: *.txt   # RIGHT: single quotes, evaluated per job

Write conflicts in a file

An >> append is only atomic for very small writes: beyond the size of a system buffer, two jobs writing at the same time produce truncated and interleaved lines, without any error message.

parallel "cat {} >> output.txt" ::: *.log   # WRONG: corrupted output file

Symptom: corrupted output, mixed lines. Cause: several processes write to the same file at the same time.

parallel "cat {} > {}.out" ::: *.log && cat *.out > output.txt   # RIGHT: one file per job, then merge

Hands-on lab

Nothing replaces practice. Create the test environment below, then go through the eight steps. At the end, clean up with rm -rf ~/parallel-lab.

Prepare the ground

This script creates a tree in ~/parallel-lab: images (some of them with spaces in the name), logs, a list of URLs and a CSV.

# Create the lab
mkdir -p ~/parallel-lab/{images,logs,data}
cd ~/parallel-lab

# Simulated image files
for i in {1..10}; do
  echo "image $i data" > images/photo_$i.jpg
done

# Files with spaces in the name
echo "data 1" > "images/mon fichier 1.jpg"
echo "data 2" > "images/photo vacances 2024.jpg"

# Log files
for i in {1..5}; do
  echo "Log content $i" > logs/app_$i.log
done

# List of URLs
cat > urls.txt << 'EOF'
https://example.com/file1.txt
https://example.com/file2.txt
https://example.com/file3.txt
EOF

# CSV file for the multi-column tests
cat > data/batch.csv << 'EOF'
fichier1,output1
fichier2,output2
fichier3,output3
EOF

echo "Correct : Lab créé dans ~/parallel-lab"
ls -la ~/parallel-lab/

The eight steps

Each step isolates one option at a time and runs on the files created above. None of them is destructive: the only deletions go through --dry-run.

1. First parallel command. Run echo on several arguments with :::.

   parallel echo ::: un deux trois quatre

   ::: passes the arguments to parallel; each echo runs in parallel (the order varies).

2. Use the {} placeholder. Print "Traitement de …" for each image.

   ls ~/parallel-lab/images/*.jpg | parallel echo "Traitement de {}"

   {} is replaced with each line of standard input.

3. Test without running. Check the deletions that would be made, without deleting anything.

   ls ~/parallel-lab/logs/*.log | parallel --dry-run rm {}

   --dry-run prints the commands without launching them, essential before an rm.

4. Limit the parallelism. Launch 6 one-second sleep, 2 at a time.

   parallel -j 2 'sleep 1; echo {}' ::: {1..6}

   -j 2 limits to 2 simultaneous jobs, the whole run takes about 3 seconds (3.07 s measured). The quotes matter: parallel -j 2 sleep 1 ::: {1..6} runs sleep 1 1, sleep 1 2… because parallel appends the argument at the end, and sleep adds up its arguments: 15 seconds instead of 3.

5. Transform files. Simulate the conversion of the .jpg files into .png.

   ls ~/parallel-lab/images/*.jpg | parallel --dry-run echo convert {} {.}.png

   {.} removes the extension, ideal for format conversions.

6. Handle file names safely. Process every image, whatever its name.

   find ~/parallel-lab/images -name "*.jpg" -print0 | parallel --null echo "Fichier : {}"

   -print0 and --null preserve every name, including those containing a newline.

7. Show the progress. Launch tasks with a progress bar.

   parallel --bar 'sleep 0.5; echo {}' ::: {1..10} > /dev/null

   --bar shows the progress; --eta adds the estimated remaining time.

8. Stop on error. Make parallel stop at the first error.

   parallel --halt now,fail=1 'test {} = 3 && exit 1; echo {}' ::: {1..5}

   --halt now,fail=1 stops everything as soon as a job returns an error code.

Progressive exercises

Eight exercises to validate your mastery. They work on the ~/parallel-lab lab. Search on your own first, then unfold the solution to check yourself.

Foundation level

These three exercises cover the three basic moves: :::, compression in parallel and the simulation with --dry-run.

Exercise 1, first steps with :::. Print the numbers 1 to 5 in parallel. Hint: ::: passes the arguments to parallel.

Show the solution

parallel echo ::: 1 2 3 4 5

::: separates the command from the arguments; each number is passed to echo.

Exercise 2, parallel compression. Compress every .log file of the lab with gzip. Hint: ::: with the *.log glob.

Show the solution

parallel gzip ::: ~/parallel-lab/logs/*.log

Each .log file is compressed simultaneously.

Exercise 3, simulation mode. Check the commands that would delete the .bak files, without deleting anything. Hint: --dry-run.

Show the solution

parallel --dry-run rm ::: *.bak

--dry-run is essential before any destructive command.

Composition level

These two exercises bring in the job limit set by -j and the placeholder {.}.

Exercise 4, limit the parallelism. Launch 6 one-second sleep, but only 2 at a time. Hint: -j limits the number of jobs.

Show the solution

time parallel -j 2 'sleep 1; echo {}' ::: {1..6}

-j 2 limits to 2 simultaneous jobs: 6 jobs = 3 waves of 2, so ≈ 3 seconds. Without the quotes, each job would run sleep 1 N and last 1 + N seconds.

Exercise 5, file conversion. Simulate (with echo) the conversion of the .jpg images into .png. Hint: {.} gives the name without extension.

Show the solution

ls ~/parallel-lab/images/*.jpg | parallel --dry-run convert {} {.}.png

{.} removes the extension, perfect for format conversions.

Industrialisation level

These two exercises cover safe file names and CSV input split into columns.

Exercise 6, safe file names. List every .jpg of the lab safely, whatever the name. Hint: find -print0 + parallel --null.

Show the solution

find ~/parallel-lab/images -name "*.jpg" -print0 | parallel --null echo "Fichier : {}"

-print0 and --null handle every character, including the newline.

Exercise 7, multi-column CSV. From batch.csv, simulate source → destination copies. Hint: --colsep and {1}, {2}.

Show the solution

parallel --colsep ',' --dry-run cp {1} {2} :::: ~/parallel-lab/data/batch.csv

--colsep sets the separator; {1} and {2} reference the columns.

Safety level

The last exercise covers error handling, so that a failed job stops the whole batch.

Exercise 8, error handling. Launch commands that stop at the first error. Hint: --halt now,fail=1.

Show the solution

parallel --halt now,fail=1 'if [ {} = 3 ]; then exit 1; fi; echo {}' ::: {1..5}

--halt now,fail=1 stops everything as soon as a job returns an error code.

Troubleshooting

When parallel misbehaves, isolate the cause with these four checks first, then read the error messages listed below to diagnose the problem.

Diagnostic methods

The four commands below each answer a different question: what parallel will run, what it actually runs, what the command receives as an argument, and whether the problem comes from the parallelism itself. Always start with -j 1 when a job behaves randomly: if the error disappears, it is a concurrency issue, not a command bug.

# See the commands without running them
parallel --dry-run commande ::: arguments

# Print each command before running it
parallel --verbose commande ::: arguments

# Replace the command with echo to observe
parallel echo "Je ferais : cmd {}" ::: args

# Run sequentially to isolate a problem
parallel -j 1 commande ::: arguments

Frequent errors

These messages come from parallel itself or from the command it launches, and the first one is the most confusing: parallel waits for an input that never comes, so it stays waiting without running anything.

Error message	Probable cause	Fix
Warning: Input is read from the terminal	No input (no ::: and no pipe)	Add ::: with arguments, or a pipe
command not found	Command missing from the PATH	Check with which, or use an absolute path
Permission denied	Script not executable, insufficient rights	chmod +x on the script, check the rights
ssh: Connection timed out	Server unreachable or firewall	Test ssh serveur echo OK, check the network

Swipe to see more

parallel cheat sheet

The three tables that follow condense what you will reopen most often: the call forms, the options and the placeholders. The rest of the options of parallel is documented in man parallel, which is large but organised by theme.

General syntax

The choice of form only depends on where the arguments come from; the behaviour of parallel is the same afterwards, whatever the source.

Form	Role	Example
parallel commande ::: arg1 arg2	Arguments on the command line	parallel gzip ::: *.log
cat liste | parallel commande {}	Arguments from stdin	find . | parallel echo {}
parallel commande :::: fichier.txt	Arguments from a file	parallel wget :::: urls.txt
parallel --colsep "," cmd {1} {2} :::: data.csv	Multi-column arguments	parallel --colsep "," cp {1} {2} :::: batch.csv

Swipe to see more

Main options

All of them go before the command. --dry-run and -j are the two to remember first: the first avoids damage, the second avoids blocking the machine.

Option	Role	Example
-j N	N jobs in parallel (default: nproc)	parallel -j 4 gzip ::: *.log
--dry-run	Prints without running	parallel --dry-run rm ::: *.bak
--verbose	Prints each command	parallel --verbose cmd ::: args
-k (--keep-order)	Keeps the order of outputs	parallel -k echo ::: 1 2 3
--bar	Progress bar	parallel --bar cmd ::: {1..100}
--eta	Estimated remaining time	parallel --eta cmd ::: args
--halt now,fail=1	Stops at the first error	parallel --halt now,fail=1 cmd ::: args
--null	Inputs separated by a null character	find -print0 | parallel --null cmd

Swipe to see more

Placeholders

They combine freely in the same command, and can appear several times. Useful reminder: {1} and {2} only work with --colsep or several argument sources.

Placeholder	Role	Example
{}	Full argument	echo {}
{.}	Without extension	convert {} {.}.png
{/}	Base name (without path)	echo {/}
{//}	Directory path	mkdir -p {//}/out
{/.}	Base name without extension	echo {/.}
{#}	Job number	echo "Job {#} : {}"
{1}, {2}	Columns (with --colsep)	cp {1} {2}

Swipe to see more

Mastery checklist

You master parallel when you can tick each of these points, from ::: to --halt and --bar:

• I can use ::: to pass arguments
• I can test with --dry-run before running
• I can limit the jobs with -j
• I can use {} and {.} as placeholders
• I can handle any file name thanks to --null
• I can process a multi-column CSV file with --colsep
• I can handle errors with --halt
• I can show the progress with --bar and --eta

Advanced use cases

These four scenarios illustrate the most useful sizing rule in practice: the right number of jobs depends on the saturated resource, not on the number of files. A video encode saturates the CPU and handles two jobs; a download waits for the network and handles many more.

Bulk file conversion

Multimedia jobs consume several cores per job: beyond two simultaneous jobs, the total time stops going down.

# Convert every image in parallel
find . -name "*.jpg" -print0 | parallel --null --bar convert {} {.}.png

# Compress videos (limit to 2 jobs because it is heavy)
parallel -j 2 --eta ffmpeg -i {} -c:v libx264 {.}.mp4 ::: *.avi

Parallel downloads

--limit-rate applies per job, not globally: with -j 4, the total throughput reaches four times the given value.

# Download a list of URLs
parallel --bar wget :::: urls.txt

# With a bandwidth limit per job
parallel -j 4 wget --limit-rate=1M {} :::: urls.txt

Execution on remote servers

With -S, parallel spreads the jobs over the listed machines and applies the -j limit to each server, not to the whole set.

# Run the same command once on each server
parallel --nonall -S srv1,srv2,srv3 hostname

# Copy a file to every server (one scp job per server)
parallel scp config.yml {}:/tmp/ ::: srv1 srv2 srv3

--nonall runs the command on each server without an argument; without it, parallel would wait for a list on its standard input, which triggers the classic Input is read from the terminal warning.

Prerequisite: passwordless SSH configured (ssh-copy-id).

Processing CSV files

--colsep expects a regular expression, not a mere character: a separator such as | must therefore be escaped. This form does not handle quoted fields that themselves contain the separator.

# batch.csv file: source,destination
# fichier1.txt,/backup/fichier1.txt
# fichier2.txt,/backup/fichier2.txt

parallel --colsep ',' cp {1} {2} :::: batch.csv

parallel vs xargs: which one to choose?

The decisive criterion is not speed but availability: xargs belongs to findutils and is on every machine, parallel must be installed, from EPEL on RHEL and its derivatives. On a server you do not control, or in a portable script, xargs -P remains the reasonable choice despite its limits.

Criterion	parallel	xargs
Native parallelism	Yes, optimised	Watch out: limited (-P)
Progress bar	Yes (--bar, --eta)	No
Error handling	Yes, --halt	No, basic
Placeholders	Yes, {}, {.}, {/}, {#}	Watch out: {} only
Remote servers	Yes, -S	No
Availability	Watch out: to install	Yes, preinstalled
Complexity	Watch out: more options	Yes, simpler

Swipe to see more

In short:

• xargs for simple cases where the command is preinstalled
• parallel as soon as you need performance, progress or robustness

Key points

1. ::: passes inline arguments, :::: reads from a file
2. {} = full argument, {.} = without extension, {/} = base name
3. -j N limits the parallelism, do not saturate the system
4. --dry-run is MANDATORY before any destructive command (rm, mv)
5. find -print0 | parallel --null = safe pattern for files
6. --bar and --eta to monitor progress
7. --halt now,fail=1 to stop at the first error
8. -k keeps the order of outputs when it matters

Knowledge check

Check that the essentials of this guide are acquired. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →