Use uniq command to delete duplicate lines

by Daniel Pham
Published: Updated:
This entry is part 5 of 10 in the series Linux commands: Part 6 - Redirection

In this article, we will talk about using the uniq command to delete duplicate lines. This is one of the commands that is also used quite a lot while working on Linux systems.

Uniq command and sort command

In the article about sort command, we know that the sort command will help you output an sorted list. This uniq command is often used in conjunction with the sort command.

In sorted list data, duplicate data may exist. This uniq command will work to delete duplicate lines, leaving only one line.

Use the uniq command to delete duplicate lines

The uniq command can use sorted list data from the standard input or from a single file argument.

In the example below, we create a text file with unsorted data. Next, we will use both sort and uniq commands to see how the results are.

$ cat demo-uniq.txt | sort | uniq
use uniq command to delete duplicate lines
Use uniq command to delete duplicate lines.

Look at the image above, we have 6 data lines. After using the sort command, we can have 2 duplicate lines 1 - line number 1.

Recommended Reading: What is pipeline and how to use it

Then, we use the uniq command in the pipeline, the uniq command is right next the sort command. You can see in the result after using the uniq command, there is only one line 1 - line number 1.

If you don’t want to delete the duplicate line from the standard output, you just want to display the duplicate line that tells you. Use the -d option as the example below.

$ cat demo-uniq-txt | sort | uniq -d
use uniq command to view duplicate lines or count number duplicate lines
Use uniq command to view duplicate lines or count number duplicate lines.

If you want to count the number of duplicate lines, use the -c option (count).

$ cat demo-uniq.txt | sort | uniq -c


Through this article, you have learned another command to process the pipeline data. It is very useful in filtering data.

(This is an article from my old blog that has been inactive for a long time, I don’t want to throw it away so I will keep it and hope it helps someone).

0 0 votes
Article Rating
Series Navigation«« Previous part: Use sort command to sort pipeline dataNext part: Use wc command to count number lines »»

You may also like

Notify of
Inline Feedbacks
View all comments

DevOps Lite is a personal blog specializing in technology with main topics about DevOps, DevSecOps, SRE and System Administrator. Articles are shared for free and contributed to the community.



Subscribe my Newsletter for new blog posts. Stay updated from your inbox!

© 2021-2024 – All rights reserved.

Please write sources “” when using articles from this website.

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More

Would love your thoughts, please comment.x

Adblock Detected

Please support us by disabling your AdBlocker extension from your browsers for our website.