In this article, we will talk about using the uniq command to delete duplicate lines. This is one of the commands that is also used quite a lot while working on Linux systems.
Uniq command and sort command
Table of Contents
In the article about sort command, we know that the sort command will help you output an sorted list. This uniq command is often used in conjunction with the sort command.
In sorted list data, duplicate data may exist. This uniq command will work to delete duplicate lines, leaving only one line.
Use the uniq command to delete duplicate lines
The uniq command can use sorted list data from the standard input or from a single file argument.
In the example below, we create a text file with unsorted data. Next, we will use both sort and uniq commands to see how the results are.
$ cat demo-uniq.txt | sort | uniq
Look at the image above, we have 6 data lines. After using the sort command, we can have 2 duplicate lines 1 - line number 1.
Recommended Reading: What is pipeline and how to use it
Then, we use the uniq command in the pipeline, the uniq command is right next the sort command. You can see in the result after using the uniq command, there is only one line 1 - line number 1.
If you don’t want to delete the duplicate line from the standard output, you just want to display the duplicate line that tells you. Use the -d option as the example below.
$ cat demo-uniq-txt | sort | uniq -d
If you want to count the number of duplicate lines, use the -c option (count).
$ cat demo-uniq.txt | sort | uniq -cConclusion
Through this article, you have learned another command to process the pipeline data. It is very useful in filtering data.
(This is an article from my old blog that has been inactive for a long time, I don’t want to throw it away so I will keep it and hope it helps someone).
