linux uniq命令_如何在Linux上使用uniq命令

linux uniq命令

A shell prompt on a Linux computer. — Fatmawati Achmad Zaenuri/ShutterstockFatmawati Achmad Zaenuri / Shutterstock

The Linux uniq command whips through your text files looking for unique or duplicate lines. In this guide, we cover its versatility and features, as well as how you can make the most of this nifty utility.

Linux uniq命令在您的文本文件中快速查找唯一或重復的行。在本指南中，我們介紹了它的多功能性和功能，以及如何充分利用這個漂亮的實用程序。

在Linux上查找文本的匹配行 (Finding Matching Lines of Text on Linux)

The uniq command is fast, flexible, and great at what it does. However, like many Linux commands, it has a few quirks—which is fine, as long as you know about them. If you take the plunge without a bit of insider know-how, you could well be left scratching your head at the results. We’ll point out these quirks as we go.

uniq命令快速，靈活，并且功能強大。但是，就像許多Linux命令一樣，它也有一些怪癖-只要您了解它們就可以了。如果您在沒有任何內幕專業知識的情況下進行嘗試，那么您很可能會為結果感到撓頭。我們將指出這些怪癖。

The uniq command is perfect for those in the single-minded, designed-to-do-one-thing-and-do-it-well camp. That’s why it’s also particularly well-suited to work with pipes and play its part in command pipelines. One of its most frequent collaborators is sort?because uniq?has to have sorted input on which to work.

uniq命令非常適合那些專一，一勞永逸的人。這就是為什么它也特別適合與管道一起工作并在命令管道中發揮作用的原因。它的一個的最頻繁的合作者是sort因為uniq必須有排序的文件在其上工作。

Let’s fire it up!

讓我們點火！

不帶選項運行uniq (Running uniq with No Options)

We’ve got a text file that contains the lyrics to Robert Johnson’s song I Believe I’ll Dust My Broom. Let’s see what uniq makes of it.

我們有一個文本文件，其中包含羅伯特·約翰遜(Robert Johnson)的歌曲《我相信我會除塵我的掃帚》的歌詞。讓我們看看uniq是如何構成的。

We’ll type the following to pipe the output into less:

我們將鍵入以下內容以將輸出傳遞到less ：

uniq dust-my-broom.txt | less

We get the entire song, including duplicate lines, in?less:

我們得到的整首歌曲，包括重復的線條， less ：

The output from the "uniq dust-my-broom.txt | less" command in less in a terminal window.

That doesn’t seem to be either the unique lines nor the duplicate lines.

似乎既不是唯一的行也不是重復的行。

Right—because this is the first quirk. If you run uniq with no options, it behaves as though you used the -u (unique lines) option. This tells uniq to print only the unique lines from the file. The reason you see duplicate lines is because, for uniq?to consider a line a duplicate, it must be adjacent to its duplicate, which is where sort comes in.

正確-因為這是第一個怪癖。如果運行不帶任何選項的uniq ，它的行為就好像您使用了-u (唯一行)選項一樣。這告訴uniq僅打印文件中的唯一行。之所以會看到重復的行，是因為，為了使uniq認為行是重復的，它必須與它的重復相鄰，這就是sort來源。

When we sort the file, it groups the duplicate lines, and uniq?treats them as duplicates. We’ll use sort?on the file, pipe the sorted output into uniq, and then pipe the final output into less.

當我們對文件進行排序時，它會將重復的行進行分組，并且uniq將它們視為重復的行。我們將對文件使用sort ，將排序后的輸出uniq給uniq ，然后將最終輸出uniq給less 。

To do so, we type the following:

為此，我們鍵入以下內容：

sort dust-my-broom.txt | uniq | less

A sorted list of lines appears in less.

行的排序列表出現在less 。

Output from sort dust-my-broom.txt | uniq | less in less in a terminal window

The line, “I believe I’ll dust my broom,” definitely appears in the song more than once. In fact, it’s repeated twice within the first four lines of the song.

這首歌“我相信我會把掃帚除塵”肯定在歌曲中多次出現。實際上，它在歌曲的前四行中重復了兩次。

So, why is it showing up in a list of unique lines? Because the first time a line appears in the file, it’s unique; only the subsequent entries are duplicates. You can think of it as listing the first occurrence of each unique line.

那么，為什么它顯示在唯一行列表中？因為第一次在文件中出現一行，所以它是唯一的。僅后續條目重復。您可以將其視為列出每個唯一行的第一個匹配項。

Let’s use sort again and redirect the output into a new file. This way, we don’t have to use sort in every command.

讓我們再次使用sort并將輸出重定向到一個新文件。這樣，我們不必在每個命令中都使用sort 。

We type the following command:

我們輸入以下命令：

sort dust-my-broom.txt > sorted.txt

Now, we have a presorted file to work with.

現在，我們有了一個預分類的文件可以使用。

計數重復 (Counting Duplicates)

You can use the -c (count) option to print the number of times each line appears in a file.

您可以使用-c (計數)選項來打印每行出現在文件中的次數。

Type the following command:

鍵入以下命令：

uniq -c sorted.txt | less

Each line begins with the number of times that line appears in the file. However, you’ll notice the first line is blank. This tells you there are five blank lines in the file.

每行以該行出現在文件中的次數開頭。但是，您會注意到第一行是空白。這告訴您文件中有五個空白行。

Output from the "uniq -c sorted.txt | less" command in less in a terminal window.

If you want the output sorted in numerical order, you can feed the output from uniq into sort. In our example, we’ll use the -r (reverse) and?-n (numeric sort) options, and pipe the results into less.

如果要按數字順序對輸出進行排序，可以將uniq的輸出輸入sort 。在我們的示例中，我們將使用-r (反向)和-n (數字排序)選項，并將結果傳遞給less 。

We type the following:

我們輸入以下內容：

uniq -c sorted.txt | sort -rn | less

The list is sorted in descending order based on the frequency of each line’s appearance.

該列表根據每行出現的頻率以降序排列。

Output from uniq -c sorted.txt | sort -rn | less in less in a terminal window

只列出重復的行 (Listing Only Duplicate Lines)

If you want to see only the lines that are repeated in a file, you can use the -d (repeated) option. No matter how many times a line is duplicated in a file, it’s listed only once.

如果只想查看文件中重復的行，則可以使用-d (重復)選項。無論文件中一行被重復多少次，它只會被列出一次。

To use this option, we type the following:

要使用此選項，我們輸入以下內容：

uniq -d sorted.txt

The duplicated lines are listed for us. You’ll notice the blank line at the top, which means the file contains duplicate blank lines—it isn’t a space left by uniq to cosmetically offset the listing.

為我們列出了重復的行。您會注意到頂部的空白行，這意味著該文件包含重復的空白行-這不是uniq用來修飾列表的空白。

Output from the "uniq -d sorted.txt" command in a terminal window.

We can also combine the -d (repeated) and -c (count) options and pipe the output through sort. This gives us a sorted list of the lines that appear at least twice.

我們還可以結合使用-d (重復)和-c (計數)選項，并通過sort傳遞輸出。這給了我們至少出現兩次的行的排序列表。

Type the following to use this option:

輸入以下內容以使用此選項：

uniq -d -c sorted.txt | sort -rn

The "uniq -d -c sorted.txt | sort -rn" command in a terminal window.

列出所有重復的行 (Listing All Duplicated Lines)

If you want to see a list of every duplicated line, as well as an entry for each time a line appears in the file, you can use the -D (all duplicate lines) option.

如果要查看每個重復行的列表，以及每次在文件中出現一行的條目，則可以使用-D (所有重復行)選項。

To use this option, you type the following:

要使用此選項，請鍵入以下內容：

uniq -D sorted.txt | less

The listing contains an entry for each duplicated line.

該清單包含每個重復行的條目。

Output from uniq -D sorted.txt | less in less in a terminal window

If you use the --group?option, it prints every duplicated line with a blank line either before (prepend) or after each group (append), or both before and after (both) each group.

如果使用--group選項，它將在每個組之前( prepend )或之后( append )，或在每個組之前和之后( both )都在每行重復的行上打印空白行。

We’re using append?as our modifier, so we type the following:

我們使用append作為修飾符，因此我們輸入以下內容：

uniq --group=append sorted.txt | less

The groups are separated by blank lines to make them easier to read.

這些組用空行分隔，以使它們更易于閱讀。

Output from the "uniq --group=append sorted.txt | less" command in less in a terminal window.

檢查一定數量的字符 (Checking a Certain Number of Characters)

By default, uniq checks the entire length of each line. If you want to restrict the checks to a certain number of characters, however, you can use the -w (check chars) option.

默認情況下， uniq檢查每行的整個長度。但是，如果要將檢查限制為一定數量的字符，則可以使用-w (檢查字符)選項。

In this example, we’ll repeat the last command, but limit the comparisons to the first three characters. To do so, we type the following command:

在此示例中，我們將重復最后一個命令，但將比較限制為前三個字符。為此，我們鍵入以下命令：

uniq -w 3 --group=append sorted.txt | less

The results and groupings we receive are quite different.

我們收到的結果和分組是完全不同的。

Output from the "uniq -w 3 --group=append sorted.txt | less" command in a terminal window.

All lines that start with “I b” are grouped together because those portions of the lines are identical, so they’re considered to be duplicates.

所有以“ I b”開頭的行都被分組在一起，因為這些行的那些部分是相同的，因此它們被認為是重復的。

Likewise, all lines that start with “I’m” are treated as duplicates, even if the rest of the text is different.

同樣，以“ I'm”開頭的所有行都被視為重復行，即使文本的其余部分不同。

忽略一定數量的字符 (Ignoring a Certain Number of Characters)

There are some cases in which it might be beneficial to skip a certain number of characters at the beginning of each line, such as when lines in a file are numbered. Or, say you need uniq to jump over a timestamp and start checking the lines from character six instead of from the first character.

在某些情況下，在每一行的開頭略過一定數量的字符可能是有益的，例如對文件中的行進行編號時。或者，說您需要uniq跳過時間戳并開始檢查從字符6而不是從第一個字符開始的行。

Below is a version of our sorted file with numbered lines.

下面是帶有編號行的已排序文件的版本。

A numbered and sorted file of duplicate lines in less in a terminal window.

If we want?uniq to start its comparison checks at character three, we can use the -s (skip chars) option by typing the following:

如果我們希望uniq在第三個字符處開始比較檢查，則可以通過鍵入以下命令使用-s (跳過字符)選項：

uniq -s 3 -d -c numbered.txt

The "uniq -s 3 -d -c numbered.txt" command in a terminal window.

The lines are detected as duplicates and counted correctly. Notice the line numbers displayed are those of the first occurrence of each duplicate.

將這些行檢測為重復行并正確計數。注意，顯示的行號是每個重復項第一次出現的行號。

You can also skip fields (a run of characters and some white space) instead of characters.?We’ll use the -f (fields) option to tell uniq which fields to ignore.

您也可以跳過字段(一系列字符和一些空白)來代替字符。我們將使用-f (字段)選項來告訴uniq要忽略哪些字段。

We type the following to tell uniq to ignore the first field:

我們輸入以下內容告訴uniq忽略第一個字段：

uniq -f 1 -d -c? numbered.txt

The "uniq -f 1 -d -c? numbered.txt" command in a terminal window.

We get the same results we did when we told?uniq to skip three characters at the start of each line.

當我們告訴uniq在每一行的開頭跳過三個字符時，我們得到的結果相同。

忽略大小寫 (Ignoring Case)

By default,?uniq is case-sensitive. If the same letter appears capped and in lowercase, uniq?considers the lines to be different.

默認情況下， uniq區分大小寫。如果相同的字母顯示為大寫且小寫，則uniq認為行是不同的。

For example, check out the output from the following command:

例如，檢查以下命令的輸出：

uniq -d -c sorted.txt | sort -rn

The "uniq -d -c sorted.txt | sort -rn" command and output in a terminal window.

The lines “I Believe I’ll dust my broom” and?“I believe I’ll dust my broom” aren’t treated as duplicates because of the difference in case on the “B” in “believe.”

由于“相信”中“ B”的大小寫不同，因此“我相信我會掃帚”和“我相信我會掃帚”這行不會被重復。

If we include the -i (ignore case) option, though, these lines will be treated as duplicates. We type the following:

但是，如果我們包括-i (忽略大小寫)選項，則這些行將被視為重復行。我們輸入以下內容：

uniq -d -c -i sorted.txt | sort -rn

The "uniq -d -c -i sorted.txt | sort -rn" command in a terminal window.

The lines are now treated as duplicates and grouped together.

現在，這些行被視為重復行并分組在一起。

Linux puts a multitude of special utilities at your disposal. Like many of them, uniq isn’t a tool you’ll use every day.

Linux提供了許多特殊實用程序供您使用。像其中許多工具一樣， uniq并不是您每天都會使用的工具。

That’s why a big part of becoming proficient in Linux is remembering which tool will solve your current problem, and where you can find it again. If you practice, though, you’ll be well on your way.

這就是為什么精通Linux的很大一部分原因是要記住哪種工具可以解決您當前的問題，以及在哪里可以找到它。但是，如果您練習的話，將會很順利。

Or, you can always just search?How-To Geek—we probably have an article on it.

或者，您始終可以只搜索How-To Geek-我們可能在上面有一篇文章。