bash - 如何从文本文件中删除部分重复项？

Question

如何使用 awk、grep 或 sort 删除 bash 中的部分重复项？

输入:

"3","6"
"3","7"
"4","9"
"5","6"
"26","48"
"543","7"

预期输出:

"3","6"
"3","7"
"4","9"
"26","48"

Answer 1

您可以尝试关注并告诉我这是否对您有帮助。

awk -F'[",]' '!a[$5]++'   Input_file

输出如下。

"3","6"
"3","7"
"4","9"
"26","48"

编辑:也在此处添加解释。

awk -F'[",]' '   ##Setting field separator as " or , for every line of Input_file.
!a[$5]++         ##creating an array named a whose index is $5(fifth field) and checking condition if 5th field is NOT present in array a, so when any 5th field comes in array a then increasing its count so next time it will not take any duplicates in it. Since awk works on condition and then action, since here no action is mentioned so by default print of current line will happen.
' Input_file     ##Mentioning the Input_file here too.