我有一个从 Zotero 导出的 CSV 文件与我的图书馆条目的元数据。我知道它包含很多重复项,但要摆脱它们并不容易:
| Year | Author | Title |
+------+-------------------------------+--------------+
| 2016 | Jones, Erik | Book Reviews |
| 2016 | Hassner, Pierre; Jones, Erik | Book Reviews |
| 2010 | Adams, Laura L.; Gagnon, Chip | Book Reviews |
| Author | Title |
+---------------+-----------------------------------------------+
| Tichý, Lukáš; | Can Iran Reduce EU Dependence on Russian Gas? |
| Tichy, L.; | "can iran reduce eu dependence onrussian gas" |
这是一个极端的例子(通常差异并不大),但是正如您所看到的,预清洁并不能完全解决这个问题;所以这个想法是消除在两列以上具有相似值的行 - 例如“作者”和“标题”。
到目前为止我尝试/浏览过的内容:
首先在“作者”列的 for 循环中使用 agrep 来获取重复行的索引;然后对“标题”列执行相同的操作;然后比较向量并对值重合的行进行重复数据删除。不用说,我无法超越第 1 步:
titles <- unlist(corpus$"Title")
for (i in 1:length(titles)){
Title_dupe_temp <- agrep(titles[i], titles[i+1:length(titles)],
max.distance = 1, ignore.case = TRUE, fixed = FALSE)
Title_dupes[i] <- paste(i, Title_dupe_temp, sep = " ")
}
结果是(几乎)完全是胡言乱语;加上我收到警告消息:
In Title_dupes[i] <- paste(i, Title_dupe_temp, sep = " ") :
number of items to replace is not a multiple of replacement length
我也通读了 fuzzywuzzyR文档,但没有找到任何可以提供帮助的功能。
最后,我试过了 RecordLinkage包裹。尽管如此,我还是无法超越基础知识。 The documentation对所有事情都相当沉重且不明确;指南很少,我发现的指南(例如 this )使用了准备好身份向量的示例数据集 - 所以我无法弄清楚如何在我的数据上复制它。
所以在这一点上我不在乎是否在 OpenRefine/R/Py/SQL/whatever 中做,只是以任何方式做。
最佳答案
解决方案一:
使用循环和库 stringdist
:
library(stringdist)
zotero<-data.frame(
Year=c(2016,2016,2010,2010,2010,2010),
Author=c("Jones, Erik","Hassner, Pierre;","Adams, Laura L.;","Tichý, Lukáš;","Tichý, Lukáš;","Tichy, L.;"),
Title=c("Book Reviews","Book Reviews","Book Reviews","Can Iran Reduce EU Dependence on Russian Gas?","Can Iran Reduce EU Dependence on Russian Gas?","can iran reduce eu dependence onrussian gas")
)
zotero$onestring<-paste0(zotero$Year,zotero$Author,zotero$Title)
zotero<-zotero[order(zotero[,1],zotero[,2]),]
atot<-NULL
for (i in 2:dim(zotero)[1]){
a<-stringdist(zotero$onestring[i-1],zotero$onestring[i])/(nchar(zotero$onestring[i-1])+nchar(zotero$onestring[i]))
atot<-rbind(atot,a)
}
zotero<-cbind(zotero,threshold=c(1,atot))
zotero[zotero$threshold>0.15,]
解决方案二:用矩阵计算它可能比用循环更快:首先我根据你的数据样本创建一个数据帧,其次我删除非 UTF 字符,第三我使用库
stringdist
计算距离矩阵。您可以轻松地将这些转换为相似度的百分比。zotero<-data.frame(
Year=c(2016,2016,2010,2010,2010,2010),
Author=c("Jones, Erik","Hassner, Pierre;","Adams, Laura L.;","Tichý, Lukáš;","Tichý, Lukáš;","Tichy, L.;"),
Title=c("Book Reviews","Book Reviews","Book Reviews","Can Iran Reduce EU Dependence on Russian Gas?","Can Iran Reduce EU Dependence on Russian Gas?","can iran reduce eu dependence onrussian gas")
)
zotero$onestring<-paste0(zotero$Year,zotero$Author,zotero$Title)
Encoding(zotero$onestring) <- "UTF-8"
zotero$onestring<-iconv(zotero$onestring, "UTF-8", "UTF-8",sub='')
library(stringdist)
stringdistmatrix(zotero$onestring)
结果:
> stringdistmatrix(zotero$onestring)
1 2 3 4 5
2 11
3 13 14
4 47 45 44
5 47 45 44 0
6 47 45 42 13 13
关于r - 基于 2 列中的值对表进行重复数据删除 + 模糊匹配,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/54895604/