r - 从杂乱的 .csv 文件创建数据框列表(或其他结构)

Question

;博士——我相信这实际上是一个简单的问题，只需要详细的解释来建立上下文。传递一次文件并构建一个 data.frames 列表

我有一个凌乱的 .csv 文件，如下所示。它包含许多“垃圾”行 -- 包含很少用/感兴趣的数据的行或带有嵌入空格、制表符等的行。有值(value)的行包括:

(a) 细节
(b) 子细节
(c) DETAIL 和 Sub-Detail 行之后的“类数据框”对象。

但是，(a)、(b) 和 (c) 之间的垃圾行数量可能会有所不同，例如示例 (testing.csv)。我想返回的是 data.frame 对象的列表，例如 results 或非常相似的东西(例如，我考虑了 Detail 的结果 和 SubDetail 被捕获为 data.frame 中的附加列):

df1 <- data.frame(Item = 1:3, Val1 = c(50, 20, 30), Val2 = c(100, 30, 50))
df2 <- data.frame(Item = 1:2, Val1 = c(20, 30), Val2 = c(30, 50))
df3 <- data.frame(Item = 1:2, Val1 = c(10, 30), Val2 = c(20, 40))
df4 <- data.frame(Item = 1:3, Val1 = c(50, 30, 70), Val2 = c(30, 40, 80))

# One possible desired result structure
results <- list(list(Detail = "01", SubDetail = "ABC", data = df1),
                list(Detail = "01", SubDetail = "XYZ", data = df2),
                list(Detail = "02", SubDetail = "ABC", data = df3),
                list(Detail = "02", SubDetail = "XYZ", data = df4))

str(results)

示例 .csv 文件 (testing.csv) 类似于以下代码段:

xxx
  
xx
DETAIL: Detail 01
Sub-Detail: ABC
x
xxxx
x
Item, Val1, Val2
1, 50, 100
2, 20, 30
3, 30, 50
x
xx
xxx
x
  
    
DETAIL: Detail 01
Sub-Detail: XYZ
x
Item, Val1, Val2
1, 20, 30
2, 30, 50
x
x
DETAIL: Detail 02
Sub-Detail: ABC
Item, Val1, Val2
1, 10, 20
2, 30, 40
xxx
xx
x
x

DETAIL: Detail 02
Sub-Detail: XYZ
Item, Val1, Val2
1, 50, 30
2, 30, 40
3, 70, 80
x
xx

---

假设我已经有办法识别文件中的“坏行”。这意味着，我可以像这样有效地打印这些行:

badLine <- function(line) grepl(pattern = "^$|^\\s|^\\t|^x", line)

con <- file("testing.csv", open = "r")

while (length(oneLine <- readLines(con, n = 1, warn = FALSE)) > 0) {
  if (badLine(oneLine)) next else print(oneLine)
} 

close(con)

产生:

# [1] "DETAIL: Detail 01"
# [1] "Sub-Detail: ABC"
# [1] "Item, Val1, Val2"
# [1] "1, 50, 100"
# [1] "2, 20, 30"
# [1] "3, 30, 50"
# [1] "DETAIL: Detail 01"
# [1] "Sub-Detail: XYZ"
# [1] "Item, Val1, Val2"
# [1] "1, 20, 30"
# [1] "2, 30, 50"
# [1] "DETAIL: Detail 02"
# [1] "Sub-Detail: ABC"
# [1] "Item, Val1, Val2"
# [1] "1, 10, 20"
# [1] "2, 30, 40"
# [1] "DETAIL: Detail 02"
# [1] "Sub-Detail: XYZ"
# [1] "Item, Val1, Val2"
# [1] "1, 50, 30"
# [1] "2, 30, 40"
# [1] "3, 70, 80"

如何在不再次传递文件的情况下构建results 对象(或类似对象)？

可以安全地假设可以利用以下辅助函数来识别它们各自的行:

detailLine <- function(line) grepl(pattern = "^DETAIL: ", line)
subDetailLine <- function(line) grepl(pattern = "^Sub-Detail: ", line)
dfHeaderLine <- function(line) grepl(pattern = "^Item", line)
dfLine <- function(line) grepl(pattern = "^[[:digit:]]", line)

Answer 1

最好先读入数据然后应用过滤器，而不是逐行阅读时应用它们。

#Read in data
alltext <- readLines("testing.csv")

#Apply filter to isolate headers and lines
onedf <- read.csv(text=alltext[dfHeaderLine(alltext) | dfLine(alltext)], stringsAsFactors=FALSE, header=FALSE)

#Split by header
alldfs <- split(onedf, cumsum(dfHeaderLine(onedf[,1])))

#Correct column names
alldfs <- lapply(alldfs, function(x) {names(x) <- unlist(x[1,]);x[-1,]})

#Make Detail and Subheader lists
dtl <- as.list(alltext[detailLine(alltext)])
sub <- as.list(alltext[subDetailLine(alltext)])

#Combine all lists
results <- Map(list, dtl, sub, alldfs)
# [[1]]
# [[1]][[1]]
# [1] "DETAIL: Detail 01"
# 
# [[1]][[2]]
# [1] "Sub-Detail: ABC"
# 
# [[1]][[3]]
# Item  Val1  Val2
# 2    1    50   100
# 3    2    20    30
# 4    3    30    50
# 
# 
# [[2]]
# [[2]][[1]]
# [1] "DETAIL: Detail 01"
# 
# [[2]][[2]]
# [1] "Sub-Detail: XYZ"
# 
# [[2]][[3]]
# Item  Val1  Val2
# 6    1    20    30
# 7    2    30    50