scala - 从RDD获取值

Question

我使用Scala使用以下格式创建了RDD:

Array[(String, (Array[String], Array[String]))]

如何从此Array[1]获取RDD的列表？

第一条数据行的数据是:

// Array[(String, (Array[String], Array[String]))]
Array(
  (
    966515171418,
    (
      Array(4579848447, 4579848453, 2015-07-29 03:27:28, 44, 1, 1, 966515171418, 966515183263, 420500052424347, 0, 52643, 9, 5067, 5084, 2, 1, 0, 0),
      Array(4579866236, 4579866226, 2015-07-29 04:16:22, 37, 1, 1, 966515171418, 966515183264, 420500052424347, 0, 3083, 9, 5072, 5084, 2, 1, 0, 0)
    )
  )
)

Answer 1

假设您有类似这样的东西(只需粘贴到spark-shell中):

val a = Array(
  ("966515171418", 
    (Array("4579848447", "4579848453", "2015-07-29 03:27:28", "44", "1", "1", "966515171418", "966515183263", "420500052424347", "0", "52643", "9", "5067", "5084", "2", "1", "0", "0"),
     Array("4579866236", "4579866226", "2015-07-29 04:16:22", "37", "1", "1", "966515171418", "966515183264", "420500052424347", "0",  "3083", "9", "5072", "5084", "2", "1", "0", "0")))
  )

val rdd = sc.makeRDD(a)

然后您使用第一个数组

scala> rdd.first._2._1
res9: Array[String] = Array(4579848447, 4579848453, 2015-07-29 03:27:28, 44, 1, 1, 966515171418, 966515183263, 420500052424347, 0, 52643, 9, 5067, 5084, 2, 1, 0, 0)

这意味着第一行(它是一个Tuple2)，然后是该元组的第二个元素(又是一个Tuple2)，然后是第一个元素。

使用模式匹配

scala> rdd.first match { case (_, (array1, _)) => array1 }
res30: Array[String] = Array(4579848447, 4579848453, 2015-07-29 03:27:28, 44, 1, 1, 966515171418, 966515183263, 420500052424347, 0, 52643, 9, 5067, 5084, 2, 1, 0, 0)

如果要获取所有行，只需使用map():

scala> rdd.map(_._2._1).collect()

它将所有行的结果放入数组。

另一种选择是在map()中使用模式匹配:

scala> rdd.map { case (_, (array1, _)) => array1 }.collect()