scala - 将时间戳与 Spark 和 Scala 中的特定日期进行比较

Question

我有以下数据框:(数据框的名称是:df)

+---------------+-----------+-------------+------+----+-----+--------------------+-------------------+------+------+------+-----+--------------------+--------------------+------------------+------------------+------+
|   time_stamp_0|sender_ip_1|receiver_ip_2| count|rank|   xi|                  pi|                  r|   ip5|   ip4|   ip3|  ip2|            variance|             entropy|    pre_chi_square|  total_chi_square|attack|
+---------------+-----------+-------------+------+----+-----+--------------------+-------------------+------+------+------+-----+--------------------+--------------------+------------------+------------------+------+
|07:19:00.005763|   10.0.0.2|     10.0.0.1|509286|   1|92055|1.963533260289896E-6|0.18075305427598637|111891|115199|190028|92055|1.317855896447428...|2.580232918985576E-5|3.7131630265751565|14.852652106300626|     1|
|07:19:00.005788|   10.0.0.2|     10.0.0.1|509286|   2|92055|3.927066520579792E-6|0.18075305427598637|111891|115199|190028|92055|6.498626409377348E-6|4.888262329310028E-5|18.310392943472664|14.852652106300626|     1|
|07:19:00.005807|   10.0.0.2|     10.0.0.1|509286|   3|92055|5.890599780869688E-6|0.18075305427598637|111891|115199|190028|92055|1.560646344288706E-5|7.093550226267817E-5|  43.9724428049685|14.852652106300626|     1|

如果时间戳大于“07:19:00.005788”并且sender_ip_1<，我需要为attack字段设置零值 等于10.0.0.3。

但是，我不知道如何处理与 scala 条件下的特定数据的时间戳比较。这是我的代码:

val df_attack = df
  .withColumn("attack",
    when($"sender_ip_1" === "10.0.0.3"
      and ($"time_stamp_0").cast(TimestampType) > "07:19:00.005788", 0)
      .otherwise(1))

有人可以帮助我吗？

Answer 1

简单的字典比较也适用于 time_stamp_0 列。

import org.apache.spark.sql.functions._
import spark.implicits._

val final_add_count_rank_xi_pi_r_attack = Dataframe_add_rank_count_xi_pi_final_chi_square
  .withColumn("attack",
    when($"sender_ip_1" === "10.0.0.3"
      && $"time_stamp_0" > "07:19:00.005788", 0)
      .otherwise(1))