linux - 显然由于 "spark-submit"参数(具有 * 通配符)未扩展，无法通过系统调用从 scala 中调用 "--jars"

Question

“spark-submit”调用在 shell 中运行良好

/bin/bash -c '/local/spark-2.3.1-bin-hadoop2.7/bin/spark-submit --class analytics.tiger.agents.spark.Orsp --master spark://analytics.broadinstitute.org:7077 --deploy-mode client --executor-memory 1024m --conf spark.app.id=Orsp --conf spark.executor.extraJavaOptions=-Dconfig.file=/home/unix/analytics/TigerETL3/application.conf --conf spark.driver.extraJavaOptions=-Dconfig.file=/home/unix/analytics/TigerETL3/application.conf --jars "/home/unix/analytics/TigerETL3/spark-jars/*.jar" /home/unix/analytics/TigerETL3/spark-agents.jar'

然而，当我只是将它转换为 Scala 中的系统调用时:

val cmd = Seq("/bin/bash", "-c", s"""/local/spark-2.3.1-bin-hadoop2.7/bin/spark-submit --class analytics.tiger.agents.spark.Orsp --master spark://analytics.broadinstitute.org:7077 --deploy-mode client --executor-memory 1024m --conf spark.app.id=Orsp --conf spark.executor.extraJavaOptions=-Dconfig.file=/home/unix/analytics/TigerETL3/application.conf --conf spark.driver.extraJavaOptions=-Dconfig.file=/home/unix/analytics/TigerETL3/application.conf --jars "/home/unix/analytics/TigerETL3/spark-jars/*.jar" /home/unix/analytics/TigerETL3/spark-agents.jar""")
import scala.sys.process._
val log = cmd.lineStream.toList
println(log.mkString)

抛出一个错误

Warning: Local jar /home/unix/analytics/TigerETL3/spark-jars/*.jar does not exist, skipping.
Exception in thread "main" java.lang.NoClassDefFoundError: scalikejdbc/DB
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
        at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
        at java.lang.Class.getMethod0(Class.java:3018)
        at java.lang.Class.getMethod(Class.java:1784)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:739)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: scalikejdbc.DB
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 10 more

异常表明 *.jars 模式由于某种原因没有被扩展(即使它在 shell 中工作正常)。 枚举 CSV 列表中的所有 jar 不是很吸引人，这将是一个怪物——187 个 jar 。 我已经尝试过任何我能想到的技巧，但都惨遭失败，好久没有这么沮丧了。

感谢帮助! 谢谢

Answer 1

您需要在指定 --jars 时删除双引号 ""。你能试试这个吗？

val cmd = Seq("/bin/bash", "-c", s"""/local/spark-2.3.1-bin-hadoop2.7/bin/spark-submit --class analytics.tiger.agents.spark.Orsp --master spark://analytics.broadinstitute.org:7077 --deploy-mode client --executor-memory 1024m --conf spark.app.id=Orsp --conf spark.executor.extraJavaOptions=-Dconfig.file=/home/unix/analytics/TigerETL3/application.conf --conf spark.driver.extraJavaOptions=-Dconfig.file=/home/unix/analytics/TigerETL3/application.conf --jars /home/unix/analytics/TigerETL3/spark-jars/*.jar /home/unix/analytics/TigerETL3/spark-agents.jar""")
import scala.sys.process._
val log = cmd.lineStream.toList
println(log.mkString)