sql - 在Sspark SQL中创建表不支持not null

Question

我运行了查询

CREATE TABLE IF NOT EXISTS OMIDimensionshql.DimPrimaryProduct (PrimaryProductKey int, abc STRING, SCDStartDate timestamp NOT NULL, SCDEndDate timestamp, OMIDQFailedFlag boolean, OMIComputeDeletedFlag boolean NOT NULL, OMIComputeCreatedDate timestamp NOT NULL, OMIComputeModifiedDate timestamp NOT NULL ) Using delta LOCATION 'adl://psinsightsadlsdev01.azuredatalakestore.net//PPE/Compute/OMIDimensions/DimPrimaryProductGrouping/Full/'

使用spark.sql()但它给出以下错误-

Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException: 
no viable alternative at input 'CREATE TABLE IF NOT EXISTS OMIDimensionshql.DimPrimaryProduct (PrimaryProductKey int, abc STRING, SCDStartDate timestamp NOT'(line 1, pos 121)

== SQL ==
CREATE TABLE IF NOT EXISTS OMIDimensionshql.DimPrimaryProduct (PrimaryProductKey int, abc STRING, SCDStartDate timestamp NOT NULL, SCDEndDate timestamp, OMIDQFailedFlag boolean, OMIComputeDeletedFlag boolean NOT NULL, OMIComputeCreatedDate timestamp NOT NULL, OMIComputeModifiedDate timestamp NOT NULL ) Using delta LOCATION 'adl://psinsightsadlsdev01.azuredatalakestore.net//PPE/Compute/OMIDimensions/DimPrimaryProductGrouping/Full/'
-------------------------------------------------------------------------------------------------------------------------^^^

at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:239)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:115)
at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:69)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:638)
at com.ms.omi.meta.execute.Execute$$anonfun$createSubjectAreaTables$1.apply(Execute.scala:55)
at com.ms.omi.meta.execute.Execute$$anonfun$createSubjectAreaTables$1.apply(Execute.scala:46)
at scala.collection.immutable.List.foreach(List.scala:381)
at com.ms.omi.meta.execute.Execute$.createSubjectAreaTables(Execute.scala:46)
at com.ms.omi.meta.entry.EntOmiMetaStore$.main(EntOmiMetaStore.scala:21)
at com.ms.omi.meta.entry.EntOmiMetaStore.main(EntOmiMetaStore.scala)

Process finished with exit code 1

当我在 Databricks 集群上的 Spark SQL 笔记本中执行相同的查询时，它可以工作，但当我使用 spark.sql() 在 Scala 中本地执行它时，它就不起作用。

Answer 1

标准 Spark 运行时不支持 NOT NULL 约束。

Databricks 使用自己的运行时，具有大量专有扩展，因此其中存在的功能不一定在开源 Spark 发行版中可用。

事实上，您尝试使用的另一个功能 - Databricks Delta - 也是一个专有扩展。