python - Pyspark 错误 - Py4JJavaError : An error occurred while calling o731. 加载

Question

我正在学习 pyspark 并在 Azure Synapse Notebook 中执行下面提到的简单代码。我在执行时遇到以下错误。有谁知道如何解决吗？

<小时/>

##########代码##########

***

%%pyspark
df=spark.read.load(path='https://adls4acc.dfs.core.windows.net/filesys/NYCTripSmall.parquet',format='parquet')
display(df.limit(10))
spark.sql("create database if not exists TEMPDB")
df.write.mode("overwrite").saveAsTable("TEMPDB.NycData")

<小时/>

##########错误消息##########

Py4JJavaError: An error occurred while calling o731.load. : java.lang.UnsupportedOperationException at org.apache.hadoop.fs.http.AbstractHttpFileSystem.listStatus(AbstractHttpFileSystem.java:94) at org.apache.hadoop.fs.http.HttpsFileSystem.listStatus(HttpsFileSystem.java:23) at org.apache.spark.util.HadoopFSUtils$.listLeafFiles(HadoopFSUtils.scala:225) at org.apache.spark.util.HadoopFSUtils$.$anonfun$parallelListLeafFilesInternal$1(HadoopFSUtils.scala:95) at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at scala.collection.TraversableLike.map(TraversableLike.scala:286) at scala.collection.TraversableLike.map$(TraversableLike.scala:279) at scala.collection.AbstractTraversable.map(Traversable.scala:108) at org.apache.spark.util.HadoopFSUtils$.parallelListLeafFilesInternal(HadoopFSUtils.scala:85) at org.apache.spark.util.HadoopFSUtils$.parallelListLeafFiles(HadoopFSUtils.scala:69) at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$.bulkListLeafFiles(InMemoryFileIndex.scala:158) at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.listLeafFiles(InMemoryFileIndex.scala:131) at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.refresh0(InMemoryFileIndex.scala:94) at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.<init>(InMemoryFileIndex.scala:66) at org.apache.spark.sql.execution.datasources.DataSource.createInMemoryFileIndex(DataSource.scala:565) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:409) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:281) at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:253) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:253) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:190) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748)

Answer 1

您不应使用存储帐户 URL 作为路径。

改为使用以下格式的路径

'abfss://<FileSystemName>@<StorageName>.dfs.core.windows.net/sample/<FileName.parquet>'

尝试下面的代码来创建数据框

df = spark.read.load(path='abfss://<FileSystemName>@<StorageName>.dfs.core.windows.net/sample/<FileName.parquet>',format='parquet')

引用 - https://docs.databricks.com/data/data-sources/azure/azure-storage.html