默认情况下,spark_read_jdbc()
将整个数据库表读取到 Spark 中。我使用以下语法来创建这些连接。
library(sparklyr)
library(dplyr)
config <- spark_config()
config$`sparklyr.shell.driver-class-path` <- "mysql-connector-java-5.1.43/mysql-connector-java-5.1.43-bin.jar"
sc <- spark_connect(master = "local",
version = "1.6.0",
hadoop_version = 2.4,
config = config)
db_tbl <- sc %>%
spark_read_jdbc(sc = .,
name = "table_name",
options = list(url = "jdbc:mysql://localhost:3306/schema_name",
user = "root",
password = "password",
dbtable = "table_name"))
但是,我现在遇到了这样的情况:我在 MySQL 数据库中有一个表,并且我只想将该表的一个子集读入 Spark。
如何让 spark_read_jdbc
接受谓词?我尝试将谓词添加到选项列表但没有成功,
db_tbl <- sc %>%
spark_read_jdbc(sc = .,
name = "table_name",
options = list(url = "jdbc:mysql://localhost:3306/schema_name",
user = "root",
password = "password",
dbtable = "table_name",
predicates = "field > 1"))
最佳答案
您可以用查询替换dbtable
:
db_tbl <- sc %>%
spark_read_jdbc(sc = .,
name = "table_name",
options = list(url = "jdbc:mysql://localhost:3306/schema_name",
user = "root",
password = "password",
dbtable = "(SELECT * FROM table_name WHERE field > 1) as my_query"))
但对于像这样的简单条件,Spark 应该在您过滤时自动推送它:
db_tbl %>% filter(field > 1)
只需确保设置:
memory = FALSE
在spark_read_jdbc
中。
关于r - 从 JDBC 连接读取数据时如何使用谓词?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/45420958/