java - Apache Kylin Cube 构建失败(使用 mysql JDBC 连接器时)

Question

即使错误日志中标记为不存在的表存在，Apache kylin 也会失败并出现 MySQLSyntaxErrorException。 多维数据集构建在第一步“SQoop 到平面配置单元表”中失败 我在运行在 docker 上的 cloudera cdh 5.7 沙箱中使用 Apache Kylin 2.5。

麒麟错误日志

java.io.IOException: OS command error exit with return code: 1, error message: SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/apache_kylin/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/01/09 00:26:17 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.13.0
19/01/09 00:26:17 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/01/09 00:26:17 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
19/01/09 00:26:17 INFO manager.SqlManager: Using default fetchSize of 1000
19/01/09 00:26:17 INFO tool.CodeGenTool: Beginning code generation
19/01/09 00:26:18 INFO manager.SqlManager: Executing SQL statement: SELECT TABLE1.PID as TABLE1_PID FROM SPRINGDB.TABLE1 as TABLE1  INNER JOIN SPRINGDB.TABLE2 as TABLE2 ON TABLE1.PID = TABLE2.PID WHERE 1=1 AND  (1 = 0) 
19/01/09 00:26:18 ERROR manager.SqlManager: Error executing statement: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'SPRINGDB.TABLE1' doesn't exist
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'SPRINGDB.TABLE1' doesn't exist
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
    at com.mysql.jdbc.Util.getInstance(Util.java:360)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:978)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
    at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2030)
    at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:777)
    at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786)
    at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289)
    at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
    at org.apache.sqoop.manager.SqlManager.getColumnTypesForQuery(SqlManager.java:253)
    at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:336)
    at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1858)
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1657)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:494)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
19/01/09 00:26:18 ERROR tool.ImportTool: Import failed: java.io.IOException: No columns to generate for ClassWriter
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1663)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:494)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:252)

The command is: 
/usr/lib/sqoop/bin/sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true  -Dmapreduce.job.queuename=default --connect "jdbc:mysql://localhost:3306/springdb" --driver com.mysql.jdbc.Driver --username root --password cloudera --query "SELECT TABLE1.PID as TABLE1_PID FROM SPRINGDB.TABLE1 as TABLE1  INNER JOIN SPRINGDB.TABLE2 as TABLE2 ON TABLE1.PID = TABLE2.PID WHERE 1=1 AND \$CONDITIONS" --target-dir hdfs://quickstart.cloudera:8020/kylin/kylin_metadata/kylin-bac2bbda-f64d-4e90-0cac-beb2902412e4/kylin_intermediate_testcube2_01c6159a_68c8_6da9_a88f_7a3e34f140b1 --split-by TABLE1.PID --boundary-query "SELECT min(TABLE1.PID), max(TABLE1.PID) FROM "SPRINGDB".TABLE1 as TABLE1" --null-string '' --fields-terminated-by '|' --num-mappers 4
    at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:95)
    at org.apache.kylin.source.jdbc.CmdStep.sqoopFlatHiveTable(CmdStep.java:49)
    at org.apache.kylin.source.jdbc.CmdStep.doWork(CmdStep.java:60)
    at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
    at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
    at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
    at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Answer 1

Nissanka Seneviratne，我发现你的日志表明sqoop找不到你的表“SPRINGDB.TABLE1”，看起来就是这个原因。请尝试确认您的特定数据库表可以从提供的用户/帐户(并且主机可访问)访问。

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'SPRINGDB.TABLE1' doesn't exist

如果您发现这不是根本原因，请回复我。