mysql - 使用pig将数据插入mysql表

Question

我有 2 个文本文件。我计划使用 pig latin 将数据转储到 mysql 表中。

有办法吗？

我编写了以下代码:

register '/homes/rdheeraj/pig-0.10.0/code/mysql-connector-java-5.1.17-bin.jar'
register '/homes/rdheeraj/pig-0.10.0/code/piggybank.jar';
a = load 'one.txt' using PigStorage('|') as (name:chararray, age:int);
b = load 'two.txt' using PigStorage('|') as (name:chararray, desg:chararray);
c = cogroup a by name, b by name;
d = foreach c generate flatten(a), flatten(b);
e = foreach d generate $0,$1,$3;
store e into 'TEST' using  org.apache.pig.piggybank.storage.DBStorage('com.mysql.jdbc.Driver','jdbc:mysql://127.0.0.1/sftool','root','password','insert into TEST (name,age,desg) values (?,?,?)');

我收到的错误如下

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Could not resolve org.apache.pig.piggybank.storage.DBStorage using imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
        at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1597)
        at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1540)
        at org.apache.pig.PigServer.registerQuery(PigServer.java:540)
        at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:970)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
        at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
        at org.apache.pig.Main.run(Main.java:555)
        at org.apache.pig.Main.main(Main.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: Failed to parse: Cannot instantiate: org.apache.pig.piggybank.storage.DBStorage
        at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
        at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1589)
        ... 14 more
Caused by: java.lang.RuntimeException: Cannot instantiate: org.apache.pig.piggybank.storage.DBStorage
        at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:510)
        at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:791)
        at org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.java:780)
        at org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.java:4583)
        at org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator.java:6225)
        at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1335)
        at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:789)
        at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:507)
        at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:382)
        at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:175)                                                                                

Answer 1

一些建议。

1. 我认为您的 jar 路径存在错误 - 它不应该是家而不是家吗？

2. 您的路径不需要引号

3. 您在注册 mysql 驱动程序后缺少一个分号。

所以它应该看起来更像这样:

REGISTER /home/rdheeraj/pig-0.10.0/code/mysql-connector-java-5.1.17-bin.jar;
REGISTER /home/rdheeraj/pig-0.10.0/code/piggybank.jar;