我正在尝试将 XML 数据加载到 Hive 中,但出现错误:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"xmldata":""}
我使用的 xml 文件是:
<?xml version="1.0" encoding="UTF-8"?>
<catalog>
<book>
<id>11</id>
<genre>Computer</genre>
<price>44</price>
</book>
<book>
<id>44</id>
<genre>Fantasy</genre>
<price>5</price>
</book>
</catalog>
我使用的配置单元查询是:
1) Create TABLE xmltable(xmldata string) STORED AS TEXTFILE;
LOAD DATA lOCAL INPATH '/home/user/xmlfile.xml' OVERWRITE INTO TABLE xmltable;
2) CREATE VIEW xmlview (id,genre,price)
AS SELECT
xpath(xmldata, '/catalog[1]/book[1]/id'),
xpath(xmldata, '/catalog[1]/book[1]/genre'),
xpath(xmldata, '/catalog[1]/book[1]/price')
FROM xmltable;
3) CREATE TABLE xmlfinal AS SELECT * FROM xmlview;
4) SELECT * FROM xmlfinal WHERE id ='11
直到第二个查询一切正常,但是当我执行第三个查询时,它给了我错误:
错误如下:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"xmldata":"<?xml version=\"1.0\" encoding=\"UTF-8\"?>"}
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:159)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"xmldata":"<?xml version=\"1.0\" encoding=\"UTF-8\"?>"}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:675)
at org.apache.hadoop.hive.ql.exec
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
那么到底哪里出错了呢?此外,我正在使用正确的 xml 文件。
谢谢,
史瑞
最佳答案
错误原因:
1) 案例 1 :( 你的情况 ) - xml 内容被逐行提供给 hive。
输入xml:
<?xml version="1.0" encoding="UTF-8"?>
<catalog>
<book>
<id>11</id>
<genre>Computer</genre>
<price>44</price>
</book>
<book>
<id>44</id>
<genre>Fantasy</genre>
<price>5</price>
</book>
</catalog>
checkin hive :
select count(*) from xmltable; // return 13 rows - means each line in individual row with col xmldata
错误原因:
XML 被读取为 13 个不统一的部分。如此无效的 XML
2) 案例 2 : xml 内容应该作为 singleString 提供给 hive - XpathUDFs 作品
引用语法:所有函数都遵循以下形式:xpath_(xml_string, xpath_expression_string).* source
input.xml
<?xml version="1.0" encoding="UTF-8"?><catalog><book><id>11</id><genre>Computer</genre><price>44</price></book><book><id>44</id><genre>Fantasy</genre><price>5</price></book></catalog>
checkin hive :
select count(*) from xmltable; // returns 1 row - XML is properly read as complete XML.
意思是:
xmldata = <?xml version="1.0" encoding="UTF-8"?><catalog><book> ...... </catalog>
然后像这样应用你的 xpathUDF
select xpath(xmldata, 'xpath_expression_string' ) from xmltable
关于hive - 将 xml 数据加载到 hive 表 :org. apache.hadoop.hive.ql.metadata.HiveException,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/24607685/