我是 Mongodb 和 Hadoop 的新手。我正在尝试访问 mongodb 数据作为 hadoop mapreduce 作业的输入。我不太清楚如何指定使用哪个集合从中获取数据。这是我试过的:
hadoop jar/usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar
-input user/test/input/
-output user/test/output/
-inputformat com.mongodb.hadoop.mapred.MongoInputFormat
-outputformat com.mongodb.hadoop.mapred.MongoOutputFormat
-io mongodb
-D mongo.input.uri=mongodb://localhost/my_dbs.collectionName
-D stream.io.identifier.resolver.class=com.mongodb.hadoop.streaming.io.MongoIdentifierResolver
-mapper /Users/wordcountMapper.py
-reducer /Users/wordcountReducer.py
-libjars /usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/mongo-hadoop-streaming.jar
但是我得到以下错误:
ERROR streaming.StreamJob: Unrecognized option: -D
Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
当我尝试这个时,我得到了另一个错误:
hadoop jar /usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar
-input user/input/
-output user/test/output
-inputformat com.mongodb.hadoop.mapred.MongoInputFormat
-outputformat com.mongodb.hadoop.mapred.MongoOutputFormat
-io mongodb -jobconf mongo.input.uri=mongodb://localhost/my_dbs.collectionName
-jobconf stream.io.identifier.resolver.class=com.mongodb.hadoop.streaming.io.MongoIdentifierResolver
-mapper /Users/wordcountMapper.py
-reducer /Users/wordcountReducer.py
-libjars /usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/mongo-hadoop-streaming.jar
`ERROR streaming.StreamJob: Unrecognized option: -libjars
Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]`
请帮忙。
最佳答案
请查看this有关如何将 MongoDB 连接到 Hadoop 的更好想法的链接。
编辑:
或者,
你可以直接在你的驱动程序中写成:
args.add("-libjars");
args.add("/some/path/to/your/jar");
关于mongodb - Mongo-Hadoop 流式传输,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/29553720/