我一直在研究 MRUnit,并尝试按照 wordcount 的教程为 hadoop wordcount 示例运行它和 unit testing
虽然不是粉丝,但我一直在使用 Eclipse 运行代码,但我一直收到 setMapper 函数的错误
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mrunit.mapreduce.MapDriver;
import org.apache.hadoop.mrunit.mapreduce.MapReduceDriver;
import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.junit.Before;
import org.junit.Test;
public class TestWordCount {
MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable> mapReduceDriver;
MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
@Before
public void setUp() throws IOException
{
WordCountMapper mapper = new WordCountMapper();
mapDriver = new MapDriver<LongWritable, Text, Text, IntWritable>();
mapDriver.setMapper(mapper); //<--Issue here
WordCountReducer reducer = new WordCountReducer();
reduceDriver = new ReduceDriver<Text, IntWritable, Text, IntWritable>();
reduceDriver.setReducer(reducer);
mapReduceDriver = new MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable>();
mapReduceDriver.setMapper(mapper); //<--Issue here
mapReduceDriver.setReducer(reducer);
}
错误信息:
java.lang.Error: Unresolved compilation problems:
The method setMapper(Mapper<LongWritable,Text,Text,IntWritable>) in the type MapDriver<LongWritable,Text,Text,IntWritable> is not applicable for the arguments (WordCountMapper)
The method setMapper(Mapper<LongWritable,Text,Text,IntWritable>) in the type MapReduceDriver<LongWritable,Text,Text,IntWritable,Text,IntWritable> is not applicable for the arguments (WordCountMapper)
查找这个问题,我认为它可能是一个 API 冲突,但我不确定在哪里寻找它。以前有其他人遇到过这个问题吗?
编辑 我正在使用一个用户定义的库,其中包含 hadoop2 jar 和最新的 Junit(4.10) jar。
编辑 2 这是 WordCountMapper 的代码
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
public class WordCountMapper extends Mapper<Object, Text, Text, IntWritable>
{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context)throws IOException, InterruptedException
{
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens())
{
word.set(itr.nextToken());
context.write(word, one);
}
}
}
最终编辑/有效
原来我需要设置
WordCountMapper mapper = new WordCountMapper();
到
Mapper mapper = new WordCountMapper();
因为泛型存在问题。还需要将 mockito 库导入我的用户定义库。
最佳答案
这是你的问题
public class WordCountMapper extends Mapper<Object, Text, Text, IntWritable>
....
MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
您的 WordCountMapper
输入类型 (Object
) 与 MapDriver
输入类型 (LongWritable
) 不兼容。将您的 Mapper
定义更改为
class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable>
您可能还想将 map
方法参数从 Object key
更改为 LongWritable key
。
关于java - 尝试运行 MRUnit 示例时出现冲突的 API,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/24084265/