1

Is the Combiner logic in Hadoop MapReduce always the same as the Reducer logic? Or is there any scenario where the Combiner logic can be different from the Reducer logic?

Ravindra babu
  • 42,401
  • 8
  • 208
  • 194
raju mail
  • 21
  • 3
  • 1
    Possible duplicate of [combiner and reducer can be different?](http://stackoverflow.com/questions/11731770/combiner-and-reducer-can-be-different) – Manjunath Ballur Nov 23 '15 at 11:00

3 Answers3

1
  1. Combiner class & Reducer classes may or may not be same depending on your requirement.

  2. But both Combiner & Reducer have to implement reduce() method by extending Reducer

    e.g.

    public void reduce(Key key, Iterable<IntWritable> values,
              Context context) throws IOException, InterruptedException {

  3. If both Combiner & Reducer are different, set those classes as below

     job.setCombinerClass(YourCombiner.class);
 job.setReducerClass(YourReducer.class);
Ravindra babu
  • 42,401
  • 8
  • 208
  • 194
0

No. You can specify whichever combiner function you want. I suggest you to read a Mahout in action book, on page 32 you may find some info on that. Example:

public class MaxTemperatureWithCombiner {
public static void main(String[] args) throws IOException {
if (args.length != 2) {
System.err.println("Usage: MaxTemperatureWithCombiner <input path> " +
"<output path>");
System.exit(-1);
}
JobConf conf = new JobConf(MaxTemperatureWithCombiner.class);
conf.setJobName("Max temperature");
FileInputFormat.addInputPath(conf, new Path(args[0]));
FileOutputFormat.setOutputPath(conf, new Path(args[1]));
conf.setMapperClass(MaxTemperatureMapper.class);
conf.setCombinerClass(MaxTemperatureReducer.class);
conf.setReducerClass(MaxTemperatureReducer.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
JobClient.runJob(conf);
}
}

Notice that you call conf.setCombinerClass(MaxTemperatureReducer.class); So you can specify any combiner function you want. Often combiner and reducer have same logic, but not always.

Maksim Khaitovich
  • 4,360
  • 3
  • 31
  • 65
-1

Combiner logic is same as reducer logic in most of the cases. Although we may have combiner logic different from reducer too.

Combiner must have same input type as reducer.

Combiner will combine results per map output. Due to this some values are combined in the mapper phase itself and due to this network traffic reduce dramatically since less values need to traverse from mapper to reducer.

rraghuva
  • 131
  • 10