I am trying to read multiple lines in mapper. For that I started using NLineInputFormat class. While using this, I am getting GC limit error. For reference, the error code is:
16/02/21 01:37:13 INFO mapreduce.Job: map 0% reduce 0%
16/02/21 01:37:38 WARN mapred.LocalJobRunner: job_local726191039_0001
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1019)
at java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:1084)
at java.util.concurrent.ConcurrentHashMap.<init>(ConcurrentHashMap.java:852)
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:713)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:442)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.<init>(LocalJobRunner.java:217)
at org.apache.hadoop.mapred.LocalJobRunner$Job.getMapTaskRunnables(LocalJobRunner.java:272)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:517)
16/02/21 01:37:39 INFO mapreduce.Job: Job job_local726191039_0001 failed with state FAILED due to: NA
For reference, please find the code snippet below.
public class JobLauncher {
public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "TestDemo");
job.setJarByClass(JobLauncher.class);
job.setMapperClass(CSVMapper.class);
job.setMapOutputKeyClass(NullWritable.class);
job.setMapOutputValueClass(NullWritable.class);
conf.setInt(NLineInputFormat.LINES_PER_MAP, 3);
job.setInputFormatClass(NLineInputFormat.class);
NLineInputFormat.addInputPath(job, new Path(args[0]));
job.setNumReduceTasks(0);
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
I just have simple CSVMapper mapper. Why I am getting this error ? Please help me resolve this error.
Thanks in advance.