Hadoop Jar FileSystemNotFoundException

Question

I am running a hadoop cluster and attempting to run a mapreduce job using a jar file I send to the cluster. The problem is that on certain nodes it succeeds and others it fails. The error I am seeing on certain nodes is:

Exception in thread "main" java.nio.file.FileSystemNotFoundException
        at com.sun.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:183)
        at com.sun.nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:169)
        at java.nio.file.Paths.get(Paths.java:154)
        at customLib.readFile(CustomClass2.java:254)
        at customLib.access$000(CustomClass2.java:210)
        at customLib.getFieldOrder(CustomClass2.java:591)
        at customLib.run(CustomClass1.java:177)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
        at customLib.main(CustomClass1.java:136)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:508)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:236)

I see that others have had similar problems such as this questions with solutions and links to other questions and soltuions: java.nio.file.FileSystemNotFoundException when getting file from resources folder. However, none of these solutions could explain why it would work on one node and not another... this must mean there is some config I have wrong somewhere.

I have checked all the hadoop xml config files as well as the $CLASSPATH, $PATH, and java runtime version and everything seems to be the same. If anyone has any other ideas on what I could check, that would be greatly appreciated - thanks.

EDIT: To clarify, the file it is trying to get is a resource that is packaged within the JAR.

score 0 · Accepted Answer · answered Dec 22 '20 at 18:36

This answer helped me out: https://stackoverflow.com/a/14607953/7969573. It looks like the nodes that were failing had a copy of the jar with the class I was running in the #HADOOP_HOME/share/hadoop/mapreduce/ folder which was in my CLASSPATH. Deleting the jar from the nodes fixed the issue as I run the jar from a shell invoked by my application. I could have just updated that jar and it would have worked as well, but it made more sense to delete it under my current workflow's architecture.

Hadoop Jar FileSystemNotFoundException

1 Answers1