0

I am working on a spark application . I am using a third party jar. I added it in my sbt file. The compilation was successful

> libraryDependencies += "org.springframework.security" %
> "spring-security-web" % "3.0.7.RELEASE

But when i submit spark job 

spark-submit --master yarn-client --class MyClass target/scala-2.10/my-app.jar

It fails with following error:

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to 
stage failure: Task 0 in stage 48.0 failed 1 times, most recent failure: Lost task 0.0 in stage 48.0 (TID 3216, localhost): 
`java.lang.NoClassDefFoundError:` org/springframework/security/web/util/IpAddressMatcher

I also tried explicitly passing jar in spark-submit jar 

spark-submit --master yarn-client --jars third-party-jars/spring-security-web-3.0.7.RELEASE.jar --class MyClass target/scala-2.10/my-app.jar

But then it fails with following error:

java.lang.NoClassDefFoundError: org/springframework/util/StringUtils
    at org.springframework.security.web.util.IpAddressMatcher.<init>(IpAddressMatcher.java:33)

Its similar error but on another class.

Any suggestion on how to resolve this?

Thanks

Alok
  • 1,174
  • 2
  • 12
  • 38
  • 1
    Try making an "uberjar" using the assembly plugin from sbt – rhernando Feb 08 '16 at 11:27
  • as @rhernando mentioned, try to build a fat jar following this instructions - http://stackoverflow.com/questions/28459333/how-to-build-an-uber-jar-fat-jar-using-sbt-within-intellij-idea . All your dependencies should be then placed inside a jar. – TheMP Feb 08 '16 at 13:43

1 Answers1

1

Also you can use IntelijIDE. You will not need to adjust it always. You can download here for Ubuntu. https://apps.ubuntu.com/cat/applications/intellij-idea-ce/

Thoram Mastero
  • 159
  • 1
  • 1
  • 11