I'm trying to ingest data through spark and push it to ElasticSearch.
I've following the basic tutorial to ingest from Oracle, put it in memory thru Spark and push it to ElasticSearch.
When I try to call the JavaEsSparkSQL.saveToEs(jdbcDF, "spark/test")
. jdbcDF is the DataSet loaded from Oracle. I'm simplying reading from my Oracle DB and calling saveToEs.
I am getting the following error:
java.lang.NoClassDefFoundError: org/elasticsearch/spark/sql/api/java/JavaEsSparkSQL
I read that it could be because of the differing version of spark as the Hadoop library is using a separate driver? Here are my dependencys:
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-hadoop</artifactId>
<version>7.3.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>2.4.3</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>2.4.3</version>
</dependency>