How to build and use flink-connector-kinesis?

Question

I'm trying to use Apache Flink with AWS kinesis. The document says that I have to build the connector on my own.

Therefore, I build the connector and added the jar file for my project and also, I put the dependency on my pom.xml file.

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kinesis_2.11</artifactId>
    <version>1.6.1</version> 
</dependency>

However, when I tried to build using mvn clean package I got an error message like this

[INFO] -----------------------< kkaldk:flink-kinesis >-----------------------
[INFO] Building Flink Quickstart Job 0.1
[INFO] --------------------------------[ jar ]---------------------------------
[WARNING] The POM for org.apache.flink:flink-connector-kinesis_2.11:jar:1.6-SNAPSHOT is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.458 s
[INFO] Finished at: 2018-12-19T17:45:43+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project flink-kinesis: Could not resolve dependencies for project kkaldk:flink-kinesis:jar:0.1: Failure to find org.apache.flink:flink-connector-kinesis_2.11:jar:1.6-SNAPSHOT in https://repository.apache.org/content/repositories/snapshots/ was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced -> 
[Help 1]
[ERROR]

Could you help me to fix this?

---------added----------

This is what I've done (which is wrong procedure)

started project following this document
build connector from flink master source (which is 1.8-snapshot)
add dependency with 1.6.1 version.

What did you put in your pom to add the kinesis jar file to your project? — 0x26res, Dec 19 '18 at 09:59
So when you say you've added 'the jar file for my project', you mean you called this? `mvn install:install-file -Dfile=flink-connector-kinesis_2.10-1.6.1.jar -DpomFile=flink-connector-kinesis_2.10-1.6.1.pom.xml`? The difficulty here is that you trying to use a dependency that is not in the maven central repo. I think this link may help: http://www.mkyong.com/maven/how-to-include-library-manully-into-maven-local-repository/ — 0x26res, Dec 20 '18 at 10:04
@BumhwanKim how did you build and add the dependency of the conector you build? I also need to use the Kinesis conector but as they said [here](https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kinesis.html) — IoT user, Dec 27 '18 at 11:58

Bumhwan Kim · Accepted Answer · 2019-01-22T00:38:48.653

10

This is what I've done for solution.

I downloaded 1.6.1 release from here.
unzip the file (the archive will be unzip in .../flink-release-1.6.1/
built it by mvn clean install -Pinclude-kinesis -DskipTests

Then I have flink-connector-kinesis_2.11 version 1.6.1 in my local maven repository

(for me, path was like this): ~/.m2/repository/org/apache/flink/flink-connector-kinesis_2.11

After then, I could use this connector at any project by adding dependency with this code. (I don't have to add the jar file anymore.)

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kinesis_2.11</artifactId>
    <version>1.6.1</version> 
</dependency>

edited Jan 22 '19 at 00:38

answered Jan 01 '19 at 15:33

Bumhwan Kim

282
3
15

1

Thanks, this was helpful – IoT user Jan 08 '19 at 15:42
i have done the same steps and in my local repo it showing the connector jar exist in the below path org/apache/flink/flink-connector-kinesis_2.11/1.9.0/flink-connector-kinesis_2.11-1.9.0.jar.lastUpdated But still my project is not able to build.i am getting below error Could not resolve dependencies for project org.data.bounce:BounceProcessor:jar:1.0-SNAPSHOT: Failure to find org.apache.flink:flink-connector-kinesis_2.11:jar:1.9.0 in https://repo.maven.apache.org/maven2 was cached in the local repository, – Anuj jain Dec 08 '19 at 04:45
@Anujjain I am getting a very similar error, did you ever resolve this? – sedavidw Feb 12 '20 at 16:56

How to build and use flink-connector-kinesis?

1 Answers1

Linked