how create datasource programmatically in spring batch?

Question

i want to copy many data from serveral db which on diff machine to a centre db. i think the spring batch may be a choice to fit my requirement. so. should be make a lot of job to accomplish the whole task, the jobs will like this:

job A: copy from db1 to db111;
job B: copy from db2 to db111;
job C: copy form db3 to db111;

etc... and the tables in db1, db2, db3...is quite different.

so far, i know how to create datasources at spring boot startup time, but i don't know how to create datasource in job instance at runtime. is any idea about this? (if can support spring data jpa will be better)

or is any other way better then spring batch?

thanks.

You must use JDBC directly https://docs.oracle.com/javase/tutorial/jdbc/basics/connecting.html — Simon Martinelli, Aug 29 '19 at 07:16
@SimonMartinelli,yes, it can be, but i really doesn't like hard code sql. thanks — sendreams, Aug 29 '19 at 07:54
But that's what you are want to do. Btw. there are many tools that can do database migration maybe you even have not to develop that at all — Simon Martinelli, Aug 29 '19 at 07:55
Yes or https://nifi.apache.org/ or Talend or any other ETL tool — Simon Martinelli, Aug 29 '19 at 08:43
@SimonMartinelli thank you very much. which tool may be the better?my etl processor has special features that require compare between source and target data to find the modify,new,delete record before write to target db. — sendreams, Aug 29 '19 at 09:26
A ETL tool like Talend can do all you need. But you should start with comparing the options — Simon Martinelli, Aug 29 '19 at 14:34

score 0 · Answer 1 · answered Aug 29 '19 at 12:44

A datasource is a set of connections to a DB so in your scenario , there are just multiple kinds of DBs or multiple DBs of same kind & for both scenarios - you will have to create one datasource for each db & then use that in piece of code wherever you need it.

Step 1 - So you write one configuration class for each database to set up one datasource. At propertly file level, you won't be able to use default properties but your custom ones where you prefix properties with db names to distinguish.

You need to define transaction managers etc for each datasource & you uniquely name each datasource.

Step 2 : Next step is to use appropriate datasource with appropriate dao classes. In above configuration class, if you use JPA , those configs would already be there including entity packages, repository packages etc etc. JdbcTemplate takes datasource in constructor etc.

All in All - scenario is similar to single datasource scenario & you will have to set up all datasources in advance at app start up but with appropriate qualified bean names & then use those data-sources wherever you need.

This Answer is what works for me

if i have 50 datasource, i should config 50 datasource in properties file, then when the spring boot startup, all of the datasource will be create, it may be not a good idea. thanks. — sendreams, Aug 31 '19 at 16:05
Do you really have those many data sources or are you just talking theoretically ? Any low level programming framework like Spring Boot is not suitable for scenario that you describe , you need to go for higher level tools as described by others. — Sabir Khan, Sep 02 '19 at 14:08
yes, i really has so many datasource, may be more. so i should found a tool to handle it. i need a schedule to manager many tasks, each one task connect two database, excute all of task one by one. — sendreams, Sep 04 '19 at 15:32

how create datasource programmatically in spring batch?

1 Answers1