39

I am trying to find an equivalent of Celery project for Java environment, I have looked at Spring Batch, but are there any better alternatives for distributed task queues.

Thanks.

Zakiullah Khan
  • 1,308
  • 2
  • 12
  • 26
  • Actually a duplicate of this: https://stackoverflow.com/questions/9577012/whats-the-equivalent-of-pythons-celery-project-for-java (but this dupe has better answers by now...) – fnl Nov 11 '16 at 08:30

8 Answers8

30

What Celery is doing is very much akin to EIP, and SEDA with convenient task scheduling... (all you have left to do is add some DB, and async HTTP networking and you have got a complete enterprise quality stack).

Basically in Java there is the Spring way, the Java EE way, and the Hadoop way:

  • Spring: Spring Integration + Spring Batch + RabbitMQ
  • Java EE: Mule + Quartz or EJB Scheduling + HornetMQ
  • Hadoop: Capacity + ZooKeeper

Those are roughly in order of ease of setting up.

wolf97084
  • 260
  • 4
  • 21
Adam Gent
  • 44,449
  • 20
  • 142
  • 191
9

Jesque (https://github.com/gresrun/jesque) is a Java distributed task queue library. It is a Java port of the Resque library (https://github.com/defunkt/resque), which is described like this on its GitHub page:

Resque (pronounced like "rescue") is a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later."

xnickmx
  • 957
  • 1
  • 10
  • 16
5

Quartz has worked for me in the past. It's integrated with Terracotta now, so it should be easy to distribute. http://quartz-scheduler.org/

ConnorWGarvey
  • 705
  • 4
  • 7
  • 2
    Is it possible to do on-demand job execution with Quartz scheduler, rather than executing a job on a specific schedule? – Zakiullah Khan Mar 06 '12 at 02:28
  • @ZakiullahKhanMohamed I'm _very_ late but yes, this is possible. You can schedule a job to be run immediately. – GuiSim Jul 29 '16 at 21:01
  • 2
    Just a heads up: the open source 'free' version does not have any support for distribution. Terracotta claims the enterprise version does. – Adam Marcionek Sep 27 '16 at 20:07
  • @AdamMarcionek not sure what you mean by that. all i see is apache license and i thought that means you're basically free to distribute your software even if you're using quartz – dtc May 22 '20 at 20:39
  • ive only taken a short look at Quartz but the user seems to have to do a lot of the manual work to setup monitoring the system – dtc May 22 '20 at 20:40
  • it's important to mention that quartz is also not a job queue system. – dtc May 29 '20 at 23:07
  • 1
    @dtc Not distribution of code, but distribution of tasks, which is locked behind the enterprise version. And yes, as you say, its not the same as the OP. – Adam Marcionek Mar 23 '21 at 18:58
3

Celery is primarily based on Erlang/RabbitMQ. RabbitMQ has a Java client library that might be useful. Also, there is octobot which has a RabbitMQ backend.

enderskill
  • 6,068
  • 3
  • 21
  • 23
  • If I am not wrong Celery is largely written in Python and not Erlang, I would agree on the note that RabbitMQ is erlang based. Looking at octobot, thanks. – Zakiullah Khan Mar 06 '12 at 14:44
  • 1
    I think he means that RabbitMQ is written in Erlang, so you need to install that to use it. – fabspro Jun 01 '14 at 09:41
2

Distributed Java tasks scheduling and execution https://redisson.org/

Daniel Dai
  • 907
  • 8
  • 23
0

The closest thing I've found is Octobot: https://github.com/cscotta/Octobot Not so much documentation though... there used to be a website for it at octobot.taco.cat, but I haven't seen that load lately. I haven't personally used Octobot, but I've often seen it recommended as a Celery for Java.

nairbv
  • 3,618
  • 1
  • 21
  • 25
0

I haven't been able to find anything as easy to use as Celery for Java. Most of the solutions recommended to use a message queue. But Celery sits one level of abstraction higher than the queue. Instead of messages and consumers, you can think in terms of tasks and workers, results, retries etc.

I also needed to implement some bridge for a company using both Java and Python so I started this project:

celery-java - Celery client and worker in Java, compatible with their Python counterparts.

Beware, it's very immature as of now.

Krab
  • 1,983
  • 12
  • 21
-5

Apache ActiveMQ http://activemq.apache.org/

Apache Kafka http://kafka.apache.org/

soulmachine
  • 3,046
  • 3
  • 37
  • 52
  • 4
    I dont think ActiveMQ and Kafta are Worker/Task Queue like celery. They are Message brokers and can be compared with RabbitMQ if you mean that. – Rupesh Feb 16 '16 at 08:46