jms - ActiveMQ - worker threads and pool of tasks -

jms - ActiveMQ - worker threads and pool of tasks -

i have next scenario:

some operation invoked , result multiple tasks created (thousands) they (their status , details) stored in database table i need somehow execute these tasks on multiple nodes

so, obvious way send jms message each created task (10000 tasks - 10000 messages) , pool (per node) of consumers perform them - looks overkill - listener needs fetch task db anyway, plenty notify it should work next task, if - send 1 message ("work next task") , somehow set delivered 10000 times.

how approach this? best thought send 10000 messages?

also, activemq has message scheduling thing [1] - maybe set period 0 , repeat 10000? work correctly? there improve way other mentioned above?

[1] http://activemq.apache.org/delay-and-schedule-message-delivery.html

actually, can utilize redis storing ids of completed tasks(1..n). , utilize "set" info structure.

keys looks tasklist:

each key contains ids of completed tasks.

so when started job 1000 async tasks. generate unique id job, , send task id, when task completed worker(who process task) should add together id in db(redis). , when lastly worker save id in db , size of set equal size of tasks (you should check after each task) — tasks completed can send message or create else.

redis provide fast access , fast few info structures.

jms activemq worker