architecture - Using Azure Storage Tables as Queues with multiple Worker Roles processing it? -

architecture - Using Azure Storage Tables as Queues with multiple Worker Roles processing it? -

my application receiving 1000+ requests/transactions every second, via multiple instances of web role. these roles write record every transaction across multiple storage tables (randomly, spread azure's 500 transactions/sec limit). now, need reliable way process/aggregate info using multiple worker roles , write results sql database. aka, needs scale horizontally.

i need retain/archive of transactions in storage tables post-processing, go having 1 set of tables queues, , when processed move them onto archive tables, or perhaps there way on single table, not sure.

what recommend far mechanism distribute current workload in these queues across work roles? obviously, each role has aware of every other role working on, work on unclaimed transactions. think each role retrieving 1000 records queue single work load , multiple worker roles working on same queue.

should maintain worker roles "state" in cache, perhaps in sql server.

your suggestions much appreciated.

i recommend utilize proper queue service implement feature instead of trying implement queueing on table service. way won't have implement complex logic know records have been processed (logic becomes hard when consider fault tolerance , possible errors, in service such table storage has limited transaction capability). trying coordinate multiple workers reliably, accounting possible failure scenarios, , beingness scalable @ same time wouldn't effort @ application level.

for instance:

the web role receives request represents transaction; the web role writes info several tables; the web role sends message queue service representing transaction unique id (for instance request id, if there isn't suitable primary key). the worker role pulls messages queue. for each message worker role retrieves set of objects table storage corresponding unique identifier of message. the worker role aggregates info needed , writes sql database.

notes:

use either queue service (from storage) or service bus queues. spread load among many queues scalability. be sure apply proper handling @ levels business relationship transient failures. deal possibility of processing same message more 1 time (the processing should idempotent).

architecture azure windows-azure-storage