TNC has open sourced a project called Consumer Dispatcher recently at github.
We ‘ve been using Consumer Dispatcher for more than 2 years which helps to consume 100 millions jobs each day. (not a great number of jobs, but consumer dispatcher wont be your bottle neck.)
What is Consumer Dispatcher and Why
Consumer Dispatcher works as a proxy between RabbitMQ and your consumers. Let you scale out and maintain consumers easily.
It can work with all http-based interfaces in any language.
- You can utilize the functionalities Consumer Dispatcher provides (see next section).
- When you are using MVC implementation for your project, it makes you easier to write consumer code, scale them out and maintain the consumers. Since Consumer Dispatcher calls your consumer code by sending http request (how does it work, http request format), you could reuse your infrastructure (like nginx + php) and reuse most of your code to process the business logic (still in MVC way).
The functionalities it provides
Please go to What and Why part for each functionality explanation.
How it works
- Your publisher publishes jobs to a specific queue inside RabbitMQ. same as classic way
- Consumer Dispatcher fetches jobs from each queue defined in config file then dispatches each job to your consumer code by sending http requests (http request format).
- Consumer Dispatcher waits (timeout for each request is configurable) until it gets http response from your consumer code. then Consumer Dispatcher acknowledges the job to the queue if it gets ok (case insensitive) as the content of HTTP response or resends the job if it gets anything else (like 502 HTTP status, or anything other than ok).
- If you need to speed up consuming one specific queue, you can open Consumer Dispatcher admin interface to add more working executors so that more number of jobs will be fetched from that queue at a time and be consumed.
How to use it
please go to How to use it for the guide.
Consumer Dispatcher Resources
- project home page
- index of all wiki and doc
- git address: https://github.com/jackyhung/consumer-dispatcher.git
There definitely a lot of rooom to improve. Any suggestion is appreciated. Thank you for your support.