This is a synthesis of asynchronous work patterns and their trade-offs.
Patterns
- Application-local Queue (e.g. Sidekiq)
- Message Bus (e.g. Kafka)
- Global Queue (e.g. AWS SQS)
If the producer and consumer are in the same application, use a application-local queue. If there are multiple consumers of the same message, maybe use a message bus. If message order delivery is not needed, use something like AWS SQS.
App-Local Trade-offs
Pros:
- Simplest
- Configurable retry and failure handling
Cons:
- Cannot cross service boundaries
- Observability is not built-in
- Ordered job execution is not guaranteed
Message Bus Trade-offs
Pros:
- Messages can be consumed multiple times by multiple consumers
- Order of messages is guaranteed
- Messages can cross service boundaries
Cons:
- Retry and failure handling is not built-in
- Partial metrics but some must be built
- Horizontal scalability relies on configuration
Global Queue Trade-offs
Pros:
- Retry and failure handling is built-in
- Messages can cross service boundaries
- Observability is built-in
Cons:
- Ordered job execution is not guaranteed (can use FIFO queue)