Concepts

Collector Queue

A collector queue (CollectorQueue) is a simple event queue that takes input payloads and sends them to backend “event collectors” for processing. The collector queue is made up of:

• a simple publisher that sends events the queue,

• a consumer pool that ingests input events and sends them to backend processors,

• a router that handles the routing logic of where to send the events,

• and an administrative layer that monitors the load ratio of the system and adjusts the consumer pool accordingly.

Publisher

The collector queue provides an event publisher (EventPublisher) for sending payloads and manages a pool of consume tasks that pulls the payloads from the queue and routes them to collectors.

Consumer

The collector queue implements the competing consumer pattern. The collector queue manages a consumer pool based on input parameters for minimum and maximum numbers of consumers. It will scale the consumer pool according to the input load.

Use cases

The collector queue and its event collectors are agnostic to how payload processing is implemented. So we are using the terms “event queue,” “message queue,” and “task queue” here interchangeably to mean any queue to which we can send an “event payload” and from which we can fetch the payload for processing.

To support basic use cases as either an event, message, or task queue, the collector queue implements queue types (FIFO queue, Priority Queue, LIFO queue) and deliver modes (see Router & Payload Delivery below).

Note

Internally, the collector queue uses a standard asyncio.Queue (which can be a FIFO, Priority Queue, or LIFO type) for exchanging data between producer and consumer.

Collector

An event collector or simply “collector” (see Collector) contains a simple function that processes incoming events or messages. It’s up to the collector to decide how to process the payload. Collectors can send data downstream to other services, or they can return a value to the caller via the collector queue, or both.

Routing & Delivery

A router (see Router) is a class that implements the logic for collector queue consumers to handle event dispatching to collectors.

The collector queue uses a default, simple routing logic where event payloads sent to the collector queue are delivered using one of 3 delivery modes (see Payload and Delivery):

1. Direct Key delivery

   In direct key delivery mode, the router will dispatch the event payload to the collectors with those keys.

2. Topic delivery

   In topic delivery mode, the router will dispatch the event payload to the collectors that are listening to topics matching the event topic(s). Topic matching is by exact keyword only (no patterns or regular expressions).

3. Broadcast delivery

   In broadcast delivery mode, the router will send the event payload to all collectors associated with the collector queue.

The default router (SimpleRouter) is designed to work for most use cases but the routing logic can be changed using a custom router class.