Concepts
========

.. image:: images/event-collector.svg

Collector Queue
-------------------------------------

A collector queue
(:py:class:`~event_collector.queue.CollectorQueue`) is a simple event
queue that takes input payloads and sends them to backend "event
collectors" for processing. The collector queue is made up of:

* a simple publisher that sends events the queue,
* a consumer pool that ingests input events and sends them to backend processors,
* a router that handles the routing logic of where to send the events,
* and an administrative layer that monitors the load ratio of the system and adjusts the consumer pool accordingly.

Publisher
---------

The collector queue provides an *event
publisher*
(:py:class:`~event_collector.queue.EventPublisher`) for
sending payloads and manages a pool of consume tasks that pulls the
payloads from the queue and routes them to collectors.


Consumer
--------

The collector queue implements the `competing consumer
pattern
<https://learn.microsoft.com/en-us/azure/architecture/patterns/competing-consumers>`__.
The collector queue manages a consumer pool based on input parameters for minimum and
maximum numbers of consumers. It will scale the consumer pool according to the input load.

Use cases
---------

The collector queue and its event collectors are agnostic to how
payload processing is implemented. So we are using the terms "event
queue," "message queue," and "task queue" here interchangeably to
mean any queue to which we can send an "event payload" and from which we
can fetch the payload for processing.

To support basic use cases as either an event, message, or task
queue, the collector queue implements queue types (FIFO queue,
Priority Queue, LIFO queue) and deliver modes (see *Router & Payload
Delivery* below).

.. note::

   Internally, the collector queue uses a standard `asyncio.Queue
   <https://docs.python.org/3/library/asyncio-queue.html>`__ (which can
   be a FIFO, Priority Queue, or LIFO type) for exchanging data between
   producer and consumer.


Collector
---------

An event collector or simply "collector" (see
:py:class:`~event_collector.collector.Collector`) contains a simple
function that processes incoming events or messages. It's up to the
collector to decide how to process the payload. Collectors can send
data downstream to other services, or they can return a value to the
caller via the collector queue, or both.


Routing & Delivery
------------------

A router (see :py:type:`~event_collector.router.Router`) is a class
that implements the logic for collector queue consumers to handle
event dispatching to collectors.

The collector queue uses a default, simple routing logic where event
payloads sent to the collector queue are delivered using one of 3
delivery modes (see :py:class:`~event_collector.data.Payload` and
:py:type:`~event_collector.data.Delivery`):

1. Direct Key delivery

   In direct key delivery mode, the router will dispatch the event
   payload to the collectors with those keys.

2. Topic delivery

   In topic delivery mode, the router will dispatch the event payload
   to the collectors that are listening to topics matching the event
   topic(s). Topic matching is by exact keyword only (no patterns
   or regular expressions).

3. Broadcast delivery

   In broadcast delivery mode, the router will send the event payload
   to all collectors associated with the collector queue.

The default router (:py:class:`~event_collector.router.SimpleRouter`)
is designed to work for most use cases but the routing logic can be
changed using a custom router class.