Tuning the Queue

The collector queue runs a reporter background task that prints runtime statistics on the standard log output at regular intervals (the default is every 5 seconds). These runtime metrics reports can be useful in helping us understand the characteristics of our input and output flows.

The event_collector library uses Little’s Law as the basic model for analyzing how the producer-consumer queue is performing over time. Little’s Law states that, for any stable system:

\[L = \lambda * W\]

where:

  • \(L\) = events waiting in the queue or being worked on (the “work-in-progress”)

  • \(\lambda\) = average arrival rate of events into the queue

  • \(W\) = service time (i.e., the event handler’s processing time for each event)

In other words, if we know the average service time of the event handler (\(W\)) and the average arrival rate of events into the queue (\(\lambda\)), then we can calculate the average work in progress (\(L\)) we can expect to see in the system.

An example collector queue report log (prettified) looks like:

{
  "logger": "CollectorQueue-bd5c4713-8157-4337-9169-bb95b8fc5f48",
  "busy": 23,
  "qsize": 0,
  "consumers": 25,
  "load": 0.92,
  "avg_load": 0.96,
  "service_time": 0.251,
  "wait_time": 0.251,
  "arrival_rate": 94.765,
  "departure_rate": 94.765,
  "wip": 23.786,
  "event_count": 9976,
  "event": "collector queue metrics",
  "level": "info",
  "timestamp": "2026-03-30T02:16:34.576247Z"
}

By looking at the arrival_rate, service_time, and wip values, we get:

wip = arrival_rate * service_time

or:

23.786 = 94.765 * 0.251

This means that when events are arriving into the queue at about 95 events per second, and with an average processing time of 0.25 seconds (250 ms) per event, we can expect the queue size to be about 24 “work in progress” events (i.e., waiting to be processed or currently being worked on).

The above measurement data tells us that in order to keep up with the arrivals, the collector queue must have a consumer pool of at least 24 workers (or background tasks). In the example report log entry, the consumer count is 25. This means the collector queue is autocaling the consumer pool to 25 “workers” so the load is balanced.

Balanced means that in the collector queue, the arrival rate is (more or less) matched by the departure rate.

If you are seeing a big difference between the arrival rate and the departure rate – for example, if the arrival rate is several magnitudes greater than the departure rate – then the collector queue is unable to create a large enough pool of consumer tasks to keep up with the input. The solution is to set a higher max_consumers value that is greater than the average “work in progress” value the collector queue is reporting (preferably with an extra margin of about 10-15%).

The collector queue is designed to run in a predictable, optimal manner when the input rate can be balanced by the output rate. The metrics log entries can provide us with insights and starting points for finetuning the parameters so the collector queue runs in a stable state and has enough capacity to handle our input load.

Note

The collector queue reporter runs as a background task alongside the consumer pool.

If you want to adjust the reporter log interval to a longer interval than the default, set the report_interval to the number of seconds appropriate for your setup.

Example:

collector_queue = CollectorQueue(
    collectors,
    min_consumers=1,
    max_consumers=100,
    report_interval=60.0,
)

If you do not need or want the reporting statistics, you can disable the reporter by setting the report_interval parameter to 0.

Example:

collector_queue = CollectorQueue(
    collectors,
    min_consumers=1,
    max_consumers=100,
    report_interval=0,
)

References