event-collector

event-collector is a toolkit for creating lightweight event processing pipelines.

This library implements an in-memory competing consumer queue using only Python standard library functions and asyncio primitives. The “collector queue” runs as asyncio background tasks and can autoscale according to load. Using collector queues you can set up data processing workflows quickly without having to deploy a heavy stack of external services.

Because the collector queue runs alongside your data-processing application or service – i.e., in the same process – it has the same fault tolerance and latency guarantees as the rest of your code.

Repository

• codeberg.org/kchan/event-collector

Requirements

• >= Python 3.13

event-collector uses the following packages:

• orjson (for structlog JSON logging)

• structlog (for JSON logging)

How to install

Using pip:

pip install event-collector

Or, if using uv:

uv add event-collector

Contents

• Concepts
  • Collector Queue
  • Publisher
  • Consumer
  • Use cases
  • Collector
  • Routing & Delivery
• Getting Started
  • 1. Create Collector
  • 2. Configure a CollectorQueue
  • 3. Run the CollectorQueue
  • Complete CollectorQueue Example
• Event Collector Workflow
  • Event Collector Setup Overview
  • A. Defining Collectors
  • B. Configuring the CollectorQueue
  • C. Sending event payloads
  • D. Creating Payloads & Specifying Delivery
  • E. Customizing CollectorQueue
  • F. Using a Custom Router
  • G. Handling Return Values
  • H. Error handling
• Tuning the Queue
  • References
• event-collector API
  • Collector
  • Router
  • EventPublisher
  • CollectorQueue
  • Message Bus
  • Data Structures
  • Logger
  • Exceptions

Note

This project is under active development.