Checkout the slides

This proposal outlines how to use PostgreSQL as a message broker, addressing memory issues encountered with AmazonMQ and observability limitations in Redis. The system design incorporates a robust database schema, efficient worker processes, and additional features for task scheduling and management. Key components include:

Database Schema: Utilizes tables for queues, tasks, task instances, and workers, with a visibility timeout mechanism to prevent duplicate processing. Worker Design: Implements concurrent task processing using "SELECT FOR UPDATE SKIP LOCKED" queries, complemented by a sidecar process for monitoring and task recovery. Metrics: Focuses on tracking total messages, worker count, consumption rate, and ingestion rate for comprehensive system observability. Additional Features: Incorporates a cron task scheduler for periodic tasks and supports ETA-based task execution.

This architecture aims to provide a scalable, observable, and reliable message brokering solution using PostgreSQL's robust features and transactional capabilities.