Postgres as a Message Broker to handle asynchronous task at scale
Presented by:
Hemendra Sharma
Platform engineer who loves building and optimizing systems to keep things running smoothly
No video of the event yet, sorry!
This proposal outlines how to use PostgreSQL as a message broker, addressing memory issues encountered with AmazonMQ and observability limitations in Redis. The system design incorporates a robust database schema, efficient worker processes, and additional features for task scheduling and management. Key components include:
Database Schema: Utilizes tables for queues, tasks, task instances, and workers, with a visibility timeout mechanism to prevent duplicate processing. Worker Design: Implements concurrent task processing using "SELECT FOR UPDATE SKIP LOCKED" queries, complemented by a sidecar process for monitoring and task recovery. Metrics: Focuses on tracking total messages, worker count, consumption rate, and ingestion rate for comprehensive system observability. Additional Features: Incorporates a cron task scheduler for periodic tasks and supports ETA-based task execution.
This architecture aims to provide a scalable, observable, and reliable message brokering solution using PostgreSQL's robust features and transactional capabilities.
- Date:
- Duration:
- 45 min
- Room:
- Conference:
- PGConf India, 2025
- Language:
- Track:
- Application Developer
- Difficulty:
- Medium