Postgres as a Message Broker to handle asynchronous task at scale

Presented by:

4612dc2ec79cdb9694a6f317d1f70b6a

Ujjwal Gupta

from SquadStack

I've been working at SquadStack for more than three years, and as of right now, I hold the position of Senior Product Engineer. Being the team's Platform Engineer is my primary responsibility in this situation. As a platform engineer, my responsibilities are to make sure to provide a rich experience to the team and make sure their productivity won’t be hampered by any reason, some of the domains where I worked as a part of platform engineer:

  • Infra Migration from one AWS region to another
  • Platform Cost Optimization
  • Handle security and get relevant VAPT Certifications
  • Setting up Observability for Developers
  • Setting up Data Warehousing

Prior to becoming a platform engineer, I was a member of a product team, where we had to develop features in accordance with the needs of the product. I, therefore, had the opportunity to work on Postgres, Redis, and Django.

No video of the event yet, sorry!

This proposal outlines how to use PostgreSQL as a message broker, addressing memory issues encountered with AmazonMQ and observability limitations in Redis. The system design incorporates a robust database schema, efficient worker processes, and additional features for task scheduling and management. Key components include:

Database Schema: Utilizes tables for queues, tasks, task instances, and workers, with a visibility timeout mechanism to prevent duplicate processing. Worker Design: Implements concurrent task processing using "SELECT FOR UPDATE SKIP LOCKED" queries, complemented by a sidecar process for monitoring and task recovery. Metrics: Focuses on tracking total messages, worker count, consumption rate, and ingestion rate for comprehensive system observability. Additional Features: Incorporates a cron task scheduler for periodic tasks and supports ETA-based task execution.

This architecture aims to provide a scalable, observable, and reliable message brokering solution using PostgreSQL's robust features and transactional capabilities.

Date:
Duration:
45 min
Room:
Conference:
PGConf India, 2025
Language:
Track:
Application Developer
Difficulty:
Medium