Drop-in SQS replacement based on SQLite

Hacker News

Share card

Drop-in SQS replacement based on SQLite

Hi! I wanted to share an open source API-compatible replacement for SQS. It's written in Go, distributes as a single binary, and uses SQLite for underlying storage. I wrote this because I wanted a queue with all the bells and whistles - searching, scheduling into the future, observability, and rate limiting - all the things that many modern task queue systems have. But I didn't want to rewrite my app, which was already using SQS. And I was frustrated that many of the best solutions out there (BullMQ, Oban, Sidekiq) were language-specific. So I made an SQS-compatible replacement. All you have to do is replace the endpoint using AWS' native library in your language of choice. For example, the queue works with Celery - you just change the connection string. From there, you can see all of your messages and their status, which is hard today in the SQS console (and flower doesn't support SQS.) It is written to be pluggable. The queue implementation uses SQLite, but I've been experimenting with RocksDB as a backend and you could even write one that uses Postgres. Similarly, you could implement multiple protocols (AMQP, PubSub, etc) on top of the underlying queue. I started with SQS because it is simple and I use it a lot. It is written to be as easy to deploy as possible - a single go binary. I'm working on adding distributed and autoscale functionality as the next layer. Today I have search, observability (via prometheus), unlimited message sizes, and the ability to schedule messages arbitrarily in the future. In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues. I'd love your feedback!