Sklad, a key-value database in Zig

Hacker News

Share card

Sklad, a key-value database in Zig

I’ve been building Sklad, a small key-value database in Zig. Right now it supports basic set/get/delete operations, TTL, and range queries. The main idea behind it is to use lock-free data structures and to build the engine around an asynchronous task queue processed by a small pool of worker threads. The task queue is a lock-free MPMC queue. Memtable is also a lock-free skip-list. I also use lock-free append-only queues and stacks throughout the code. The only place where I currently use locking is writing to WAL. The request processing pipeline is split into small tasks. A single I/O worker accepts incoming requests. When a new request arrives, the I/O worker publishes a read-request task, which is then picked up by one of the generic workers. From there, the request continues through a series of smaller tasks: a query-processing task parses the query, an execution task performs the operation, and finally a write-response task sends the result back to the client. Each stage publishes the next task to the queue, and that task may be picked up by a different worker. The maximum number of workers is controlled by a configuration parameter, and there is simple logic to retire idle workers and spawn new ones as the workload increases. Sklad also collects internal metrics such as request latency, task latency, queue wait time, and the number of pending memtables and active workers. Right now these metrics are exposed through a dedicated metrics request, but I’d like to use them as inputs for adaptive behavior in the engine, for example scaling the worker pool, deciding when to run compaction, and potentially tuning SSTable parameters such as memtable size or Bloom filter bits per key. GitHub: https://github.com/sklad-dev/Sklad