I recently revisited the blog post series by the DragonflyDB team. In my view, DragonflyDB is the best Redis implementation available, surpassing both Valkey and Redis itself.
Key Architectural Insight
The main takeaway is that instead of having a multithreaded I/O TCP server and single-threaded command processing, it
spawns a pool of threads, each having its own store; that is, the threads share no data, no pointers. Therefore, there
are almost no locks. You still need locks for multi-key commands like MGET, MSET, SCAN *, etc.
Implementation Experience
I began implementing this architecture in Zedis. The implementation is far from complete, but I learnt a lot. The performance-complexity tradeoff becomes severe beyond a certain threshold. The complexity increase may not justify the performance gains in many contexts.
For Zedis, my personal project, I intend to complete the implementation as a learning exercise. However, the production viability of such complexity remains questionable.
What Next?
I’m reaching a natural stopping point with Zedis. The optimizations I implemented were valuable learning experiences, and the project served its purpose well. I still want to do the geospatial module, that sounds pretty interesting.
Distributed filesystem in Zig?
I haven’t touched disk much working with Redis. Obviously, since Redis is in-memory. But disk is where the truth lives for basically every persistent database, like Postgres, FoundationDB, RocksDB, Tigerbeetle. My goal for the next two months: fully understand how to implement a disk-based B+ tree.
To those who don’t know, MinIO is a S3-compatible object storage, basically a
distributed filesystem. It is in maintenance mode, which means that it won’t take any more changes. There are other
similar projects that uses the same underlaying algorithm, like horcrux.
horcrux allows you to split a file into encrypted horcruxes; if you split it into 5, with 3 parts you can get the
original file.
My initial idea is to learn a consensus algorithm like Raft of Viewstamped Replication protocol, and build something like horcrux on top of that. Easier said than done! By the way, Joran from Tigerbeetle made a great talk about VSR (Viewstamped Replication protocol).
Most likely, that’s what’ll tackle next, if you want to join me in this adventure, feel free to reach out!