cl-naive-store

In 2016 I decided to have another go at writing my own database because I wanted a database that was blindingly fast for queries but with files that are machine and human-readable. Not to mention that I have a fondness/weakness for plists. Thus cl-naive-store was born.

Like any pet project, it took on a life of its own and has become a bit of a beast (by 2022) with the best of intentions. The reason is that you can customize just about anything (you could say it was designed to be customized) and it has a lot of additional packages that can be loaded to layer on more and more functionality. The “base API“, creating, deleting, querying etc stays the same no matter how many layers of functionality you load. So you can incrementally use more and more of the possible functionality without rewriting code. That means quick proto-type code can be fleshed out later if needed.

In its simplest form cl-naive-store is a collection of plists with or without keys. In its fully loaded form, it’s a lazy-loaded, in memory, persistent, hierarchical, declaratively described, document universe with indexing and sharding thrown in for good measure. You can choose what you want to use and what not its all up to you.

Query speed was always the number one priority, though with lazy loading and sharding the loading and persistence of data runs at a respectable clip. It was also never meant to be a database that would host terabytes of data in one collection. That being said, it can deal with millions of records without too much effort and I would wager that it could deal with tens of millions if sharding is used properly.

One peculiarity of cl-naive-store is that it does not use data definitions (like table definitions found in SQL), a package for such definitions is available (thus the “declaratively described”) but the inner workings of the cl-naive-store don’t need them to load, query or persist data. Collections of data only need a name and maybe keys to function. That means what a document looks like today compared to yesterday has no consequences as far as cl-naive-store is concerned. Such power could come at the cost of your sanity if you abuse it, so a framework for data definitions is supplied but not enforced.

I think cl-naive-store is pretty well documented and there are simple but comprehensive examples.

I have been blessed to have been assisted on this project by some really exceptional people, so it has been a really fulfilling experience in more than one way.

Try it out and let me know what you think.

Advertisement